Optimizing AX Batch Performance - Batch Thread Configuration

Imparted from here

Are you pushing a lot of transactions through the AX batch framework?  Are you wondering if you've optimized the system for your workload?  We recently did some testing in our lab to illustrate some of the performance and concurrency characteristics of the AX 2009 batch framework. Our goal was to show the impact of two commonly overlooked configuration options (batch groups & batch threads) that can greatly impact the throughput of your system.  In this post we'll address the topic of batch threads.   Batch groups are addressed in Tom Stumpf's blog post, 'Optimizing AX Batch Performance - Batch Group Configuration.

Test Environment
In our AX 2009 test environment we have 1 SQL Server hosting the AX database and 2 AOS servers configured for batch processing.  Two separate companies were created within AX and each has 10,000 orders that need to be invoiced.  A separate invoicing batch job was setup for each company to process the orders.  In order to understand the results below it's important to note that the invoicing operation in AX is multi-threaded.   This means it can leverage multiple batch threads for parallel processing.  To illustrate how the batch threads and batch server configuration affect batch job performance, we ran three different tests:
In scenario 1, two AOS instances and batch groups are used.  1 AOS instance is assigned only to batch group A and the other AOS instance is assigned only to batch group B.  Company A is configured to use batch group A, and Company B is configured to use batch group B.  Both AOS instances are using the default of 8 batch threads.
In scenario 2, two AOS instances and batch groups are used. 1 AOS instance is assigned only to batch group A and the other AOS instance is assigned only to batch group B. Company A is configured to use batch group A, and Company B is configured to use batch group B.  Both AOS instances are using 12 batch threads.
In scenario 3, two AOS instances and batch groups are used. 1 AOS instance is assigned only to batch group A and the other AOS instance is assigned only to batch group B. Company A is configured to use batch group A, and Company B is configured to use batch group B. Both AOS instances are using 16 batch threads.

Test Results
Scenario 1 - (8 Threads)
The red line represents the number of invoicing batch tasks completed each minute for company A. The green line represents the number of invoicing batch tasks completed each minute for company B. Both jobs were started at the same time.

%Processor Time for AOS instance 1

%Processor Time for AOS instance 2

Scenario 2 - (12 Threads)
The red line represents the number of invoicing batch tasks completed each minute for company A. The green line represents the number of invoicing batch tasks completed each minute for company B. Both jobs were started at the same time.

% Processor Time AOS instance 1

% Processor Time AOS instance 2

Scenario 3 - (16 Threads)
The red line represents the number of invoicing batch tasks completed each minute for company A. The green line represents the number of invoicing batch tasks completed each minute for company B. Both jobs were started at the same time.

%Processor Time for AOS instance 1
%Processor Time for AOS instance 2

• Because each job was in a separate batch group dedicated to a separate AOS instance, each had access to a set of dedicated batch threads. 
• When using the default thread of 8, the jobs took about 13 minutes to complete while with 12 threads took about 12 minutes and 16 threads took 11 minutes.
• CPU utilization will increase as more threads are processing on any one AOS instance.  From the tests above, the average CPU usage for 8 threads was in the 72-74%, for 12 threads was 84-89%, and 16 threads was 90-94%.

Conclusions
Based on what we saw in the test results, I think you'll agree it's important to consider the different workloads you have and how the maximum batch threads setting can affect overall performance.  The test we did was meant to illustrate CPU usage versus throughout and how tasks for multi-threaded jobs are affected by the maximum batch threads count.   If both jobs were single-threaded, only 2 of the available threads would have been consumed.   Here are some things to consider when setting the maximum batch threads value:

• You can increase batch processing throughput by increasing the maximum thread count and adding more AOS servers
• Increase the thread count if the AOS is underutilized such as less than 70% 
• Add more AOS servers if the AOS is over utilized such as more than 80%
• When increasing the thread count a good rule of thumb is to increase in intervals of 4-8 and then test the results
• Your goal is the find the most optimal setting as the AOS will be underutilized if the thread count is too low and overutilized if the thread count is too high.   If overutilized, you may actually see a performance decrease as thrashing may occur on the CPU.    For example, in our tests when set to 8 we could see that increasing the thread count would more efficiently utilize the AOS servers, and then by increasing to 16 we were about at the top end for optimal performance.  If we were to increase to 24 we may have likely saw a decrease in performance as the processors would most likely be maxxed out.

The concept of batch thread configuration is important to understand so you can get the most out of your batch processing.   If you use the information listed above as a guide, you can rest easy knowing that your jobs will execute at the optimal rate.