A common question that RedLine13 users ask is how many vCPUs needed for load testing. One example of such a question might be:
I want to do a load test with 500k users. I’m trying to decide how many vCPUs I would need for this, and therefore which plan to go for with Redline13. Please could you advise me on how many users you’d recommend per vCPU.
AWS has an article in their documentation which speaks about how to optimize CPU options that serves as a good introduction to the topic. Specifically, you will find they state the definition of a vCPU as it pertains to AWS servers is a single thread running on a CPU core.
Most test usage for vCPUs is dependent on your test and performance of your endpoints. Therefore, your vCPU requirements will be highly dependent on how your particular test is configured. For most users this will mean how their JMeter test is set up as that is by far our most popular test type.
Factors Influencing CPU Utilization
There are a number of factors which will influence CPU utilization for your particular load test. As the complexity of your test increases, so does CPU demand. If your test has any script logic (e.g., JavaScript elements, or a JSR232 sampler) this can quickly and drastically impact CPU utilization depending on the efficiency of our test logic. The actual response time of your endpoints can also impact the load test. For example, if your endpoint takes 500ms vs. 333ms means you will need more concurrent virtual users to simulate the load. Additionally, the type of testing you are performing can also potentially impact test performance. If your test streams video content for instance, it is likely to consume more CPU resources (as well as data bandwidth) than a test which is designed to fill out new user registration forms.
Special Cases with High CPU Utilization
While a simple scripted sequence of requests to basic HTML pages likely will have relatively low CPU usage, a JMeter test using the Selenium WebDriver plugin will almost certainly demand much more resources. This is due to several factors in addition to the complexity of your test plan. Certain plugins will have their own resource requirements separate from JMeter requirements. Selenium WebDriver is a particular example where the load generator needs to load a full web browser on each load generator. This is in addition to the actual JMeter test process and its associated JVM, which constitute resource overhead in ordinary JMeter tests. The web browser instance that Selenium loads is resource-intensive and tends to make the test overhead much higher for these types of tests. To ensure that there is ample processing power left for your test logic, a much larger EC2 instance size is required to support your test.
Monitoring CPU Utilization
On the RedLine13 results page for your test, you can monitor the CPU usage for each load generator server under the “Agent Metrics” section. Below is a good example where the test only hits about 10% peak average CPU utilization:
In the above case, we could have added many more users to this test – or alternatively selected a smaller instance size to optimize cost. Here is a different example where multiple load generators became overloaded:
If your CPU utilization graph looks similar to the one above, it is clear that the instance size you have selected is too small for the performance demanded by your test. In fact, anytime your CPU utilization hits 100% it is a strong indicator to select a larger instance size as the test results are at risk of becoming unreliable at that point.
Selecting Optimal Instance Types
Sometimes people ask about AWS EC2 Instance Types and what will work best for your load test. Determining this often requires some experimentation, as every load test is different. We suggest running your test on a single server, something like a m5.xlarge, which has 4 vCPUs, or m5.2xlarge, which has 8 vCPUs. From here you could get a sense of how many vCPUs you might need depending on the performance of that test as you monitor CPU usage from the “Agent Metrics” section of the load test results page. We stream this data in real time as you run your test. Therefore if you notice that utilization is less than ideal you may be able to cancel your test early and try a different configuration.
As a general rule of thumb, you should strive for a target of about 80% peak average CPU utilization as displayed on our load agent CPU usage graph. If you are exceeding this value, you are at risk of experiencing momentary peaks approaching 100% usage. As mentioned above, once your load test is demanding 100% of the load generator processing power, you may experience unreliable results. Demands below 80% are acceptable, however if you are far below this target you may benefit from a smaller instance size to optimize your AWS costs. Related to this, you may also be interested in reading a related post about AWS cost estimation.
You can try your own test with different vCPUs for free with RedLine13 here. You’ll find that it’s not that hard to figure out how many vCPUs needed for load testing.