Load testing with a high concurrent user count can be challenging in many ways. This post will tell you about how one team did it with one million user testing with JMeter and the challenges they faced.

Requirements

Thinklogic, a Los Angeles based consulting firm specializing in web development and custom business software solutions, was tasked with quickly building and load testing a highly scalable web-based application that would handle one million concurrent users.

  • The high level application functionality consisted of:
    • User account creation
    • 3rd Party API calls
    • Multivariable form imports
    • Document upload
  • In two months they needed to build and test a web-based application
  • Supporting 200 million users in the first month
  • Supporting 1 million active users at any one time
  • Technologies: Amazon CloudFront, Microsoft Azure, AWS EC2 (for testing)

The Challenges

Here were some of the challenges Thinklogic faced along the way:

  • Getting the test to provision to all the EC2 instances before a timeout
    • The larger the sample upload file, the more likelihood of a timeout
    • The Ohio region was specifically prone to time outs
  • Often times the volume of EC2 request would be so large that it would exhaust the available on-demand compute capacity available for that datacenter
  • AWS CloudFront and Azure both had automatic DoS prevention measures that would kick in
  • The default AWS CloudFront limits were exceeded (max requests per second), and it wasn’t easy to get this limit increased
  • Azure App Services would experience thread pool exhaustion long before CPU or RAM scaling limits were hit
  • The CSV file had unrecognizable end of line characters which would run correctly in JMeter but failed to run in Redline13

The Solution

Chris Adams, President of ThinkLogic:

Monthly EC2 testing servers costs were in the 10’s of thousands of $. We looked into testing with other web-based testing tools, but they were not feasible due to max vUsers limits and cost. So we chose RedLine13.

In addition, to assist with logging and troubleshooting, AWS/CloudFront asked them to capture CloudFront request IDs when they fail. Request id is returned in a response header. JMeter lets you include that in the script (see example). Redline13 handles this without any problem because it doesn’t alter the JMeter test to run. 

One of the tests required testing 1 million unique, 1 time use codes. This required using Redline13 to split the CSV file and distribute to 24 servers. The CSV file provided had records that could only be put through the script once as a new record took you to one page and a used record took you to a different page. They were testing the new record process so the step would fail if the “used” record page loads. (They did this by searching for a certain text string on that page.) You can read more about JMeter testing with CSV files in a post titled Data Driven Testing with JMeter.

Some tests required utilization of up to 750 Large (8-core) AWS EC2 cloud servers spread across 5 North American regions.

Chris Adams again:

The initial tests used so many servers and had so much data, we had to contact RedLine13 support in which they quickly optimized their Redline13 interface. Thanks for that!

Success

Once they had everything working as needed, they ran the full Redline13 test and selected the split file option so the 1 million records in the CSV files were split between all 24 servers. Success!

You can try your own test, whether it is one user or one million concurrent user load testing, on RedLine13 for free.