RedLine13 has made changes to handle the cases where it is necessary for merging very large JTL files, which are output by JMeter. RedLine13 has for years been able to handle large JTL files needed to support high volume load tests with no problem. Now RedLine13 can handle JTL files of 70GB and greater. We’ve included details on why we made the change for those that want to understand more.
A customer was running tests which generated JTL files in excess of 70GB across multiple servers. To give you some idea of the size of their test, it ran for over 2 hours, generated over 4.8 million requests at an average rate of more than 55,000 requests per second. The JTL files that were generated were as large as 70GB. While our system handled the process of collecting and merging the JTL files, there were some external factors which caused issues. Here we will explain what the nature of the problem was, how we solved it, and ultimately what we learned to make RedLine13 even better.
Issues that we overcame:
The primary issues were as follows:
- We were unable to write the merged JTL file back to S3 because of the massive file size. This is due to the limitations of the S3 “putObject” method which only permits files up to about 5GB to be uploaded.
- This issue was addressed by using the “upload” method of getting files into S3. Using this method raises the upload limit to the maximum S3 object size of 5TB.
- Generating the JMeter Dashboard report took more than 2 hours to process. However, our watchdog monitoring is configured to terminate worker instances taking longer than 2 hours to complete.
- This issue was addressed by dynamically optimizing this timeout value based on how large the generated JTL file is.
The ultimate result:
When the customer reported the issue, we diagnosed the problem and made the changes. As a result, RedLine13 now supports scaling tests even bigger. Nobody had previously tried using our service for such a massive scale test. However, with a few small improvements to our architecture we have proven we can support it.
Did you know that RedLine13 offers a free trial account? Sign up and start testing today!