JMeter is one of the most popular load testing frameworks. However, to use JMeter at scale requires supporting infrastructure. In this post, we will be comparing and contrasting managed versus do-it-yourself (DIY) JMeter load testing.
With the rise of cloud-based architectures, it has become increasingly the standard to test using cloud-based resources. This means writing scripts to provision servers, coordinate load generators, and handle data collection. Load testers using JMeter have two general options. They can opt to design and build the entire stack themselves as a “DIY” approach. Alternatively testers can use ready-made solutions such as RedLine13 to abstract away everything except for the actual load test.
RedLine13 is Fully Managed
One advantage to using RedLine13 over a DIY approach is that it is fully managed. You can access your tests from anywhere and at any time through an easy to use web portal. Support exists for multiple test types “out-of-the-box”, including JMeter, Gatling, custom scripting, and more.
JMeter itself is a command-line tool, and though there is a GUI mode it is not optimal to generate load in this way. This necessitates coordinating often multiple instances across multiple load generating servers, which requires significant infrastructure programming. With some time and effort invested this is entirely possible with the DIY approach, however RedLine13 makes it easy by providing all of this functionality. This can be a significant advantage for small teams who might want to focus on writing actual tests versus investing in a custom test infrastructure.
There is also the concern about keeping a DIY stack up to date with the latest software releases. RedLine13 manages this as well, keeping testing software, operating systems, and dependencies updated with the latest releases and security fixes.
Integrating with AWS
There are multiple cloud providers supplying commodity compute instances. Utilizing these resources has become the de facto standard in cloud-based load testing. When tasked with a DIY approach, it is up to the test administrator to provision and configure these instances, run tests, collect results, and ultimately dispose of these resources. This can add a significant amount of work to a load testing project. It also adds complexity which can require some degree of the testing of the test architecture itself. Though it confers the advantage of flexibility, taking this approach can increase time and cost.
Often complexity, time, and cost are sought to be minimized. RedLine13 can help testers achieve this by providing a vetted platform which tightly integrates with AWS. Running a JMeter test on RedLine13 is as simple as uploading the test file, selecting the number and specifications of the EC2 instances to be provisioned, and clicking “Start Test”.
Hidden Costs of Load Testing
With cloud-based load testing, there are finite resource costs which are known. Primarily these are server costs for running load generators, and associated networking resources. However, with a DIY setup one has to be cognizant of additional hidden costs. A significant investment in infrastructure is required to coordinate load generators and collect results. Similarly maintenance of that infrastructure also requires time and attention in keeping that software up to date with the latest releases and security patches. There are also hidden costs in testing that infrastructure itself to ensure that the results obtained are reliable.
Comparing this to RedLine13, most of these hidden costs no longer apply. With our service you have access to a vetted platform that does not require you as the tester to address these issues. This allows you or your team to focus on what matters most — running your load tests.
Convenience Features
When developing your own testing architecture, inevitably one would seek to automate common and repetitive tasks. However, this comes only after identifying these repetitive tasks, and then requires a development investment to create these shortcuts.
RedLine13 has already identified many of these convenience features, which are available on the platform for your use today:
- Duplicate a load test with a single click
- Automatically download logs and JTL files from load generators
- Automatically merge JTL files across multiple load generators
- Generate JMeter dashboard reports with a single click
- Organize, search, and sort from all previously run load tests
Technical Support
Redline13 offers a few distinct advantages over a purely DIY load testing architecture. One of these is through our excellent technical support. Apart from your actual load test, all components of the architecture responsible for running your test are backed by our responsive technical support team. We offer email support with all of our plans and usually same-day responses within normal business hours. When compared to a DIY approach, this can offload a significant burden from your own technical team.
Cost of RedLine13 vs. DIY
It is possible to build your entire test infrastructure from scratch. Doing so would allow fine-grained and complete control over every aspect. However, in many cases the test architecture does not need to be so specialized as to justify this. More commonly it is advantageous to utilize a ready-build load testing platform that performs all of the required tasks. This is the dichotomy of “build” vs. “rent”.
RedLine13 provides a robustly featured platform for a low monthly subscription. On our pricing and features page, you can compare our three different plans (Free, Basic, and Premium). Each plan provides increasingly more features that otherwise you would have to build for yourself.
Ready to compare running your test on RedLine13? Sign-up for a fully featured free trial and start testing today!