How many businesses or organizations actually do continuous load testing?

There are typically two purposes for load tests: event-based and continuous load tests.

The first is the one we normally think of: load tests that are run due to some external event. It could be a new marketing campaign that is going to drive additional volume. It could be a new version of software and wanting to make sure that it will handle the volume. Or it could be Black Friday and the increased volume.

The second one is not so common: continuous load test. This could be run every day or week or even longer timeframe. Or potentially shorter if it makes sense. The purpose of this test is to ensure that there are good and consistent results. There are innumerable internal and external factors that could change and lead to unacceptable performance. By running tests on a regular basis, we are hopefully finding out issues or seeing trends before our users are.

As for tools, you can load test your project continuously with Jenkins. Jenkins is one of the most popular open source CI/CD tools used to build, deploy and test your project continuously. RedLine13 supports integration with Jenkins which means you can build, deploy and load test your project continuously with Jenkins. 

But what does it cost?

Most people would agree that this all makes sense. The concern often turns to the cost to do this. How could cost be an issue for RedLine13 users when we say it’s Almost Free? The AWS cost per test is miniscule. There is a RedLine13 subscription cost, but if you have an Annual Premium subscription for $2,500 and run just one test a day, it is $6.85 per test. And if you only need an Annual Basic, it’s $1.37 per test. So what does that mean? 

It means that it is cheap but they aren’t the only costs. Drew Batshaw, Co-Founder and CTO of Waggl and RedLine13 customer, explained when they run event based tests:  “Whenever we make changes that we think might impact load or are needing to increase our load.”

Waggl is efficient and has also automated the work with AWS scripts and CloudFormation templates that they use for building and tearing down the environment they point the load testing at.

They see the benefits in continuous load testing but it would be expensive for them. Drew pointed something out:  “The biggest cost is not Redline13 or the AWS costs but the cost to run the environment that gets load testing. For us (and I imagine most people), that is an environment that is as expensive to run as our real production environment.  It is not unusual for us to spend thousands of dollars a month on that if we are doing a lot of load testing (and that is not running that environment all the time).” 

Is there a solution for continuous load testing? What do you do?
In the meantime, you can try your own test, event-based or continuous, with RedLine13 here.