BlazeMeter is one of the leading load testing platforms. RedLine13 has claimed that it provides (Almost) Free Load Testing. This post will first take a look at a BlazeMeter pricing comparison vs. RedLine13. Then it will take a look at a feature comparison because price isn’t everything.
In addition, there is the increasing realization by businesses with the value of shift left testing. So this blog post will also look at pricing related to continuous integration / continuous development (CI/CD).
BlazeMeter Pricing
BlazeMeter Pro Plan is their most popular plan according to BlazeMeter. It allows up to 5,000 concurrent users. The Pro plan costs $499/month or $5,988/year. The Pro Plan also allows up to 80,000 Virtual User Hours/Year. What does that mean? BlazeMeter explains that “Each Virtual User (Thread) in your test will consume 1 VUH for tests of 1-60 minutes, 2 VUH of tests for 61-120 minutes, etc.” So if you run a 5,000 concurrent user test for 1 hour, that will use 5,000 VUH. That means you can run a one hour 5,000 user test 16 times in one year. Since the Pro Plan costs $5,988/year, this results in each one hour test costing $374.25.
If you want to run more tests, you can’t use the Pro Plan and have to upgrade.
RedLine13 Pricing
RedLine13 also has a Pro Plan which will also allow up to 5,000 concurrent users. The Pro plan costs $500/year. This post details testing 5,000 concurrent users for one hour on Amazon Web Services (AWS) with an AWS charge of $10. If you run 16 tests like above, that would be $10 x 16 tests which is $160. Then add in $500 for the Pro plan for a total of $660. Therefore each one hour test costs $41.25.
If you want to run more tests on RedLine13, that’s no problem. There is no limit on the number of tests. Your only additional cost will be the AWS cost. In this case it is $10. That means your cost per test will go down. So if you do twice the number of tests, the cost per test of the 32 tests is $25.63.
In many cases, the cost per test is less than $10 per test. There are many examples such as a 10,000 user test for $6.71. Therefore the RedLine13 cost per test would be even less than the amounts above.
Pricing Summary
Let’s wrap up the pricing comparison with some tables to see how BlazeMeter pricing compares with RedLine13.
Our first comparison: 16 tests with 5,000 concurrent users
| Blazemeter | RedLine13 | |
| Test Size Limit – concurrent users | 5,000 | unlimited |
| #tests/year with 5,000 concurrent user tests | 16 | unlimited |
| cost/year | $6,000 | $660 |
| cost/test | $374.25 | $41.25 |
Our second comparison doesn’t work since the BlazeMeter Pro plan doesn’t support 32 tests of 5,000 concurrent users. RedLine13 does and the cost/test is reduced to $25.63.
Continuous Testing: BlazeMeter Pricing vs. RedLine13 Pricing
If you’re not familiar with Continuous Testing and what is referred to as Continuous Integration / Continuous Development, read this blog post.
Are 16 tests in a year enough for businesses that are serious about Continuous Testing? Not for most businesses. It even seems contradictory to BlazeMeter’s objective of providing a “continuous testing solution”. Businesses are now running multiple tests per day. Let’s keep it simple and look at just one test per business day for a total of 250 tests per year.
The BlazeMeter Pro plan will not allow for that number of tests. That means you will have to go with their Unleashed plan. There is no pricing available publicly for the Unleashed plan. If we use the cost above of $374.25 x 250 tests, we get $93,562.50.
The RedLine13 Pro plan does allow for that number of tests. Actually the number of tests allowed is unlimited. That means staying with the Pro plan at $500. Let’s stay with the AWS cost per test of $10 which comes out to $10 x 250 tests for $2,500. Add in the Pro plan price of $500 and the total is $3,000.
Continuous Testing of 250 tests per year with 5,000 concurrent users
| Blazemeter | RedLine13 | |
| Test size of concurrent users | 5000 | unlimited |
| #tests/year | 250 | unlimited |
| cost/year | Unknown, but extrapolating the Pro plan is $93,562.50 – call BlazeMeter | $3,000 |
| cost/test | $374.25 | $12 |
BlazeMeter Features vs. RedLine13
Pricing isn’t everything. So, let’s look at what features you need for your testing.
| Feature | BlazeMeter | RedLine13 |
| Use JMeter Scripts | yes | yes |
| API Testing | yes | yes |
| AWS Cloud Testing | yes | yes |
| Real-time Reporting | yes | yes |
| CI/CD* with Jenkins | yes | yes |
| Use Gatling, PHP, Node, Python, Selenium, WebDriver, API | yes | yes |
Redline13 just announced CI/CD integration with Codeship.
Conclusion
This post shows that there is a BlazeMeter competitor that costs significantly less and that RedLine13 has comparable features needed for load testing.
Are you married to an expensive load testing tool? If so, RedLine13 is a BlazeMeter alternative.
