Measure True Peak Performance Testing by Removing Outliers

Sometimes we test a system with a specific goal of measuring sustained performance capability. For that we want to measure true peak performance testing by removing outliers at the beginning or end of our tests that may otherwise influence performance characteristics. In this post we will describe how to accomplish this using a basic JMeter post-processor script.

Suppose for a moment I were a cyclist, and I was attempting to calculate my average biking speed. For very long rides, this might be as straightforward as dividing distance by time. However, if I were to calculate it the same way over a series of short rides my average speed would be slower. This is because a greater portion is spent accelerating at the beginning, and decelerating at the end.

With a cloud-based load test (especially those of shorter duration) the amount of time spent ramping up a test can be significant. Depending on the statistics we are collecting, we may want to exclude results collected during these times as they will be influenced by extraneous factors (internal server processes, setup, etc.) and not be indicators of true performance. We will show you a simple way to measure the true peak performance of your test by removing these outliers.

Removing Outliers

We will use JMeter for our example because it has become by far the most popular load testing framework on RedLine13. Using JMeter, we can add a basic post-processor script (BeanShell or JSR223 + Groovy) that will automatically exclude all results outside a predetermined time frame. We will illustrate this here by adding a JSR223 post-processor to our Thread Group:

We can set the “Language” property of this post-processor to either “groovy” (recommended) or “beanshell”, and to execute the following code:

if(args[0].toBoolean()) {
    int before_ms = (args[1].toInteger() * 60 * 1000);
    int after_ms = (args[2].toInteger() * 60 * 1000);

    long start_time_ms = vars.get("TESTSTART.MS").toLong();
    long current_duration_ms = (new Date().getTime() - start_time_ms);

    if ((current_duration_ms < before_ms) ||
        (current_duration_ms > after_ms)) {
        prev.setIgnore();
    }
}

Finally, under the “Parameters” section of the JSR223 post-processor, we will want to specify our “before” and “after” durations which will constrain our results:

Here we have set the test to exclude results for the first 5 minutes, and then after the 30 minute mark. (These values are picked up as args[…] and converted to milliseconds in the first few lines of our above script.)  The first argument is simply a flag which we have configured to conveniently enable or disable the conditions within our script.

Alternatively we could also pass these parameters into JMeter as properties rather than hard-coding them into our test.  Our script would remain the same, but we might specify our arguments using the __P() property function.  This offers the advantage of being able to configure test timing by setting these properties in RedLine13 as described in this post.  From the perspective of our above example, this may be set as follows:

Running this test with the sampler enabled will show us a reduction in captured results:

This is both expected and desired. If we compare the output of the original test plan, a subsequent run with the post-processor enabled will only consider the area shaded below:

This effectively considers only the region of the test unaffected by confounding ramp up and ramp down factors. The effect on the average throughput of the sample test used here can be seen as follows:

Free Trial

Did you know that RedLine13 offers a full-featured free trial? Sign up today and try out this example for yourself!