#4 Load Testing is …

From the Open Source Load Testing presentation http://www.slideshare.net/richardfriedman/open-source-load-testing

If you have been through the process of build and running load tests, I am sure you have gone through all the emotions that come with it. From the fun of taking down your site or API, to the agony of not knowing where the bottleneck is, and eventually the eureka moment when you resolve the issue and run your next load test.  Here are some of my emotions.

• Fun.  The first phase of load testing is fun; the feeling of success you are doing something that will help your users into the future.
• Create.  Creating a load test can run from fun (auto-record a session and just play it back) to frustrating when you are three layers deep into your JMeter test trying to pass variables from request to request.
• Run. I always find this task enjoyable, seeing the load kick off and the metrics start flying in.  Work is being done.
• Data.  All that data coming back leads to a lot of pretty graphs.  Graphs of data always puts me in a better place, even with bad news we will have actionable data.

But then, we move to the analysis.  This can turn your fun of load testing into a hard, time consuming, and frustrating process.   You will hopefully have some quick success with low hanging fruit, a loop that can be optimized, a better function that could be used, a DB Query tuned to perfection.  Unfortunately the final gains you might need might not be as simple.  Perhaps, you need to reimplement using memcached vs memcache,  or you find you will need a schema change.  These are the issues you wanted to uncover, but they can lead to making foundational changes.  This is good stuff, it is just the hard work you will need to put in to get to scale.

• Boring.  Load tests can take time to run.  They warm up, go to steady state, and then climb down. Running load tests:  start, get coffee, drink coffee, peek at results, go back to coffee.
• Tedious.  Sometimes analysis shows that you don’t have the instrumentation in place.  Back to code, add metrics or logging, then run code again.
• Detailed.  Having access to a lot of data is great, but sifting through to make sense of issues takes looking at the details.
• Precise.  I thought I could build one general load test and be done, I was surprised.  I exposed issues but without creating some precise load tests I could not isolate my issues.

At some point we work through this and we are rewarded with the fruits of our labor. Our product, API, or site can scale to meet or exceed our requirements.  This is an ‘oh yeah’ moment. Enjoy it, good job!