From the Open Source Load Testing presentation http://www.slideshare.net/richardfriedman/open-source-load-testing
Obligatory list of load failures
- Pope Visit Fail for Septa: http://www.phillymag.com/news/2015/07/20/septa-suspends-pope-pass-sales-technical-difficulties/
- HealthCare.gov: too many links… here is a google search https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=HealthCare.gov+fail
- Your story – we have all had one.
And as we start the tick up towards holiday season and soon to start the code freeze phase we are reminded work through our checklist. In a good review post at gotgroove.com #1 on that list should be stress testing.
For most, the priority of load testing falls to the bottom of the list of things to do. Perhaps it is in your plan, but as the last thing on the list, there is a good chance it will get a quick review instead of a deep analysis. Underlying the majority of enterprise IT budgets is an upside-down triangle. At the top our $$$ go to the operational wellness while at the bottom $ is the sunk cost of activities to push to production.
- Investment in Operations and Development. Managing OpEx budgets, the biggest product spends come in running your day to day. Load Tests of large sizes are not daily operations. There are many ways to dissect technical op-ex budgets, but at a gross level, I would say it flows People, Run (Systems and Services), Monitor (Systems/Network/Storage), Deploy (Continuous Delivery), Test (QA), Load Testing.
- Invest as Needed. If we invest as needed then we will invest heavily in items that impact our day to day, less into the one-time activities.
As a CTO and VP Engineering, I have yet to see load testing truly integrated into the agile process. Instead, it is more common to be integrated into the tail of the launch process. The common issue with late phase integration as a serious performance flaw becomes uncovered we could have a major setback, and then very delayed projects.
- After QA, During Beta Launch. It is better than the alternative and it is in your process. Just set the expectation that you can learn something at this point that might set your project back by days, weeks, or more.
- When it falls over. Tough pill to swallow and we now readjust to figure out where things went wrong.
I am not suggesting Load Testing can be integrated earlier in the process, just to understand where it really sits and make sure your project gives it the due respect it deserves. There is no silver bullet to load testing and there is no product that will cure all woes, having managed a multi-million dollar data center and many large scale projects, the expense of load testing was put into the people, never the tools.
