Making sure software behaves properly under a tester’s hands is important, but it’s not enough by itself. More strenuous tests are needed to see if it works day after day in a production environment. Software that will run on heavily used websites is especially in need of that kind of testing. Not all bugs will show up in unit tests and with single users. Automated tests, simulating aspects of real-life concerns, are an essential part of determining if software is ready to deploy. A business that cares about a reliable product should run a full test suite before each release. Here are our definitions of load testing vs. scalability testing and other testing models.
Load Testing vs. Scalability Testing and Other Testing Models
- Load testing is probably the best known of the large-scale tests. An automated procedure simulates many users, interacting simultaneously and asynchronously with the software. It can catch memory usage problems, misuse of shared variables, and timing issues. Problems that only occur rarely are easier to catch under a load test.
- Scalability Testing is related to load testing, but it examines the effect of changing specific performance parameters rather than the aggregate load on the system. It runs the software under different configurations to check its behavior when scaled up or down. Some errors may occur when the available memory exceeds a certain level or too many files are in use. A bad decision to use a sixteen-bit index may lead to problems when usage is high. Scalability testing is also called performance testing.
- Endurance Testing, also known as soak testing, tests how well the software works when it runs for a long time. The test works best when it varies the load, sometimes emulating just a few simultaneous users, sometimes going for a heavy load. Small memory leaks add up over time. Database connections will run out if they aren’t closed. Data structures that keep growing can drag down performance. Hours or days of testing will turn up these problems.
- Reliability Testing will turn up rare glitches. It consists of performing a large number of operations and checking them for consistency. Race conditions may cause a problem only once in a while, and casual testing will miss them. Something that goes wrong one time in a thousand will drive users crazy if testing doesn’t catch it.
- Failure Testing, sometimes called negative testing, subjects software to intentionally invalid data. The code should not only respond by appropriate error actions, but avoid any system degradation or data corruption. Error paths usually aren’t as well-explored as normal paths. Large-scale testing with invalid data can reveal problems which single intentionally bad queries won’t turn up.
- Stress Testing deliberately pushes the software to the point of failure. Any system has limits; the question is how gracefully it fails. If the whole system crashes when it can’t allocate enough memory, that’s not graceful. Failures should produce understandable error messages, not stack dumps that leak information about internals. The system should keep running even if some actions throw errors.
- Failover Testing covers the worst-case situation. The system fails completely, perhaps because of a hardware problem. How quickly will recovery measures get functionality working again, and what will be lost in the process? Two metrics are important here. Recovery Time Objective (RTO) is the maximum acceptable amount of time till operations are restored after a failure. Recovery Point Objective (RPO) is the maximum window of time over which user actions can be lost.
Thorough testing in all of these dimensions requires a significant investment of time in devising the tests and running them. Fortunately, once they’re set up, test automation software can run without human intervention and report its results. When software has to work reliably on a large scale, this investment pays for itself. Users who rarely run into problems are happy users, and a company that avoids losing data and business to bugs has the best chance of success. A tester’s bag of performance test types are the tools they employ to predictably keep customers happy while maintaining fluid services, as companies scale up for future market demands. And with the definitions of load testing vs. scalability testing and other testing models, everyone can be on the same page with what needs to be done.