The Endless Load Test

Or, why you need to include load testing in your CI/CD pipeline

The Endless Load Test, it sounds like the title of a horror movie no one ever asked for. Probably a better way to look at this topic is: Load Testing While You Sleep. It’s a well known fact that the earlier you detect a defect that hurts performance, throughput and/or stability, the easier it is to mitigate. In my experience, 70% to 80% of the problems I’ve helped uncover during load testing are reproducible at much lower scales. It is much less painful (and less expensive) to deal with such bugs testing with 20-500 virtual users than during your seasonal load testing which may use tens to hundreds of thousands of virtual users.

You’re already performing functional tests as part of your release cycles (right?), so taking those a step further to load testing is easier than you think. There are just a few things you need to keep in mind:

Keep it to scale

Whether you’re using traditional system specifications like CPU cores and gigabytes of RAM, or cloud-native services such as Lambda, Aurora, containers as a service etc, keep your testing environment scaled. Are you running 16 application servers with a 32 core database? Then you could aim for 2 app servers with a 4 core database for example. Using cloud-native services? Then make sure you scale capacity, throughput reservation and so forth proportionally in your development environment.

Reuse test scripts as much as possible

Remember that automated functional testing you’re already doing? Look for load testing tools or services that are based on the same technology. Doing functional testing with JMeter? Then use JMeter for your load testing as well. Using real browsers from Selenium scripts? There are load test services and opensource tools for that. One caveat to remember is that most large scale load testing uses protocol emulation rather than real browsers for resource/cost reasons. While it’s admittedly a pain to maintain 2 flavors of scripts, it may be justified by the cost savings.

Define your KPI’s, then actually use them

KPI’s for load tests can be more nuanced than for a functional test, where the results are either it worked, or it didn’t. How do you know if your load tests “worked”? Try starting with your production environment as a baseline, think about KPI’s based on:

• Throughput such as requests per minute
• Loading times for pages, click-actions and even background actions that are on the critical path (this is an area where real browser tests have an advantage having real javascript engines etc as everything is automatically included in the test)
• Don’t forget to take into account or filter out as appropriate, requests to external services and third parties

Last but definitely not least, monitor your results! If your load test results are heading south in your releases, you don’t want to miss the warning signs. This may require automated alerting based on the KPI’s you’ve defined. The last thing you want is to have these results disappear into the the Inbox black hole where so many SLA reports have gone.

This article is published before on LinkedIn.