Eldad Uzman, Automation Architect at Merck, is the guest author on IoT load testing. You can read more from Eldad at Medium.
Abstract
In this article, I’ll discuss some of the challenges of load testing in the IoT world, and try to provide some practical solutions to these problems based on my personal experience.
What is IoT?
The internet-of-things refers to connecting physical devices such as sensors or computational devices to the internet.
It is often compared and contrasted with the internet-of-people (IoP) which describes the internet being used by human beings, and the combination of these two is sometimes referred to as the internet-of-everything (IoE).
Connecting devices to the internet is used widely for many different purposes: in agriculture (agri-tech), smart homes, smart cities, sports or even battlefields, to name a few.
IoT is growing rapidly.
According to statista, the number of connected IoT devices grew from 8.6 billion in 2019 to 11.28 billion in 2021, and is expected to grow all the way up to 29.42 billion in 2030 (source – https://www.statista.com/statistics/1183457/iot-connected-devices-worldwide)
This means that by the end of our decade, the number of connected devices is expected to more than triple the human population.
IoT vs IoP
There are several key differences to take into account when dealing IoT:
Protocols
On the internet of people, most of the time HTTP is the main protocol.
Sometimes we get some of the variants of HTTP such as websockets and gRPC.
IoT is much more varied.
MQTT, LoRa, LwM2M, SNMP and others.
IoT protocols are designed for resource constraint environments with arguably the most valuable resource is of course energy.
They are built to deliver data with as little energy expenditure as possible, and as little bandwidth usage as possible.
Load models
Humans are emotionally driven, so the load models regarding the internet of people should have a much greater room for change.
IoT tends to be more steady state and procedural.
Volumes
IoT traffic tends to be much greater than that of IoP.
This is quite straightforward, as the number of devices connected to the internet grows much faster than the human population.
But another reason for that is that connected devices use the internet more frequently during the day than humans.
Flows
Internet of people protocols are typically ‘client-server’ type of protocols, and most of the time the virtual user initiates a request and waits for a response.
In IoT the flow is more bi-directional, in other words, the IoT client may receive data or commands from the server at given points in time, depending on the protocol in use.
In LoRa for example, in the most energy saving configuration, the LoRa gateway might open a ‘downlink’ channel for 15 seconds, and the server must deliver the command to the gateway within that timeframe.
Failing to do so and this command will not be operated until (maybe) the next downlink window.
IoT Load Testing Strategies
The moment your product involves both IoP and IoT, the load testing strategies need to be adjusted.
You’d probably have to run the two in conjunction with one another to simulate a real-life scenario.
Despite the fact that the two flows might use different computational resources, both of them end up using the same data lake or storage.
So the main strategy would be to run an emulator(s) for all the IoT devices at the background of the test, and then to run the IoP flow in the foreground of the test.
In case your IoT flows contain a ‘downlink’ element, this would also have to be part of the emulator’s capacity, and your load test would have to be able to report the time it took for the downlink to reach the emulated gateway.
Since the typical IoT work involves much larger volumes of virtual users, and these virtual users tend to be idle most of the time, the costs of running so many emulators might be high, so your emulators better be designed for this purpose with cost effectiveness in mind.
In a previous article, I illustrated how Locust and JMeter can be used as a background emulator.
On the analysis part, correlating between IoT and IoP might be quite complex.
So after your load test is done, take your time to thoroughly examine all the data available to you and think carefully about pieces of data that are missing from the analysis.
In one of my testing bouts for example, I saw that the overall response time in one of our requests was quite good, around 1.2 seconds.
However, examination of the time series has shown that the response time had increased steadily during the test, from .2 seconds all the way up to 2 seconds.
This was due to a database competition between IoT and IoP which caused ever increasing latencies, and as a result we had to redesign our database architecture.
In the below images you can see the response time distribution at the top, and at the bottom, you can see the steady increase in average response time over time.
On A final note
IoT containing software is very challenging to develop and test, and load testing is no exception to that.
The basics of performance engineering, ie, load modeling, careful analysis, data-driven decision making, and continuous end-to-end testing strategies will lead your efforts towards better load testing even when testing such complex systems.
RedLine13 allows amazing levels of flexibility, both in its design and in its pricing model.
As such, it allows you to experiment in different ways to efficiently simulate your IoT flows.
While other tools offer a “virtual user hour” based pricing model, in RedLine13 you pay for the infrastructure, such that you’re not only incentivised to deliver efficient solutions, but your organization is rewarded for that.
I was able to run simulators in conjunction with JMeter test flows in only a fraction of the price that I’d likely pay if I used other vendors.
Try RedLine13
You can sign up and try RedLine13 today for any load testing.