In this post, we’ll go over the different factors that impact a mobile app’s performance. We’ll also explain how to test mobile app performance in the different ways that matter, tools, and key metrics.
You’ve probably interacted with an app on your phone or tablet that’s slow, takes a long time to load, freezes or even crashes on you altogether.
Frustrating, right?
On the flip side, you can probably think of an app that you love to use because from day one, it’s never given you any trouble.
Or maybe you never paid any mind to an app that works quickly, because isn’t that how it’s supposed to be?
So, what causes one app to be crash-prone and another, fast and reliable?
Whether an app has good or bad performance depends on three combined factors: the backend, the network, and how the app itself runs on the device.
A developer or mobile tester can measure the performance of an application in different scenarios.
For example, they can test for when there’s a concurrency of users on the app at the same time, on different devices (which vary in hardware resources and screen sizes), and multiple networks such as 3G, 4G, Wifi, and more.
The reality is that there are many variables that affect the performance of a mobile application.
Moreover, a user may have a very bad experience with your app and the cause might not even have anything to do with the code or its implementation.
But, by running performance tests for each of these three factors, you’ll be able to identify problems and optimize your app for the best user experience possible.
Keep reading as we’ll cover the different types of tests for each factor, what to measure, and what tools are available to help you along the way. By the end, you should have a great introduction on how to test mobile app performance!
1st Mobile Performance Factor: The Backend
A mobile app’s backend architecture generally involves an application server, a web server and a database.
When it comes to the backend, the things related to performance that are important to know when an app is under load are the server response times, database queries times, and the server’s resource usage.
Using this information, it’s easier to detect issues such as:
- High server response times
- Bottlenecks or breakpoints in the database and application server resources
- Poor implementation of escalation policies
For the last issue mentioned, poorly implementing escalation policies, what is meant by this is that with cloud servers, you can configure beforehand how you’re going to scale your servers if your app is facing a higher load. For example, when you have peaks in traffic.
By running performance tests, you can identify whether your escalation policies are actually implemented correctly and it’s even possible to find that you might be spending more money on servers than what you need, among other things.
So what kind of tests are normally run to check the app’s backend performance? Load tests.
This is when you simulate load on the backend in different ways, whether it be through stress testing, peak testing, endurance testing, load testing, etc.
In general, the objective of these tests is to understand how the backend systems of an app behave and handle a certain volume of concurrent users.
There are several tools that allow you to load test your mobile app. The most commonly used ones include:
- Apache JMeter – the number one open source load testing tool
- Gatling – a developer-friendly, open source load testing tool with scripts written in Scala
- BlazeMeter – a cloud performance testing platform that scales your JMeter or Gatling tests for a greater amount of concurrent users
2nd Mobile Performance Factor: The Network
With regards to the cellular network that the device is connected to, there are two key things to measure: latency and bandwidth.
- Latency is the time that elapses when information is sent through the network (measured in milliseconds).
- Bandwidth is the maximum capacity (the amount of data) that can be transmitted through the network (measured in bits per second).
An app’s performance can vary depending on, for example, whether it’s connected to a 3G network or a 4G network, and unfortunately, this is beyond an app developer or tester’s control.
But, it is possible to incorporate the network during the mobile app performance testing process, simulating the different types of networks and measuring their impact on the response times, both on the server side and the client side.
Luckily, the third factor is something that’s been overlooked for years and we now have ways to incorporate it into our test cycles and gain greater control over it.
3rd Mobile Performance Factor: The Client
For a mobile app, the client is the software that runs on the device, the app itself and it’s very important to know how it will behave there, as it is sharing resources with the other apps installed on someone’s phone and is a huge factor in the user experience.
At Apptim, we’ve seen firsthand, companies lose money due to poor client-side performance.
“We track everything, and we know that each year we are losing $100M a year due to app crashes and screen freezes that are not being caught by our test automation suite…”
-Mobile Team Lead at Major Rideshare company
This means that even if you’re doing functional testing, incorporate automation, or even have CI/CD in place, these bugs could still slip into production that are performance related.
It becomes really challenging thing for businesses to start thinking, “How can we improve this? What type of tests do we need to do early on in our development and testing cycles, so we can eliminate this type of regression earlier?”
So, enter client-side performance testing.
But, it’s tricky… Why?
With client-side testing, the variety of devices, operating systems and screen sizes of a device that an app is running on comes into play.
Today there are thousands of mobile devices on the market which vary by brand, hardware and operating system.
And each year, it becomes more and more difficult to ensure that an app works well on most of the devices on the market because the amount only continues to grow.
This image below shows the different Android versions in use over the past seven years:
And here are the different versions that were used in just 2019:
From these charts, you can see that in order to cover 70% of Android OS’s in 2019, you need to test on at least 6 different devices with OS’s going from Lollipop 5.0 to Pie 9.
Fortunately, unlike Android developers, iOS developers’ lives are a bit easier as the iOS community consists of fewer devices and operating systems.
Testing client-side performance allows you to understand how the app behaves on a device and how it uses the resources that it shares with other apps.
Running tests for client side performance, you’ll also be able to find areas to improve that you do have control over (unlike the network), resulting in a better user experience.
At Apptim, we consider it critical that any mobile app development team integrate client-side performance testing into its test cycle.
There are several areas with different metrics you can gather to understand client side mobile app performance:
Device resource usage (How the app uses the resources of the device it’s running on):
- % CPU
- Memory (Mb)
- Data sent and received by the application (Kb)
- % Battery usage
Rendering (Measures how the app is drawn on the device):
- Render time (ms)
- Frames per second (fps)
- Render lag (janks)
To give you an idea of what would be acceptable or at least a recommendation, in order to achieve a fluid rendering, which is set at around 60 frames per second, each frame of an app needs to be completed in less than 16 milliseconds.
If that’s not the case, the app can create a disruption in the animation and end up freezing.
Of course, it depends on what type of app.
When it comes to gaming apps, the requirements are even higher, but for other apps, you should actually measure how your app rendering times are; how many frames per second it’s drawing. With this information, you can identify if there’s any disruption in the animation and how the experience of the user is.
On the other hand, if it’s excessively drawing the screen, it will require high CPU or GPU which leads to a battery drain. This is why it’s important to be careful with managing the rendering of your app in the device itself.
Errors:
- Exceptions
- ANR (Application Not Responding) in Android
For Android specifically, if an application stops responding, typically a message will appear on the screen with the option to either wait or close the app.
This is generally due to the fact that certain events may not execute in a specific period of time. For example, if you click on an input and there’s no response after five seconds and the app takes it as an ANR, it will show you the message.
Transaction response time (how long it takes to complete an action):
- Server response times
- Database response time
- Network latency
- Screen drawing time
The transaction response time can be measured from the user perspective, but it’s not only what’s happening on the device.
It takes things like the server response time, which can be measured with a proxy, for example. It will also include, if there’s any database call, it will include the SQL time as well. It will also include any network latency and of course the rendering times.
If you think about the time that a transaction takes to be completed, you should consider that there are different parts that add to the transaction time. If you want to improve it, you need to measure each of them and see where the opportunities for optimization are.
For many teams, one key area that they focus on regarding this is the app startup time, or how long it takes to open when clicked on. This is something that you can measure improve over time.
Once you’ve got all of this information, you can uncover critical problems such as:
- Memory leaks: when an app doesn’t free up memory well, it increases until it runs out of available memory and then crashes
- Screen freezes: due to very high rendering times
- Crashes: exceptions that jump in the app and cause it to suddenly close… bummer!
Should We Use Emulators or Real Devices?
We always recommend testing mobile app performance on real devices. Why?
Because that’s when you get the actual performance of your app, if you think about emulators, you’re going to be using emulated hardware, so it’s not true to what will happen on the real devices of your users. That’s why we always recommend using real devices for client-side mobile app performance testing.
This leads us to the next question, what tools can we use for the client-side?
For native apps, there are profiling tools such as Android Studio and Instruments (for Android and iOS respectively).
These tools are of great help when debugging an app and finding the root of certain kinds of performance problems.
In addition to Android Studio and Instruments, we recommend using Apptim, the tool we’ve created to give entire development teams access to mobile app performance information and KPIs, where anyone can quickly and easily run tests.
With Apptim, you can test all of your native apps’ performance with just one tool, instead of one for Android and one for iOS and also compare performance over time.
Test Mobile App Performance as Early as Possible
Now, you may be wondering, “At what point in the development cycle should I start running client-side performance testing?”
If you ask us… start as soon as possible.
If you start testing early on, from the first version of your app, you’ll give yourself more time to understand how it behaves, get familiarized with client side metrics, and run these tests alongside functional ones in each sprint. You’ll also feel more experienced when it comes to how to test mobile app performance, if you’re just learning.
Another advantage of testing early is you can test on different devices at your disposal as well as understand how these metrics evolve over time.
It’s important to understand how the metrics change over time so you can identify what changes to the app alter its performance (find performance regressions) and have more time to remedy degradations.
This helps a developer understand how the decisions they make affect performance and ultimately, avoid disappointing users when they find that the app is running slower than usual. For other tips and tricks to improve mobile performance, check out this post.
We hope this post provided you an overview on how to test mobile app performance!
You can also watch this webinar, where I discussed all of the above and gave a demo of Apptim 🙂
Have any questions? Leave us a comment!
Give Apptim a try today and get started with mobile app performance testing.
It’s 100% free, no credit card required.
Images via unsplash.
mobile performance
Leave a Reply