We asked leaders from DoorDash, Fave, and Mercado Libre to share some of the challenges they’ve faced in the performance testing of their mobile commerce apps. See what they said they’ve done to overcome those challenges, their thoughts on the impact of 5G networking, machine learning, augmented reality, and more!
Mobile accounts for over 67% of all e-commerce sales worldwide and it’s expected to keep growing as more people are getting used to shopping on their phones. Nearly four out of five smartphone users have made at least one purchase in the last six months.
Since mobile performance directly affects user experience, retention rates, conversions, and ultimately, revenue, improving it has become top priority for mobile commerce companies globally.
With this in mind, we invited one of our customer advocates, Nithin Satheesh, QA Automation Team Lead at Fave as well as Joshua Alvarado, an iOS engineer at DoorDash, and Juan Felippo, IT Expert at Mercado Libre, to participate in a live panel discussion led by our CEO, Sofia Palamarchuk.
Below is a recap of the discussion, including questions provided by the audience:
What’s One of Your Biggest Challenges with Mobile App Performance?
The three panelists essentially agreed that the biggest challenge they face with mobile app performance involves having the right data, knowing where it’s being tracked and knowing what to do with it. It’s not only important to have a way to capture the performance data, but also understand the story behind it. What requires more attention and what doesn’t?
For Joshua Alvarado, one of the biggest challenges he faces at DoorDash with their mobile app performance project involves data visualization:
“There’s a lot of data, and you can make a lot of pretty charts, but make sure that those charts and that visualization actually tell the right story.”
He added, “Especially as a developer, you’re probably not working on creating charts a lot. But it’s just very important to start to understand that data because if you can log that data, and then make it flow into a visualization, that’s really powerful.”
Juan shared that he and his team at Mercado Libre face a similar challenge. Today, with the growing competition, it’s not enough to check crash stability, he claimed, but nowadays, teams have to monitor a lot more KPIs that directly affect the user experience. He said that the key for this is to have visibility into each metric and how they impact the user. For that, you have to generate a lot of data, so you can plan what to do with each metric. Some of the metrics he mentioned include startup time, device resource consumption, app size, frozen frames, etcetera.
He stated there are several tools that you can use (like Apptim) or if you work in a large company, your team may build a custom solution. He emphasized that it’s important to make a plan for each specific metric once you have that visibility. Also, strive to have visibility over any changes in performance that may occur from one version to another.
Bottom line: Make sure to gain visibility on the most important performance metrics that impact the end user.
In line with Juan and Joshua’s answers, Nithin described delivering a seamless user experience as his greatest challenge at Fave:
“Understanding our customers is always the key factor for success; what they want, what they are looking for… What performance metrics should we capture to ensure we’ve got the user’s attention? How big and brilliant a new feature is doesn’t really matter if we don’t deliver a seamless user experience.”
What Steps Have You Taken to Confront this Challenge?
Joshua found Google’s Data Studio to be instrumental for data visualization. He explained, “We use Firebase Performance Monitoring for app performance tracing and Data Studio is another product offering from Google. It’s an integration that you can just instantly add on to Firebase and BigQuery.”
He noted these tools’ usefulness in being able to see what other companies share on their dashboards regarding their setups. You can see what they highlight in their dashboards and that’s very helpful. He said when they got started, they did so holistically at a high level. And then with each page in the app, they went into more detail, so they could see a story and make sure that they’re just tracking the correct metrics that make sense for them.
Some of the top KPIs they’re tracking today that they’re seeing in his graphs include:
- Response time
- Payload size
- App startup time
- Rendering for certain views
Just as concerned with providing excellent user experiences, Nithin shared some of the things his team is working on at Fave:
“Initially, we were fully focused on the server side. Our developers used profiling tools in Android Studio and Xcode. So it was full dev-centric performance testing to ensure user experience, initially. And then we got an opportunity to inject QA into performance testing aspects.”
“We tried Apptim for capturing the performance-related data and metrics and it was pretty awesome with the comprehensive reports. We eventually dropped all the performance capturing done from the dev side and we do functional testing together with Apptim. We us it to capture all the data and share with our team.”
Nithin shared they not only use Apptim, but also have some tracking in CleverTap and Firebase, as Joshua mentioned.
How Will 5G Change the Mobile Commerce Landscape?
The panel agreed that 5G is already here and the effects can be seen, especially since it is adopted in the iPhone 12, which means it will become the standard moving forward. Joshua had this to say about 5G:
“It will be very, very important to double down on performance with 5G, especially on networking. It’ll be an end-to-end solution here, because on the back end, you’ll be able to communicate a lot faster, but you need to make sure that the client can also take that speed as well and interpret it and render it onto the view very quickly. It will be a whole entity that you need to double down on.”
Sofia chimed in, “It feels like the bar is going to be even higher than today, because we’re going to be able to provide faster experiences through the network, which is sometimes the bottleneck and in many countries, it’s less. But then you need to also meet those expectations from the server and client side as well.”
Nithin believes that 5G will drastically change everything as it will eventually bring more customers online. And that will in turn, bring more revenue to companies. In addition, he shared, from the user experience perspective, 5G will have a lower latency so it will deliver faster results for users and ensure that a seamless interaction happens between the client and server.
Other benefits of 5G, he continued, are that more companies in the e-commerce landscape will start leveraging AI/ML capabilities in their mobile apps. Fave recently adopted AI/ML capabilities into its app to deliver more personalized experiences. It is said that 5G will also be able to provide more accurate location-specific features for apps that use geo-tagging features. Therefore, 5G will be more helpful for teams to scale up and bring better results to their customers.
Sofia shared that she’s interested in learning more about how teams are including AI capabilities in their apps since most of Apple’s machine learning algorithms are being run on-device. This means that more processing is happening on the device which could potentially affect performance and the end user experience.
Juan also agreed that 5G is going to be a game changer, especially in emerging markets such as Latin America, where his company, Mercado Libre, operates.
He shared that the Mobile Economy Latin America Report 2020 forecasts almost 62 million connections across the region by 2025, with almost a 10% 5G adoption rate and an 80% smartphone adoption rate.
5G will also help increase the amount of apps that people download, as many users consider app size and download speed before trying a new app. Juan shared that he read an article which stated that 70% of people in emerging markets consider app size before making a download.
Juan continued, “I think that from the app size to also the network performance, with this new technology, you will have lower latency which will translate to a better product and better user experience. I expect a lot from 5G. I think that in Latin America, we are a bit behind other regions. But I think it will greatly benefit Latin America who has traditionally seen very poor connections.”
How Do You Choose the Minimum Device Set to Test the Performance of Your App?
Joshua was the first to answer this question, sharing, “There’s not exactly a hard rule but I will say when an issue arises, we definitely do our best to prioritize. Based on the severity of the issue, we look through logs, we look to performance. We actually call “dashers” (DoorDash drivers) as well and talk to them to see how widespread this issue is to make sure that they can keep going and doing deliveries and dealing with feedback on the Dash.”
Juan returned to the importance of having the right data to make these decisions. He said his team looks at the device data from their users to know which devices are the most important. He shared, “We have the visibility on the performance of each device that we have on our users. And with that data, we can see if the performance improves or worsens in each version. Also, we take the amount of devices or the majority of devices that the user has and we run our test taking in consideration that data.
Nithin echoed Juan about using data to drive this decision at Fave, “Ours is also a data-driven testing approach. It is practically impossible to test on all devices and OS combinations, so we capture the data from our customers… What is the majority of the devices they are using and what are the majority of OS solutions they’re having? And then we prioritize based on that. All our tests are designed based on that specific data.”
To find this information, Nithin’s team turns to the APM tool Sentry when there is an issue on the client side. Sentry specifies the device fragmentation and OS combination. He added they capture other data from CleverTap so all the details of the devices and the user-specific things are available to them.
Does Your Test Automation Use Machine Learning and Has it Been Effective?
Nithin was happy to take this question first, responding that he hasn’t yet implemented AI/ML capabilities into Fave’s test automation framework, but he evaluated Testim and Test Project. Both of these tools have a self-healing mechanism whenever the most common flakiness occurs as objects change. This helps ensure tests are more stable and minimizes time needed for test maintenance. One of his biggest struggles with UI automation is tests failing after objects change.
Like Nithin, Juan shared his team has not yet begun to implement AI in test automation, but it is a very interesting topic that’s worth taking a deeper dive into.
Will Apps Get Heavier with the Implementation of Capabilities like AR/VR? Will They Change Where They Do the Processing?
The next question for the panelists had to do with whether or not apps will become heavier due to new AR/VR capabilities. Will teams still try to avoid processing a lot on the client side and get most of the computation to happen on the back end? Will anything change because of 5G?
Joshua stepped up to answer from his personal experiences as an app consumer: “With 5G, what I have been seeing is that things have been moving to the server side a little bit more. The cloud is something that’s definitely heavily leveraged. Now with 5G, you can have even faster interaction with that. As far as processing the client, I think it definitely depends on the platform. I’ve seen Apple is taking the approach of doing a lot more client side processing. Even now recently, Siri is being processed on the client side, versus that being done on the cloud.”
“I think it’s just going to be half and half. There’s some apps that are very much heavily reliant on server side processing versus client side, so that makes a lot more sense. But then there’s other apps like Signal, which is very privacy-centric, and I believe that they do everything on the client side. There’s really not much that touches the server side.”
He continued that at DoorDash, he hopes their app doesn’t become heavier or bigger. Something about which they are very cautious is the binary size. But, he stated, with 5G, he doesn’t see there being a big change.
Juan shared an optimistic view of the benefits of 5G and anticipates that app creators will want to take full advantage of the added benefits it offers.
He commented, “Let’s say, for example, a few years ago, you had some devices with less than one gigabyte of RAM. Now you have a cell phone that has almost the same capability as a notebook. So I think that the more resources you have, the more you want to make benefit of. So I think that for some years, we will see a lot of benefit. But at some point in the future, we will implement a lot of technologies to make the best use of 5G.”
Nithin agreed with Juan about users in Southeast Asia also being concerned over app size, “So if it is an e-commerce app, and we’re saying that it is 100 MB in size, people won’t download it at all because we’re not delivering as many capabilities as a gaming app, let’s say. So what matters is what our app delivers. And then mainly depends on the connectivity.”
He commented that Fave tries to keep their app to a smaller size, because people often complain about how long it takes to download or update the app. He’s hopeful that connectivity issues will be rectified with 5G technology and reinforced that an app’s size really depends on what features it offers.
How far do you think we are from adopting AR and VR in mobile shopping experiences?
Nithin pointed out how we are pretty much there already. Companies like IKEA and Airbnb are already using AR to bring the offline experience online, which has especially gained importance for these companies as a consequence of COVID. Some industries are faster to adopt these technologies than others, mostly due to external pressures and the need to innovate to stay relevant.
Without spilling any secrets, Joshua shared that DoorDash is very interested in AR and VR, especially to use them in the Dasher app. They are trying to think of out-of-the-box uses for these technologies beyond the obvious in order to make a dasher’s experience better.
Do You Have Any Practical Advice for Someone Looking to Start a Performance Testing Practice?
For Juan, the best thing to do when starting out is to start small. Don’t try to make a custom solution for performance. Use a tool or platform already available like Apptim to start getting visibility into the metrics.
Once you have that, you can start understanding what the pains are for the users, the general user experience of the application, and then you can get more advanced. He warned,
“Don’t try to tackle the entirety of the metrics that you have; startup, app size, networking, all at first. When you have a basis for one metric, you can move on to the next.”
Joshua liked Juan’s advice and offered more advice to build on top of it. He said, “Before even touching the keyboard and developing anything or integrating any solutions, take a step back and look at it at a high level, and diagram it out. Involve other teams that might benefit from performance of the application and the entire platform.”
He added that accuracy is key here, so go slow, and don’t forget to plan it out.
For those getting started, Nithin believes it’s helpful to I would say, pair up with your developers and collaborate more with the infra side to get a holistic view of the entire application. Break it down and narrow the scope.
Don’t forget to watch the full webinar for more insights from our guests in this panel discussion and check out similar videos on our Youtube channel!
ecommercemetricswebinarSign up for Apptim today and get started with mobile commerce testing for free.
Leave a Reply