Check out how Globo improves streaming experience and achieves omnichannel growth with HeadSpin
Close
Strategies for choosing the right mobile devices for testing

Strategies for choosing the right mobile devices for testing

July 29, 2022
 by 
 Rajeev Ranjan Rajeev Ranjan
Rajeev Ranjan

Did you know how much time people spend on their mobile phones today? Well, as per a survey conducted in February 2021, nearly half of the respondents mentioned that they spend 5 to 6 hours on their phones daily, excluding work-related use. Further, 22 percent of the respondents said spending 3 to 4 hours on average on their phones. Of this, consumers spend the majority of their time on mobile apps. Hence, as mobile apps are a fundamental part of our lives and play a key role in how we work, communicate, and entertain ourselves, being downloaded in millions each day across a myriad of devices, assuring the quality of these apps and experience delivered is a top priority. As a result, there’s been an increased focus on mobile device testing, and testing for apps across mobile devices is quintessential now.

What is mobile app testing?

Mobile app testing refers to the process by which apps for the devices are tested for their functionality, usability, performance, and much more.

Mobile application testing can be manual and automated, which helps ensure that the app caters to all the business requirements and user expectations.

Check out: How to Test Your Mobile Apps from Anywhere

Challenges of mobile app testing

Testing apps on mobile devices can often be complex. Following are some of the reasons for this:

  • A wide range of mobile devices — Devices now have different screen sizes and hardware configurations such as a hard keypad, virtual keypad, trackball, etc.
  • Different types of models of mobile devices like Samsung, Apple, Google, HTC, Nokia, OnePlus and much more.
  • Various operating systems, including Android, iOS, and many more
  • Different versions of OS like iOS 5, iOS 15.5, Android 6 etc.
  • Multiple network types including both WiFi & SIM(5G,4G,2G etc.) Regular and frequent updates, where with each update, a new testing cycle is required to ensure that no application is impacted.
Also check: Choosing the Right Mobile App Testing Product - Challenges & Best Practices

Testing on real devices vs. virtual devices

Whether to use a real or virtual device for testing mobile apps mostly depends on what you wish to test.

Real devices

If testing your app’s performance is a priority, real devices are always the most suitable option. Real devices help provide accurate results and allow testing in the same condition as end-users.

Benefits of testing on real devices

  • Test applications across a variety of real user conditions such as network conditions, screen size, battery consumption and much more
  • Assists in recreating the customer user journeys
  • 3rd party transactions can be effectively monitored
  • Ability to assess geolocation, push notification and security features(2FA) on an app
  • Faster test execution as compared to simulators/emulators 

Several enterprises garner the benefit of real device cloud to enhance the efficiency of the testing processes and deliver the best quality apps. It offers a mobile testing environment that has a diverse collection of iOS and Android devices.

Virtual devices

Besides the real devices, QA teams utilize virtual devices for mobile testing, including emulators and simulators.

Emulators are often preferred while testing the external behavior of the mobile device, such as calculations, transactions, etc. On the other hand, simulators are less reliable than emulators and inefficient for debugging in most cases.

Don’t Rely on iOS Emulators & Android Simulators. Test on Real Devices

Device fragmentation and Android versions

As Android is supported by a large variety of mobile device manufacturers, the fragmentation or diversity of versions in use at any given time is more than in iOS. 

How does it affect the users?

  • A large section of customers judges a company’s credibility based on the range of devices and OSs supported
  • More than half of the users will uninstall or not revisit an app due to issues on mobile and visit a competitor
  • The overall experience of the end-users
Test and monitor websites and apps with our vast real local devices across the world. Know more!

Strategies for selecting the right mobile device for testing

As the dependence on mobile devices and apps has increased exponentially, it is an absolute mandate that the app meets user expectations. Under this premise, selecting the right device is a fundamental part of the entire process as the performance and functionality can be varied depending on the mobile device being tested. This is due to the fact that there is a wide range of devices with diverse screen resolutions, processing capacities, or even hardware pieces that are integrated into the apps. 

Let us take a look at specific strategies that can help you determine the right test devices for testing mobile apps.

1. Narrowing the scope

Narrowing the scope and identifying the list of test devices that are a 100 percent fit for the test requires the following steps:

  • Collecting the requirements — It is essential to collect all the requirements for the product and check if there is any device or devices that you want to include compulsorily. 
  • Identifying the market share for different platforms — It is crucial to analyze the market and your potential customers to identify the market share for different platforms easily. Every region/country might have a different market share for different devices and platforms, and from hereon, it is important to leverage the available statistics from trusted sources and identify the devices that the target audience is most interested in. With this, you’ll also be able to target the users more efficiently and smoothly.
Model Market share
Google Pixel 2 3.1%
Samsung Galaxy S21 Ultra 5G 2.8%
SM-G991U 2.7%
Samsung Galaxy S20 FE 5G 2.5%
Samsung Galaxy A32 5G 2.4%
Samsung Galaxy S9 2.4%
Samsung Galaxy A12 2.2%
Samsung Galaxy S10 2.0%
Samsung Galaxy S10+ 2.0%

  Source: Top Android phones and tablets in the United States | AppBrain

  • Establishing testing goals and constraints — Prior to starting the testing process, it is essential to clearly define and understand the testing goals and probable constraints that might hinder the process. Testing goals may include checking the usability, performance, security, network connectivity, functionality, compatibility, and many other aspects.

Once the goals are defined, you should prioritize them and assess the number of test devices.

For instance, if your app is a mobile banking app, security is one of the biggest priorities. Therefore, you will need to test the app on only a few devices since the app’s security doesn’t depend much on the firmware or hardware. On the other hand, if performance is the core priority of an app, testing across multiple hardware and firmware is an important aspect.

Leverage the HeadSpin Solution for Better Performance of BFSI Apps. Know more!

When it comes to the constraints, the budget remains one of the most significant ones. If you wish to test the app on several devices, it will require more time and result in many working hours for the QAs. Therefore, any budget limitations will directly affect the number of devices that you can test on. In this case, one can use the most popular devices for the target audience and get back to the second picks later if required.

2. Identifying parameters for choosing the devices —

  • OS versions:

In the case of the Apple products and their unified hardware and software, it is often simpler as you can consider only the latest and second latest versions to support. While for Android device testing, several OS versions are used simultaneously. For instance, Android 9.0, Pie, Android 10.0, Android 11.0, and Android 12.0 are some of the popular versions existing parallelly, and hence all of these need to be considered. Certain factors to consider while testing on various Android versions include:

  • Determining the precise target audience and their preferred version
  • The countries of residence of the target audience
  • The share of different OS versions across these countries
  • Screen sizes and resolutions

It is essential to ensure that the UI of the app is compatible with different screen sizes and resolutions. Due to a wide range of devices with multiple screen sizes and resolutions available in the Android market, it is advisable to check the statistics to observe users’ devices’ DPIs. These data are crucial when choosing the devices for testing as it helps show the preferences of real users who will be using the app.

  • Hardware

Certain applications might require specific hardware to function properly. In these cases, choosing particular devices that can provide the necessary hardware is essential. 

Further, it is also vital to focus on the processor architecture. While testing, one must ensure that the app works equally on the primary architectures.

  • Software

Certain apps depend on the specific built-in features of the devices, such as biometric support, 5G, and many others. If the app is built around these features, it is a mandate to choose devices that possess the required functionality; else, the app might not be able to deliver as intended. 

  • Considering the target users

Selecting devices based on certain factors critical to your target audience is essential to ensure broad testing coverage. Key data points to consider include:

  • Market share
Android version Market share
11.0 31.65%
10.0 21.92%
12.0 17.54%
9.0 Pie 11.06%
8.0 Oreo 5.9%
7.0 Nougat 2.64%

Source: Mobile & Tablet Android Version Market Share Worldwide | Statcounter Global Stats

  • Traffic volume
  • Popularity
  • Region

Mobile Vendor Market Share North America

Platform Share
Apple 53.51%
Samsung 29.22%
Motorola 5.42%
LG 2.51%
Google 2.2%
Xiaomi 1.53%

Mobile Vendor Market Share Asia

Platform Share
Samsung 24.91%
Xiaomi 18.1%
Apple 17.42%
Oppo 9.34%
Vivo 7.98%
Huawei 6.27%

Mobile Vendor Market Share Europe

Platform Share
Apple 32.35%
Samsung 31.98%
Xiaomi 14.05%
Huawei 8.76%
Oppo 2.21%
Motorola 1.89%

  • Industry
  • Different device configurations (as discussed priorly)
  • Company size

3. Reviewing the statistics of the most popular devices—

It is crucial to review the statistics of the most popular and widely used devices in the global and local market that can assist in narrowing down the list of devices for testing. 

Websites like StatCounter, Statista, and much more constantly update statistics related to operating systems, processors, screen resolutions, and device models that can help choose the devices for testing.

4. Selecting the right amount and variety of devices

The type of tests you’re running often determines the number of test devices required for your app. As manual testing is time-consuming, testers can select a small number of devices from the range of devices popular among the target audience. On the other hand, automated testing allows selecting from a broader range of devices.

  • Manual testing

A high-end model, a low-end model, and a tablet (depending on your app) can make the right combination for manual testing. For Android device testing, a suitable practice is to include a popular device brand like Samsung and begin with a high-end and low-end device. For better brand coverage, add other popular brands like Xiaomi, Oppo, Motorola, and many more. Google Pixel devices can also assist in commencing the Android beta version testing.

  • Automated testing

It is advisable to add a larger mix of OS versions for automated testing to achieve optimal coverage and ensure faster test execution.

  • Test speed v/s granular device selection

Based on the requirement of speed, it might be more effective to focus on choosing the right device than the granular elements of the screen size, OS, and others. However, in this case, choosing the Android and iOS devices that the target users most extensively use is vital to ensure better test coverage.

  • Parallel testing

Parallel testing allows testing multiple configurations of an app at the same time. One can seamlessly run parallel testing on a single app or several components on several platforms and allows to test the app, subcomponents such as OS, browsers, or even both. Customers subscribe for multiple device types to ensure they get the speed in testing and align well with their testing goals and metrics.

5. Reassessing the test devices periodically

With the evolving mobile devices market, new devices are emerging every now and then. Therefore, it is more crucial than ever today to focus on the current market conditions, new models, device configurations, and OS versions while planning the testing coverage. 

For choosing the device versions, evaluating the technical specifications of similar models is helpful. Again, Google Pixel devices are often the first to support the latest version of Android OS for Android devices.

Top selling smartphones 2022

  • Apple iPhone 13
  • Samsung Galaxy S21
  • Honor 50
  • Xiaomi Mi 11
  • Realme GT
  • Infinix NOTE 11 NFC
  • Redmi Note 10 Pro
  • One Plus 9 Pro
  • Huawei nova 9
  • Samsung Galaxy A72

List of best Android devices for testing

With thousands of real iOS and Android mobile and tablet devices today, one can easily. Following are a few best Android devices for testing.

Android
Samsung Version
Galaxy S22 12.0
Galaxy S22 Ultra 12.0
Galaxy S21 12.0
Galaxy S21 11.0
Galaxy S21+ 11.0
Galaxy S21 Ultra 11.0
Galaxy S20 11.0
Galaxy M32 11.0
Galaxy M52 11.0
Galaxy S20 10.0
Older Versions
Galaxy Note 9 8.1
Galaxy S9 8.1
Galaxy S9+ 8.0
Galaxy Note 8 7.1
Google Version
Pixel 6 Pro 13.0
Pixel 6 12.0
Pixel 6 Pro 12.0
Older Versions
Pixel 2 8.0
Pixel 8.0
Nexus 6 6.0
Others Version
OnePlus 9 11.0
OnePlus 8 10.0
OnePlus 7T 10.0
Redmi Note 9 10.0
Redmi Note 7 9.0
Motorola G9 Play 10.0
Huawei P30 9.0

6. Creating a device matrix

With the information acquired till this stage, one should attempt to create a device matrix that contains information related to OS version, model, resolution, CPU, memory, and the associated market distribution data.

7. Look out for similar devices

As the device matrix is ready, the next step involves finding the mobile devices that have similar characteristics and further narrowing down the possibilities of discarding devices that have characteristics already covered by a similar device.

Conclusion

However, the selection of mobile devices is not limited to a fixed list. It depends widely on the specifications and requirements of each individual project. Keeping in mind a particular strategy and the tips mentioned above can nevertheless help refine the final list of devices selected for your app testing. 

Given the extreme importance of mobile devices and their apps today, it is pivotal to ensure that these are thoroughly tested before the market release. Doing so is a major step forward toward improving the brand’s reputation as a developer and building a positive relationship.

Strategies for choosing the right mobile devices for testing

4 Parts