Video games are more popular than ever today. These flood our newsfeeds, drive social interactions and give us boatloads of fun for hours. There is a myriad of game genres to choose from, whether these are point-and-click adventures, simulations, virtual reality games, and many more. Especially with the COVID-19 pandemic, the number of users of these games has exponentially increased. According to Statista, the number of gamers increased to 201.9 million in 2020 from 188.2 million digital gamers in 2019. It is further projected to rise to 213.3 gamers by 2025.
Amidst this exceeding popularity, gaming app testing has become of increased importance to promise flawless experiences for gamers.
What is game testing and why is it crucial?
Game testing, a crucial subset of game development, is a software testing process that enables developers to discover, document, and fix software bugs to prevent their interference with the gaming experience. This assists in controlling the quality of the game by creating optimal performance across distribution channels.
Importance of gaming testing
Video games today are highly interactive with complex gaming mediums and coupled with several users consuming games over multiple platforms, which make these more susceptible to vulnerability, unexpected errors, and glitches. These might go unnoticed and unresolved without a proper testing mechanism. Hence, testing is crucial to avoid criticisms from gamers, losing out on significant sales due to poor experiences. The primary importance of game testing includes:
- Quality assessment — Every game developer wishes to deliver the best user experience to customers. Hiring a qualified game tester is one of the easiest ways to achieve this. A tester will identify the loopholes efficiently and identify the probable defects.
- Video game ranking — Game testing can directly affect the ranking of a video game. When a new game is to be launched, testing profoundly affects how customers view it. Again, when testers are employed, they generally consider a detailed list of the parameters prior to giving the final verdict.
- Identify improvement areas — Most video games are subject to frequent upgrades during their time in the market. Most are also thought of becoming series and launching sequels. Game testing allows developers to identify the areas that must be improved to give the games a long life in the market.
What are some of the bugs that drive game testing?
Some of the most common game bugs include:
- Interface bugs: These occur as game graphics are misinterpreted where the game elements are at the incorrect place, and the space allotted for the text doesn't fit appropriately.
- Technical bugs: These refer to the situations when the game doesn't work desirably due to unstable internet or network connections.
- Localization bugs: The errors found in the untranslated strings or texts constitute localization bugs.
- Performance bugs: Instances like slow performances and the game's frames per second (FPS) decreasing remarkably on high-end segment devices during a character attack animation are some of the performance bugs.
- Compatibility bugs: These refer to issues that occur when the games don't function appropriately in different devices or OS despite being built to support all.
- Balance and logic bugs: these bugs don't allow gamers to complete tasks.
What are the consequences of not fixing bugs?
Let's take a look at the impact of these bugs:
- The bugs directly affect the apps' popularity and acceptability, and ratings on app stores.
- Bugs can negatively impact the business by hindering the users' investments in those apps and also impact the revenue generated.
The risk factors of launching games in the market
When game testing is executed, developers primarily focus on factors like errors, bugs, bottlenecks, coherence, and inconsistencies. Apart from these, there exist other risks that game testing can help discover:
- Does the game deliver an exceptional experience to the target audience
- Does the software showcase a compelling player-centric design
- Is there a fun factor in the game
- Is the game easy-to-control
- Is the visual design appealing to the audience
Challenges faced by game testers
- Direct screen access
Most mobile games utilize direct screen access, which is performed by bypassing the OS-level tasks with the help of the various APIs and software frameworks like OpenGL and ActiveX. As a result, the conventional frameworks for mobile automated testing become redundant when mobile games are concerned. The automation only makes use of XY clicks as there is no access to the object-level data.
- Social media integration
Mobile games are connected to social media platforms more than ever today. This urges the game testers to pay more attention as social media can be both constructive and destructive. If implemented and used effectively, social media can help connect like-minded gamers, drive exchanging ideas, and enrich the user experience. Similarly, social media integration can bring negative consequences as well. Hence, testers need to be cautious while performing social media integration.
- Multiplayer feature and multiple locations
Testing for the multiplayer feature can be a significant challenge for the testers, especially when players are in very different locations and far apart from each other. Mobile games with multiplayer features are often difficult to test and debug and can also consist of problems that are difficult to fix. To resolve this challenge, the game testers need to check mobile games for their robust design.
- Game engines
Choosing the best engine for a game is no doubt a challenge for the developers, but it is often more difficult for QA to be prepared to test any range of engines they are given. They have to consider all the specifics of each engine and seamlessly apply different types of game testing to check it thoroughly.
- The performance of gaming apps matters more than any other app
Performance is one of the key drivers of user experience, and real performance can only be monitored for using real hardware. Frame rates also act as an important factor with the richness and diversity of mobile devices where every new model counts. Further, most games utilize sensors or other hardware features which are not consistent from one device to another. If this is left unchecked, it causes several negative effects on the overall performance.
- Diverse tools used for game testing
The range of tools, techniques, best practices, and frameworks leveraged for mobile game testing are very extensive. But certain aspects are always executed in the same way by the developers. Despite different game engines, more than 90 percent of game developers utilize some form of image recognition. This enables testing to be as easy as capturing a screenshot and analyzing it automatically or manually. It can also be completely gameplay where the game is driven by test scripts and output is logged for further inspection.
What is a game development cycle?
Before delving into the types and processes of game testing, let us first take a glance at the game development life cycle (GDLC)
1. Concept development — In this first phase, the idea or concept of the game is developed, where a document is prepared that details the concept.
2. Preproduction — This phase involves the concept finalization and creating the concept document. Besides, three other documents are created, which are the :
- Game design document
- Technical document
- Project plan
This phase also includes the designers building the prototype based on the decided concept art, gaming levels, and other details.
3. Development — Based on the prototype built, programmers start coding the game. Artists begin working on modeling, textures, and animation.
4. Alpha — In this stage, a workable product with all features and complete gameplay is developed. It uses temporary assets and placeholders, becoming the first milestone in the development process.
5. Beta — The game is completely developed with all assets in this phase.
6. Code freeze/gold — Each and every bug identified in the beta stage is solved, and the game is ready to be released in this phase.
7. Release to manufacture — In this stage, the game is released to the manufacturer.
8. Patch/upgrade — Post the game's release, the next step is maintenance. When required, the team can release a patch to solve the bugs, and further upgrades are released in case new features are added to the game.
Also check: How to Launch High Performing Mobile Gaming Apps
Types of tests and techniques for game testing
1. Combinatorial testing — This is a technique of experimental design leveraged for commercial software testing and for generating test cases. Applying combinatorial testing to game testing increases the efficiency of test execution and delivers better quality, lower costs, and better phase containment.
- Each possible combination of values of the parameters is covered using this test
- Systematically generate combination for testing
- Identify distinct attributes which can be varied either in data or configuration
2. Clean room testing — This is a software development process that intends to develop gaming software with a certifiable level of reliability.
- Programming commences after the formal specification
- Combines mathematical reasoning, statistical reasoning, and design refinement during test case generation and testing
- The key goal is to produce a minimal defect in the software
3. Functionality testing — This testing mode involves identifying bugs and other game errors that might affect the user experience.
- Determining whether the application is working according to the specifications.
- It is a complex method under the black-box testing category.
- Requires more time to execute as testers look for gameplay issues, graphics issues, and audio-visual issues
- Validates whether installation proceeds smoothly, the app functions in minimized mode, allows social networking options, supports payment gateways, and much more.
4. Performance testing — Evaluating the performance of gaming apps on different parameters like scalability, stability, speed, and responsiveness assists real user conditions for varying levels of traffic and load, which is essential for building a robust gaming app. Some of the parameters checked during mobile app performance testing are:
- Response time on client and servers, transaction completion times, peak load performance, network coverage, low memory, reliability, and many more.
- Battery consumption should be optimal, and game responses must be stable under varying loads.
- Response time across different network types like WiFi, 2G, 3G, and 4G. It also helps check connectivity between mobile devices, data centers, or the cloud, and monitor peak times, jittery connections, duplications of data, packet loss, and data fragmentation.
5. Compatibility testing — This testing type assists in finding out whether a game is functioning correctly, not concerning the hardware, software configuration, and graphics with which the device is built. This is one of the most important tests that evaluates whether the gaming application works across different devices and screen sizes without compromising the quality of the user's experience.
- Validates the user interface of the app by comparing its design, text, and functionality across different screen sizes
- Checks the performance of the app with different OS, browsers, and devices
- Ensures the stability, workability, scalability, and usability of the app on multiple platforms
6. Soak testing — This is an automated game testing technique that involves leaving the game running for a long time in various modes of operation, for instance, idling paused or at the title screen. Soaking can help identify memory leaks or rounding errors.
7. Tree testing — Such a testing technique helps organize the test cases and puts together the selection of a proper set of tests that are better suitable for the given set of code changes.
- Can be conducted prior to designing page layouts or navigation menus
- Enhances overall understanding of the complex features in the game
- Monitors potential deviations, especially when functions interact with other game rules, functions, and other elements
- Doesn't require sketching wireframe or preparing content for testing and only requires tasks and trees, i.e., instructions and menus
8. Play testing — This is a game testing technique where the testers can play the game as real users and analyze the quality of the game application. They can play the game to verify the functional workflow as well as non-functional aspects of the game like the entertainment value, difficulty level, level design, and many more.
- Focuses on judging the game instead of identifying bugs in the app
- Checks whether the game is properly structured and character-centric
- Verifies the game's storyline, challenges, and fun elements to evaluate if the game is innovative, engaging, and player-centric.
9. Recovery testing — For software, recovery testing helps check how well the app can be recovered from crashes, hardware failures, and other similar failures. The app is forced to fail, and later it is evaluated how it recovers from the failure conditions and the environment.
For instance, while the gaming app is running, suddenly restarting the gaming console and check the validation of the data integrity.
10. Regression testing — The regression testing is executed to retest the unchanged parts of the software. The test cases are checked to analyze the working of the previous functions of the app and if the new features or changes have not caused any new errors or vulnerabilities.
- Re-run the previously conducted tests
- Assists in comparing the previous results with current results and identifying errors, if any
- Critical to quality control
- Saves time by detecting bugs at the initial stage
11. Ad hoc testing — This is an unplanned testing technique that is generally utilized for breaking down the system where testers randomly test the app without test cases or any documents.
- Doesn't follow any structured way of testing; randomly executed on any part of the app
- Primarily done to find defects using random checking
- Achieved using error guessing testing method
- As the defects are not mapped to test cases, it is difficult to reproduce the defects
12. Load testing — This is a type of performance testing for determining the system's performance under real-time loads. Load testing helps to find out the reaction of an app when multiple users are simultaneously using the app.
- Determining if the current infrastructure is sufficient for the smooth running of the game
- Checking the sustainability of the app with the peak user load
- Accommodating strategies for performance management
Difference between game testing and software testing
Though game testing and software testing may be fundamentally similar as they involve testing code to deliver high-quality, working software, there are a number of significant differences in both.
Unlike game testing, software testing uses automation scripts developed pre and post-production of test cases. For setting up automation scripts, testers utilize many tools and frameworks in the software testing process.
Unlike software testing, there are a wide number of areas in game testing, especially in the case of mobile game testing that require more focus. Some of these aspects are:
- Graphics are a crucial factor in engaging the users to a particular game. Gamers usually love being involved in high graphics supported by the device and accompanied by the gaming engine. In fact, today, no game can thrive in the market without being graphically excellent. The performance metrics aren't a single parameter in this case but a collection of many that every player keeps in mind. For instance, battery drain rate—if a mobile game is causing battery drain at an unusual rate, players opt-out of such games.
- Another factor is the GPU of the mobile phone that is used most. The GPU might be running at the highest clock rate, and hence if it isn't implemented appropriately, the game can slow down the device due to excessive heating. Hence the testers should test the games on multiple devices before zeroing on the supported devices.
- Additionally, game testers must run the game on different devices and log vital device parameters like CPU consumption, device temperature, GPU consumption, data consumption, GPU consumption, and much more.
Very often, specific frameworks are required for functional testing. Several elements of the game design and behavior need to be tested manually as automation is not the most suitable option. Even if it is deployed, it needs substantial effort and test automation tools to generate the test scripts because game test use cases are usually beyond the simplicity of searching static elements on the screen.
Usually, it requires you to embed an agent into the test script for identifying objects and testing if specific actions are triggered within a game. These factors add complexities to game testing as compared to software testing.
- In the case of software testing, we require qualified professionals with Software Development Engineer in Test (SDET) certification. However, for game testing, there is no specific requirement for formal qualifications. While, in most cases, niche specialists execute the tests, gaming enthusiasts are often called to test new games.
- Software testing focuses primarily on UI/UX, functionality, security, and system, while game testing focuses on realism, AI, and multiplayer.
How to test a game?
The generic workflow of testing a game includes the following steps:
- Gather the requirement: One must collect and understand the details such as storyboards, characters, architecture involved in the game, and the concept behind the game, its rules, and stages.
- Prepare the strategy: It is essential to make the decisions and document details like the required timeline, number of testing cycles, testers, in-scope, out-scope, testing types, service level agreements, risk-based test analysis, and mitigation, defect logging process, and much more.
- Design test cases: One must consider both the positive and negative test cases for mobile games as well as desktop games. Few effective ways to test gaming apps are critical path testing, exceptional path testing, and basic black box testing techniques.
- Execute test cases: In this step, test cases are executed to uncover bugs. For even better results, alpha testing, beta testing, and respective age group testing are executed.
- Record the results: Every aspect is recorded in the form of videos and screenshots. This helps achieve a better perspective of how the application is being used. These details help analyze the behavior of the app.
- Maintain the defect log: This action helps record, review, prioritize, categorize, and efficiently track the bug.
HeadSpin and game testing
HeadSpin and its capabilities have percolated well into the gaming industry, allowing the companies to deliver flawless gaming applications for the gamers.
Prominent social gaming developers have leveraged the HeadSpin platform to solve key challenges, including:
- Hike in the number of concurrent players during the pandemic situation
- Geographically distributed user base
- Dynamic and difficult-to-automate game user journeys
- Need for quick ramp-up of testing infrastructure
As a solution to these challenges, HeadSpin is able to deploy dedicated Android, and iOS devices on HeadSpin hosted cloud across different locations in a short time frame.
Further, the HeadSpin platform is capable of identifying multiple factors that affect battery drain, which is a significant concern of game developers and testers. This helps improve the overall playtime for the users of lower-end devices.
Also check: Fundamentals of Test Harness
- Deliver test automation for the game engine used by the clients
- Deliver required infrastructure to the clients to help them better understand user experience across major market geographies
In addition to the HeadSpin platform, the company's add-on product, Create Your Own Lab (CYOL), enables businesses that have their own devices to connect to the HeadSpin platform and run automated tests on products extensively.
Other benefits that CYOL offers include:
- Allowing reverse bridging of client devices, helping client companies save their investment and also leverage HeadSpin Platform. HeadSpin offers the industry's only reverse bridge solution that allows you to connect your devices to the HeadSpin Cloud through local machines, and the test device is not required to be physically plugged into the HS appliance. You can run the tests on your devices manually or automate through the HeadSpin Platform. The platform then captures session data and analyzes the session data to get insights. simultaneously to deal with other existing challenges
- Help setup automation from processes to training in client's manual labs
As the popularity of gaming continues to increase, the focus on testing will increase too. Therefore, every gaming business today concentrates on deploying efficient testing methods and strategies to deliver the right gaming experiences for its customers. It is necessary to carefully follow the game development life cycle at the beginning and consider the various types of testing to ensure building the game just as your customers would want it to be. Moreover, game testing is a repetitive process that must be executed to identify and eliminate newer defects and bugs with every new release of the game.
1. How to write test cases for games?
There are some crucial areas to focus on during testing a gaming application that directly impacts the performance compatibility and user rating. It is hence important to write the test cases based on these factors:
- User interface and functionality
- Usability and gaming experience
- Graphics performance
- Multiplayer features
- Social media integrations
- Security and other liabilities
2. What are some of the most popular game engines across the globe?
The most popular game engines are:
Unreal Engine —This is one of the most popular and extensively used engines, owned by Epic Games. It is a game development multi-platform engine designed for businesses of all scales to help use real-time technology for transforming ideas into engaging visual content.
Unity — This is an easy-to-use game engine that caters to multiple platforms with regard to the interactive 3D gaming experience.
Amazon Lumberyard —Lumberyard is Amazon's comprehensive offering, which is a 3D game engine designed to build games and fan communities. This engine offers VR preview mode, visual scripting tools, and Twitch integration.
Godot — This is free and open-source through the MIT license that requires no royalties, no subscription fees, and no hidden fees. This engine is very efficient for making both 2D and 3D games.
3. What is the role of audio effects in mobile game testing?
Music and sound effects play a vital role in the overall experience of a game. These help project the likings and feelings of the users and build a specific atmosphere to enhance the overall aspects throughout the playthrough. As a result, the audio effects should essentially be monitored and tested for audio glitches, inconsistencies in sound effects, and other bugs that can negatively impact the user experience.
4. What are the Kleenex test and focus group test?
The Kleenex test is a one-off testing option in which a pair of players is given the task of testing a game so that developers can observe their initial reaction to the gameplay. Such testers play the game only once.
On the other hand, focus group testing is an open discussion with a sampling of players in the demographic section you wish to reach to understand their experience after playing the game.