Making Your Appium Tests Fast and Reliable - Part 8(Mocking External Services)

This article is the eighth in a multi-part series on test speed and reliability, inspired by a webinar I gave on the same subject (you can watch the webinar here). You might also want to check out the previous episodes on Test Flakiness, Finding Elements Reliably, Waiting for App States, Dealing With Unfindable Elements, Setting Up App State, Advanced Capabilities, and Disabling Animations.

Appium tests are usually written as end-to-end tests. In other words, they exercise the app from top (the UI) to bottom (backend services, which most apps employ). There is certainly value in this kind of testing. After all, your users are doing the same thing! Sometimes, though, what we care about with an Appium test is not whether the backend service works, but whether the App Under Test itself works, on the assumption that any data retrieved from an external service is valid.

The sad truth is that the Internet is not always very reliable. Requests made by an app to an external service might fail for a number of reasons, many of which would have nothing to do with the app itself. It could be a "stormy day" when it comes to "Internet weather", and requests could time out. Or the version of the backend service used for testing might not be able to handle the load of so many requests if you're running a lot of tests at once. The list goes on.

Check out: A comprehensive guide to parallel testing

Luckily, it's possible to ensure quality for both the backend service and the mobile app in a much smarter way. The very fact that the Internet lies between these two components is the key: most backend API calls are just HTTP requests. This means that the backend service can be tested "end-to-end" by generating HTTP requests from something other than the app. This is called API Testing, and it's a much faster way to test your backend service than using Appium! Similarly, all the app UI tests care about is the data that comes over the wire from the backend service: it doesn't actually matter whether it's "real" data or not, as long as it's an HTTP response of the appropriate form, with content appropriate to the test. So, rather than having our app make calls to a real running API service, we could have it make calls to a "mock server" instead!

What are the benefits of using a mock server?

1. Speed. Because the mock server can live on the same network as your device, API calls become blazingly fast.
2. Reliability. The mock server is a very simple piece of code which just replies with canned API responses. There is very little risk of it failing. Because there is also no longer an actual Internet route in between the app and the API service it relies on, a huge source of unreliability is taken out of the picture. If the backend service itself is unreliable, the benefits increase (though that service should obviously be tested independently, so any unreliability can be addressed by the owners of the service).
3. Flexibility. A mock service can even be used in the development of an app before or during the development of the backend service, allowing for greater flexibility and independence between teams.

Also check: A Comprehensive Guide to Cross-Platform Mobile Test Automation Using Appium

Using a mock server does have some downsides, however:

1. Maintenance. A mock server and the canned responses do have to be maintained separately of the service itself.
2. API Drift. Using a mock server raises the danger that what your app is being tested against is not the same as the API the backend service is actually using. The mock server and the backend service must be kept strictly in step with one another. A good practice is to have the owner of the service also provide a mock of the service for use in testing by consumers of the service.
3. App and Test Complexity. You will have to build a way for your app to select whether it makes its service calls to the real service or to the mock service, based on whether the app is being used for testing or not. You will also have to augment your tests with information about what the mock server should do, and coordinate with the app to make sure it's connecting with the mock service.

While these downsides are real, the speed and reliability gains are definitely worth it for any sizable build. So much for an introduction! Let's see how we would go about implementing this kind of strategy. Here's what we need to get it all working:

1. A mock server package of some kind (for example, the aptly named Mock Server for Java and JS).
2. Changes to our app code to allow the app to connect to the mock server and not the real API server. This can be accomplished in a variety of ways, for example by having a separate build for testing which has a host and port hardcoded for the mock server, by passing startup arguments to the app to tell it where to look for its API endpoints, or by adding a new view to the app that allows setting of server details via text fields in the app UI itself.
3. The ability to start and stop the API server before and after our testsuite, or before and after certain test classes (depending on our needs). Of course, it needs to be running on the same host and port we specified in our device.
4. The ability for the app to talk to the mock server (this might mean ensuring the app and the mock server are on the same network).
5. Appropriate "expectations" or "scenarios" we can set on the mock server. If we roll our own mock server, we can simply hard-code these responses in request handlers. If we're using a mock server package, it often comes with its own idiosyncratic way of specifying how the server should behave. Most often, we tell the mock server how to match against requests from the app, and how to respond based on the shape of the request.
6. (Maybe) Depending on the mock server project we choose to incorporate into our build, it might also come with features that allow the same requests to generate different responses based on the "scenario". For example, we could have two different login scenarios that involve the exact same requests, one of which responds with a successful response and one of which responds with an error about e-mail formatting. In this case, we'll also need a way to uniquely identify requests from a particular device, so that requests from a device running a different test at the same time don't result in the wrong response. The best candidate for this unique id is the device id, which must be available from both the app code (so it can be sent in a header to the API server in all requests), and in the test code (so that the appropriate "scenario" can be selected in advance of any given test). If you want a good description of how to implement this for React Native apps specifically, check out the slides from Wim Selles's SeleniumConf India 2018 talk.

Putting all of these pieces together, the actual test flow would look like this (in pseudocode):

Before:
    - Start mock server at certain port
    - Start Appium session with app (connected to server)

Test:
    - Set mock server expectations
    - Use Appium to drive app, which makes calls to mock server under hood
    - Verify app behaves properly given mock server responses

After:
    - Quit Appium session
    - Shut down mock server

The Mock Server library I linked above has the ability to do all of these tasks with ease. The server can be run from the command line, as part of a maven plugin, via a Java API, etc... We can set expectations in our Java code (or other language code) using a client that comes with the library. What does setting expectations look like? Here's an example that shows how we might mock a login request:

new MockServerClient("localhost", 1080)
    .when(
        request()
            .withMethod("POST")
            .withPath("/login")
            .withBody("{\"username\": \"foo\", \"password\": \"bar\"}")
    )
    .respond(
        response()
            .withBody("{"\status\": \"success\"}")
    );

Running this command at the beginning of a test (or test suite) directs the mock server to respond with a success JSON response to any login attempt with username foo and password bar. With this kind of setup, all we need to do is direct our Appium driver to enter these values into the login prompt of our app, and the mock server will receive the request and respond with the appropriate response we've entered here. As we mentioned earlier, it's important that both the request and response match the form of the real API service, otherwise we'll either run into failures or test the wrong conditions.

Read: Simulating Different Network Conditions for Virtual Devices

One difficulty in running mock API servers is encountered when utilizing a cloud service. One of the main benefits of mock servers is speed, and putting the Internet in between the app (running in the cloud) and your mock server (running locally) works against this benefit. Still, because the mock server doesn't actually have to do any work, it will be faster than a fully-fledged API server, and is worth considering even in cloud testing environments. All in all, the benefits of using mock servers are pretty compelling, and help you to reduce one of the most common types of instability in your test suite--relying on remote or 3rd-party dependencies in the form of external services.