Capturing Network Traffic in Java with Appium

April 17, 2019

Wrapping up a short series on capturing device network traffic, I'm happy to present mitmproxy-java, a small library which allows convenient access to network requests of devices made in the middle of your test runs. It has the following features which I was unable to find using other methods in Java:

Starts the proxy server as a background task.
Allows for passing in a lambda function which gets called for every intercepted message, allowing you to manage the recording of data any way you see fit.
Allows for modifying the responses from the proxy, so you can inject arbitrary data and states into your app.
Captures HTTPS traffic even when ip addresses are used instead of host names.

The first three advantages come from the wrapper code in mitmproxy-java which is basically a Java version of the great Node.js module I found for the same purpose: mitmproxy-node. The last bullet point comes from the use of mitmproxy.

Traditionally, the testing community mostly seems to use Browsermob Proxy, but I found it has not been maintained recently and can't support Android emulators due to the issue with HTTPS traffic and ip addresses. I'm hoping that people will be able to find mitmproxy-java as a suitable upgrade.

Remotely test and debug key workflows for your app as if the device were in the palm of your hand. Learn more.

But please help! I put a lot of work into it but I'm not a Java expert. The way I currently handle exceptions isn't friendly. Hop onto github and submit pull requests or make issues if you run into trouble. If the community is supportive, we can improve it further.

Oh, this should work for Selenium too, if you set up the browsers to proxy correctly.

Setup

For those just tuning in, see the past two articles on capturing network traffic to learn about what we're doing and how it works:

Those two articles also go through the setup needed for configuring devices, this post will focus on setting up mitmproxy-java and how to write the Java test code.

While mitmproxy-java will start the proxy server for us programmatically, we need to install mitmproxy ourselves, just like we did in the previous articles. Make sure to install with pip3 since installing with other methods, misses some python dependencies which we need.


sudo pip3 install mitmproxy

Don’t Rely on iOS Emulators & Android Simulators. Test on Real Devices.

When running mitmproxy-java, we need to supply it with the location of the mitmdump executable. mitmdump is installed automatically when you install mitmproxy and is a commandline version of mitmproxy which isn't interactive and runs in the background. Let's get that location and make a note of it for later.


which mitmdump

For me the output is /usr/local/bin/mitmdump.

Test and monitor websites & apps with our vast real local devices across the world. Know more.

Next, we need to install the Python websockets module. the way mitmproxy-java works, is it starts mitmdump with a special Python plugin which is included inside the mitmproxy-java jar. This plugin runs inside mitmdump and connects to a websocket server hosted by mitmproxy-java. The Python code then transfers request/response data to the Java code over the websocket.


pip3 install websockets

That should be all the setup we need on our host machine, now on to the actual test code.

Writing a Test Using mitmproxy-java

Include the mitmproxy-java jar in your project. The library is hosted on Maven Central Repository.

Add the following to your pom file:

Or in your build.gradle file, for Gradle users:


compile group: 'io.appium', name: 'mitmproxy-java', version: '1.6.1'

You can now access two classes in your test code: MitmproxyJava - The class which starts and stops the proxy. InterceptedMessage - A class used to represent messages intercepted by the proxy. A "message" includes both an HTTP request, and its matching response.

Read: Controlling Appium via raw HTTP requests with curl

The constructor for MitmproxyJava takes two arguments. The first is a String with the path to the mitmdump executable on your computer. We got this value earlier in the setup section. The second argument is a lambda function which the MitmproxyJava instance will call every time it intercepts a network request. You can do anything you like with the InterceptedMessage passed in. In the following example, we create a List of InterceptedMessage objects and instantiate a new MitmproxyJava instance. every intercepted message gets added to our list, which is in scope for the rest of the test.


List messages = new ArrayList();

// remember to set local OS proxy settings in the Network Preferences
proxy = new MitmproxyJava("/usr/local/bin/mitmdump", (InterceptedMessage m) -> {
    System.out.println("intercepted request for " + m.requestURL.toString());
    messages.add(m);
    return m;
});

Notice that we return the message from the lambda function. If we forget to return it, no worries, this is the implicit behavior. If you block or throw an error though, then the message response never completes its journey to your test device.

Also read: Why is Real Device Cloud Critical in App Testing?

You can also modify the response in the InterceptedMessage. Modifying m.responseHeaders and setting different bytes in the content of m.responseBody will result in overwriting the data which the device receives in response to its request.

Now that we've instantiated our MitmproxyJava object, all we need to do is call


proxy.start();

to start the proxy server and start collecting responses. This method call runs in a separate thread. Call


proxy.stop();

to shut down.

Check: Appium for Mobile Testing Infrastructure Setup

The proxy, by default, runs on localhost:8080 just like in the examples from the previous articles. One future feature should be to allow configuration of this port.

That's it!

Here's an example of an entire test for Android and iOS, using mitmproxy-java:


import io.appium.java_client.AppiumDriver;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.ios.IOSDriver;
import org.junit.After;
import org.junit.Test;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.io.IOException;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.*;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;

import static junit.framework.TestCase.assertTrue;


public class Edition065_Capture_Network_Requests {

    private String ANDROID_APP = "https://github.com/cloudgrey-io/the-app/releases/download/v1.8.1/TheApp-v1.9.0.apk";
    private String IOS_APP = "https://github.com/cloudgrey-io/the-app/releases/download/v1.6.1/TheApp-v1.6.1.app.zip"; // in order to download, you may need to install the mitmproxy certificate on your operating system first. Or download the app and replace this capability with the path to your app.

    private AppiumDriver driver;
    private MitmproxyJava proxy;

    @After
    public void Quit() throws IOException, InterruptedException {
        proxy.stop();
        driver.quit();
    }

    @Test
    public void captureIosSimulatorTraffic() throws IOException, URISyntaxException, InterruptedException, ExecutionException, TimeoutException {
        List messages = new ArrayList();

        // remember to set local OS proxy settings in the Network Preferences
        proxy = new MitmproxyJava("/usr/local/bin/mitmdump", (InterceptedMessage m) -> {
            System.out.println("intercepted request for " + m.requestURL.toString());
            messages.add(m);
            return m;
        });

        proxy.start();

        DesiredCapabilities caps = new DesiredCapabilities();
        caps.setCapability("platformName", "iOS");
        caps.setCapability("platformVersion", "12.0");
        caps.setCapability("deviceName", "iPhone Xs");
        caps.setCapability("automationName", "XCUITest");
        caps.setCapability("app", IOS_APP);

        driver = new IOSDriver(new URL("http://0.0.0.0:4723/wd/hub"), caps);

        // automatically install mitmproxy certificate. Can be skipped if done manually on the simulator already.
        Path certificatePath = Paths.get(System.getProperty("user.home"), ".mitmproxy", "mitmproxy-ca-cert.pem");
        Map args = new HashMap<>();
        byte[] byteContent = Files.readAllBytes(certificatePath);
        args.put("content", Base64.getEncoder().encodeToString(byteContent));
        driver.executeScript("mobile: installCertificate", args);

        WebElement picker = driver.findElementByAccessibilityId("Picker Demo");
        picker.click();
        WebElement button = driver.findElementByAccessibilityId("learnMore");
        button.click();
        WebDriverWait wait = new WebDriverWait(driver, 5);
        wait.until(ExpectedConditions.alertIsPresent());
        driver.switchTo().alert().accept();


        assertTrue(messages.size() > 0);

        InterceptedMessage appiumIORequest = messages.stream().filter((m) -> m.requestURL.getHost().equals("history.muffinlabs.com")).findFirst().get();

        assertTrue(appiumIORequest.responseCode == 200);
    }

    @Test
    public void captureAndroidEmulatorTraffic() throws IOException, URISyntaxException, InterruptedException, ExecutionException, TimeoutException {
        List messages = new ArrayList();

        // remember to set local OS proxy settings in the Network Preferences
        proxy = new MitmproxyJava("/usr/local/bin/mitmdump", (InterceptedMessage m) -> {
            System.out.println("intercepted request for " + m.requestURL.toString());
            messages.add(m);
            return m;
        });

        proxy.start();

        DesiredCapabilities caps = new DesiredCapabilities();
        caps.setCapability("platformName", "Android");
        caps.setCapability("platformVersion", "9");
        caps.setCapability("deviceName", "test-proxy");
        caps.setCapability("automationName", "UiAutomator2");
        caps.setCapability("app", ANDROID_APP);

        driver = new AndroidDriver(new URL("http://0.0.0.0:4723/wd/hub"), caps);

        WebElement picker = driver.findElementByAccessibilityId("Picker Demo");
        picker.click();
        WebElement button = driver.findElementByAccessibilityId("learnMore");
        button.click();
        WebDriverWait wait = new WebDriverWait(driver, 5);
        wait.until(ExpectedConditions.alertIsPresent());
        driver.switchTo().alert().accept();


        assertTrue(messages.size() > 0);

        InterceptedMessage appiumIORequest = messages.stream().filter((m) -> m.requestURL.getPath().equals("/date/1/1")).findFirst().get();

        assertTrue(appiumIORequest.responseCode == 200);
    }
}

Full source code for this example can be found with all our example code on Github.