Speeding Up Android Screenshots With MJPEG Servers

Previously on Appium Pro, we saw how to stream video from an iOS device, and moreover how to turn that video into an MJPEG stream for any use. In this article, we're going to focus on the utility of MJPEG streaming for Android. The sad fact about Android is that the supported method of retrieving screenshots (adb screencap) is very slow. There are ways to tweak this command to make it faster (for example by using adb exec-out and parsing the raw data from stdout), and this is indeed what Appium does to implement the "get screenshot" command on Android.

Still, even in this best case scenario, Android screenshots can take an average of 0.3 to 0.5 seconds. On my machine, an accelerated emulator was able to generate a screenshot every 0.3 seconds, and I'm using pretty beefy hardware. 0.3 seconds is pretty fast, but may not be fast enough if the reason you're taking screenshots is to perform image analysis, or maybe use Appium's find-element-by-image feature. If you're interacting with an app primarily through screenshots, then 0.3 seconds might be the window within which you need to find an element!

Luckily, there's a faster way, leveraging the mjpegScreenshotUrl we saw previously. This capability directs Appium to avoid its normal "get screenshot" implementation and instead connect to an MJPEG server which is providing a stream of the screenshots. This stream results in a buffer of screenshot images stored within Appium itself, so that screenshot retrieval is basically instant, costing you only the time it takes to make the client call and parse the image data.

Bottom line: if you care about what's happening on the Android screen from a timing perspective, you need to use mjpegScreenshotUrl. The only question is, how do you get an MJPEG stream of your Android screen going? There are different methods, but the easiest method I've found is to use the Screen Stream Over HTTP app. It's a free download, and is also open source in case you want to figure out how it works.

Basically, this app lets you turn on an MJPEG server hosted by the device itself, delivering image frames based on what's happening on the screen. You can configure the app so that it serves this stream on any port, and also direct it to serve the stream over WiFi only or also over other interfaces. Here's how I set it up:

Setting up Screen Stream Over HTTP for a real device

1. Ensure the device is connected to the same WiFi network as your Appium server's host
2. Download and install the app (using ADB, or through the Play Store)
3. Launch the app and grant it permissions
4. Press the "Start Stream" button
5. Take note of the URL shown in the app. You will need the host and port shown here to build the mjpegScreenshotUrl

Also Check: Test your mobile apps and browsers on real Android devices

Setting up Screen Stream Over HTTP for an emulator

1. Download and install the app (I've found apkpure to be a reliable host for this)
2. Launch the app, and navigate to the Settings
3. Uncheck the box that only allows streaming over WiFi (see image below). This is because the emulator doesn't actually connect to WiFi so we need to access the stream using another interface
4. Press the "Start Stream" button
5. Take note of the URL shown in the app. You will need the port shown here to build the mjpegScreenshotUrl. However, unlike for real devices, you will not use the host IP provided. Instead, you'll forward the port to the local interface on the host machine.
6. Run adb forward tcp:8080 tcp:8080 to ensure that the stream will be available on the emulator's host machine (and replace 8080 with whatever's appropriate if you changed the app's default port)

Using the mjpegScreenshotUrl

Once you have the URL from the streaming app, try to load it up in your browser. You should see your screen mirrored there! This is unfortunately not exactly what we need for Appium, since the Screen Stream app embeds the stream itself within an iframe. To get the URL of the actual stream, add /stream.mjepg to the server URL. (You can also load this up in your browser to verify it works).

To ensure Appium uses the images provided by this stream, all you have to do is make that URL the value of the mjpegScreenshotUrl capability! Pretty simple, once the app is installed and running. (Remember, if you're using an emulator, the host should be localhost, because you forwarded the port on your local machine to the port on the emulator via the adb command).

Putting it all together

To showcase the utility of this approach, I've duplicated the timing code from the previous edition of Appium Pro, so we can get a good scientific comparison between tests that don't use the MJPEG server and tests that do. The only difference between the two test cases is the use of the mjpegScreenshotUrl. Since this is the only difference, we'll run both tests just by calling this one helper function:

public long timeScreenshots (boolean useMjpeg) throws IOException {
    DesiredCapabilities caps = new DesiredCapabilities();
    caps.setCapability("platformName", "Android");
    caps.setCapability("deviceName", "Android Emulator");
    caps.setCapability("automationName", "UiAutomator2");
    caps.setCapability("app", ANDROID_APP);
    if (useMjpeg) {
        caps.setCapability("mjpegScreenshotUrl", "http://localhost:8080/stream.mjpeg");
    }

    driver = new AndroidDriver(new URL("http://0.0.0.0:4723/wd/hub"), caps);

    long startTime = System.nanoTime();
    for (int i = 0; i < 100; i++) {
        driver.getScreenshotAs(OutputType.FILE);
    }
    long endTime = System.nanoTime();

    long msElapsed = (endTime - startTime) / 1000000;
    return msElapsed;
}

Basically, it attempts to get 100 screenshots in a row, and returns the amount of time taken on average. We can either run the screenshot retrieval using the normal method or the MJPEG server method, as depicted in the tests below:

@Test
public void timeScreenshotsWithDefaultBehavior() throws IOException {
    long msElapsed = timeScreenshots(false);
    System.out.println("100 screenshots normally: " + msElapsed + "ms. On average " + msElapsed/100 + "ms per screenshot");
}

@Test
public void timeScreenshotsWithMjpegScreenshotBehavior() throws IOException {
    long msElapsed = timeScreenshots(true);
    System.out.println("100 screenshots using mjpeg: " + msElapsed + "ms. On average " + msElapsed/100 + "ms per screenshot");
}

On my machine, using the MJPEG server cut the average screenshot time by more than half, from ~350ms to ~150ms per screenshot. Your mileage may vary, depending on a lot of factors. But it's clear that if you care about shaving off those milliseconds from your screenshot time, using the mjpegScreenshotUrl will be a great strategy.

Have you done anything cool with this feature? Let us know! And meanwhile, you can always check out the full code sample for this article on GitHub.