Traditionally, Appium is used to make functional verifications of mobile apps. What I mean is that people usually check the state of an app's UI to ensure that the app's features are working correctly. Functional testing is very important, but there are other dimensions of the user experience that are equally worth checking via automated means in your build. One of these other dimensions is performance. Performance is simply how responsive your app is to the user, and can include a variety of specific factors, from network request time to CPU an d memory usage.
Mobile apps, more than desktop apps, run in very resource-constrained environments. Mobile apps also have the potential not only to create a bad experience for the user while the app is open, but by hogging CPU or memory, could shorten the battery life or cause other applications to run slowly. Engaging in performance testing is therefore a way to be a good citizen of the app ecosystem, not just to ensure the snappiest experience for your own users.
Luckily for Appium users, all kinds of interesting performance data is available to you via the Appium API---well, on Android at least. Building on the wealth of information that comes via adb dumpsys, Appium provides a succinct overview of all aspects of your app's performance via the getPerformanceData command. The client call is simple:
driver.getPerformanceData("<package>", "<perf type>", <timeout>);
Here, <package> is the package of your AUT (or any other app you wish to profile). <perf type> is what kind of performance data you want. There is another handy driver command (getSupportedPerformanceDataTypes) which tells you which types are valid. For the time being, they are: cpuinfo, memoryinfo, batteryinfo, and networkinfo. Finally, <timeout> is an integer denoting the number of seconds Appium will poll for performance data if it is not immediately available.
There's too much to dig into in one edition, so let's focus on a simple example involving memory usage. One problem encountered by many apps at some point in their history is a memory leak. Despite living in a garbage-collected environment, Android apps can still cause memory to be locked up in an unusable state. It's therefore important to test that your app is not using increasing amounts of memory over time, without good reason.
Let's construct a simple scenario to illustrate this, which is easily portable to your own Android apps. Basically, we want to open up a view in our app, take a snapshot of the memory usage, and then wait. After a while, we take another snapshot. Then we make an assertion that the memory usage outlined in the second snapshot is not significantly more than what we found in the first snapshot. This is a simple test that we don't have a memory leak.
Assuming we're using the good old ApiDemos app, our call now looks like:
List<List<Object>> data = driver.getPerformanceData("io.appium.android.apis", "memoryinfo", 10);
What is returned is a set of two lists; one list is the keys and the other is the values. We can bundle up the call above along with some helper code that makes it easier to query the specific kind of memory info we're looking for (of course, in the real world we might make a class to hold the data):
private HashMap<String, Integer> getMemoryInfo(AndroidDriver driver) throws Exception {
List<List<Object>> data = driver.getPerformanceData("io.appium.android.apis", "memoryinfo", 10);
HashMap<String, Integer> readableData = new HashMap<>();
for (int i = 0; i < data.get(0).size(); i++) {
int val;
if (data.get(1).get(i) == null) {
val = 0;
} else {
val = Integer.parseInt((String) data.get(1).get(i));
}
readableData.put((String) data.get(0).get(i), val);
}
return readableData;
}
Essentially, we now have a HashMap we can use to query the particular kinds of memory info we retrieved. In our case, we're going to look for the totalPss value:
HashMap<String, Integer> memoryInfo = getMemoryInfo(driver);
int setSize = memoryInfo.get("totalPss");
What is this totalPss business? PSS means 'Proportional Set Size'. According to the Android dumpsys docs:
This is a measurement of your app’s RAM use that takes into account sharing pages across processes. Any RAM pages that are unique to your process directly contribute to its PSS value, while pages that are shared with other processes contribute to the PSS value only in proportion to the amount of sharing. For example, a page that is shared between two processes will contribute half of its size to the PSS of each process.
In other words, it's a pretty good measurement of the RAM impact of our app. There's a lot more to dig into here, but this measurement will do for our case. All that remains is for us to tie this into the Appium boilerplate and make a real assertion on our app. Here's the full code for the test case we outlined above (also available on GitHub):
import io.appium.java_client.android.AndroidDriver;
import java.io.File;
import java.net.URL;
import java.util.HashMap;
import java.util.List;
import org.hamcrest.Matchers;
import org.junit.Assert;
import org.junit.Test;
import org.openqa.selenium.remote.DesiredCapabilities;
public class Edition005_Android_Memory {
private static int MEMORY_USAGE_WAIT = 30000;
private static int MEMORY_CAPTURE_WAIT = 10;
private static String PKG = "io.appium.android.apis";
private static String PERF_TYPE = "memoryinfo";
private static String PSS_TYPE = "totalPss";
@Test
public void testMemoryUsage() throws Exception {
File classpathRoot = new File(System.getProperty("user.dir"));
File appDir = new File(classpathRoot, "../apps/");
File app = new File(appDir.getCanonicalPath(), "ApiDemos.apk");
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("platformName", "Android");
capabilities.setCapability("deviceName", "Android Emulator");
capabilities.setCapability("automationName", "UiAutomator2");
capabilities.setCapability("app", app);
AndroidDriver driver = new AndroidDriver(new URL("http://localhost:4723/wd/hub"), capabilities);
try {
// get the usage at one point in time
int totalPss1 = getMemoryInfo(driver).get(PSS_TYPE);
// then get it again after waiting a while
try { Thread.sleep(MEMORY_USAGE_WAIT); } catch (InterruptedException ign) {}
int totalPss2 = getMemoryInfo(driver).get(PSS_TYPE);
// finally, verify that we haven't increased usage more than 5%
Assert.assertThat((double) totalPss2, Matchers.lessThan(totalPss1 * 1.05));
} finally {
driver.quit();
}
}
private HashMap<String, Integer> getMemoryInfo(AndroidDriver driver) throws Exception {
List<List<Object>> data = driver.getPerformanceData(PKG, PERF_TYPE, MEMORY_CAPTURE_WAIT);
HashMap<String, Integer> readableData = new HashMap<>();
for (int i = 0; i < data.get(0).size(); i++) {
int val;
if (data.get(1).get(i) == null) {
val = 0;
} else {
val = Integer.parseInt((String) data.get(1).get(i));
}
readableData.put((String) data.get(0).get(i), val);
}
return readableData;
}
}
I heartily recommend throwing some automated performance testing in the mix for your app. With even a basic test like the one above, you could catch issues that greatly affect your user's experiences, even if a functional test wouldn't have noticed anything wrong.
As mobile app testing becomes more and more ubiquitous, the lines between different kinds of automated testing can become blurred. For example, performance testing is becoming an integral part of the development cycle. And for good reason---users of mobile apps have very low tolerance for poorly performing apps. We've already shown how it's possible to use Appium to collect performance metrics for Android apps, and we can do something very similar for iOS.
Performance testing for iOS apps involves the Instruments utility distributed by Apple alongside Xcode. Instruments comes with a number of built-in analyses and measurements. If you open it up, you're greeted with a list of these:
Essentially, these are the various performance measurements it will be possible to initiate using Appium, so keep an eye on this list for ideas about what you might want to measure for your app, and make sure to check out Apple's docs if you want to know more about what each of these do. For our purposes in this newsletter, we're going to choose the "Time Profiler" instrument. But since we're using Appium, we don't need to click on anything in the Instruments app itself. Instead, we'll head to our code editor!
The way iOS profiling works with Appium is with two commands: one to start the profiling and one to stop it and dump the data out to us for viewing. These commands are available as of Appium 1.8, via the mobile: command interface. Essentially, it's as simple as:
driver.executeScript("mobile: startPerfRecord", args);
// here: do some stuff in the app that you want to profile
String b64Zip = (String)driver.executeScript("mobile: stopPerfRecord", args);
// here: convert the base64-encoded zip file to actual file data on your system
We use the mobile: startPerfRecord and mobile: stopPerfRecord commands to signal to Appium when during our script we'd like the profiling to occur. There is one wrinkle, however: for any of this to work, we need to have started the Appium server with the --relaxed-security flag. This is because Instruments can gather data from the system as a whole, not just the AUT. (It's thus a security risk to expose potentially sensitive system information to Appium sessions running from a remote client, for example in the context of a cloud Appium host).
There's also another aspect of the snippet above that I haven't yet touched on: what about the args parameter to these methods? The "start" method takes an argument object with three fields, for example:
HashMap<String, Object> args = new HashMap<>();
args.put("pid", "current");
args.put("profileName", "Time Profiler");
args.put("timeout", 60000);
Here we have:
For stopPerfRecord, the only argument we care about is profileName, which should have the same value as what we passed in to startPerfRecord, so Appium knows which of potentially multiple traces to stop. The other wrinkle here is the return value of stopPerfRecord; what's b64Zip supposed to mean? Well, what Appium is giving back to you when you stop performance recording is actually an Instruments Trace Document, which happens to be a directory under the hood. Since directories are impossible to send in string format, Appium zips up this .trace directory and hands it back to the client script in base64 encoding. To make use of this data, we have to decode it and dump it into a zipfile on our system, with code like the following:
File traceZip = new File("/path/to/trace.zip");
String b64Zip = (String)driver.executeScript("mobile: stopPerfRecord", args);
byte[] bytesZip = Base64.getMimeDecoder().decode(b64Zip);
FileOutputStream stream = new FileOutputStream(traceZip);
stream.write(bytesZip);
At this point, we'll have a nice little trace.zip sitting at the specified location on disk. We can now simply unzip it and double-click it to open the trace file up in the Instruments viewer:
In this Instruments UI, we can dig through the various threads that were active during the profiled portion of our Appium test, and see which routines that thread spent most of its time in (via the stacktrace snapshots taken by the profiler). This can help us to find CPU-hungry areas of our app, which we might decide to offload to a worker thread to improve the user experience, for example. There are all kinds of considerations, and potential avenues of improvement based on the data gleaned from these trace files, but that is outside the scope of this brief tutorial. What's important today is that you've figured out how to capture the data!
Since a common use case might be to profile your app over time, you might consider attaching the zipped trace files to your test report in your CI system, so that if a test fails, you also have some juicy profile data that could help in remediating the test. (There's actually an easy way to send the zip file straight to an asset manager that supports HTTP uploads; check out the Appium XCUITest performance docs for more info).
For the sake of showing a full example, the following is a simple test which lifts the actual app behavior from a different edition of Appium Pro, and simply runs those steps a number of times while bracketed by performance recording. The zip file is then written to disk, just as above, where I can happily open up the report in Instruments.
import io.appium.java_client.MobileBy;
import io.appium.java_client.ios.IOSDriver;
import java.io.File;
import java.io.FileOutputStream;
import java.net.URL;
import java.util.Base64;
import java.util.HashMap;
import org.junit.Assert;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class Edition012_iOS_Performance {
private String APP = "https://github.com/cloudgrey-io/the-app/releases/download/v1.3.0/TheApp-v1.3.0.app.zip";
private File traceZip;
private By msgInput = By.xpath("//XCUIElementTypeTextField[@name=\"messageInput\"]");
private By savedMsg = MobileBy.AccessibilityId("savedMessage");
private By saveMsgBtn = MobileBy.AccessibilityId("messageSaveBtn");
private By echoBox = MobileBy.AccessibilityId("Echo Box");
private String TEST_MESSAGE = "Hello World";
@Test
public void testAppActivity() throws Exception {
// Note: Appium server must have been started with --relaxed-security
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("platformName", "iOS");
capabilities.setCapability("platformVersion", "11.3");
capabilities.setCapability("deviceName", "iPhone X");
capabilities.setCapability("app", APP);
IOSDriver driver = new IOSDriver<>(new URL("http://localhost:4723/wd/hub"), capabilities);
traceZip = new File("/path/to/trace.zip");
try {
HashMap<String, Object> args = new HashMap<>();
args.put("timeout", 60000);
args.put("pid", "current");
args.put("profileName", "Time Profiler");
driver.executeScript("mobile: startPerfRecord", args);
performActions(driver);
performActions(driver);
performActions(driver);
args = new HashMap<>();
args.put("profileName", "Time Profiler");
String b64Zip = (String)driver.executeScript("mobile: stopPerfRecord", args);
byte[] bytesZip = Base64.getMimeDecoder().decode(b64Zip);
FileOutputStream stream = new FileOutputStream(traceZip);
stream.write(bytesZip);
} finally {
driver.quit();
}
}
public void performActions(IOSDriver driver) {
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.presenceOfElementLocated(echoBox)).click();
wait.until(ExpectedConditions.presenceOfElementLocated(msgInput)).sendKeys(TEST_MESSAGE);
wait.until(ExpectedConditions.presenceOfElementLocated(saveMsgBtn)).click();
String savedText = wait.until(ExpectedConditions.presenceOfElementLocated(savedMsg)).getText();
Assert.assertEquals(savedText, TEST_MESSAGE);
driver.navigate().back();
}
}
Ever wondered what goes on under the hood while Appium is starting up, say, an iOS session using the XCUITest driver? Sometimes this is a process that takes several minutes, and sometimes it only takes a handful of seconds. Having a look at the Appium logs can clear up this confusion (maybe you notice that the first time you run an iOS test with a new version of Appium, it spent time rebuilding WebDriverAgent--that makes sense).
Just looking at the logs doesn't necessarily tell you how long Appium spent within each zone of its startup routine, though. You could turn on timestamps in the logs (using --log-timestamp), but then you have to do a bunch of mental gymnastics to develop a picture of the startup flow over time.
For this reason, the Appium team developed something called the "Events API" (or sometimes the "Event Timings API"). Basically, the timestamps of various points of interest in the lifecycle of a session are logged automatically as "events", which can be retrieved at any point during the test.
Previously, it was possible to retrieve events using the getSessionDetails command:
Map<String, Object> details = driver.getSessionDetails();
Map<String, Object> events = details.get('events');
For this to work, you also had to use the eventTimings capability, and set it to true, in order for Appium to decorate the session details response with the appropriate timing information. However, in the W3C protocol there is no route that corresponds to getSessionDetails, and so the Appium team has added its own route, which is supported in the Java client as the getEvents command:
ServerEvents events = driver.getEvents();
What you get back from this call is a ServerEvents object encapsulating the various types of data returned with this API. Let's take a look at the (truncated) raw API response for an example call, so you can see its shape:
{
"commands": [
{
"cmd": "findElement",
"startTime": 1573001594602,
"endTime": 1573001594810
},
{
"cmd": "getLogEvents",
"startTime": 1573001599636,
"endTime": 1573001599636
}
],
"xcodeDetailsRetrieved": [
1573001580191
],
"appConfigured": [
1573001586967
],
"resetStarted": [
1573001586969
]
}
There are two main sections here, one labeled commands, and then a bunch of other names consisting of timestamps. All commands sent to the Appium server are considered events, and their start and end times are tracked. In addition, the hand-picked named events are included, for example xcodeDetailsRetrieved or appConfigured above. (In all cases, the numeric values are timestamps in milliseconds.)
The ServerEvents object for this reason has a commands field and an events field, each constituting a list of their own type (CommandEvent and ServerEvent) respectively, and containing the event name and timestamp(s).
In addition to the hand-picked events automatically included by the Appium team, you can also tell the Appium server to include your own events within the event log, so that you can have a record of when actions within the world of your test logic took place with respect to the built-in Appium commands. To do this, you need to log the event:
CustomEvent evt = new CustomEvent();
evt.setVendorName("prefix");
evt.setEventName("eventName");
driver.logEvent(evt);
In the example above, we are logging an 'eventName' event with the vendor name 'prefix'. Why do we need this vendor prefix? To ensure that custom events can never conflict with built-in ones, or with events logged by another library. These custom events will show up when you call getEvents later on! Here are some that I triggered in a test of my application:
...
"theapp:onLoginScreen": [
1573001595285
],
"theapp:testEnd": [
1573001599630
]
...
If you want, you can use the Java client to write out the raw data in JSON format somewhere on disk for use with other applications, too:
// assume we have ServerEvents object 'events'
events.save(new File('/path/to/events.json').toPath());
With all of this raw data, you could build interesting visualizations of the time Appium spent in between each of these events. But you don't really need to, because the Appium team already built a tool to do this for you! It's called the Event Parser, and it's a CLI tool you install via NPM:
npm install -g appium-event-parser
You can use the Event Parser to generate a timeline of all the events that are described in a JSON file, for example. (Did you notice the save command above? It gets you exactly what you need to make this happen).
appium-event-parser -t -i /path/to/events.json
(The -i flag is followed by the path to the data file, and -t tells the program you want a timeline printed out. You can also pass an -l flag followed by a number to change the number of lines the timeline takes up). Once you run this command, you'll be greeted by a nice little timeline:
You can also check out a full sample test using these features!
Did you know that Appium is just a web server? That's right! Appium is just like your backend web or API server at your company. In fact, you could host Appium on a server connected to the public internet, and give anybody the chance to run sessions on your devices! (Don't do this, of course, unless you're a cloud Appium vendor).
You may be used to writing Appium code in Java or some other language, that looks like this:
driver = new IOSDriver<WebElement>(new URL("http://localhost:4723/wd/hub"), capabilities);
WebElement el = driver.findElement(locator);
System.out.println(el.getText());
driver.quit();
This looks like Java code, but what's happening under the hood is that each of these commands is actually triggering an HTTP request to the Appium server (that's why we need to specify the location of the Appium server on the network, in the IOSDriver constructor). We could have written all the same code, for example, in Python:
driver = webdriver.Remote("http://localhost:4723/wd/hub", capabilities)
el = driver.find_element(locator)
print(el.text)
driver.quit()
In both of these cases, while the surface code looks different, the underlying HTTP requests sent to the Appium server (and the responses coming back) are the same! This is what allows Appium (and Selenium, where we stole this architecture from) to work in any programming language. All someone needs to do is code up a nice little client library for that language, that converts that language's constructs to HTTP requests.
What all this means is that we technically don't need a client library at all. It's convenient to use one, absolutely. But sometimes, we want to just run an ad-hoc command against the Appium server, and creating a whole new code file and trying to remember all the appropriate syntax might be too much work. In this case, we can just use curl, which is a command line tool used for constructing HTTP requests and showing HTTP responses. Curl works on any platform, so you can download it for your environment if it's not there already (it comes by default on Macs, for example). There are lots of options for using curl, and to use it successfully on your own, you should understand all the components of HTTP requests. But for now, let's take a look at how we might encode the previous four commands, without any Appium client at all, just by using curl!
# 0. Check the Appium server is online
> curl http://localhost:4723/wd/hub/status
# response:
{"value":{"build":{"version":"1.17.0"}},"sessionId":null,"status":0}
You can see above that we can make sure we have a connection to the Appium server just by running curl and then the URL we want to retrieve, in this case the /status endpoint. We don't need any parameters to curl other than the URL, because we're making a GET request, and so no other parameters are required. The output we get back is a JSON string representing Appium's build information. Now, let's actually start a session:
# 1. Create a new session
> curl -H 'Content-type: application/json' \
-X POST \
http://localhost:4723/wd/hub/session \
-d '{"capabilities": {"alwaysMatch": {"platformName": "iOS", "platformVersion": "13.3", "browserName": "Safari", "deviceName": "iPhone 11"}}}'
# response:
{"value":{"capabilities":{"webStorageEnabled":false,"locationContextEnabled":false,"browserName":"Safari","platform":"MAC","javascriptEnabled":true,"databaseEnabled":false,"takesScreenshot":true,"networkConnectionEnabled":false,"platformName":"iOS","platformVersion":"13.3","deviceName":"iPhone 11","udid":"140472E9-8733-44FD-B8A1-CDCFF51BD071"},"sessionId":"ac3dbaf9-3b4e-43a2-9416-1a207cdf52da"}}
# save session id
> export sid="ac3dbaf9-3b4e-43a2-9416-1a207cdf52da"
Let's break this one down line by line:
Running this command, I see my simulator pop up and a session launch with Safari. (Did the session go away before you have time to do anything else? Then make sure you set the newCommandTimeout capability to 0). We also get a bunch of output like in the block above. This is the result of the new session command. The thing I care most about here is the sessionId value of ac3dbaf9-3b4e-43a2-9416-1a207cdf52da, because I will need this to make future requests! Remember that HTTP requests are stateless, so for us to keep sending automation commands to the correct device, we need to include the session ID for subsequent commands, so that Appium knows where to direct each command. To save it, I can just export it as the $sid shell variable.
Now, let's find an element! There's just one element in Appium's little Safari welcome page, so we can find it by its tag name:
# 2. Find an element
> curl -H 'Content-type: application/json' \
-X POST http://localhost:4723/wd/hub/session/$sid/element \
-d '{"using": "tag name", "value": "h1"}'
# response:
{"value":{"element-6066-11e4-a52e-4f735466cecf":"5000","ELEMENT":"5000"}}
# save element id:
> export eid="5000"
In the curl command above, we're making another POST request, but this time to /wd/hub/session/$sid/element. Note the use of the $sid variable here, so that we can target the running session. This route is the one we need to hit in order to find an element. When finding an element with Appium, two parameters are required: a locator strategy (in our case, "tag name") and a selector (in our case, "h1"). The API is designed such that the locator strategy parameter is called using and the selector parameter is called value, so that is what we have to include in the JSON body.
The response we get back is itself a JSON object, whose value consists of two keys. The reason there are two keys here is a bit complicated, but what matters is that they each convey the same information, namely the ID of the element which was just found by our search (5000). Just like we did with the session ID, we can store the element ID for use in future commands. Speaking of future commands, let's get the text of this element!
# 3. Get text of an element
> curl http://localhost:4723/wd/hub/session/$sid/element/$eid/text
# response:
{"value":"Let's browse!"}
This curl command is quite a bit simpler, because retrieving the text of an element is a GET command to the endpoint /session/$sid/element/$eid/text, and we don't need any additional parameters. Notice how here we are using both the session ID and the element ID, so that Appium knows which session and which element we're referring to (because, again, we might have multiple sessions running, or multiple elements that we've found in a particular session). The response value is the text of the element, which is exactly what we were hoping to find! Now all that's left is to clean up our session:
# 4. Quit the session
> curl -X DELETE http://localhost:4723/wd/hub/session/$sid
# response:
{"value":null}
This last command can use all the default curl arguments except we need to specify that we are making a DELETE request, since that is what the WebDriver protocol requires for ending a session. We make this request to the endpoint /session/$sid, which includes our session ID so Appium knows which session to shut down.
That's it! I hope you've enjoyed learning how to achieve some "low level" HTTP-based control over your Appium (and Selenium) servers!
.png)
