This article is the ninth in a multi-part series on test speed and reliability, inspired by a webinar I gave on the same subject (you can watch the webinar here). You might also want to check out the previous episodes on Test Flakiness, Finding Elements Reliably, Waiting for App States, Dealing with Unfindable Elements, Setting Up App State, Advanced Capabilities, Disabling Animations, and Mocking External Services.
We come finally to the last episode in this not-so-little series on test speed and stability. This edition is not about specific speed and reliability techniques; instead, it's about what to do when none of the previously-discussed suggestions have been helpful. There are no doubt lots of other useful tips hiding in the wings, and they'll make their appearance here in due time. But even if there's not an obvious solution to a particular problem, it's very useful to be able to pinpoint the problem so that you can look for a reasonable workaround or notify the appropriate authorities (Appium, Apple, Google, etc...).
In order to pinpoint any kind of problem with an Appium test (whether it's a straight-up error, or undesired slowness or instability), it's necessary to understand something about the complexity of the technology stack that comes into play during test execution. Take a look at this diagram of (most of) the pieces of Appium's XCUITest stack:
You've got your test code, the Appium client code, the Appium server code, WebDriverAgent, XCUITest itself, and who knows how much proprietary Apple code shoved in between XCUITest and the various bits of iOS. A problem on any one of these layers will eventually manifest itself on the level of test behavior. It's not impossible to figure out what's going on, and there are typically lots of helpful clues lying about, but my point is that a little work is required to get to a useful answer about your problem. So, here is what I recommend doing to diagnose issues, in roughly this order:
Test Code Exceptions
The first investigation should always be into your own test code. Oftentimes an exception arises that has nothing to do with Appium whatsoever, and is a simple programming error in your test code itself. Always read your test code exceptions in detail! Since your test code imports the Appium client in your language, this is also where you will get the first indication of any other errors. Whenever possible, Appium puts useful information in the error messages it returns to the client, and these messages should end up in your IDE or console.
In the case of exceptions generated by the Appium client, the exception will often belong to a certain class (NoSuchElementException, for example), which will give you a clue as to what has gone wrong. In the case of a generic exception, look for the message itself to see if the Appium server has conveyed anything about what could be wrong. Often, system configuration errors will be noticed by the Appium server and expressed in this manner, and a quick look at the message may even give you specific instructions as to how to fix the issue.
Appium Logs
The Appium server writes a ton of messages to the console (or to a file if that's how you've set it up). In the case of normal test execution, this information is mostly superfluous. It tells you all about what Appium is doing, and gives you a very detailed picture of the flow of HTTP requests/responses, commands Appium is running under the hood, and much more. Check out this article on how to read the Appium logs (by Appium maintainer Isaac Murchie) for a more detailed introduction to the logs. Here are some of the things I usually look at as a first quick pass of the logs:
- The session start loglines, to double-check that all the capabilities were sent in as I expected.
- The very end of the log, to see the last things Appium did (especially in the case of an Appium crash).
- Any stacktraces Appium has written to the logs (since these will often have information directly related to an error).
- Any log lines rated "warn" or "error".
If nothing pops out after following those steps, I read through the logs in more detail to try and match up my test steps with loglines, and then look for the area in the log matching the problematic area in my test.
Log Timestamps
In the case of slowness issues, it's essential to see how long different commands take. It's possible to determine this on the client side by adding time measurements before and after individual commands. Often, however, this doesn't reveal anything useful. We already know that a certain command is slow---that's why we're having trouble! But using the Appium logs, we can dig deeper and try to isolate the slowness to a particular Appium subroutine, which can be much more useful.
By default, the Appium log does not show timestamps, in order to keep the log lines short and uncluttered. But if you start the Appium server with --log-timestamp, each line will be prepended with a timestamp. You can use these to good effect by isolating the section of the log that matches the problematic area in your test, and start looking for large jumps in time in the log. You might find a single line or a small set of lines which are taking a long time. If it is surprising to find such a lag with those particular lines, you can always reach out to the Appium issue tracker and share your findings with the maintainers to see if there's a potential bug causing the slowness.
Device Logs
When it comes to problems whose root cause lies with your app, the mobile OS, or vendor-owned frameworks (like XCUITest or UiAutomator2), Appium is typically unable to provide any insight. This doesn't mean we're out of options, however. On both iOS and Android, apps can write to system-level logs, and often the system logs will also capture any exceptions thrown in your app (which might have led to a crash, say).
So, if the Appium logs don't provide any help in your investigation, the next place to go is to these logs. On Android, you can run adb logcat to access this log stream, and on iOS you can simply tail the system log (for iOS simulators) or use a tool like idevicesyslog for real devices. In fact, you can also set the showIOSLog capability to true, and Appium will scrape data from these logs and interleave it with the regular Appium logs, which often helps to put device problems into the context of the Appium session.
Use a Different Appium Client
If you don't notice anything untoward in the Appium server logs, and suspect that the issue might lie with the Appium client you are using, you could always try to rewrite your test script (or the problematic portion of it) using a different client (either in the same language or a different language). If after doing this the problem goes away, you have pretty strong evidence that the problem was with the Appium client.
Happily, the Appium team is often able to fix bugs with the Appium client very quickly (as long as the client is officially maintained by the project). Sometimes, though, a bug might exist in the underlying Selenium WebDriver client library which the Appium client is built on, and then the bug will need to be reported to the Selenium project. The Appium team can assist in that process, too.
Use the Underlying Automation Engine Directly
Appium is built on top of other automation technologies, like XCUITest as we saw in the diagram above. In many cases, issues that appear to be related to Appium are actually issues with the underlying automation technology. Before deciding that the Appium code is at fault for errors or slowness and creating a bug in the Appium issue tracker, it can be useful to write up a small testcase using the underlying engine, expressing the same test steps as with your Appium script. If you notice the same kind of problem (slowness, errors, etc...), then it's unlikely Appium will be able to do anything to directly fix the issue, and it should instead be reported to Apple or Google (and great---now you have a perfect minimal repro case for them!)
It's still useful to ask around the Appium forums in such a case, because there are often workarounds to these kinds of problems. If the workaround is useful enough, we can also build it into Appium as a first-class citizen, making life easier for all Appium users (at least until Apple or Google fixes the original problem).
Ask for Help
If none of the strategies above have resulted in clarity on the problem or a satisfactory resolution, it's time to reach out for help. A good first step is always searching the Appium issue tracker, StackOverflow, or the Internet more generally for any sign of your problem. Maybe someone else has already discussed a workaround for it.
If that's not the case, it's a good idea to create an issue at the Appium issue tracker. At this point, if you've followed all the steps above, prepare to be showered with love and praise by the Appium maintainers, who will appreciate the extremely detailed report you will be able to include in your issue.
Hopefully the Appium maintainers will be able to provide a resolution, especially if it's determined that your issue reflects an Appium bug. If not, then I suppose you could always put on your crazy hacker hat and start decompiling some XCUITest source code! Yeah---that's no fun. Anyway, hopefully you'll find resolution somewhere else along the way, using the methods we've discussed here.
I hope you've enjoyed this series on test speed and reliability. If in the course of your Appium test writing you come across any other good suggestions, please send them along to me, and I'll be happy to feature them in a future article. Even though the series is over, I'll certainly continue to cover tips to do with speed and reliability moving forward.
-1280X720-Final-2.jpg)
