Appium Tools for Rapid Development of Functional & Performance Tests

A Brief Overview of Functional and Performance Testing

So, I just want to give a little bit of a refresher on the types of testing that we’re considering today. First of all, we have functional testing, which I’m sure many of you are quite familiar with already. But if you are kind of wondering what’s underneath this term of functional testing, we can go back to the word function in its mathematical sense. So, if you’ve ever, you know, done algebra or calculus you’ve seen, you know, formulas or equations that look like this, you know, f (x) equals something else (equals why in this case). In this schema, f is the name of a function and functions take an input, or one or more inputs, but from a mathematical perspective usually one input, and then you can compose multiple functions together to deal with multiple inputs. Anyway, you’ve got your input and then you’ve got your results. So, you know different functions are defined by the different ways that inputs lead to outputs.

Check out: Designing an Automated Functional Testing Strategy

So, from the perspective of functional testing, we might re-categorize this as saying functional tasks is asking, you know, does f (x) actually equal y? So, when I take my application, which we can consider to be a function or a set of functions, and when we put in an input of a certain kind, whether that’s a tap or a whole series of actions, is the correct output produced?

So, this is what we mean by functional testing. It’s basically, given the right inputs, does my application behave correctly? And, sometimes given the wrong inputs, does my application behave correctly dealing with those wrong inputs. Are the error conditions handled appropriately making sure the app doesn’t crash or something like that.

Test your mobile apps on thousands of real devices. Know more!

There are a bunch of aspects of testing which people don’t usually include when they’re thinking about functional testing.

One of these is time. You know, before we asked does f (x) equal y. Well, we didn’t ask “and how long did it take to equal y?” You know, this is something which is important for users but not really an aspect of functional testing if it takes 10 seconds or 20 seconds. The functionality still worked.

Another aspect of an application that doesn’t usually show up in functional test is appearance. Does an app look good is it usable from a visual perspective? Does it contain any visual regressions – things like that? As long as the functionality of the app isn’t impacted, usually appearance isn’t considered as part of functional testing. Security is another issue. Security and functionality go hand-in-hand obviously. Security could be viewed as an element of functionality, but very often it’s not and so very often, for security testing you get to a different layer of the test stack and consider it as a separate thing that is a responsibility of another team than the people writing functional test. Things like that. So, there are these aspects that aren’t often considered in functional testing.

Also check: All you need to know about application security testing

This is often reflected in the usage of Appium, because Appium is fundamentally just a UI automation tool. It’s not a test framework. It doesn’t give you all the features that you need for writing and running tests. Its primary focus is to let you automate the use of a user interface.

So, in other words, it’s kind of like in the mathematical formula we had earlier. We had this function, which you could think of as your application, and we had this way of passing in an input which we called X and then determining an output which we call Y. And Appium is just a way of letting you input X into your application and letting you figure out what Y is, what happened on your application after we input X. So, in other words, it kind of provides the I/O, or the input and output, for functional testing. That’s why Appium is, I would say, primarily used for functional testing, even though it can also be used in other kinds of testing as well.

One of the most important other kinds of testing is what we call performance testing. So, if we go back to our little schema here, which we were using for functional testing, f(x) equals Y, we could ask additional questions beyond does f(x) equal Y and that gets us into the domain of performance testing. So, these are some of the questions we might ask within the domain of performance testing.

Okay, so f(x) was Y. We determine that with our functional testing, but did it become Y fast enough? Are we going to meet the users’ expectations of speed and responsivity? Did f(x) maybe consume too much memory? Now this is something that the user might not see themselves, but might impact the operation of the system, might impact the further operation of the app, or might cause a crash down the road. This is something internal to the operation of the function, not something that is visible within the UI, at least not until it gets to a crisis point.

See: Facts to Note About Using Appium for Desktop Applications

On Android, you might have an out-of-memory killer that shuts the app down. Things like that might happen. We might ask: what are the various kinds of bottlenecks inside the operation of the application? We can look at the application from an internal perspective and ask, you know, when I tap this button and when I waited for this result to appear on the screen, what were all the parts of my application internals that that were navigated through that were called, and how much time did the app spend in each of those? We can figure out bottlenecks on our application as a way of trying to point out problems or give us areas to potentially work on for improvement of the performance.

We could ask beyond the app itself, did running our application have any consequences on the device at large or the system at large? What could these negative consequences be? Could we be using way too much CPU, such that other applications are negatively impacted or that the battery is drained too quickly? Things like that.

We might also ask about how much data our app consumed, because if we’re developing our apps with the assumption that all of our users are on Wi-Fi speeds, we might run into an issue where our app is downloading so much data over the network that it becomes unusable in cellular conditions, 3G conditions, or 2G conditions.

So, this would show up to the user as something like slowness, but asking how much data was drawn down over a period of time is a really good way at figuring out again where we could optimize or where we’re making assumptions about our users’ experiences. So, these are the kinds of things that we talked about in performance testing.

Now since we’re talking about HeadSpin today in this webinar and it’s being hosted by my friends there, I took a screenshot of the HeadSpin application.

This is a UI, which is given to you after you run one of your Appium tests on HeadSpin. I know that the slide is probably showing you a bunch of stuff that’s too small to read and it’s not important for us to go through this in detail. My point is that all of these little metrics and graphs and potential issues that you see here are the kinds of things that you care about more in performance testing than functional testing.

You could have an app work totally fine on a functional level but two runs of the same application and two different conditions might give you a very different set of performance data here. In some situations, it might be really concerning and some situations might not be.

For example, in this example, HeadSpin has automatically found some issues that we might want to take a look at. So, it’s evaluated various aspects of the performance data of the application that we didn’t have to capture ourselves. It was kind of capturing them automatically as we are running our tests, but it’s sharing that this app – it’s the Walmart app in fact – had a loading animation on the screen for more than a second, which is known to be a potential spot of frustration for users. So, it’s saying we’ve got 11 counts of this error, so hopefully the Walmart team is taking a look at this kind of thing. I know from experience, having played around trying to automate this app, that it is painfully slow.

We can look at a lot of different issues not just as a bear warning but over time as well. This is the kind of stuff that we’re talking about when we talk about performance. Most of these bits of data pertain to network data and how long it took things to download and how long it took things to display on the screen.

We could also dig even more deep into the application internals. Here’s a bunch of timelines and some graphs of internal system resource usage. So, we’ve got a graph of the CPU usage over time.

We could see for example, and of course if we were actually on the website, we could drag the mouse over here and be greeted with a screenshot of the application at this particular time, so if we saw a particular spike in CPU, we could put our mouse over it and say, what’s happening here within the application? There are other ways to do this looking at process graphs and things like that too that are pretty interesting, especially when you’re digging into the nitty-gritty of the performance of your app. We can look at memory as well as CPU. Things like that. IO.

Also see: Best Practices for Application Performance Testing

So, there’s all kinds of pretty standard types of performance data that is really interesting to look at once you kind of dig a little deeper than the functional level.

Appium actually lets you retrieve some raw performance data just using the regular old Appium APIs, and I recommend checking that out. Some of the early editions of Appium Pro were on this topic. There’s an article on Android and an article on iOS. If you just search from appiumpro.com you can go and find them or I’m sure Google will take you there directly as well. So, check that out.

As I showed you in the screenshots, one of the cool things about HeadSpin is that they actually capture all this performance data automatically while you’re just running your functional tests, so you don’t have to do anything to capture their performance data. That’s all done kind of under the hood transparently to you. So, it’s kind of nice because you can focus on writing your functional test asking does f(x) equal y, but then in addition to that just by running on HeadSpin you get a report back that says by the way, here’s how long it took for y to show up on screen and here’s all your bottlenecks and here’s all these potential issues.

Read: How to Identify Client-Side Performance Bottlenecks

So that’s a really great thing about their service that I really appreciate. Something else to mention is that I mean this is true for all kinds of testing, but I think it’s especially relevant for performance testing. You can run one-off performance tests, but it’s often more useful to think of performance as something that shouldn’t just be tested at discrete points at time but should be monitored and should be automatically checked. You know, you want to make sure that there is nothing maybe in the back-end of your application that’s changing that is causing performance issues. Things like that.

‍

‍

So that’s just a brief overview of functional performance testing. I thought it’s interesting to dig a little deeper into that because I know that we can often throw the words around but may not know the concepts super clearly. Without further ado, let’s move on to some examples of some of the tools that I use to develop Appium tests and to make sure that they perform well.

Appium Desktop

So first thing I want to talk about is Appium Desktop. Appium Desktop is a user interface for Appium that you can use without even needing to download anything from the command line or open up your terminal. It’s a totally self-contained application.

So, if you’ve never used Appium Desktop, check it out. It’s at github.com/appium/appium-desktop and you can download binary releases for Mac, Windows, and Linux there. Rather than show you a bunch of slides, I’ll just open it up and do a little demo of some of the things that I use it for when I’m developing Appium tests.

Also read: Facts to note about using Appium for desktop applications

So, I apologize – things probably will show up kind of small on my screen. I’ll maximize things as much as I can. Actually, I wonder if I can. I can’t change the zoom within the application here, because it’s not actually a web browser. So, this is Appium Desktop. When you open it up, it greets you with this kind of screen that says hey, I’m Appium what host support do you want to run me on. I’m just going to go ahead and close down some other windows too so that we can see it in all its beauty.

Here we are. You can see my very dirty desktop as well. So, here’s Appium. What we can do once we’ve got it downloaded is we can actually start an Appium server. So, an Appium server is running. It says welcome to Appium version 1.15.1. So, I don’t need to run Appium from the command line if I don’t want to, and I could run an Appium test from my IDE or anything else. It says its running on Port 4723, which is what I selected, but usually what I use acting desktop for is not just as the server, but I use it to inspect my application so that I know how to find the elements in the most efficient way possible. To do that I can go over here and start an inspector session.

Once this loads up, I basically have a choice of server here. I can choose an automatic server, which is the one that’s already running. I can choose a custom server, and I can even select cloud providers to run on including HeadSpin. Then I basically set up an inspector session by typing in the desired capabilities, or I can use a saved capability.

I’m going to choose a set of capabilities that I implemented for this webinar so we can see we’re going to run on iOS 11.4 using an iPhone 8 simulator and I’ve got this application that I developed that I’m testing, and I’ve got this other capability here which I’ll explain a little bit later on. Once I’ve got my capability set, I can go ahead and start session.

I’ll go ahead and throw up the iOS simulator so you can see what stuff is happening. Obviously the Appium logs are scrolling by in the background.

So once the session loads what we have is a screenshot of the app here and I can hover over all the different elements, just kind of like Chrome or Safari or Firefox devtools. I can tap on one of these elements that Appium knows about and I can see a whole bunch of interesting metadata about it like its name, its label, its position.

But the most interesting thing is that Appium Desktop actually gives me suggestions of how to find this element in my test. So for this element, I can actually find it by accessibility ID, which is preferred and here’s the selector that I can use to find that.

So, I can actually test this out if I want just by tapping on the element here so I can click tap and we can see that we got to a new screen, new elements, new UI tree here, and the tap was actually successful. So let’s try and find something else.

Here’s a save button. Now, it does have an accessibility ID, but as you can see in the XML tree, it’s accessibility ID is not unique, because it’s the same as the element that’s its parent, and that’s because I’m using react native to develop my application. So, it just kind of does this – whether I like it or not.

Also check out: How React Native Testing Works?

So, the selector that Appium has recommended in this case is X path, but it hasn’t given me a kind of unintelligent XPath selector. It’s given me one that has some unique information in it namely. Here’s the name of the element. The name itself isn’t unique, but the fact that it’s the second one of those on the screen makes it unique.

Now, some feature that I use that I know a lot of people don’t really know about is something called the search for element dialogue. So let’s say I didn’t really know that this XPath was the suggested one or I was trying one out in my test and my code, but I’m getting an “element not found” error. So I can load the app up in this screen. I can click search for element. I can choose my locator strategy.

So, let’s choose accessibility ID, and I’m going to go ahead and type in what I thought maybe was the name of that particular button. Then I can click search, and Appium will come back with all the elements that it thinks correspond to this element. So actually, there’s two – so they sort of occur in more or less the same space on the screen. So, the fact that there’s two elements here could indicate to me the cause of a bug in my test script.

Don't forget to check: How to Integrate Audio APIs into Your Appium Automation Scripts

Maybe I assumed there was only one or maybe I was trying to interact with one of them, but I was actually interacting with the other one. Of course, with this selected, I can also tap it or do different things here. So that’s kind of a fun little tool within Appium Desktop.

Something else you can do within Appium Desktop that some people don’t know about – this is with more recent versions – is we actually have an actions tab over here.

This is when we want to perform Appium commands that aren’t really related to elements per se. So Appium has all these different actions that do with the device or the session. So, let’s just pick session here and you can see we have all these different groups of commands. What should we try?

Recommended post: Batching Appium Commands Using Execute Driver Script to Speed Up Tests

Let’s try orientation. Let’s get the orientation of the app. So, we get the result commanders return result portrait. Okay, that’s cool. I wonder if we can even set the orientation. Let’s try setting it to be landscape and voila. So, I’m going to change it back to portrait now.

So, you can do a lot of things with Appium beyond just find and interact with element specifically. So have a look at this actions tab. There’s a bunch of interesting stuff in there.

Another kind of fun tool, that’s especially useful when you’re developing tests is the recorder. So, we got something here already. I’m going to clear out. It’s actually saved my last action. But if I turn on the recorder, I can pause it or start it again, and if I perform some actions like for example “go back”.

We can see that that command has been added here in text, but this is the command in a particular language so I could if I wanted kind of copy and paste this into my test code. This is python. We could use Java or anything else, and if you want to take this and actually kind of make a full test out of it, that you could copy into your ID and execute. You can show and hide boilerplate code.

So, now I’ve got all of this in the context of the correct capabilities that I use to start the test and actually even puts it into a nice test method for me. Here’s the back command that we used. Let’s actually go to the web view page here. So, I'll tap this.

You can see that the element was found and clicked in the code as well. So that’s a pretty useful feature of Appium Desktop. Currently, there is no ability to play this back within Appium Desktop, but that’s something that we’ve been working on and we’ll release at some point.

So, one of the things that – I’m going to go ahead and close out the recorder here – one of the things you’ll notice that if you’ve got a Webview that basically it just shows up as this view element here. It doesn’t give you a bunch of context underneath it even though there might be elements within this.

So, Appium Desktop doesn’t actually support inspecting within a Webview, at least any elements that aren’t automatically promoted to native elements like this one here. Apple is promoted to a native element even though it’s actually a web element. But, we can’t use the contacts API within Appium Desktop, and the reason is because you can actually already do that really easily using Safari for iOS and Chrome for Android. So just as a way of highlighting that let’s go ahead and open up Safari.

I could make it a little smaller. So, here’s Safari. If I go up to develop, I can see that there’s actually an iOS simulator in this list, and there are two web contexts within it about:blank and JS context. Now, I’m not exactly sure which one of these is this, so I’m going to first of all type into the box and go to AppiumPro.com.

Sadly, with the most recent version of that code, there’s a bit of a delay in sending keystrokes, and this is a bug and XCUITest which hopefully Apple will be fixing soon. Maybe I’ll just type it in manually instead of waiting here.

Oh, it looks like the whole simulators is occupied. Well, that’s not very happy is it. Now it’s actually making my system slow. Okay. Well. We’ll go ahead and just pick one of these web views, and we’ll see which one is the right one.

So, I’m going to pick about:blank, and now I get actually a web inspector here that shows me everything that’s available within sight within the Webview. So, this one doesn’t actually have any elements in it. So, potentially this was not the one that I wanted to work with. Let’s try the other one.

This is just the JavaScript context for that one as well. So, I think that probably because the simulator is hanging, it’s not actually showing me the element within this Webview.

Anyway, this is how you would do it. I apologize on behalf of Apple for their untested software. So yeah, you can do the same thing with Chrome on Android as well. I actually have some Appium Pro articles on how to do that. So, this was all basically how to inspect your applications, how to exercise them using a visual interface, and you can do both the native aspect your application and the web aspect using Safari Chrome. So, we’ll come back to this. I might have to quit the simulator. See if that works. Okay, we will try that again later.

So, back to the presentation here. Something else that I want to talk about this morning is a new tool that has been released recently called Web2driver. So, Web2driver is a new kind of Appium client. So, you’re thinking Appium client, you’re thinking okay, there’s the java client, the python client, .NET, ruby, you know couple JavaScript clients. These all run from your test scripts, which you run on the command line, right? Or from your editor like IntelliJ or whatever.

Web2Driver

So, this is an assumption which has been true for a long time – until now. So, Web2driver is actually an Appium client that runs within a web page. It’s located at HeadSpin’s GitHub organization which is called “projectxyzio/web2driver” on GitHub. So, you can check it out. Here’s a little image of the readme.

As it says, Web2driver is a pure JavaScript web driver client that can run in a browser. One of the interesting facts about Web2driver is that because it runs in a browser, it kind of depends on the browser’s cross-site request to speak to the Appium server. But, because of browser standards to do with security and making sure the bad people can’t do things you don’t want, your browser can’t just use JavaScript to talk to a different server than the one that served it the webpage to begin with.

So if you want Web2driver to be able to speak to your Appium server, you actually have to start the Appium server with a special flag, which is called allow CORS, which is the cross-origin request security. I don’t remember what ‘S’ stands for – something about cross-origin requests, I think.

So, Appium doesn’t have this turned on by default, because it is again potentially a security risk. You could go to a web page, and somebody could start running tests on your Appium server – running on your machine without your knowledge. That’s pretty scary. So, you have to start the Appium server with that special flag to make sure that that’s able to happen. Otherwise, you can bundle this web driver package into your own JavaScript projects that you can run within a web page. So, that’s pretty amazing.

The Web2driver code looks very much like something you’d expect from WebDriver IO, and in fact, we built Web2driver on top of the same code that WebDriver IO is built off of. So, it uses all that logic, which has obviously been quite battle-tested over the years. So, running a test looks like defining your capabilities, starting a session, finding elements. Everything is promise-based, so you can use async/await to keep your code clean.

So, it’s pretty much a standard WebDriver and Appium client. I say Appium client, but really could use this for Selenium too. Although Selenium doesn’t support the allow CORS headers. So, you’re kind of out of luck, unless you put a proxy in front of Selenium that you built.

Why do we want a web-based Appium client?

Well, what if we could build Appium tools that require nothing but a browser – no downloads, no command lines – just your browser. This is, I think, pretty interesting.

So, this is exactly what we’ve been working on at HeadSpin. Here’s a little video of something called the HeadSpin Recorder, which is I guess in beta now. So, if you’re a HeadSpin user – talk to your account people and see if you can get access to this soon. But, the idea is that you load your app, you define the type of device you want to use, and then this recorder interface pops up.

The interface looks different, but it’s actually very similar to Appium Desktop, in the sense that it gives you a kind of point-and-click interface to use with your application. But, what’s happening is that this is all just taking place in your browser.

So, you just go to your HeadSpin account, go to your recorder tab, and you can load it up. You don’t have to download Appium Desktop or anything else. And, what’s really interesting about it is that not only does the recorder record your actions, but then you can have them play back and you can even set them to play back at set intervals. So, you can essentially define your own Appium-based test without using any code. So obviously there are limitations to any kind of no-code approach, but this can get you pretty far and it’s been pretty stable so far as I’ve been playing with it.

So, this is just kind of showing you some examples of you know, clicking on things and finding an element – very similar to Appium Desktop. You can see the actions proceeding here. So, there’s all kinds of different interactions you can use. You can even do swipes. So, here’s a swipe happening, and then you can even find elements by image. This is one of the cool things about the HeadSpin Recorder. You can select a region of the screen that may or may not be an element, and then that screen reading will be passed to Appium as an image, and it can be tapped during later runs.

So, in the video, the tests are just kind of finished and said, you know, I’m done, and now the recorder is automatically moving into the review phase where it’s loading up the device again with a new session. It’s going to replay all of these steps for you to kind of see – oh, yeah, this works like I expect. I’m going to sign off on this and now can schedule the test.

So, it’s going to skip ahead, you know, so here we are walking through the different steps for the application. We’re in the middle of a swipe here. Then, again once we’re done, we can create a report and decide how frequently we’d like this report to be regenerated by re-running all these commands.

So again, this is kind of an example of the kind of thing you could build with Web2driver because it’s a full-fledged Appium client. So, you can build these great experiences on top of Appium just in a browser that make it much more accessible, require no special software, are accessible from Mac and Linux and Windows and any operating system that can run Firefox. That kind of thing. So that is the recorder.

Last thing I want to talk about is something called Event Timings. Event Timings is an Appium feature that can be helpful for digging a bit deeper into the performance of your test itself, kind of giving you a little idea of what’s going on under the hood of the test- the different steps that Appium is taking that you may not even know it’s taking – that can help you discover where there are potential bottlenecks within your test code.

Events API

So, the fundamental idea here is that Appium takes time to do stuff.

This is pretty obvious. Not exactly mind-blowing, but a question we can ask is well, how long? How long does it take to do those different things? We’re just sitting there running a test, and it looks like it’s taking forever. Obviously, something’s happening. How many things are happening, and when are they happening? And what is the thing that is causing the holdup?

So, the Appium team has developed something called the Events API to help us determine this. Now, when you start a normal Appium test, you use capabilities like this. This is the kind of stuff that we’ve been using today.

To turn on the Events API, basically you add a new capability called “eventTimings” and just set it to true. Now, when “eventTimings” is set to true, then you have the ability to get a bunch of interesting timing data from Appium just by calling the “getSessionDetails” command. So, this is I think in the Java client it’s “driver.getSessionDetails” – something like that.

Now that response will have a bunch of other information like your capabilities and things like that, but there will be this key in the response called “events”. Within the events object here, we have basically two main parts.

One is a list of commands. So, every command that you send during the course of your test will get added here, including the command name and the start time and the end time, which you could obviously use to figure out how long did each command take. And then there are a bunch of other kind of special events that we’ve built into the Appium startup flow.

For example, this is an iOS test that I got this data from. So, there’s an event in the startup of an iOS test called ‘xcodeDetailsRetrieved’. So, we log that when you have the ‘eventTimings” API turned on, so that you can kind of see when did this happen, and then you can see the next thing that happened, and so on. You can see that the last thing that happened within this particular test that I ran was that I set the orientation, or the orientation was set automatically, I guess, by the Appium server when I start it up.

So, we could plot all of these on a map or on a timeline and figure out: where are the spaces in between these different events? But you don’t actually have to do that manually, because we have something called the Appium Event Parser that will generate a nice timeline for you from that data. So, you can take that data, use it on your own, whatever you want to do with it, but we can also generate a nice timeline.

So, here’s an example of the output for the Event Parser for some of this event data that I pasted into a file and ran through the Event Parser. So, you can see that all of these different events happened – xcodeDetails were retrieved, the app was configured, the reset was started, the reset was complete. Then we actually see this is all to scale. We can see there’s a gap here of almost three seconds – between resetComplete and logCaptureStarted.

So, what was going on between here? Did it actually take three seconds for the log to start or maybe the Appium team needs to add a little bit more granularity in here so we can see what’s happening? Or maybe simultaneously the simulator was starting? Maybe that’s what took some time.

Maybe we’ve got other gap of almost five seconds for the app to be installed and for us to start launching WebDriverAgent? Then by far the biggest amount of time taken here was the WebDriverAgent start. So, between start attempted and session started, it was almost seven, or a little over six, seconds.

So, this is pretty interesting. Let’s run a quick demo of this actually. So, to run this demo, I’m going to go ahead and start up a test again, and I’ve got the eventTimings capability turned on here.

The simulator is booting because it was frozen for

FAQs

Q1: Can Appium be used for cross-platform testing?

Appium is a cross-platform tool, which means that QA teams can use Appium to write test cases for iOS, Android, and Windows with the same code.

Q2: What are some popular load testing tools?

Some of the most commonly used load testing tools are:

1. LoadRunner
2. Apache JMeter
3. HeadSpin
4. Gatling
5. LoadNinja

Q3: What are some alternatives to Appium?

While Appium is a great tool, it might not fit all applications. Some alternatives to Appium are

1. Selenium: Beneficial for web apps
2. Microsoft Visual Studio App Center
3. Cypress
4. qTest by Tricentis 

Q4: What is the difference between scalability testing and stress testing?

Scalability and stress testing are part of performance testing. Scalability testing tests the application's capacity to handle the gradually increasing or decreasing load. On the other hand, stress testing identifies the breaking point of an application.