Using XCTest

Using XCTest with HeadSpin

Overview

XCTest is a testing framework that allows users to create and run unit tests, performance tests, and UI tests for Xcode projects (i.e., Apple-specific applications on Apple devices). You will need an iPhone or iPad running iOS 10 or later and an installation of Xcode 8 or later, as well as an installation of the HeadSpin CLI in order to connect to a device. This document will show you how to easily execute an XCTest test on a Headspin-hosted Apple device.

Setup

You will need to download and install the Headspin CLI.

1. Click on your name in the upper right hand corner of the Headspin UI.

2. Click on Settings.

3. Scroll down to the Downloads section and then select the appropriate CLI installation package. Supported platforms are Mac, Linux, and Windows. Download and install.

4. If you haven’t already, you will also need to download Xcode. You can download it and learn more about it from Apple’s Xcode page, located here.

Connect Your Apple Device

Once you have successfully installed the CLI, now you can connect a HeadSpin device to your computer using the Headspin CLI command ‘hs connect’.

1. Go to the Remote Control page from the Nav Bar. Start the device that you would like to connect from the HeadSpin UI’s Devices list.

2. Once the device has opened its live Remote Control view, you will automatically have the ‘Overview’ tab of the interactive window open (to the right of the live device view).

3. On the Overview page, scroll down to the Remote Debug section. Click on the clipboard icon to copy the <code>hs connect</code> command for this device. (Note that the clipboard icon does not immediately appear on the page; hover over the space to the right of the <code>hs connect</code> command to make the icon appear.)

4. Open a terminal window, paste the command, and hit ‘Enter’.

5. Verify that the connection has been made between your machine and the device by executing <code>idevice_id</code>, as demonstrated below.

Execute Your XCTest Test from Xcode

To run a test, open Xcode and navigate to Product > Test. Move your pointer over the test case in the Xcode Test navigator, then click the gray play icon shown in the figure below. This method can also be used to run multiple tests/a subset of test cases.

To run a single test method in Terminal, run <code>xcodebuild</code> by giving the test method’s identifier as the parameter to the -only-testing option. (If you are unfamiliar with <code>xcodebuild</code>, running the man <code>xcodebuild</code> command in Terminal will show you the available command options on your machine and their details; you can also read more about using Xcode in the command line in general in Apple's documentation (here)[https://developer.apple.com/library/archive/technotes/tn2339/_index.html].) The identifier for a test method has the form target name/test case name/test method name.

% xcodebuild test -scheme SampleApp -only-testing SampleAppTests/SampleAppTests/testEmptyArrayWhenNoOverlappingNotes

To run all of the tests for your project or workspace, you have two options: you can run them in Xcode in the method outlined above using the Xcode Test navigator, or you can run the following <code>xcodebuild</code> command in Terminal:

<code>% xcodebuild test -scheme SampleApp</code>

You can run tests repeatedly for more accurate results as well as determining reliability. To do this, control-click the icon next to a test method or test case and select “Run Repeatedly”. In the panel that appears, choose when to stop running the test. The options are:

• Stop after failure: Run the test until the first time it fails, or the maximum repetitions are reached.
• Stop after success: Run the test until the first time it passes, or the maximum repetitions are reached.
• Stop after maximum repetitions: Run the test the specified number of times, regardless of its outcome.

Choose the maximum number of times to repeat the test, whether to pause execution if the test fails, and whether to restart the test runner for each repetition. Finally, click Run to repeatedly run the test.

To repeatedly run test cases or test methods in Terminal, specify the number of repetitions with the -test-repetitions option. You can optionally specify whether the tests should repeat until success or failure, and whether to restart the test runner for each repetition:

% xcodebuild test -scheme SampleApp -only-testing SampleAppTests/SampleAppTests/testEmptyArrayWhenNoOverlappingNotes -run-tests-until-failure -test-iterations 20

When Xcode runs the selected test/tests, it updates the test’s icon to indicate the outcome. The possible states are:

• green checkmark: test method passed
• red X: test method failed (the failure may also be due to a test method that is expected to fail (marked by XCTExpectFailure(_:options:)) that didn’t fail when it ran)
• grey X: test method failed with an expected failure
• grey arrow: Xcode skipped this test method

Additionally, when running multiple tests consecutively, you may see these icons:

• green minus: a green icon with a minus sign (–) indicates that the test case had a mixed outcome: some test methods passed, while others failed with expected failures or were skipped
• red minus: a red icon with a minus sign (–) indicates that the test case had a mixed outcome; some of the test methods failed while others failed with expected failures or were skipped

You can watch the execution of your test on the device in real time from the HeadSpin UI. In the Remote Control Devices list, click on the blue button beside your selected device. This will pull up the view of the device within HeadSpin as your test runs on it.

Capturing a Session - UI and API

A session, to clarify, represents a single touchpoint between a user and your application. It is a recording of this touchpoint’s actions taken on a device as part of an automated or manual test. You can use the UI to start or stop a session, but it may be more convenient or reliable for you to use API to do this while you are using Espresso for your testing framework. You can use API to gather data on the sessions your organization performs, as well as starting or stopping a session.

Within the UI, to start a session, you must first go to the Remote Control page and begin viewing the device on which you intend to test. Within the device’s Remote window view, click on the blue camera icon in the upper-right area of the Remote view controls. This will create a session and begin recording on the device.

To stop recording, click that same button, which will now be colored green.

When you finish recording a session, the HeadSpin UI will prompt you to view your session’s data in either Waterfall or Issue UI, and provide links to those pages.

API - Create a session

To start a session on a device, you must first lock that device. Lock the device through the HeadSpin UI or using the Android Device API. Browsers require starting and stopping a capture session using the HeadSpin Selenium capture capabilities. Please refer to Selenium Capabilities and Chrome DevTools.

Request Body

The request body must contain a JSON object with a minimum of:

{
    "session_type": "capture",
    "device_address": ""
}

For optional session configuration parameters, please review our Session API documentation.

[???]

Example

curl -X POST https://@api-dev.headspin.io/v0/sessions -d '{"session_type": "capture","device_address": "{android-device-serial}@{hostname}"}'

Response

If the request is successful, the response is a JSON object with details of the created session, including <code>session_id</code>.

{
    "companion_id": null,
    "session_type": "capture",
    "start_time": 1571175936.264,
    "endpoints": [],
    "state": "active",
    "session_id": "",
    "device_id": ""
}

If the request fails, you may get:

• <code>{"status": "Device is currently in use by someone else.", "status_code": 403}</code>: Someone else is currently using the device. Choose another device to create a session, or wait until this device is free.
• <code>{"status": "Invalid device ID.", "status_code": 404}</code>: The device ID specified doesn't exist. Double check that the device ID is correct.
• <code>{"status": "Device must be locked before capture.", "status_code": 403}</code>: The device must be locked by you before a session can be created. Lock the device through the HeadSpin UI or an API call.

Stop a session

Browsers require starting and stopping a capture session using the HeadSpin Selenium capture capabilities. Please refer to Selenium Capabilities and Chro

‍

me DevTools.

Request Body

The request body must be this JSON object:

{ "active": false }

Example

curl -X PATCH -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/{session_id} -d '{"active": false}'

Response

• A <code>HTTP 200</code> OK response with <code>{"msg": "Video uploaded to https://api-dev.headspin.io/v0/sessions/{session_id}.mp4"}</code> is returned. This message contains a link to the captured video. The name of the video is the session ID.
• A <code>HTTP 500</code> response if the request fails. Check that you have the right JSON object in the request body.
• A <code>HTTP 404</code> Not Found response if the request fails. Make sure that the session id in your request is valid.

Optional Parameters

• <code>?include_all={true|false}</code>: If <code>true</code>, this will return both live (active) sessions and ended (inactive sessions). If this value is <code>false</code>, only live sessions will be included. The default value is false.
• <code>?num_sessions={n}</code>: The number of sessions you want the API request to return, sorted by most recent. The default value is <code>10</code> and will return up to 10 sessions.
• <code>?tag={key}:{value}</code>: Select sessions that contain the tag defined by tag key <code>key</code> and tag value <code>value</code>. Spaces in either the tag key or value may be represented using <code>+</code>. For example, sessions containing the tag <code>my example: this is an example tag</code> could be selected using the query argument <code>?tag=my+example:this+is+an+example+tag</code>. This query argument may be provided multiple times to select sessions containing at least one of the tags provided.

Example

To get a list of the most recent 20 sessions, both active and inactive:

curl 'https://@api-dev.headspin.io/v0/sessions?include_all=true&num_sessions=20'

To get sessions containing at least one of the tags <code>my</code>:<code>tag</code> or <code>my other</code>:<code>example tag</code>:

curl 'https://@api-dev.headspin.io/v0/sessions?include_all=true&num_sessions=20&tag=my:tag&tag=my+other:example+tag'

Response

The response is a JSON object with the sessions key, and it contains a list of <code>sessions</code> in your org matching the optional parameters <code>include_all</code> and <code>num_sessions</code>.

{"sessions":
    [
        {
            "companion_id": null,
            "session_type": "capture",
            "start_time": 1565291652.17,
            "state": "ended",
            "session_id": "",
            "device_id": ""
        },
        {
          "companion_id": null,
          "session_type": "capture",
          "start_time": 1539154924.161,
          "state": "ended",
          "session_id": "",
          "device_id": ""
        }
    ]
}

Adding a Session to a User Flow - UI and API

User Flows are a grouping tool HeadSpin uses to give you powerful data analysis options by comparing any number of sessions across any time scale. A User Flow is really just a collection of sessions, typically grouped according to a user intention. As an example, you could create a "Checkout" User Flow and add sessions that test out the various steps of your application's checkout experience.

Again, you can explore the creation of User Flows in our UI within the Performance Sessions and Performance Monitoring tabs, but it may be more convenient for you to use API to add a session to a User Flow.

Within the UI, to add a session to a User Flow, go to the Performance Sessions tab to view a list of sessions. You can search for a specific session in the search bar at the top of the page if you do not see the session you want to add to a User Flow. Click on either the Waterfall or Issue UI icons beside your selected session to view that session’s info.

On the session’s information page, in either Waterfall or Issue UI, you will see a button in the upper-left corner of the screen labeled “Add to User Flow”. Clicking this button will bring up the ‘Add Session to a User Flow’ modal.

Within this modal you can select a previously created User Flow from the dropdown list or create a new one if desired. You can also write out a description for the session that will be attached to it within the assigned User Flow, and you can select a status (passed, failed, excluded) for the session’s outcome. Note that a User Flow and status assignment are mandatory.

API - Attaching a Session to a User Flow

Body

{
  "session_id": "..." // Session UUID ID, required
  "status": "..." // One of "Passed", "Failed", "Excluded", optional
  "status_message": "..." // Optional status message
}

Response

<code>200</code>: Updated session info.

• <code>user_flow_id</code>
• <code>perf_event_id</code>
• <code>status</code>
• <code>status_message</code>
• <code>host</code>
• <code>device_id</code>
• <code>event_time</code>: ISO format timestamp.

<code>400</code>:

• Missing argument (<code>session_id</code>).
• Invalid arguments (<code>bad session_id</code>).
• Invalid status (Must be one of "Passed", "Failed", "Excluded").
• Session already belongs to another user flow.

<code>404</code>: Session or user flow does not exist.

Example

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/userflows/{user_flow_id}/sessions \
     --data '{"session_id": "{session_id}"}'
# Output
{
  "user_flow_id": "f24cbec7-b04c-11ec-97a0-06c4c57ba6cd",
  "perf_event_id": "f527cb02-d8ce-471d-9660-689bff371b02",
  "session_id": "9f527e3f-daa1-11ec-ac12-06c4c57ba6cd",
  "status": "Not Set",
  "status_message": null,
  "host": "proxy-us-mvo-6.headspin.io",
  "device_id": "00008030-000559D902F0C02E",
  "event_time": "2022-05-23 10:07:04"
}

For more information on interacting with User Flows through the API, check out our documentation here.