iOS Mirror Automation

Contents:

1. Introduction
2. Robot Machine Configuration
3. iOS Device Configuration
4. Automation

1. Introduction

The "iOS Mirror" connection allows Robot to test iOS applications on non-jailbroken (non-rooted) devices:

• The T-Plan iOS Plugin must be compiled to or injected into the tested application to enable touch screen and keyboard control.
• The device screen is mirrored either via the plugin (the "iOS Plugin" mirror) or using the Mac OS system API ("Lightning USB cable").
• Legacy options include control over the Veency VNC server and screen mirroring via 3rd party mirroring applications. This functionality is considered to be obsolete and it is not maintained any more. For details see the iOS Mirror Legacy Functionality document.

Note that regardless of the selected screen mirror type the device must be still connected to a WiFi at least in the connection init phase to ensure a correct start up of the network interfaces.

WiFi connection to the T-Plan iOS Plugin can be optionally tunneled through the USB cable. This allows to avoid unstable or slow WiFi networks.

Parallel connections (i.e. automation of multiple devices from one machine in parallel) is only supported when tunneling is not used.

iOS Device Requirements:

• An Apple iOS device equipped with the Lightning connector.
• The device must be connected to the network (usually over WiFi).

Robot Machine Requirements:

• Use of the "Lightning USB cable" screen mirror requires a compatible Java version.

---

2. Robot Machine Configuration

MS Windows Set Up

Automation of an iOS device from a MS Windows machine is supported only for the "iOS Plugin" mirror. No configuration steps are required.

Mac OS X Configuration

• Connect the iOS device over the Lightning USB cable to the Mac machine. This is not necessary if you select to control the device control over TCP/IP together with the "iOS Plugin" mirror type.

• Give the Robot process the necessary access rights.

  • Start Robot, select the "iOS Mirror" connection in the Login window, populate its options and click Connect. The connection will fail complaining that Accessibility is not enabled.
  • Open Apple→Settings and select Security & Privacy
  • Click Accessibility in the left column and enable the access in the right part of the window. The Robot process may appear as "java" or "sh" depending on how it was started.
  • Click Connect in the Robot's Login window again. A pop up asking for Camera permission should appear. Approve it. Alternatively select the Camera item in the Security & Privacy and grant the permissions there.
  • Similarly you may have to enable the access for Screen Recording and Automation too (these may be missing in older Mac OS X releases).

  

• You may have to repeat this step after each Robot or Java upgrade.

---

3. iOS Device Configuration

1. Device Configuration

It is recommended to set off the Auto Lock feature to prevent the device from going to sleep after the specified idle time:

• Tap the Settings icon in the iOS home screen.
• Go to General -> Auto Lock and select Never.

2. Configure Device with T-Plan iOS Plugin

The application under test must contain the T-Plan iOS Plugin.

• This will give you control over the application as long as it is displayed on the screen.
• If you need a full iOS control use the iOS Over Xcode connection instead.

---

4. Automation

Connection from Robot

The device must be connected to the network over the WiFi or through the Lightning USB cable to the driving Mac machine.

• Make sure that the application with the iOS Plugin is running and displayed on the device screen. The connection will not work if the application is not running or if it has been pushed to the background.
• If you opt for the TCP/IP control you will have to enter the device IP address and the port of 5909, for example "192.168.1.10:5909".  To find out IP address of your device open Settings->WiFi in the iOS and tap the little 'i' circle.

Start Robot and select the "iOS Mirror"connection type. Then choose the parameters as follows:

Device Control defines how the device touch screen and keyboard will be controlled:

• T-Plan Server/Plugin over TCP/IP will connect to the T-Plan plugin component over the WiFi network. This option requires you to enter the IP address of the device followed by a colon and the port number of 5909. See the image below for an example.
• T-Plan Server/Plugin over USB will tunnel the TCP/IP connection with the T-Plan plugin through the USB cable. The device must be connected to a WiFi at least when the application under test (AUT) gets started to ensure a correct start up of the network interfaces. This option is supported only on Mac OS X.
• VNC Server (Veency) will connect to the VNC server running on the device over the WiFi network. You will have to enter the IP address of the device followed by a colon and the VNC port number, typically 5901. This functionality is obsoleted and it is not being maintained any more. For details see the iOS Mirror Legacy Functionality document.

Screen Mirror specifies how the device screen will be transferred to the PC/Robot:

• Lightning USB Cable will mirror the screen through the Lightning USB cable connected to Mac OS X. You must specify the name of the device. There is a selector allowing you to list devices connected to your Mac.
• iOS Plugin will mirror the screen through the plugin. This option together with the TCP/IP device control enables automation from MS Windows or Unix/Linux machines. It is recommended to use the iOS plugin library version 1.11 or newer to get a stable connection.
• Mirror app will use the mirror application of your choice. This functionality is obsoleted and it is not being maintained any more. For details see the iOS Mirror Legacy Functionality document.

To connect to a device from a test script use the Connect command (TPR scripts) or the connect() method (Java test scripts). The argument URL must be in form of "apple://<address>:<port>". For example, let's have an Apple device with the IP of 192.168.100.1:

Example 1: Connect to the T-Plan Plugin/Server over TCP/IP using a mirror app:
TPR test scripts:     Connect apple://192.168.100.1:5909
Java test scripts:    connect("apple://192.168.100.1:5909");

Example 2: Connect to the T-Plan Plugin/Server over USB using a mirror app:
TPR test scripts:     Connect apple://localhost:5909
Java test scripts:    connect("apple://localhost:5909");

Example 3: Connect to the VNC Server using a mirror app:
TPR test scripts:     Connect apple://192.168.100.1:5901
Java test scripts:    connect("apple://192.168.100.1:5901");

If the screen mirror is the Lightning USB cable you must provide the device name. Since v5.0.6 you may retrieve the list of available devices through a call of the _IOS_DEVICES dynamic variable.

Example 4: Connect to the T-Plan Plugin/Server over TCP/IP using Lightning USB:
TPR test scripts:     Connect apple://192.168.100.1:5909 device="My iPhone 5"
Java test scripts:    connect("apple://192.168.100.1:5909", null, false, "My iPhone 5");

Example 5: Connect to the T-Plan Plugin/Server over USB using Lightning USB:
TPR test scripts:     Connect apple://localhost:5909 device="My iPhone 5"
Java test scripts:    connect("apple://localhost:5909", null, false, "My iPhone 5");

To force the iOS Plugin mirror use the reserved device name of "plugin":

Example 4: Connect to the T-Plan Plugin/Server over TCP/IP using iOS Plugin (suitable for automation from MS Windows/Linux/Unix and/or devices which are NOT attached to the Mac via USB):
TPR test scripts:     Connect apple://192.168.100.1:5909 device="plugin"
Java test scripts:    connect("apple://192.168.100.1:5909", null, false, "plugin");

Automation Tips

As the iOS device screen is mirrored using H.264 video with a lossy image compression format, the artifacts (components) on the screen change at the pixel level. To search for them using Robot template images use the search2 comparison method with the pass rate of 50% or lower. 

The preferred way of automation of multiple devices from a single PC is the Lightning USB connection on Mac OS X. Avoid using the USB tunneling so that the devices can be identified by the IP address. The steps:

• Make sure to have a license with 2+ seats (one seat per each connection). Insufficient number of seats will make the exceeding scripts wait for a free connection (sequential execution).
• To achieve parallel testing execute a workitem with two or more parallel scripts in the CLI mode. Alternatively start multiple Robot instances in the GUI mode and initiate the testing as usual (one instance per one test script).