Continuous Integration

Contents: 

1. Introduction
2. CI Installation
3. Create Project
4. Integrate Project
5. Design Automation
6. Integrate Automation
7. Advanced Tips
8. Conclusion

1. Introduction

This document describes how to set up a Continuous Integration job that builds an application and executes T-Plan Robot Enterprise automated testing. The guide uses Jenkins as the tool of choice. Integration with Hudson is not covered but the steps are very similar.

2. CI Installation

Download Jenkins from:

http://jenkins-ci.org

The native release is the easiest way to go. In this guide we will install native Windows release of Jenkins 1.563 on a Windows 8.1 system.

The native installer sets up Jenkins to start as a Windows service. This is not suitable for integration with Robot relying on the Local Desktop or iOS Mirror connections because the service won't have sufficient privileges to start local UI applications. For the needs of our example integration we will have to disable the Jenkins service:

1. Open Services:
• Windows 8: Go to the Start screen and type "view local services"
• Windows 7 and older: Open the Control Panel, search for Services and open "View local services" from the "Administrative Tools" group
2. Double click the Jenkins service, stop it and set the Start Type to "Disabled"

To start Jenkins manually open a command prompt, change to the Jenkins install directory (typically C:\Program Files\Jenkins or C:\Program Files (x86)\Jenkins) and execute:

jre\bin\java -jar jenkins.war

To open the Jenkins web interface open http://localhost:8080/ in a web browser. 

3. Create Project

For the purpose of this guide we will use a simple Java application developed with Netbeans. It displays a simple message window with a Close button. As the application is a Java one it will work fine on any Java supported platform such as MS Windows, Mac OS X, Linux, Unix or Solaris OS. You can download the application source code and its project from the following link:

You don't need to have NetBeans installed for this example. You must only install Ant and Java Development Kit (JDK) if you don't have it. These are needed for compilation and building of the sample Java project.

Install alternative #1: Use the Jenkins tools

1. Click "Manage Jenkins" in the Jenkins home (http://localhost:8080/)
2. Click Configure
3. Use the "Add JDK" and "Add Ant" buttons to install/configure JDK and Ant

Install alternative #2: Manual installation and configuration

1. Download JDK and install it: http://www.oracle.com/technetwork/java/javase/downloads/index.html
2. Download the Ant binary distribution (ZIP) from http://ant.apache.org and unzip it to the local hard drive, for example to "C:\Program Files\Ant"
3. Put the "C:\Program Files\Ant\bin" folder to the system path and configure the JAVA_HOME variable:
   • Windows 8: go to the Start screen and type "system"
   • Windows 7 and older: open Control Panel and select System
   • Select Advanced System Settings -> Environment Variables, double click the Path variable from the System variables table and append a semicolon ; and the path to the variable value:

<previous_path>;C:\Program Files\Ant\apache-ant-1.9.4\bin

• Add a new System variable, the name is JAVA_HOME and the value is the JDK install path (for example C:\Program Files\Java\jdk1.7.0_51)

• Restart Jenkins to pick up the system changes. If you started Jenkins from a command prompt close it and start a new one.
• To verify the result open a command prompt (Windows+R, "cmd") and execute "ant.bat". The system must recognise the command. If there's a message like "tools.jar not found" you haven't set the JAVA_HOME variable properly.

4. Integrate Project

The next step is to integrate the project of your application with CI (Jenkins). For simplicity we presume that the application is not under a version control (CVS, SVN etc). 

1. Click "New Item" in the Jenkins home (http://localhost:8080/)
2. Give the item a name, such as "DemoApp", choose "Build a free-style software project" and click OK. 
   
   
   
   
3. Leave the Source Code Management option on "None". You may change it to CVS, SVN or any other supported system as needed. 
4. Click "Add build step". As Netbeans use Ant select the "Invoke Ant" option. Add two targets separated by a space:
   • clean jar
5. Click Save

The project is now set. To initialize the local project workspace we have to build it:

1. Go to Jenkins home (http://localhost:8080/), click DemoApp and select "Build Now" from the left menu.
2. In the "Build History" select the link to the build log.
3. Select "Console Output" from the left menu.
• If the project is under a version control it should show success because Jenkins pulls the project from the server. 
• As our example does not use any version control the build will fail with a message that Ant can not find the build.xml file. Download the sample project and unzip it into the location mentioned in the log (typically <user_home>\.jenkins\jobs\DemoApp\workspace). Then go back to the project page and click "Build Now" again. This time it should succeed.

Congratulations, your demo Java project is integrated with Jenkins now! The further configuration is up to your needs now. You may set up Jenkins to build the project on a regular schedule. If the project is under version control you may also trigger builds after every source code change.

The next step is to design a simple T-Plan Robot automated test and integrate it into the build process.

5. Design Automation

Before we design the automated testing we have to choose the test environment configuration. T-Plan Robot supports a number of architectures that involve one or more machines and/or mobile devices. For an overview see the Supported Configurations tutorial topic.

For the purpose of this guide we will choose the simplest single-machine scenario. We will test the application on the Jenkins machine using the Local Desktop connection. The T-Plan Robot project with the test code can be downloaded here:

The test project is cross platform and it will work on any Java supported platform (MS Windows, Mac OS X, Unix, Linux, Solaris OS). Unzip the project archive and open the folder it contains using File->Open Project under the T-Plan Robot GUI. The project contains one test script (tpr/Jenkins.tpr) and one template image of the icon used by the Close button of the test application.

The script tests a very simple scenario:

1. Start the demo application.
2. Verify that the application started properly. This is achieved though searching the screen for the Close button icon.
3. Click the Close button.
4. Verify that the application shut down properly. The script ensures that the Close button is no longer visible.

If you set the variable APP_HOME variable (line #3) to the location where you unzipped the demo application you can execute the script right away. Requirements:

• Java installed and put on the system path (presume you've already done it under 3. Create Project)
• T-Plan Robot 3.5.2 or higher (or at least 3.5.2Beta from 19 May 2014 or newer) which introduced support of the "Exec kill" command. The script will also execute on older releases but it will not kill the application if it for some reason fails to close. The Jenkins build will then hang because it will wait for all processes to complete.

In order to produce a reasonable test report the script creates a few result objects. It creates an XML report at line 17. It gives the script a name and a number (line 23) and it creates two step results (lines 40 and 54) and a screen shot (line 41). To view the resulting report see the Test Results tree node of the Robot project or open the online copy here.

In the next step we will integrate the test script into Continuous Integration.

6. Integrate Automation

Integration with CI is enabled thanks to the T-Plan Robot command line interface (CLI). The most important options:

To create the start command suitable for CI integration use the CLI Wizard tool. To start it select Tools->CLI Wizard in Robot:

• "Test script" -> choose Jenkins.tpr from the demo project
• "File paths" -> absolute
• OPTIONAL: Set "Connect to" to "java://localhost". This is not really needed because our script calls the Connect command to start the connection on its own.
• OPTIONAL: Set on "Execute the script in the CLI mode"
• OPTIONAL: We will further instruct Robot to store the report (test results) to a unique directory under the Jenkins workspace. To do so we will set the _REPORT_DIR Robot standard variable to a value like:

%WORKSPACE%\reports\%BUILD_NUMBER%

The WORKSPACE system variable will be set by Jenkins to the application workspace path. The BUILD_NUMBER one will contain the unique build number. For example, if the workspace is C:\Users\JohnDoe\.jenkins\jobs\DemoApp\workspace and the build number is 3 the report will be stored to:


C:\Users\JohnDoe\.jenkins\jobs\DemoApp\workspace\reports\3

This will make the report available at the following URL. You may also navigate to the report from Jenkins through the Workspace link in the project's (job's) menu.

http://localhost:8080/job/DemoApp/ws/reports/3/results.xml

This allows you to send the test result URL by email after every build. This can be achieved through the Email-ext (Email Extension) Jenkins plugin. To enable valid URLs you will also have to go to Manage Jenkins -> Configure and set the Jenkins Location to a valid host name. For other report options see the Advanced Tips chapter.

• Once you are done with the CLI Wizard click the Copy To Clipboard button.
  
  

To integrate the automation to the build process simply add a new build step:

1. Go to Jenkins home (http://localhost:8080/), click DemoApp and select "Configure" from the left menu.
2. Click "Add Build Step" in the Build section and choose "Execute Windows batch command"
3. Paste the CLI command into the Command field.
   
   

1. Save the changes and select Build Now. Jenkins will now build the application AND run the automation.
2. The Jenkins build will succeed only if the build and automation finish successfully. It will fail if the build or automation fails. The test results will be stored to a unique build-specific folder in the Jenkins workspace and can be viewed in the web browser.

7. Advanced Tips

Automation under Jenkins running as a service

Jenkins running as a service is typically restricted in two areas:

1. The process doesn't have permissions to start GUI applications. This prevents it from integration with Robot automation based on the Local Desktop and iOS Mirror connections. Other "remote" connections such as VNC Server (except for Local Desktop over VNC) and Android Over ADB can be integrated with Jenkins in the service mode provided that you start Robot with the -n/--nodisplay CLI option.
2. The process usually runs under a user account with limited file system permissions. Impacts on Robot:
• If there's no valid user home folder the Robot process will fail to save changes to the user configuration file. You will see a few exceptions in the log but the program will continue using the default factory settings. As Robot stores the license key location to the configuration file you may see the process fail for a missing license. To work around this save the key file to the Robot's installation directory from where it will be picked up automatically. Alternatively add a CLI option like:
  -o "config.licenseKeys=<license_key_path>"
  The -o/--option option may be specified repeatedly to customize preferences from the configuration file. Robot version 4.0 will address these limitations with the new --cfg and --licensekey options.
• If the process has no permissions to create folders in the default system temporary location the application log service (Help->Log Viewer in the Robot's GUI) will be disabled. All logs will be printed out to the terminal (command prompt) only.
• If the process has no permissions to create files in the system temporary location the Robot process will crash or not work properly. It is necessary to grant the process at least these permissions.

Automated processes wrapped into a custom Java code (example) will require an extra step. Due to a limitation in Robot 3.5.2 and earlier it is necessary to set the cliMode field of the ApplicationSupport class as follows before the class is instantiated for the first time. When it's omitted the ApplicationSupport constructor will throw a java.lang.ExceptionInInitializerError. This will be fixed in release 4.0.

import com.tplan.robot.*; 
import java.lang.reflect.Field;

...

try { 
    Field field = ApplicationSupport.class.getDeclaredField("cliMode"); 
    try { 
        field.setAccessible(true); 
        field.set(null, true); 
    } finally { 
        field.setAccessible(false); 
    } 
} catch (Exception e) { 
    e.printStackTrace(); 
} 
ApplicationSupport a = new ApplicationSupport();

...

Linking reports to the Jenkins UI

The latest Robot report can be optionally linked to the project's menu and header page:

The steps:

1. Install the HTML Publisher Plugin
2. The plugin unfortunately can't handle report paths containing Jenkins build variables such as %BUILD_NUMBER%. It can only link to static locations in the project workspace. To work around this we have to make the Robot script create an HTML copy of the report in a stable directory. To achieve it add the following command to the Jenkins.tpr script:
   • Report "..\latest\results.html" desc="Testing of the Jenkins demo application, build {_ENV_BUILD_TAG} ({_ENV_BUILD_ID}) located at {_ENV_WORKSPACE}"

3. Select Configure in the DemoApp menu. Select "Add post-build action" and choose "Publish HTML reports". Configure the parameters as follows:

   HTML directory to archive:	reports\latest
   Index page[s]:	results.html
   Report title:	Latest T-Plan Robot Report

   Save the changes and execute the build to make the link appear in the menu.

8. Conclusion

With the steps above allow you will be able to integrate T-Plan Robot automated testing with the Continuous Integration tool of your choice. The principles of the CLI integration can be applied to integration with virtually any other framework. 

The document does not cover other test environment configurations involving multiple test machines. These will be discussed later. 

Should you have any question please contact the T-Plan Support team.