Program Level Execution and Load Testing

Program Level

By program level execution we understand running of automated process(es) from a custom Java program or a third party Java application. As T-Plan Robot's role is in this scenario reduced to a Java library, this mode is supported on the level of Java APIs only and there are no GUI or CLI features supporting development of the related Java code. Use a third party tool (an IDE) such as NetBeans or Eclipse for this purpose.  

T-Plan Robot Enterprise supports program level execution through the AutomatedRunnable and ApplicationSupport Java interfaces. The API allows to create and start either a single process (effectively simulating a single automatic execution in the GUI or CLI mode) or an unlimited number of processes (executed as Java threads) running in CLI mode (until the system resources are exhausted). Each such a process seamlessly accepts the same set of options as the CLI and executes one test script against one test environment (desktop) at a time. 

Program level execution consists of three steps: 

  1. Instantiate the ApplicationSupport class. This initialises the necessary framework and shared services. 
  2. Use the instance to create an AutomatedRunnable. Test script name (file) as well as other options compatible with the CLI specification are accepted through one of the createAutomatedRunnable()method arguments.
  3. Either call the run() method of the runnable (single process scenario) or encapsulate it in a java.lang.Threadand start it as a thread (multi-threaded scenario).

The following example starts two automated threads. The first one connects to a VNC server running on port 5900 of machine called testmachine1 and executes the C:\TestScripts\test1.tpr test script . The other one connects to a VNC server running on port 5901 of machine called testmachine2 and executes the C:\TestScripts\test2.tpr test script. Note that both the threads will be executed simultaneously and the program will exit when the last thread finishes. 

TwoTasks.java
import com.tplan.robot.ApplicationSupport;
import com.tplan.robot.AutomatedRunnable;
/**
*ExampleoftwosimultaneoustestprocessesstartedattheJavacodelevel.
* (C)T-PlanLimited,http://www.t-plan.com
*/
publicclass TwoTasks {
publicstaticvoid main(String[] argv) {
        // Step 1: Instantiate the ApplicationSupport class.
        ApplicationSupport robot = new ApplicationSupport();
        // Step 2: Create AutomatedRunnable instances and initialize them with CLI-compatible options.
        String args1[] = {"-c", "rfb://testmachine1:5900", "-p", "welcome", "-n", "-r", "C:\\TestScripts\\test1.tpr"};
        AutomatedRunnable runnable1 = robot.createAutomatedRunnable("cli-1", args1, System.out, false);
        String args2[] = {"-c", "rfb://testmachine2:5901", "-p", "welcome", "-n", "-r", "C:\\TestScripts\\test2.tpr"};
        AutomatedRunnable runnable2 = robot.createAutomatedRunnable("cli-2", args2, System.out, false);
        // Step 3: Encapsulate the runnables with Thread and start them.
        Thread t1 = new Thread(runnable1);
        t1.start();
        Thread t2 = new Thread(runnable2);
        t2.start();
    }
}

This multi-threaded approach can be also employed in load testing scenarios. T-Plan Robot can measure the application response time, and it is an excellent engine for the simulation of a number of users accessing the tested application or service in order to generate load. When this model is combined with another tool for performance measuring, it may represent a very cost effective solution.

In addition to supporting the automation of the local desktop, T-Plan Robot utilises the RFB (VNC) technology, and in this mode each thread must automate its own VNC desktop server instance. Such an approach is very efficient with Unix or Linux where a single platform (OS) may run as many VNC servers, as the system resources allow (such as RAM and CPU). This scenario may be exploited for testing of applications running on Unix/Linux and/or for web application testing where the Unix/Linux machine hosts web browser instances. As MS Windows systems may run just one VNC server per OS, load testing on these environments is not so efficient. In these instances it would be better to employ T-Plan Robot ability to automate the local desktop.

T-Plan Robot does provide tools to measure application response time, and produce performance results. As automation is performed on the desktop level, such testing will have an extra added value because it will reflect the true time elapsed between the user request and result. This is in contrast with many other load performance testing tools which typically focus on measuring of one critical phase of the request-response cycle (for example database or web/application server response time). Such results however usually do not correspond with the end user experience, where the time is further on affected by the environment.

Load Testing

Robot can be successfully used to generate load for stress testing. The idea is to start many parallel testing processes to simulate a number of users accessing a service or using an application. As Robot simulates a real user behaviour in a graphical environment, such a scenario will  create load on the tested system which is very similar to real production environment. 

Robot supports has a Java API which allows to create multiple testing threads within a single Java Virtual Machine. Each thread may independently execute one test script on a single VNC server instance. See the RobotThread interface of theRobot API  for more information. Note that Unix/Linux systems are preferred to host the load tests because unlike Windows they are capable of  running many VNC server instances on a single machine. 

Let's create an example program generating load on a tested application. We need to create three components: 

  1. Test script in the Robot scripting language which will represent a typical user session with the tested application. 
  2. Java wrapper which will take advantage of the Robot Open API to create and start parallel test threads.
  3. Shell wrapper which will start and eventually shut down the VNC servers (Linux/Unix only).

1. Test Script

There is no test script included in this example. You need to create it before and integrate it with the Java and shell wrappers described below. See the previous examples in this document on how to create a test script in the Robot scripting language. You may easily use the Record & Replay feature to do so. It is highly recommended to use a small remote desktop size (e.g. 800x600 pixels) and avoid image comparison and taking of screenshots. All these factors significantly increase the amount of required memory and CPU time. Another good idea is to set your remote desktop background (wallpaper) to a solid colour. In the next steps we will suppose that the test script already exists and is saved in a file called client.txt.

2. Java Wrapper

Java wrapper is a program which creates and starts the automated testing threads using the Robot API. A universal wrapper code is displayed below:

LoadExample.java
import com.Robot.Robot;
import com.Robot.api.RobotThread;

/**
 * LoadExample.java
 *
 * This is an example of load testing based on Robot.
 * 
 * (C) 2007, Robert Pes
 */ 
public class LoadExample {
   // Default number of threads.
 final static int DEFAULT_THREAD_COUNT = 50;
   // Main method.
 public static void main(String[] argv) {
 // Number of threads
    int threadCount = DEFAULT_THREAD_COUNT;
 // Password
        String password = null;
 // Script path
        String script = null;
 // Parse the program arguments. We expect an integer
 // indicating how many threads we will start followed by script name.
 // The last argument, VNC password, is not mandatory because some 
 // VNC servers can be configured not to require authentication.
    if (argv.length > 1 && argv.length < 4) {
     try {
                threadCount = Integer.parseInt(argv[0]);
                script = argv[1];
 if (argv.length > 2) {
                    password = argv[2];
                }
            } catch (NumberFormatException ex) {
                System.out.println("Invalid number of threads - default of "
                         + DEFAULT_THREAD_COUNT+" will be used.");
            }
        } else {
            System.out.println(
 "Usage: LoadExample <number_of_threads> <script_path> [<VNC_password>]");
      return;
        }
 // Thread arguments. They are identical with Robot CLI options.
        String[] threadArguments;
    if (password == null) {
            threadArguments = new String[] {
 "--connect",        // Connect option
 "",                 // VNC host name and port - will be generated dynamically
 "--run",            // Run option
                script,             // Name and path of the script to run
 "--nodisplay"// CLI execution mode option
            };
        } else {
            threadArguments = new String[] {
 "--connect",        // Connect option
 "",                 // VNC host name and port - will be generated dynamically
 "--run",            // Run option
                script,             // Name and path of the script to run
 "--nodisplay",      // CLI execution mode option
 "--password",       // Password option
                password            // Password
            };
        }
 // Create an instance of Robot
        Robot robot = new Robot();

        RobotThread runnable;
        Thread thread;
// Loop for index from 1 to specified number of threads
    for (int i=1; i<=threadCount; i++) {
	    // Populate the VNC host name and port in the thread arguments
            threadArguments[1] = "localhost:" + i;
            // Create a new automated thread and start it
            runnable = robot.createRobotThread(
                   threadArguments[1], threadArguments, System.out, false);
            thread = newThread(runnable);
            thread.start();
        }
    }
}

To run the wrapper compile the Java code to .class. The program accepts up to three parameters - desired number of threads to be started (mandatory), path to the testing script (mandatory) and VNC password (optional). The command should look like:

java -classpath .:./Robot.jar LoadExample <thread_number> <script_path> [<password>]

3. Shell Wrapper

Before you start Robot load testing, you always need to start all the necessary VNC servers. You should also verify whether any application involved in the testing allows only one instance per user. A typical example is Firefox which will not allow a single user to run more than one browser. To resolve this you will need to create a different user for each test thread and start the VNC server under his account.

All these necessary overhead tasks can be efficiently automated on Linux/Unix by simple shell scripts. Let's suppose that our example application doesn't have any instance limitations. All we need to do then is to start the VNC servers, execute the Java load application and kill the VNC servers after the testing gets finished. A Bash script handling these tasks follows:

runLoad.sh
#!/bin/bash 

# Number of threads to be executed 
THREADCOUNT=10 

# Path to the automated script 
SCRIPT=client.txt 

# VNC password. If your VNC server doesn't require password, 
# leave the variable empty. 
PASSWORD=welcome 

# This 'for' statement will start VNC servers on local machine 
# on ports 1 to THREADCOUNT 
for ((i=1; i<=$THREADCOUNT; i++)) 
do 
   vncserver :$i 
done 

# The following command will execute the Java wrapper with Robot 
# automated threads 
java -Xmx512m -classpath .:./Robot.jar LoadExample $THREADCOUNT $SCRIPT $PASSWORD 

# This 'for' statement will kill all the VNC servers started 
# at the beginning of this script 
for ((i=1; i<=$THREADCOUNT; i++)) 
do 
   vncserver -kill :$i 
done

Before you execute it, set correct values of the THREADCOUNT, SCRIPT and PASSWORD variables. If you define the number of threads too high, some of them may start to fail with java.lang.OutOfMemoryError.

To fix this issue increase the JVM memory heap size specified by the -Xmx Java option in runLoad.sh (default value is set to 512MB).