Image Search ("search")

4.2.2 Image Search ("search")

DESCRIPTION

The Image Search (code "search") searches the desktop screen for a component or an object represented by one or more template images and/or image collections. It is typically employed for verification of the desktop screen contents ("make sure that an object is visible") or to locate a component for a subsequent mouse or keyboard operation ("find and click or drag", "find and press/type", etc.).

The method uses a plain pixel comparison with three independent mechanisms of tolerance:

1. Pixel-based tolerance makes the algorithm search for occurrences which have a certain amount of pixels different from the template image. This is achieved through the Pass Rate "passrate" parameter which defines the percentage of pixels required to match. If your template image size is, for example, 10x10 (100 pixels) and you specify the pass rate of 99%, the algorithm will find all matching locations having up to 1 different pixel. Note that the lower pass rate you specify, the lower the performance will be and the longer the search will take.
2. RGB tolerance has been introduced in v2.2. It is activated by the tolerance parameter which is an integer number between 0 and 256. It indicates how much the Red, Green and Blue components of a desktop pixel may differ at a maximum to consider the colour be equivalent to the corresponding template pixel. This value allows dealing with images whose pixels are changing slightly, for example as a result of blurring or merging of the image with the background. This behaviour has been reported for Flash applications where decorative texts and even some images are not being rendered in a constant way as the application refreshes. Be aware that the higher the tolerance value, the higher the probability of false match detections is. In most scenarios, the value should be in the [0, 100] range depending on the blurring level. If the parameter is not specified, it defaults to zero and the algorithm compares pixels using an exact colour match.
3. Transparency/translucency based tolerance allows ignoring certain pixels of the template image. The image search algorithm by default counts as "passed" all transparent or translucent (partially transparent) pixels having the translucency component Alpha lower than the limit specified by the Minimum Alpha parameter (minalpha). For example, if a template with 100 pixels contains 90 transparent pixels and the minimum alpha is set to 0xFF (forcing to search for fully opaque pixels only), only the remaining 10 non-transparent pixels will be compared against the desktop image. 
   
   Transparency is a powerful method allowing to build image comparison which is robust against a background or component colour changes. When you for example search for an icon rendered on your remote desktop, marking the background colour pixels transparent in the icon template will make sure that the image comparison passes regardless of the remote desktop colour changes. The template image may be further on doctored with common image editors to remove insignificant pixels and leave just a skeleton of the searched object. This is often the only possible way to test certain types of applications with unstable object rendering, such as for example Geographic Information Systems (GIS).

T-Plan Robot Enterprise version 2.1 and higher supports automatic background transparency through the removebg and bgcolor parameters. It allows ignoring the template image background without having to edit the image with a third-party editor. The algorithm accepts an opaque template image on the input (meaning an image with no transparency) and applies an internal transparency filter to remove all pixels equal to or reasonably similar to the background. The background colour defaults to the first template image pixel but it may be also specified explicitly through the bgcolor parameter. Automatic transparency may be comfortably configured through the GUI, in particular with the Template Image Editor and the comparison GUI component described in the CompareTo window.

Transparency may be also introduced into template images using third party image editors.

For example, Windows and Linux/Unix users may take advantage of e.g. Gimp to set up transparency as follows:

1. Open the template image with e.g. Gimp
2. Select Layer->Transparency->Color To Alpha
3. Select the transparent colour. It usually works fine if you leave it on the defaults provided by Gimp (white background colour).
4. Save the template image to the file.
5. Retest with the Template Image Editor. This is necessary because if the image contains just colours which are very close to the background colour, Gimp may turn all pixels transparent or translucent which will make the template unusable for image search with the default minalpha value. The Template Image Editor is able to detect such cases and suggest changes to the image search parameters.

Coordinates of the resulting match locations are exposed to the calling script in the form of a set of _SEARCH_ prefixed variables:

To get the width and height of the match location use the _COMPARETO_TEMPLATE_WIDTH and _COMPARETO_TEMPLATE_HEIGHT variables which are populated by the hosting command or Java method call.

Note that the number of match locations is limited by a parameter in the Compareto command preferences and the search will stop when the maximum gets reached.

OPTIONS

The method requires one or more template images and/or image collections specified through the hosting command or Java method call. The parameter of "passrate" specifies in the context of the "search" method the minimum ratio of pixels that must match exactly between the template image and the screen match location. The default value is 100% (no pixel diff tolerated). The "cmparea" parameter is optional and defaults to the full screen when omitted.

Supported specific parameters:

tolerance=<0-255>

- The tolerance parameter controls the RGB tolerance to minor colour changes. It must be an integer number between 0 and 255 where 0 stands for no tolerance (exact matching) and 255 meand maximum tolerance. It indicates how much the Red, Green and Blue components of a desktop pixel may differ at a maximum to consider the colour be equivalent to the corresponding template pixel. High tolerance values increase the probability of false match detections.

This parameter allows dealing with images whose pixels are changing slightly, for example as a result of blurring or merging of the image with the background. This behaviour has been reported for Flash applications where decorative texts and even some images are not being rendered in a constant way over time. In most scenarios, the value should be in the [0, 100] range depending on the blurring level. If the parameter is not specified, it defaults to zero and the algorithm compares pixels using the exact colour match.

removebg=<true|false>

The removebg parameter switches the automatic background transparency on ("true") or off ("false"). The value of "true" applies automatic transparency to ignore the template image background colour. If the template image (or template images) already contain transparent or translucent pixels, the background filter is not applied and this parameter, as well as the bgcolor one, are ignored. The default value is "false" (background filter is off).

bgcolor=<HTMLColor>

The bgcolor parameter is optional and it may define custom background colour for the automatic background transparency mode activated through the removebg parameter. It accepts RGB in the HTML-like notation, such as for example "ffffff" for white or "000000" for black. The value must be 6 characters long where each of the R, G, B components are specified in the form of a 2-digit hexadecimal number (00 to ff). When the bgcolor is not specified the method selects the background colour from the very first template image pixel (top left image corner).

minalpha

The minalpha parameter sets translucency tolerance. It may be applied to images filtered through the removebg filter as well as to templates with already existing transparent or translucent pixels. The default value of 255 instructs the search algorithm to compare just the fully opaque pixels and ignore all transparent or translucent (semi-transparent) ones. Values lower than 255 will make the algorithm to compare even translucent pixels with the alpha component equal to or greater than the specified limit.

The minimum alpha parameter is intended to solve situations where the background colour filter makes too many pixels or even all of them transparent or translucent. This leaves the search algorithm comparing just against a small number of opaque pixels which usually leads to unexpected match results. As lowering the alpha limit increases the number of comparable pixels, it may be used to improve the accuracy of the image search algorithm. Be however aware that comparison of translucent pixels is based on colour similarity, it has significantly slower performance than the classic image search and extremely low values of minimum alpha may produce unexpected match results.

RETURNS

The method makes the calling command (method call) return 0 (success) if at least one matching location is found for at least one of the input template images. Otherwise, it returns the value of 1.

 TROUBLESHOOTING

To deal with a failing "search" comparison:

• If the script is designed to terminate after a failed search it is a good practice to create a screenshot in a lossless format (PNG or BMP). An example of such a code is shown in the example below. This will allow you to reproduce and debug the failed comparison after you load the screenshot either through the "Static Image Client" in the Login Dialog or using the "Load RD Image" button in the comparison command/method call property window.
• Decrease the pass rate or increase the RGB tolerance parameter ("tolerance") and test it through the "Compare" button until you get a match. If the parameters don't appear to be able to fix the broken search create an alternative template image and add it to the template list or image collection.
• For the most common factors causing comparison failures to see the Image Comparison Recommendations chapter.
• Should the search fail for no obvious reason please mail the screenshot together with the image template(s) to the T-Plan Support.

EXAMPLES

Compareto buttonOk.png  method="search" tolerance="50"

if ({_EXIT_CODE} > 0) {
  Screenshot failure.png desc="Failed to find the OK button."
  Exit 1 desc="Failed to find the OK button."
} else {
  Mouse click to=x:{_SEARCH_X},y:{_SEARCH_Y}
}

- Search for the OK button and click it. If the button is not found take an exit screenshot and terminate the script with the exit code of 1.

See the "search2" method specification for more component search examples. Or see the excerpt below: