Image Search V2 ("search2")

4.2.1 Image Search V2 ("search2")

DESCRIPTION

The Image Search v2 (code "search2", since v3.0) searches the desktop screen for a component or an object represented by one or more template images and/or image collections. It is typically employed for verification of the desktop screen contents ("make sure that an object is visible") or to locate a component for a subsequent mouse or keyboard operation ("find and click or drag", "find and press/type", etc.).

The "search2" algorithm is a new generation of the older Image Search method ("search") one with the following improvements:

1. Easy to use - The only parameter controlling the tolerance to minor desktop image changes is the standard "passrate" one. Its default value is set to 50% to perform an out-of-the-box reliable search for most environments including those considered as difficult to automate at the desktop image level, such as for example Flash applications.
2. Fast performance - The search operates significantly faster even for low values of the passrate parameter. Also as the algorithm avoids copying of the image pixels and works directly with the desktop buffer it consumes up to 90% less memory than the other similar methods.
3. Result sorting - The default sorting of the results (matching locations) by the best match ensures that even if the passrate value is too low and the method produces multiple fuzzy matches, the best matching location is always reported in the first place. This allows to work with the first match (which is the most likely the correct one) and ignore the other remotely similar ones.
4. Search of scaled images - The scale parameter introduced in release 4.0 searches for scaled instances of the input component image. This allows reusing a single search across devices with various resolutions (Android, iOS ...).

Coordinates of the resulting match locations are exposed to the calling script in the form of a set of SEARCH prefixed variables. This system is compatible with the older "search" method:

To get the width and height of the match location use the _COMPARETO_TEMPLATE_WIDTH and _COMPARETO_TEMPLATE_HEIGHT variables which are populated by the hosting command or Java method call.

This method is not capable of working with transparent/translucent template images and with the "search" specific parameters of "removebg", "bgcolor", "minalpha" and "tolerance". Should you need this functionality use the "search" method instead.

Since v4.4 the method allows setting a limit on the number of matching locations. It is intended to support scenarios such as "find first N occurrences and stop" where the full desktop search would yield too many matches or take too long. To set the limit from a script populate the _SEARCH2_MATCH_LIMIT variable using the Var command. To reset it set the variable to an empty value. For example, the value of 1 will make the search stop on the first matching location:

// Set the match limit to 1 
Var _SEARCH2_MATCH_LIMIT=1 
Compareto comp.png method=search2 

// Reset the match limit 
Var _SEARCH2_MATCH_LIMIT=1

Starting with v6.1 the maximum number of matching locations was internally limited to 1,000 to prevent performance issues. The limit can be changes in the method preferences.

OPTIONS

The method requires one or more template images and/or image collections specified through the hosting command or Java method call. 

In the context of the "search2" method, the parameter of "passrate" specifies the required minimum similarity between the template image and the matching location on the screen. The default value is 50% which ensures a good performance/accuracy ratio in most cases. If you experience too many matches, for example on low contrast environments, set the pass rate to a higher level. To get the maximum performance set the pass rate to 100%.

The "cmparea" parameter is optional and defaults to the full screen when omitted.

The method supports two specific parameters as follows:

sort=<best|none|top|bottom|left|right>

- Result sort mode (optional), one of:

• "best" - Sort by the best match (the best matching location with the lowest difference first). Locations having the same difference value will be sorted in the natural reading order (left to right, top to bottom). This mode is the default one when the parameter is not specified.
• "none" - Don't sort and leave the results in their natural "reading" order (left to right, top to bottom). This mode will produce the same order as the legacy "search" method.
• "top" - Sort from the top to the bottom (the topmost first).
• "bottom" - Sort locations from the bottom to the top (the bottommost first).
• "left" - Sort locations from the left to the right (the leftmost first).
• "right" - Sort locations from the right to the left (the rightmost first).

scale=<float_number(s)>

- The scale is optional parameter supported since 4.0. It allows to search for scaled instances of the input image(s). The value may be a single float number (scale factor) or a semicolon ';' separated list of numbers. When there are more numbers they are processed in the specified order until a match is generated or the list is exhausted.

To set off scaling use the value of 1. When the value is greater than zero it is being handled as a scale ratio. For example, the value of 2.0 will search for the component whose width and height are magnified twice. 

There are two negative constants which can be specified to employ dynamic scaling. They will scale the input image(s) with regard to the difference between the current desktop resolution and size of the desktop the template (component) image was created from. As Robot 3.x and older did not save the desktop resolution to the template metadata, the image(s) must be created or updated with Robot 4.0 to enable this operation. To update older images connect to the original desktop, right-click the template(s) in the Project View and select the Update image properties item.

The supported scale modes are:

• -1 ("stretch mode") will scale the input image following the desktop width and height changes. The resulting image may or may not have the same width and height proportions (the width/height ratio).
• -2 ("proportional scale mode") will pick up the smaller relative change of the desktop width and height and use it for both width and height scaling. The resulting image will have the same proportions (the width/height ratio).

RETURNS

The method makes the calling command (method call) return 0 (success) if at least one matching location is found for at least one of the input template images. Otherwise, it returns the value of 1. 

TROUBLESHOOTING

To deal with a failing "search2" comparison:

• If the script is designed to terminate after a failed search it is a good practice to create a screenshot in a lossless format (PNG or BMP). An example of such a code is shown in the example below. This will allow you to reproduce and debug the failed comparison after you load the screenshot either through the "Static Image Client" in the Login Dialog or using the "Load RD Image" button in the comparison command/method call property window.
• To achieve higher tolerance for the image changes simply decrease the pass rate. Don't be afraid to set it really low to the Fuzzy level, but make sure to test through the "Compare" button in the GUI that the correct component location is reported first. If you don't achieve to fix the broken search through the pass rate create an alternative template image and add it to the template list or image collection.
• For the most common factors causing comparison failures, see the Image Comparison Recommendations chapter.
• Should the search fail for no obvious reason please mail the screenshot together with the image template(s) to the T-Plan Support.

 EXAMPLES

Compareto buttonOk.png  method="search2" passrate="90" sort="bottom"

if ({_EXIT_CODE} > 0) {
  Screenshot failure.png desc="Failed to find the OK button."
  Exit 1 desc="Failed to find the OK button."
} else {
  Mouse click to=x:{_SEARCH_X},y:{_SEARCH_Y}
}

- Search for the bottommost OK button and click it. If the button is not found take an exit screenshot and terminate the script with the exit code of 1.

// Search for the buttons and click every single one 
Compareto "button.png" passrate="100" method="search2" 
if ({_EXIT_CODE} == 0) { 
    for (i=1; {i}<{_SEARCH_MATCH_COUNT}+1; i={i}+1) { 
        Mouse click to=x:{_SEARCH_X_{i}},y:{_SEARCH_Y_{i}} wait=1s 
    } 
} 

// Wait 15 seconds for the buttons to disappear or change their state
Waitfor "mismatch" method="search2" passrate="100" template="button.png" timeout="15s" 
if ({_EXIT_CODE} > 0) { 
    Exit 1 desc="Some buildings failed to turn the state!" 
}

- Search the screen for multiple buttons and click every single one. Then check the screen if all the buttons disappear or change its state.

// Find the scroll button 
Compareto "scrollbutton.png" method="search2" 
if ({_EXIT_CODE} > 0) { 
    Exit 1 desc="Failed to locate the scroll button!" 
}

// Save its coordinates to the X, Y variables 
Var X={_COMPARETO_CLICK_X} Y={_COMPARETO_CLICK_Y}

// Iterate 100 loops 
for (i=0; {i}<100; i={i}+1) { 
    // Click the scroll button 
    Mouse "click" to="x:{_COMPARETO_CLICK_X},y:{_COMPARETO_CLICK_Y}" 

    // Check if the component is visible on the screen. We use Waitfor 
    // because the page may take time to scroll and update on the screen 
    Waitfor match template="component.png" method="search2" timeout=3s 
    if ({_EXIT_CODE} == 0) { 
        // Found -> break the for loop 
        break 
    } 

    // Last loop #99 -> component not found, terminate the script 
    if ({i} == 99) { 
        Exit 2 desc="Failed to scroll to the component!" 
    } 
} 

- An example of how to search for a component which is displayed in a scrolled page (window). The example keeps clicking the scroll down button and checks for the existence of the component in a loop. The task of clicking onto the scroll button could be eventually replaced with the pressing of the PgDown key.