Text OCR ("tocr")

4.3.1 Text OCR ("tocr")

DESCRIPTION

The Text OCR method (code "tocr") allows calling an external Optical Character Recognition (OCR) engine to extract text from the remote desktop image. The preferred OCR engine must be selected and configured in the preferences. Robot 5.x supports:

• Tesseract OCR is the default one. It is free and supports all major platforms (MS Windows, Linux/Unix, Mac OS X). The disadvantages are lower performance (speed) and accuracy.
• ABBYY Fine Reader is a commercial OCR engine available at an extra cost. It is faster and more accurate than Tesseract. It is supported on MS Windows only.
• Google Vision is a cloud service performing OCR. As the desktop image gets uploaded through a secure connection (HTTPS) to the Google site please consider your privacy and security before choosing this method. The OCR process is fairly fast and accurate. The biggest factor in the overall performance is your network speed.

The recognized text may be optionally tested for the presence of a phrase using the "text" (plain text search), "text" and "distance" (tolerant text search) or "pattern" (regular expression matching) parameters.

Regular expression matching matches the recognized text against the specified java.util.regex.Pattern compliant regular expression. Up to version 4.1.3 the expression must match the whole text and no searching for a matching substring is performed. Version 4.1.4 supports searching of the text for matching locations and returns their text and coordinates the same way as the text parameter.

The tolerant (fuzzy) text search is based on the Levenshtein distance. It is defined as the minimum number of edits needed to transform one string into the other, with the allowable edit operations being insertion, deletion, or substitution of a single character. This metric is enabled through the "distance" parameter which is an integer number specifying how many characters may be omitted or not recognized properly at a maximum to consider the sample text provided in the "text" parameter equivalent. Unlike the regular expressions, the tolerant matching always searches the recognized text for any occurrence of a string matching the given text and distance. There's no need to specify that the string is preceded or trailed by another text.

The method stores the OCR results into a set of TOCR prefixed variables as follows:

 When the text or pattern parameter is specified to search the recognized text for the given string the method creates result variables as follows:

OPTIONS

The method accepts no template images. The parameter of "passrate" is not applicable in the context of OCR and it is ignored. The parameter of "cmparea" is optional and defaults to the full screen when omitted.

Supported OCR parameters:

text=<textToSearchFor>

Optional text to search the recognized text for. The hosting command will then return either 0 when the OCR was performed correctly and the recognized text contains the given string or 1 otherwise. Indices and screen coordinates of the text locations are then stored to the _TOCR script variables and may be farther processed. This parameter can not be used together with the pattern one.

distance=<0-[textLength]>

Optional Levenshtein distance used in conjunction with the "text" parameter to perform tolerant text matching. See the method specification for details.

pattern=<regularExpression>

Optional java.util.regex.Pattern compliant regular expression to test the recognized text against. This parameter cannot be used together with the text one.

• • Up to version 4.1.3 the expression must match the whole text (no searching for a matching substring is performed). 
  • Since version 4.1.4 the parameter will search the text for matching locations the same way as the text parameter. Indices and screen coordinates of the matching text locations are then stored to the _TOCR script variables and maybe farther processed. This enhancement allows creating actions like "find a matching text location and click it" within a single "Click ocr" command.  

See the Tesseract OCR help page for details on its specific parameters.

RETURNS

The method makes the hosting command throw a runtime error on misconfiguration or an I/O error. Text of the error is made available through the _TOCR_ERROR variable.

The return code otherwise depends on the input parameters. If the method is called to perform just OCR with no result testing it always returns 0 even if no text was recognized. If the method is called with the parameters of "text" or "pattern" it returns 0 if the recognized text matches the given string/regular expression or 1 otherwise.

EXAMPLES

Var _TOCR_LINE_COUNT=0
Compareto method="tocr" cmparea="x:33,y:2,w:200,h:22"
for (i=1; {i}<{_TOCR_LINE_COUNT}+1; i={i}+1) {
   Typeline "{_TOCR_LINE{i}}"
} 

- Recognize text in the specified desktop area and type it onto the desktop.


Compareto method="tocr" cmparea="x:33,y:2,w:200,h:22" pattern="[aA]pplication"
if ({_EXIT_CODE} > 0) {
  Exit 1
}


- Exit the script when the recognized text doesn't match the 'Application' or 'application' word.

Compareto method="tocr" cmparea="x:33,y:2,w:200,h:22" pattern=".*[aA]pplication.*"
if ({_EXIT_CODE} > 0) {
  Exit 1
}

- The previous example modified to exit the script when the recognized text doesn't contain 'Application' or 'application'.

Compareto method="tocr" cmparea="x:33,y:2,w:200,h:22" text="apple" distance="2"

- Recognize text on the screen and verifies whether it is like and/or it contains a string like 'apple':

• If the OCR recognizes text like 'There is an apple', the command will report success (the exit code of 0) because an exact match is found.
• If the OCR recognizes text like 'There is an Apple', the command will report success because word 'Apple' can be changed to 'apple' by substitution of one character (A->a) which is within the tolerated distance of 2.
• If the OCR recognizes text like 'There are Appls', the command will report success because word 'Appls' can be changed to 'apple' in two operations of (1) substitution (A->a) and (2) addition (+e) or substitution (s->e) which are still within the tolerated distance of 2.
• If the OCR recognizes text like 'There is Doppler', the command will report success because word 'Doppler' contains a sequence matching 'apple' after a single substitution (o->a).
• If the OCR recognizes text like 'There are Appis', the command will report failure (the exit code of 1) because word 'Appis' requires at least three operations (A->a, i->l, s->e) to be turned into 'apple' which is more than the tolerated distance of 2.
  
  When the text is matched, the index of the match location in the text is stored to the TOCR_MATCH_INDEX variable. The index starts from 0 which represents the text beginning. The matching substring is further on saved under the _TOCR_MATCH variable. For example, the example above with the recognized text of _'There are Appls' will create the variables as _TOCR_MATCH=Appls and _TOCR_MATCH_INDEX=10.