Desktop automation

Botream automates Windows programs through Windows UI Automation: you name a window with a window variable and work with its buttons, fields, lists and menus through the desktop command.

#Window variables

Every desktop command works on a window, given as --window:$name$. There are three ways to get a window variable:

#Wildcards

Title and path both accept * wildcards:

• Title: *Notepad*, * - Google Chrome, Report*
• Path: erp.exe, C:\Program Files\*\EXCEL.EXE

Use the program name rather than a full path where you can: the task then also works on machines where the program is installed in another folder. Store apps have no program path; identify them by title.

#Patterns and handles

A variable from window --action:open or window --action:attach pins one specific window (its handle). This tells two windows of the same program apart, for example two documents of the same editor. Using such a variable after its window closed fails with "has been closed". A set --type:window variable stays a pattern and finds whichever matching window is open.

window --action:open --path:"notepad.exe" --output:editor
desktop --action:settext --window:$editor$ --by:classname --target:"Edit" --value:"Hello"
window --action:close --window:$editor$

window --action:open waits up to --timeout for the window; for launchers that open another program, give the --title of the window to wait for.

#Creating window variables in the editor

• Next to a Window field, New creates a window variable from one of the windows open on the agent machine.
• Next to a Title field, Choose lists the open windows and fills Title and Path (the path is reduced to the program name).
• When you capture an element and the command has no window yet, the editor reuses a window variable for that program or declares a new one ($window1$, $window2$…) at the top of the task.

#Window actions

window also manages the window itself: activate, close, maximize, minimize, restore, resize (--width --height), move (--x --y), exists, wait, getproperty and list.

window --action:wait --window:$erp$ --timeout:30
window --action:maximize --window:$erp$
window --action:getproperty --window:$erp$ --property:Title --output:erpTitle

getproperty reads Title, Path, ProcessId, X, Y, Width, Height, IsMinimized, IsMaximized or Handle.

#The desktop command

desktop finds an element in the window with --by and --target and acts on it:

desktop --action:click --window:$erp$ --by:automationid --target:"btnSave"
desktop --action:settext --window:$erp$ --by:name --target:"Customer" --value:$customer$
desktop --action:getproperty --window:$erp$ --by:automationid --target:"statusBar" --property:value --output:statusText
desktop --action:exists --window:$erp$ --by:name --target:"Save" --output:saveExists

getproperty reads value, name, automationid, classname, controltype, isenabled, isoffscreen, ischecked or helptext. Element actions wait up to --timeout (30 seconds by default) and can be repeated with --retry and --retryinterval.

#Locator strategies

#Capturing desktop elements

Select a desktop command and click Capture next to Target. The Capture desktop element dialog asks you to switch to the application and click the element.

• If the command already has a Window, that window is brought to the front.
• A click-through overlay highlights the element under the mouse.
• The click that picks the element is not passed to the application, so nothing happens in the program.
• A right-click cancels. Esc asks Cancel the capture? first: Enter cancels, Esc again keeps capturing.

The capture fills By and Target with the recommended locator (a unique AutomationId, then Name, then class name, else an XPath from the window), the Object properties table, and a screenshot of the window with the element framed in red.

Note: Desktop capture needs the Botream Agent tray app running on the computer you are working on.

#UI technology

Windows shows the elements of a program through UI Automation, in two versions. Choose the version next to the Capture button before you capture: once the picker runs, every click picks an element.

The choice is saved with the command (--technology:uia2), and the task finds the element with the same technology when it runs; with Auto, UI Automation 3 first, then 2. When Auto took an element from UI Automation 2 at capture, the command keeps UI Automation 2, since its path only exists there. Anchors and fallbacks use the command's technology. Object properties shows the technology next to the object type.

Note: UI Automation 2 needs the current Botream Agent. An older agent refuses a task that chooses it, with a message to update the agent.

#Object properties for desktop elements

As for web elements, every checked property must be equal to the element's, * is a wildcard (Save*), $variables$ work, and Target is rebuilt from the table on every change: a single exact AutomationId, Name or class name is used directly, anything else becomes an XPath such as //Button[starts-with(@Name, "Save")]. Editing Target by hand detaches it from the table; Use properties rebuilds it. See Finding elements reliably for anchors, fallbacks and self-healing.

#Sending keys

sendkeys types text and special keys into a window:

sendkeys --window:$erp$ --keys:"$orderNo$[TAB]$amount$[ENTER]"
sendkeys --window:$erp$ --keys:"[CTRL+S]" --interval:50

• Special keys go in brackets: [ENTER], [TAB], [ESC], [F5], [CTRL+C], [ALT+F4]. Use [[ for a literal [. Insert key next to Keys to send lists the common ones.
• --activate (default true) brings the window to the front first.
• --interval waits between keystrokes, in milliseconds, for programs that miss fast typing.

#Mouse

mouse clicks, drags and scrolls at pixel positions, for programs whose elements cannot be captured (remote desktops, drawings, canvas views):

mouse --action:click --window:$erp$ --x:120 --y:80
mouse --action:drag --window:$erp$ --x:100 --y:200 --tox:400 --toy:200
mouse --action:scroll --window:$erp$ --amount:-3 --x:300 --y:250
mouse --action:getposition --output:pos

• Actions: click, doubleclick, rightclick, middleclick, move, drag, scroll, getposition.
• With --window, positions count from the window's top-left corner and the window is brought to the front. Without it, they count from the screen's corner.
• Clicks and scroll without a position act where the pointer is.
• scroll --amount is in wheel steps: positive up, negative down.
• Pick next to X lets you click the point on the agent's screen to fill both coordinates.

Warning: Pixel positions break when the layout changes. Prefer desktop with a captured element wherever it works.

#Screenshots

screenshot saves a picture of the whole screen (all monitors), of a window, or of an area of either:

screenshot --file:"C:\Shots\error-$orderNo$.png"
screenshot --file:"C:\Shots\erp.jpg" --window:$erp$ --x:0 --y:40 --width:800 --height:500 --output:saved

The file extension picks the format: .png, .jpg or .bmp. The window is brought to the front first. For a web page, use browser --action:screenshot instead (the whole page, also when headless).

#Reading text with OCR

ocr reads text with Windows' built-in OCR, for applications whose elements cannot be captured (terminal emulators, Citrix or remote desktop sessions, scanned documents):

ocr --action:read --window:$terminal$ --x:0 --y:60 --width:900 --height:400 --language:en --output:screenText
ocr --action:find --window:$erp$ --text:"Total" --output:pos
if --condition:"$pos.Found$ = true"
    mouse --action:click --window:$erp$ --x:$pos.CenterX$ --y:$pos.CenterY$
endif
ocr --action:read --file:"C:\Scans\invoice.png" --output:scanText

• The source is --window, an area of it or of the screen (--x --y --width --height), or a picture --file (.png, .jpg, .bmp, .tif). The window is not brought to the front.
• Positions count like --x/--y, so you can click them with mouse.
• --language takes a tag such as en, de or tr; empty uses the Windows display languages. Install more languages in Windows under Settings → Time & language → Language (the "Optical character recognition" feature).

#Finding pictures on the screen

image finds a picture on the screen — a button, an icon, a logo — and clicks it or tells where it is. It works where nothing else can: Citrix and remote desktop sessions, canvas and game-like applications, anything drawn as pixels.

image --action:click --image:4f1d2c3b-… --window:$citrix$
image --action:find --image:"C:\Pictures\approve.png" --similarity:85 --output:pos
if --condition:"$pos.Found$ = true"
    mouse --action:click --window:$citrix$ --x:$pos.CenterX$ --y:$pos.Y$
endif
image --action:waitvanish --image:4f1d2c3b-… --timeout:60

• The picture: in the properties panel, Capture next to Image dims the agent's screen; drag a rectangle tightly around the thing to find. The picture is kept with the task (the field holds its id) and shown under the field; runs take it along. A .png, .jpg or .bmp file on the agent machine works too.
• Where: --window limits the search to that window (and brings it to the front); without it the whole screen is searched. Positions count from the window's top-left corner (or the screen's), like mouse and ocr.
• Similarity (--similarity, 50–100, default 90): how alike a spot must be. Brightness and contrast changes are forgiven; lower it for pictures that change a little (anti-aliasing, a slightly different colour), raise it when look-alikes are found instead.
• Waiting: commands wait up to --timeout seconds (10 by default) for the picture to appear; exists does not wait unless you give a timeout.

Note: The picture is found at the size it was captured. Capture it on a machine with the same display scaling (and resolution, for remote sessions) as the agent that runs the task.

#Waiting for the screen

wait has desktop types too:

wait --type:windowexists --window:$erp$
wait --type:windownotexists --window:$progress$ --timeout:300
wait --type:desktopelement --window:$erp$ --by:name --target:"Save"
wait --type:desktopelementgone --window:$erp$ --by:name --target:"Loading"
wait --type:screenchange --window:$erp$ --x:0 --y:40 --width:800 --height:500
wait --type:screenchange --until:stable --stablefor:3 --tolerance:1
wait --type:image --image:"C:\Pictures\ok-button.png" --window:$citrix$ --timeout:60
wait --type:imagegone --image:"C:\Pictures\spinner.png" --timeout:120

image and imagegone wait for a picture to appear on the screen or to go away, with the --image, --window and --similarity of image; Capture picture marks it.

screenchange watches the screen, a window or an area of it:

• --until:change (default) waits until it looks different; --until:stable waits until it stays the same for --stablefor seconds (default 2).
• --tolerance is the percentage of pixels that may change without counting (default 0).

#Mark area

Instead of typing coordinates, click Mark area under Window. The screen is dimmed on the agent machine; drag a rectangle around the part that changes (a status text, a progress area). Esc or a right-click cancels. The area is made relative to the command's window; without one, the editor points Window at the window under the area, or keeps it relative to the screen over the bare desktop. Clear watches the whole window or screen again. screenshot and ocr offer Mark area in the same way.

When a wait's time is up, the task goes on with a warning (--output receives false), unless --failontimeout:true makes the command fail.

#Checking the screen in a condition

To branch on what the screen shows, give if or elseif a screen condition: one block instead of desktop --action:exists (or image --action:exists) followed by if --type:boolean.

if --type:desktopelementexists --window:$erp$ --by:name --target:"Save" --timeout:5
    desktop --action:click --window:$erp$ --by:name --target:"Save"
elseif --type:imageexists --image:"C:\Pictures\save.png" --window:$citrix$
    image --action:click --image:"C:\Pictures\save.png" --window:$citrix$
else
    throw --message:"No Save button on the screen"
endif
if --type:desktopelementnotexists --window:$erp$ --by:name --target:"Loading" --timeout:60
    log --message:"The ERP is ready"
endif

The properties panel shows the same Capture button and object properties table as desktop, and Capture picture with its preview as image. --timeout (default 0) waits up to that many seconds for the element or picture to appear (…exists) or go away (…notexists) before deciding. A missing element, window or picture is an answer, never an error. See if / elseif / else for all condition types.

#Clipboard

clipboard reads and writes the text on the Windows clipboard of the signed-in user:

sendkeys --window:$erp$ --keys:"[CTRL+A][CTRL+C]"
clipboard --action:gettext --output:copied
clipboard --action:settext --text:$orderNo$
clipboard --action:clear

gettext returns an empty value when the clipboard holds no text.

#Desktop automation needs a signed-in session

Windows services run in Session 0, which has no desktop: they cannot see or click windows. That is why the Botream Agent runs tasks in its tray app, on the desktop of the signed-in user:

• On installed machines, the BotreamAgent Windows service supervises the tray app: it starts it in the active user session and restarts it after a crash.
• The machine must therefore have a user signed in, with the session unlocked, for desktop and visible-browser steps to work.
• Unattended robot machines sign in to Windows automatically (auto-logon), keep the display on without a screen saver, and move a Remote Desktop session that was disconnected back to the console so it stays unlocked. See Install agents.
• Element capture, the Mark area and Pick tools and the desktop recorder also run in the tray app, so they need it on the computer you are working on.

#Related pages

• Finding elements reliably
• Recording: record clicks and typing in Windows programs.
• Agents