Control a desktop through observations and bounded actions

A desktop agent operates in a partially observed environment. A screenshot shows visible pixels, but not necessarily the focused control, clipboard contents, pending background saves, or hidden modal windows. The same keyboard shortcut can perform different actions depending on focus. Before typing a filename, the agent needs evidence that the intended dialog and field are active. Treat focus as a precondition rather than assuming the last click established it forever.

Define an observation record containing the active application, window identity, screen dimensions, observation revision, visible regions, and available semantic controls. A planner proposes a short action with expected preconditions and postconditions. An executor validates these facts immediately before acting. This architecture turns a vague instruction such as export the chart into an inspectable series of state transitions that can stop when the interface differs from expectations.

Desktop accessibility frameworks can expose programmatic properties and actions for controls. Microsoft UI Automation, for example, lets applications provide and consume information about desktop interfaces and supports automated interaction. [Microsoft UI Automation documentation](https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32). These properties can help identify a Save button even when its screen position changes, but coverage depends on the application and control implementation.

A screen reader tree is not a perfect copy of the screenshot. A custom canvas may expose little semantic structure, while a hidden control may remain represented in an automation tree. Fuse the views deliberately: use semantic identity where available, use pixels for appearance and spatial context, and verify that both refer to the same window revision. Platform accessibility APIs differ, so the standard-library examples here use explicit teaching records rather than inventing a universal desktop SDK.

A desktop may have multiple monitors, mixed display scaling, and windows that move. A point expressed relative to a cropped screenshot is different from a point in global screen coordinates. Suppose a window begins at 100, 60 and its captured image is displayed to a model at half size. A model point of 80, 50 corresponds to screen point 260, 160 if the crop and action coordinates share the stated scale. A second monitor with another origin requires another mapping.

Store transforms with the observation, not in a global constant remembered from startup. Recompute after resizing, monitor changes, or remote-session reconnects. Validate bounds and test known corners. The seed maps a local rectangle into an explicit window frame; it does not infer scaling, inspect a real display, or dispatch mouse input.

Desktop operations range from reading a chart to overwriting a file. Use small reversible steps until a concrete action is ready, then apply the authorization appropriate to that action. A dialog showing a proposed export path is useful review evidence. It does not justify sending unrelated messages or replacing an unrelated document. Keep the target, content, and relevant user intent attached to the proposed operation.

Verify outcomes through the strongest available observation. A file export can be checked through the application's export status and, when permitted, the resulting file's existence and basic metadata. A closed dialog alone is weak evidence because Cancel also closes a dialog. If a timeout occurs after an action that might already have completed, inspect state before repeating it. Blind retries can create duplicate exports or apply a keyboard shortcut in the wrong application.

OSWorld evaluates multimodal agents on tasks in real computer environments, including interactions that span applications. It motivates measuring completed tasks rather than only isolated clicks. [OSWorld paper](https://arxiv.org/abs/2404.07972). For your own workflow, enumerate assumptions the agent relies on: current window, language, focus, scaling, selected document, and save behavior. Then create fixture cases that violate one assumption at a time.

A useful test set includes a moved window, an unexpected confirmation dialog, a stale observation, and a target whose visible label differs from its accessible name. Record whether the agent detects the mismatch before acting and whether its final artifact meets the task contract. Keep perception accuracy, executor reliability, and business success as separate measures. This separation tells you whether to improve grounding, state tracking, or the plan instead of treating every failure as a generic model limitation.

This teaching simulation checks window, revision, and focus before mapping a local point to screen coordinates. All frame dimensions are invented and share one coordinate system. It performs no desktop actions and has no operating-system dependencies.