Combine visual grounding with fresh browser state
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Map screenshot coordinates into an action coordinate frame.
- Handle visual ambiguity and stale observations.
- Design fallbacks between semantic and pixel evidence.
Why pixels sometimes matter
Semantic browser observations work well for labeled controls and structured text. They may omit a canvas diagram, the identity of a product pictured in a card, or a color legend needed to interpret a chart. Visual grounding links a language description to a visible region: the red triangle, the leftmost matching shoe, or the point representing August. A screenshot supplies appearance but not automatically the hidden meaning or interactability of every region.
VisualWebArena studies web tasks that require visual understanding, while WebArena provides realistic task environments and outcome evaluation. These research settings motivate testing visual requirements explicitly; they do not establish that a particular agent will succeed on your application. [VisualWebArena paper](https://arxiv.org/abs/2401.13649), [WebArena paper](https://arxiv.org/abs/2307.13854). Build fixtures matching the actual visual ambiguity and navigation behavior your workflow encounters.
Name every coordinate frame
A screenshot may be resized before a model sees it. The browser action API may expect viewport CSS coordinates, while the image contains device pixels or a cropped region. Write the mapping explicitly. If a 1600 by 900 screenshot is represented to the model at 800 by 450, a point at 300, 200 in the model image maps to 600, 400 in the original screenshot. That still is not necessarily a desktop click until the screenshot's relationship to viewport and device scale is known.
Cropping adds an offset before or after scaling depending on the defined frames. For a crop starting at original pixel 200, 100 and scaled by one half, model point 50, 40 maps to original point 300, 180. Test the transform with corners and known landmarks. Never infer a universal device-pixel ratio from a single successful click.
Freshness is part of grounding
A coordinate can be numerically correct and operationally wrong. Suppose a screenshot shows a Delete control at 600, 400. Before the action executes, a notification banner shifts content downward or the user scrolls. The same coordinate now targets a different control. Store the observation revision with the proposal, and reject or reground after layout-changing events. A recent screenshot is evidence, not a lock on the interface.
Use semantic locators for a target once visual identification has narrowed the region, when the application exposes an appropriate element. If pixel interaction is necessary, verify that the expected window, page, and region remain present immediately before the action, then inspect the result. Repeating the old click after a timeout is especially risky because the first action may already have succeeded and moved the interface into a new state.
Fuse modalities without inventing certainty
Consider two product cards with identical names but different images. The DOM can supply stable card identifiers and price text, while the screenshot distinguishes a blue mug from a green mug. Associate visual evidence with the card's region and then act on the corresponding semantic control. Preserve that association in the trace so a reviewer can see why card B was selected.
When modalities disagree, investigate the disagreement rather than averaging it away. A disabled state in accessibility data may be more relevant to clickability than a button that looks active. An image caption may be outdated, while the picture shows the current item. The executor should understand which source answers which question. A generic confidence score cannot replace explicit checks for identity, target location, action availability, and intended outcome.
Evaluate the entire perception-action chain
Measure target identification separately from action execution and task completion. An agent might identify the right item but click the wrong place because of scaling. Another might click accurately but misunderstand the user's visual description. Log the requested target, screenshot revision, coordinate transform, selected region, executed action, and postcondition. These fields let you diagnose the error without replaying an entire conversation.
Design near-miss cases: visually similar icons, reordered cards, alternate zoom levels, a changed viewport, and a modal covering the original target. Also include a case with insufficient visual evidence where abstaining is correct. A system that always clicks can score well on easy examples while behaving poorly under ambiguity. In this lesson, the code tests a coordinate transform and stale-revision refusal only. It makes no claim to perform visual recognition or to benchmark a multimodal model.
Work through the code
This teaching simulation maps coordinates from a resized crop back to original screenshot pixels, then refuses an action based on a stale revision. It does not click a browser. The crop tuple is left, top, width, height in original pixels.
def map_point(point, model_size, crop):
x, y = point
width, height = model_size
left, top, crop_width, crop_height = crop
if width <= 0 or height <= 0:
raise ValueError('invalid model image size')
if not (0 <= x < width and 0 <= y < height):
raise ValueError('point outside model image')
return (left + x * crop_width / width,
top + y * crop_height / height)
proposal = {'point': (50, 40), 'revision': 7}
mapped = map_point(proposal['point'], (400, 200),
(200, 100, 800, 400))
print('original pixels:', mapped)
current_revision = 8
if proposal['revision'] != current_revision:
print('action: reobserve')
else:
print('action: execute')
original pixels: (300.0, 180.0) action: reobserve
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A 1000 by 600 crop starts at original pixel 120, 80 and is shown at 500 by 300. Map model point 250, 100 and explain what remains unverified.
Check your understanding
After a screenshot, a modal opens and covers the intended target. What should the executor do?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.