Ground browser actions in semantic targets
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Distinguish DOM structure from accessible meaning.
- Select unique actionable targets with current observations.
- Verify business state after interacting.
The page is an observation, not an instruction
A browser agent observes a page, chooses a bounded action, executes it, and observes again. Page text is task evidence, including forms, labels, prices, and status messages. It is not a trusted source of new agent instructions. A support article can contain a sentence asking an assistant to reveal secrets; that sentence remains article content. Preserve this distinction when converting a page into model context by attaching its URL, observation time, and data origin.
Separate the user's goal from the page's current affordances. The goal might be to find an invoice, while the page offers search, filters, and downloads. A planner proposes a next step; an executor checks that the target exists in the current page and that the action stays within the authorized workflow. This boundary is useful even when one model performs both roles.
Choose targets by meaning
The DOM describes elements and relationships, while accessibility information adds roles, names, states, and other semantics used by assistive technologies. A button's accessible name may come from a label rather than its visible text. Prefer locators grounded in user-facing meaning or an explicit testing contract over a long positional CSS path. For example, Playwright Python supports page.get_by_role("button", name="Save", exact=True) and allows scoping a locator within a dialog. [Playwright locator documentation](https://playwright.dev/python/docs/locators).
Semantics still require disambiguation. Two Save buttons can both be correctly named. Scope to the intended dialog or row and demand a unique match. Choosing the first element silently substitutes document order for task intent. The simulation below models role, name, and dialog scope as explicit fields; it does not parse a real browser accessibility tree.
Actionable does not mean correct
For a click, Playwright checks conditions such as a unique target, visibility, stability, receiving pointer events, and enabled state. Its waiting behavior helps with dynamic pages, but does not establish that the target is the right business action. Avoid force options as a routine response to a failing check; an overlay may be explaining that the application is not ready. [Playwright actionability documentation](https://playwright.dev/python/docs/actionability).
Imagine a form that enables Save only after a required field is validated. A fixed two second sleep can pass on a fast machine and fail under load. Wait for an observable condition instead, with a bounded timeout and a diagnostic snapshot on failure. Then inspect the resulting state. An enabled Save button proves only that clicking is possible, not that the invoice was saved under the intended account.
Use preconditions and postconditions
Before an action, assert the page identity, target uniqueness, and relevant business preconditions. After it, look for an outcome that distinguishes success from a mere animation. Saving a draft could be verified by a persisted draft identifier and the updated field value after reload. A toast saying Saved may be useful evidence, but it can also reflect a different background operation. Choose checks that are specific to the user's goal.
In a test fixture, an agent opens invoice 42, changes its reference, and sees a success banner. If the underlying record still contains the old value, the task failed despite the visible message. This is why browser evaluation benefits from state-based checks when available. In real systems, read-only verification through an authorized API may complement UI evidence, provided it does not bypass the workflow's access controls.
Keep the action loop small
A long speculative sequence becomes fragile when the first click changes layout or opens a dialog. Observe after transitions that can invalidate targets, including navigation, modal opening, list filtering, and tab switching. Stable locators can be re-evaluated against current page state; captured element identifiers or coordinates may refer to an older observation. Include a revision number in the action proposal when implementing a simulator or custom executor.
Minimize observation noise without dropping important constraints. Send the current dialog, relevant rows, and nearby labels instead of every hidden script and menu. Retain enough context to distinguish similar controls. If no unique valid target exists, return an explicit blocked result with the ambiguity, then gather more evidence. That behavior is a successful application of the executor contract, even though the business task is not yet complete.
Work through the code
This standard-library teaching simulation filters a supplied semantic snapshot and checks an in-memory record. It is not a Playwright API implementation. Add a second visible invoice Save button to see the ambiguity rejection.
nodes = [
{'id': 'a', 'role': 'button', 'name': 'Save', 'scope': 'profile',
'visible': True, 'enabled': True},
{'id': 'b', 'role': 'button', 'name': 'Save', 'scope': 'invoice-42',
'visible': True, 'enabled': True},
{'id': 'c', 'role': 'button', 'name': 'Save', 'scope': 'invoice-42',
'visible': False, 'enabled': True},
]
def select(role, name, scope):
matches = [node for node in nodes
if (node['role'], node['name'], node['scope'])
== (role, name, scope)
and node['visible'] and node['enabled']]
if len(matches) != 1:
raise ValueError('target must be uniquely actionable')
return matches[0]['id']
print('target:', select('button', 'Save', 'invoice-42'))
record = {'reference': 'OLD'}
record['reference'] = 'REVIEWED'
print('postcondition:', record['reference'] == 'REVIEWED')
target: b postcondition: True
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A page has Save buttons in a profile form and an invoice dialog. The first match is the profile button. Specify the locator strategy and a meaningful postcondition.
Check your understanding
A click returns successfully but the record remains unchanged. How should the agent classify the task?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.