Understand image-text alignment without confusing it with grounding
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Compute cosine similarity for normalized feature vectors.
- Explain contrastive image-text learning conceptually.
- Distinguish semantic similarity from spatial localization.
Shared representations make comparison possible
An image encoder and a text encoder can map different modalities into vectors with comparable geometry. If a photo of a red bicycle and the phrase red bicycle have nearby representations, retrieval can rank that photo above a picture of a kettle. CLIP studies learning transferable visual representations from image and natural-language supervision using a contrastive objective. [CLIP paper](https://arxiv.org/abs/2103.00020). The vectors in this lesson are hand-created so the geometry is easy to inspect; they are not outputs from a trained model.
Representations compress information. A vector useful for object category may not preserve the exact position of a small button or a serial number. Before choosing a model, identify the information the action requires. Matching a screenshot to the concept spreadsheet is much easier than locating the specific cell containing a negative value in row seventeen.
Normalize before comparing direction
Cosine similarity is the dot product divided by the product of vector lengths. For vectors (3, 4) and (6, 8), the dot product is fifty and the norms are five and ten, so cosine similarity is one. Their magnitude differs, but their direction is identical. For (3, 4) and (4, -3), the dot product is zero, so their directions are orthogonal. Zero vectors require an explicit policy because normalization would divide by zero.
Similarity is not a calibrated probability that a caption is true. A score of 0.82 cannot by itself mean eighty-two percent confidence. Ranking also depends on the candidate set: the best of three poor captions is still a poor caption. Include an abstention rule based on validation data or require another check when the top candidates are too close for the task's consequences.
Contrastive learning needs useful negatives
A contrastive training example pairs a matching image and caption and compares them with mismatched candidates. The learning signal encourages the matching pair to score higher. If all negative captions describe completely unrelated objects, the model may learn coarse distinctions while failing on subtle ones such as red mug versus blue mug. Harder negatives test whether the representation captures the relevant attribute, but incorrect negative labels can damage learning.
Consider three captions for a screenshot: settings dialog, settings dialog with notifications disabled, and settings dialog with notifications enabled. A global image representation may strongly recognize the first phrase while barely separating the other two. If the task is to change notification state, crop the relevant region or use a model and observation method suited to fine detail. A representation's training objective shapes what it preserves, but does not guarantee every downstream distinction.
Grounding requires an additional association
Image-text retrieval answers which image or region is semantically related to a description. Spatial grounding also needs coordinates or a region identifier. If two identical icons appear in different rows, a global similarity score cannot identify which belongs to the requested account. Associate candidate regions with semantic context such as row label, nearby text, and stable element identity, then select within that constrained set.
A practical pipeline can first identify relevant regions, then compare descriptions within each region, then validate the chosen element against accessibility data. Keep uncertainty attached to each stage. Cropping might omit a label needed for disambiguation; OCR might misread a minus sign; the selected region might move before the click. A high score at one stage must not erase uncertainty introduced elsewhere. The executor still needs current-state checks and a task-specific postcondition.
Measure the representation where it will be used
Build evaluation examples around the actual distinction your agent needs. A gallery of common objects does not test whether a desktop assistant can distinguish a selected checkbox from a disabled one. Include visually similar alternatives, text-rich screens, small objects, and cases outside the training distribution you expect. Separate retrieval rank, attribute correctness, localization error, and final action success when reporting results.
The numerical demo ranks three synthetic candidate vectors against a query, and illustrates that direction rather than magnitude drives cosine similarity. Changing one coordinate reveals how rankings respond. It does not load an encoder, recognize an image, or establish a benchmark result. This small calculation is valuable because it exposes assumptions that a large SDK can hide. Once the mechanism is understood, a real model integration should preserve the same explicit inputs, ranking interpretation, and limits.
Work through the code
This deterministic teaching simulation ranks hand-authored vectors using cosine similarity. It does not run CLIP or any vision model. Change the candidate vectors to explore direction, zero-vector rejection, and ties; do not interpret the scores as probabilities.
import math
def cosine(left, right):
if len(left) != len(right):
raise ValueError('dimension mismatch')
norm_left = math.sqrt(sum(x * x for x in left))
norm_right = math.sqrt(sum(x * x for x in right))
if norm_left == 0 or norm_right == 0:
raise ValueError('zero vector')
return sum(x * y for x, y in zip(left, right)) / (norm_left * norm_right)
query = (3, 4)
candidates = {'same_direction': (6, 8),
'partial_match': (4, 3),
'orthogonal': (4, -3)}
ranked = sorted(((name, cosine(query, vector))
for name, vector in candidates.items()),
key=lambda row: (-row[1], row[0]))
for name, score in ranked:
print(f'{name}: {score:.3f}')
print('probabilities: no')
same_direction: 1.000 partial_match: 0.960 orthogonal: 0.000 probabilities: no
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A global screenshot embedding ranks settings dialog highly, but you need the location of its enabled notifications checkbox. What information is missing?
Check your understanding
Two vectors are (1, 2) and (10, 20). What is their cosine similarity?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.