Workspace/Mini projects
Loading progress
All mini projects
MODULE 16 · 5 HOUR BUILD

Multimodal command evidence ledger

Build a timeline that joins spoken commands to screen observations and emits reviewable action proposals, retaining uncertainty when evidence is stale or missing.

Build evidence Record your actual checks, results, and limitations.

Build it in stages

  1. Define frame, utterance, pointer, and proposal schemas with event and receive times.
  2. Implement offline and causal temporal joins with explicit freshness limits.
  3. Attach semantic control identity and window revision to matched commands.
  4. Create fixtures for delayed audio, clock offsets, missing frames, and ambiguous references.
  5. Report alignment results and abstentions in a reproducible JSON artifact.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

  • Every proposal records the supporting frame and utterance interval.
  • Commands outside the freshness bound are left unresolved.
  • Offline and causal modes differ on a fixture containing a future frame.
  • No simulated command directly executes a desktop action.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py
python
import json

FRAMES = [
    {'time': 100, 'id': 'f1', 'window': 'chart-A', 'revision': 1},
    {'time': 500, 'id': 'f2', 'window': 'chart-B', 'revision': 2},
]
COMMANDS = [
    {'start': 440, 'end': 560, 'text': 'export this chart'},
    {'start': 1200, 'end': 1300, 'text': 'export this chart'},
]

def propose(command, frames, max_gap=150):
    midpoint = (command['start'] + command['end']) / 2
    nearest = min(frames, key=lambda row: (abs(row['time'] - midpoint),
                                          row['time'], row['id'])) if frames else None
    proposal = {'utterance': command, 'status': 'needs-evidence',
                'frame': None, 'action': None}
    if nearest is None or abs(nearest['time'] - midpoint) > max_gap:
        return proposal
    proposal.update({'status': 'ready-for-review', 'frame': nearest['id'],
                     'window': nearest['window'], 'revision': nearest['revision'],
                     'action': 'export-chart'})
    return proposal

def main():
    proposals = [propose(command, FRAMES) for command in COMMANDS]
    for proposal in proposals:
        print(json.dumps(proposal, sort_keys=True))
    ready = sum(row['status'] == 'ready-for-review' for row in proposals)
    print('reviewable:', ready)
    print('unresolved:', len(proposals) - ready)

if __name__ == '__main__':
    main()

Push it further

Add recorded media from a controlled local task and compare semantic, temporal, and spatial errors independently before integrating an operating-system executor.