Handle realtime voice as a playback state machine
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Separate generated, buffered, and played audio.
- Invalidate stale response chunks after interruption.
- Reconcile conversation state with what the user heard.
A voice response has several positions
A model can generate audio faster than a listener hears it. Track at least three positions: generated audio, audio queued for playback, and audio actually played. If the user interrupts after hearing one second of a five second response, the remaining four seconds must not remain scheduled for playback. The conversation state should also avoid assuming that the user heard the unplayed portion.
This distinction is not specific to one model provider. It follows from asynchronous generation and playback. A text interface often renders tokens immediately, while an audio interface has a physical playback rate. Network buffering and device latency add another gap. Design the conversation controller around measured playback progress and response identity, rather than treating a generation-complete event as evidence that the whole response reached the user.
Transport changes who owns media state
OpenAI's Realtime documentation distinguishes server-managed output buffering for WebRTC and SIP from client-managed playback over WebSocket. With WebSocket playback, interruption handling includes stopping local audio and sending conversation.item.truncate using the played offset. The exact event sequence and configuration depend on the transport and turn-detection settings. [OpenAI Realtime conversation documentation](https://developers.openai.com/api/docs/guides/realtime-conversations).
Treat that as a versioned integration contract, not a reason to hide state inside an SDK callback. Your application still needs to connect the interrupted response to its UI transcript, pending tools, and new user input. A canceled audio response does not automatically cancel a business operation already sent to another service. Distinguish media cancellation, reasoning cancellation, and external-effect cancellation in the orchestration layer.
Use response epochs to discard late chunks
After an interruption, old audio chunks may still arrive from network buffers or pending callbacks. Assign every response a stable identifier or generation epoch. The playback consumer accepts chunks only for the currently active response. On interruption, invalidate that response and clear its unplayed queue. A later response starts with a new identity, so delayed chunks cannot accidentally become part of the new answer.
Suppose response R1 has chunks A, B, and C. A plays, then the user speaks. B and C are cleared, and a delayed D from R1 is ignored. Response R2 begins with E. The listener has heard A and E, not A through E. The simulation represents chunks as short strings to make this trace visible. Production playback usually tracks sample counts or milliseconds and must account for the device buffer's actual progress.
Turn detection is a policy with tradeoffs
Voice activity detection can make interaction feel natural by noticing when the user starts and stops speaking. It can also react to background noise, short pauses, or another speaker. Push-to-talk provides a clearer boundary but requires an explicit gesture. Neither choice removes the need to support interruption and recovery. Choose based on the environment and task, then evaluate false starts, missed interruptions, and response latency.
Avoid interpreting a partial transcript as a final irreversible command. The phrase do not send can initially appear as do while more audio is arriving. Maintain provisional text until a defined turn boundary, and use explicit action proposals when the request requires review. For low-risk conversational replies, early generation may be acceptable if the controller can cancel it. Different actions deserve different commitment rules even within the same voice session.
Test conversations as event traces
A deterministic trace can reveal bugs that are hard to reproduce with a microphone. Include chunk arrival, playback advancement, user-speech start, cancellation, late old chunks, and a new response. Assert the final heard sequence and remaining queue. Also test interruption before any audio plays and after generation finishes but while playback remains buffered. The latter case is common when a short answer is generated quickly.
Measure time to first audible response, interruption-to-silence delay, and stale-audio leakage separately. A low generation latency does not compensate for an assistant that keeps talking over the user. When exporting traces, distinguish synthetic test audio from recordings and retain only the media necessary for debugging. The lab implements the small state reducer at the center of this design. It gives you a testable contract before you add codecs, microphones, networking, and provider-specific events.
Work through the code
This teaching simulation uses strings as audio chunks. It demonstrates response identity, queue clearing, and late-chunk rejection only. It does not implement Realtime API events, audio timing, codecs, playback, or transcription.
active = 'R1'
queue = []
heard = []
def receive(response, chunk):
if response == active:
queue.append(chunk)
receive('R1', 'A')
receive('R1', 'B')
heard.append(queue.pop(0))
active = None
queue.clear()
receive('R1', 'late-C')
active = 'R2'
receive('R2', 'E')
heard.extend(queue)
queue.clear()
print('heard:', heard)
print('queued:', queue)
print('active:', active)
heard: ['A', 'E'] queued: [] active: R2
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
The model finished generating R1, but three seconds remain queued when the user interrupts. Is canceling generation sufficient?
Check your understanding
A chunk tagged R1 arrives after R2 becomes active. What should the playback reducer do?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.