TTS

Streaming (WebSocket)

The realtime protocol: every message, in both directions.

One socket, JSON text frames in both directions, and audio as binary. The connection is opened with a URL from the session route. See Authentication for why it is not opened directly.

WSSwss://<gateway>/v1/stream

Session properties#

Everything that describes the session (the voice, the format, the sample rate, the sampling knobs) is a query parameter on that URL, put there by the session route from the body you posted. Which means all of it is immutable for the life of the connection. Changing a voice means opening a second socket; there is no message that changes one mid-stream.

wss://<gateway>/v1/stream
  ?token=zwst_1…            the short-lived connect token
  &voice_id=maichi          a built-in voice, a system voice, or one you cloned
  &format=pcm_s16le         the wire format of the audio frames
  &sample_rate=24000        ask explicitly: a player built at the wrong rate is silently wrong
  &normalize_text=true      expand digits, dates and acronyms before synthesis
  &word_timestamps=true     opt in to the timestamps message

Contexts#

A context_id is one utterance, or one turn of an agent. It is a queue: several synthesize messages on the same context are synthesised in order, and their audio streams out as one continuous timeline with monotonic presentation timestamps. A listener cannot hear where one ended and the next began.

That is the mechanism a voice agent is built on. Send a sentence the moment your language model has finished writing it, rather than waiting for the paragraph, and the model starts speaking while the rest is still being written.

One sentence per synthesize beats one paragraph. The splitter packs sentences toward a 15-second budget rather than emitting one per sentence, so a whole paragraph can arrive as a single long segment, and prefill time, which grows with segment length, sits directly on the time-to-first-audio path.

Sending#

All client messages are JSON text frames.

synthesizeclient → server

Submit text on a context. The server segments it and synthesises the parts in order.

typestringrequired

"synthesize"

context_idstringrequired

Names the queue. Reuse it to append to the same utterance; use a new one for a new turn.

textstringrequired

May span several sentences.

segment_mode"auto" | "none"optionaldefault "auto"

auto lets the server split the text. none says the text is exactly one segment. Over the model’s input limit that fails the connection with 4003 segment_too_long.

{
  "type": "synthesize",
  "context_id": "turn-7",
  "text": "Xin chào. Tôi có thể giúp gì cho bạn?",
  "segment_mode": "auto"
}

endclient → server

No more text is coming for this context. Optional, but send it: it lets the server release the slot immediately rather than waiting for a timeout, and on a metered socket a timeout is money.

{ "type": "end", "context_id": "turn-7" }

cancelclient → server

Barge-in. Stop this context now, drop everything still queued on it, and free the slot. This is what you send when the user starts talking over the agent.

{ "type": "cancel", "context_id": "turn-7" }

Audio frames for a cancelled context can still arrive for a moment afterwards, because they were already in flight. Drop frames whose context you have cancelled rather than assuming the stream stops at the same instant your message is sent.

configclient → server

Adjust delivery for a context that has not started yet.

{ "type": "config", "context_id": "turn-8", "speed": 1.1 }

pingclient → server

An application-level heartbeat, answered with pong. Send one every 20 seconds: intermediaries kill idle sockets, and you want to discover that before your user speaks rather than during.

{ "type": "ping" }

Receiving#

Audio frameserver → client

A binary frame: a 12-byte header followed by the audio payload in the format the session asked for. Not base64 inside JSON, which costs a third of the bandwidth and a parse on every chunk.

byte  0     version (0x01)
byte  1     flags: bit 0 is set on the final chunk of a context
bytes 2-3   context ordinal (uint16, in the order contexts were seen)
bytes 4-7   chunk sequence  (uint32, per context)
bytes 8-11  presentation timestamp in ms (uint32, per context)
bytes 12+   audio payload: mono 16-bit little-endian PCM at sample_rate

Read the samples out explicitly little-endian. Aliasing the buffer with an Int16Array takes the platform’s byte order, and the wire format is pcm_s16le whatever the platform happens to be.

context_startserver → client

The server has accepted a context and begun work on it.

{ "type": "context_start", "context_id": "turn-7" }

segment_startserver → client

One segment of a context has been accepted, carrying queue_depth, the gateway’s backlog for that context.

This is the signal worth wiring up. A growing queue_depth means you are feeding segments faster than they can be synthesised, which is what tells an agent to trim its response rather than keep talking into a queue the listener will wait through.

{ "type": "segment_start", "context_id": "turn-7", "queue_depth": 2 }

timestampsserver → client

Word timings, when the session was opened with word_timestamps=true. They are context-relative: the offset of every preceding segment is already folded in, so lipsync and captions never do the arithmetic.

{
  "type": "timestamps",
  "context_id": "turn-7",
  "words": ["Xin", "chào"],
  "start": [0.04, 0.31],
  "end": [0.29, 0.72]
}

segment_doneserver → client

Every frame of one segment has been delivered. The next segment follows seamlessly.

{ "type": "segment_done", "context_id": "turn-7" }

context_doneserver → client

The context is finished and its audio is complete. This is the message that ends a turn, and it arrives after the final audio frame, so it is safe to treat as the end of the utterance.

{ "type": "context_done", "context_id": "turn-7" }

errorserver → client

Something went wrong mid-stream. The socket may survive it; a fatal problem arrives as a close code instead. See Errors and retries.

{ "type": "error", "context_id": "turn-7", "message": "…" }

pong and resumed also arrive as JSON. resumed means a worker was lost and the gateway rebound the session to another one. See recovery. Neither needs handling; ignore any message type you do not recognise rather than treating it as an error, so a new one can be added without breaking your client.

A turn, end to end#

client                                    server
  │
  ├─ synthesize  context_id=turn-7 ───────▶
  │                              ◀───────── context_start
  │                              ◀───────── segment_start  queue_depth=0
  ├─ synthesize  context_id=turn-7 ───────▶   (the next sentence, queued)
  │                              ◀───────── ▪ audio frame  seq=0  pts=0
  ├─ end         context_id=turn-7 ───────▶
  │                              ◀───────── ▪ audio frame  seq=1  pts=40
  │                              ◀───────── segment_done
  │                              ◀───────── segment_start  queue_depth=0
  │                              ◀───────── ▪ audio frame  seq=2  pts=980
  │                              ◀───────── segment_done
  │                              ◀───────── context_done
  │
  ├─ (next turn on the same socket, or close 1000)

Rules worth following#

Reuse one socket for a whole conversation. Reconnecting per turn costs 40–100 ms of handshake on every utterance, which is the same order as the latency this API exists to deliver.
Consume audio as it arrives. Each session’s outbound queue is bounded at about two seconds. A client that stops reading has frames dropped, and after a sustained run the socket closes with 1008.
Heartbeat every 20 seconds. Silent sockets get killed by intermediaries.
Close when the conversation ends. Billing runs on open-socket time, silence included.

Ordering guarantees#

Within	Guaranteed
One context	Audio frames arrive in sequence order, with monotonic timestamps.
One context	Segments are synthesised and delivered in the order their text was submitted.
Across contexts	None. Use the context ordinal in the frame header to route audio, not arrival order.