Renderer

Renderer-only flow

Drive the avatar engine directly from your own application.

@zeroweight/renderer draws an avatar on a canvas and plays its actions. It is framework-agnostic, has no opinion about where your audio comes from, and is the whole of what we ship for rendering. This page is the advanced path, and it is now the only one.

The hosted conversation flow is gone. The Chat API and the managed React session component are no longer served. What remains is this: you fetch a bundle, you drive the engine, and the conversation loop is yours.

Install#

npm install @zeroweight/renderer

A minimal integration#

Three steps: get the bundle, initialise against a canvas, play an action. The bundle comes from the bundle endpoint, and it should be fetched through your own backend so your API key stays there.

avatar.tsts

import { ZeroWeightRenderer } from "@zeroweight/renderer";

// Your backend proxies this and injects the API key. See the bundle endpoint.
const { payload } = await fetch("/api/avatar-bundle/avatar_123").then((r) => r.json());

const canvas = document.getElementById("avatar") as HTMLCanvasElement;
const renderer = new ZeroWeightRenderer();

renderer.on("ready", () => console.log("engine ready"));
renderer.on("dimensions", (width, height) => console.log(`avatar is ${width}×${height}`));
renderer.on("error", (error) => console.error(error));

await renderer.init(canvas, { payload });

renderer.play("speaking");                 // a looping action
renderer.play("wave_hand", "listening");   // a oneshot, then fall back to a loop

// Later, when the view goes away. The engine holds WASM memory and a render
// loop; neither is reclaimed on its own.
renderer.destroy();

Engine API#

init(canvas, config)#

payloadstringrequired

The bundle payload from the API.

fullWidthbooleanoptionaldefault false

Scale the avatar to cover the full canvas width, top-aligned. The bottom may be clipped on a ratio mismatch, like CSS object-fit: cover with object-position: top.

scalenumberoptional

Override the computed scale factor.

forceMobilebooleanoptionaldefault false

Override the built-in mobile detection.

Methods and properties#

Member	What it does
`play(actionId, fallback?)`	Play an action. With a fallback, a oneshot transitions into that action when it finishes.
`interrupt()`	Stop the current action immediately.
`onOneshotComplete(cb)`	Subscribe to oneshot completions; returns an unsubscribe function.
`destroy()`	Release the render loop, the WASM memory and the canvas.
`state`	`idle` · `loading` · `ready` · `error`
`isReady`	True once the engine can play actions.
`dimensions`	The avatar’s intrinsic { width, height }.

Events#

ready, actionLoaded, allActionsLoaded, dimensions, oneshotComplete, error, stateChanged.

renderer.on("allActionsLoaded", () => enableUi());

const unsubscribe = renderer.onOneshotComplete((actionId) => {
  console.log(`finished: ${actionId}`);
});

Syncing to speech#

Two helpers ship alongside the engine for the case everyone hits: making the avatar look like it is saying what it is saying.

ActionQueue dispatches actions with turn-based queuing, so a gesture requested mid-sentence lands at a sensible moment rather than cutting the mouth off.
VoiceActivityDetector does amplitude-based detection over the Web Audio API, with hysteresis, so the mouth follows the audio instead of a timer.

speech-sync.tsts

import {
  ZeroWeightRenderer,
  ActionQueue,
  VoiceActivityDetector,
} from "@zeroweight/renderer";

const queue = new ActionQueue((actionId, fallback) => renderer.play(actionId, fallback));

const vad = new VoiceActivityDetector({
  threshold: 0.008,
  speechStartFrames: 1,
  speechPauseFrames: 30,
  turnEndFrames: 50,
});
queue.attachVoiceActivityDetector(vad);

// Drive the mouth from whatever is producing audio: a microphone track, or a
// MediaStreamTrack you built from the speech API's PCM frames.
queue.startVoiceActivityDetection(mediaStreamTrack);

queue.setTurnActive(true);      // the agent has the floor
queue.dispatch("wave_hand");    // executed now, or queued until the turn allows
queue.forceListening();         // back to the idle loop

queue.stopVoiceActivityDetection();

Pairing this with the streaming speech API is the whole integration: PCM frames feed both your player and the detector, and the detector drives the avatar. The mouth then follows the audio the listener is actually hearing, jitter buffer included, rather than the moment a frame arrived.

Two ways to use it#

Level	What you use	When
Raw	`ZeroWeightRenderer`	You have your own action scheduling and your own idea of when the avatar should move.
Queued	`+ ActionQueue` `+ VoiceActivityDetector`	You want speech-synced behaviour without writing the state machine.