Summary
Animate the mascot's mouth while the realtime ElevenLabs voice agent is speaking, so the Human tab's realtime path has lip-sync the way the classic tap-and-speak path already does.
Problem
The mascot sits with a frozen mouth for the entire realtime call. Tap-and-speak animates it; realtime does not, so the surface we are moving users onto is the one that looks broken — the mascot appears asleep while the agent talks.
The two paths get their audio differently, which is why the existing lip-sync does not simply carry over:
- Tap-and-speak synthesises through our own TTS proxy, so we own the audio element and hold a viseme timeline — frames of
{viseme, ms}. useHumanMascot samples that timeline every animation frame against playbackRef.current.currentMs() (findActiveFrame).
- Realtime hands playback to the ElevenLabs SDK. There is no audio element of ours to read, so there is no
currentMs() to sample and no timeline to sample against — the two inputs the existing lip-sync needs are both absent.
What the SDK does expose is the output signal itself (getOutputVolume()), plus character-level timings via the onAudioAlignment callback. useRealtimeVoiceSession already forces connectionType: 'websocket' specifically "so the per-audio-event character alignment is available for mascot lip-sync" — the intent was designed in and never wired.
Solution
Two stages, because they differ sharply in cost and risk.
Stage 1 — amplitude-driven (this issue). Drive the mouth from getOutputVolume() on an animation-frame loop. Less accurate than visemes — it opens and closes with the envelope rather than forming phonemes, so no M/F closures — but in sync by construction, because it is the audio being played rather than a prediction of it. A frozen mouth reads as broken; an approximate one reads as alive.
Stage 2 — alignment-driven visemes (follow-up). Build a viseme timeline from onAudioAlignment and reconstruct a clock from wall time anchored at the alignment event, matching tap-and-speak quality. Riskier: wall-clock drift when the SDK buffers or the user interrupts. Most of the mapping already exists — visemeMap.ts maps ElevenLabs/Oculus viseme codes to mouth shapes, and ttsClient.ts already derives rough visemes from char-level alignment when the backend omits them. Stage 1 stays as its fallback for when alignment is absent or the timeline runs dry, mirroring the hedge the TTS path already makes.
Scope: app only. No core, backend, or API change.
Acceptance criteria
Related
Summary
Animate the mascot's mouth while the realtime ElevenLabs voice agent is speaking, so the Human tab's realtime path has lip-sync the way the classic tap-and-speak path already does.
Problem
The mascot sits with a frozen mouth for the entire realtime call. Tap-and-speak animates it; realtime does not, so the surface we are moving users onto is the one that looks broken — the mascot appears asleep while the agent talks.
The two paths get their audio differently, which is why the existing lip-sync does not simply carry over:
{viseme, ms}.useHumanMascotsamples that timeline every animation frame againstplaybackRef.current.currentMs()(findActiveFrame).currentMs()to sample and no timeline to sample against — the two inputs the existing lip-sync needs are both absent.What the SDK does expose is the output signal itself (
getOutputVolume()), plus character-level timings via theonAudioAlignmentcallback.useRealtimeVoiceSessionalready forcesconnectionType: 'websocket'specifically "so the per-audio-event characteralignmentis available for mascot lip-sync" — the intent was designed in and never wired.Solution
Two stages, because they differ sharply in cost and risk.
Stage 1 — amplitude-driven (this issue). Drive the mouth from
getOutputVolume()on an animation-frame loop. Less accurate than visemes — it opens and closes with the envelope rather than forming phonemes, so noM/Fclosures — but in sync by construction, because it is the audio being played rather than a prediction of it. A frozen mouth reads as broken; an approximate one reads as alive.Stage 2 — alignment-driven visemes (follow-up). Build a viseme timeline from
onAudioAlignmentand reconstruct a clock from wall time anchored at the alignment event, matching tap-and-speak quality. Riskier: wall-clock drift when the SDK buffers or the user interrupts. Most of the mapping already exists —visemeMap.tsmaps ElevenLabs/Oculus viseme codes to mouth shapes, andttsClient.tsalready derives rough visemes from char-level alignment when the backend omits them. Stage 1 stays as its fallback for when alignment is absent or the timeline runs dry, mirroring the hedge the TTS path already makes.Scope: app only. No core, backend, or API change.
Acceptance criteria
.github/workflows/ci-lite.yml).Related