Skip to content

Chunk Kokoro text up front and fix the Nemotron parity metric - #531

Merged
ivan-digital merged 3 commits into
mainfrom
fix/kokoro-chunk-hard-cap
Oct 24, 2026
Merged

ivan-digital merged 3 commits into
mainfrom
fix/kokoro-chunk-hard-cap

Conversation

@ivan-digital

Copy link
Copy Markdown
Member

Summary

The packed-LSTM Kokoro models (#518) speak at upstream Kokoro's rate, so one pass of the 5 s graph holds about 56 tokens. Two things still assumed the faster models:

  • synthesizeChunks first tried any text up to 128 tokens in one pass and discarded the result on overrun (Render long Kokoro text chunk by chunk #516). Most two-sentence replies now overrun, so their first audio waited for a pass that was thrown away.
  • E2ENemotronHarshAudioTests.testStreamingMatchesBatchOnCleanLongUtterance has failed on main since the Core ML re-export (recall 0.923 and 0.949 against a 0.95 bar). CI skips the E2E suites, so it didn't show.

What changed

  • Chunk up front. Text goes straight to the chunker, as in speech-core. Text within the 56-token budget is still one chunk and one call. A short last chunk now merges only within 64 tokens (chunkTokenHardCap), the hard cap of speech-core's 5.0 s profile (Pin Kokoro's packed-LSTM graphs and size chunks for their speech rate speech-core#160), instead of within the 128-token input. The plan lives in KokoroTTSModel.chunkPlan, and three tests pin its budget and cap.
  • Parity metric. The Nemotron test now scores both transcripts against the paragraph Kokoro spoke. The stream must stay within 0.05 of the batch pass, about two words, and the batch pass must reach 0.85, so a broken fixture still fails. The old metric scored the stream against the batch transcript, so every word the batch pass misheard counted against the stream. On main the two passes only misheard "costs declined. Our" differently ("Ledecland" against "Ledeckland", "Our" against "are").
  • testLongTextIsNotClippedToOnePass accepts a peak of exactly 1.0, which the ±1 clamp from Size Kokoro chunks and trimming for the packed-LSTM models #518 can produce.
  • docs/models/kokoro-tts.md gives the current budget (56) and cap (64). It also notes that a sentence longer than one chunk is split between words.

Measurements

M5 Pro, re-exported aufklarer/Kokoro-82M-CoreML, generateStream, median of 7:

First audio main This branch
Two-sentence reply, 119 tokens 273 ms 131 ms
Its first sentence alone, 47 tokens 135 ms 131 ms

The Nemotron parity paragraph is now 22.8 s of audio. On this branch, batch and stream both recover 0.949 of the spoken text.

Tests

  • KokoroTTSTests: 102 tests, 0 failures, 1 skipped (env-gated German parity). This includes the three new plan tests and the E2E tests against the re-exported models.
  • Nemotron E2E suites (E2ENemotronHarshAudioTests, E2ENemotronLangTagLeakTests, E2ENemotronStreamingASRTests): 18 tests, 0 failures, 2 skipped (English-only bundle not installed), in two full runs.
  • CI

testNoLangTagOnOverlappedSpeech returned an empty transcript in one of five runs. Its 4.05 s two-voice mix is the same on main and here, and its transcript varies from run to run ("Please verify your credentials and confirm the booking time", "Sidensify your credentials"). That flake predates this change and isn't addressed here.

Related: #516, #518, soniqo/speech-core#160.

The packed-LSTM models speak at upstream Kokoro's rate, so one 5 s pass
holds about 56 tokens. Trying any text up to 128 tokens whole first now
overruns for most two-sentence replies, and the discarded pass delayed
the first audio: a 119-token reply streamed its first chunk after 273 ms
instead of 131 ms. Text is now chunked the way speech-core does it, with
a short tail merged only within the 64-token hard cap of the same
profile. Text within the budget is still one chunk and one call.

The long-text test accepts a peak of exactly 1.0, which the output clamp
can produce, and the Kokoro page gives the current budget and cap.
The test measured how many batch-transcript words the stream also had,
so every word the batch pass misheard counted as a streaming loss. On
the re-exported Kokoro audio it fell under its 0.95 bar (0.923 and
0.949) with nothing missing from the stream: the two passes only
misheard "costs declined. Our" differently ("Ledecland" against
"Ledeckland", "Our" against "are"). Both transcripts are now scored
against the paragraph Kokoro spoke, and the stream must stay within
0.05 of the batch pass, about two words of it. The batch pass must
recover 85% of the paragraph, so a broken fixture still fails.
@ivan-digital
ivan-digital merged commit 6ba2866 into main Oct 24, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant