Skip to content

feat: voice dictation to the chat composer - #321

Open
BartSu wants to merge 1 commit into
alphaXiv:mainfrom
BartSu:feat/composer-voice-input
Open

feat: voice dictation to the chat composer#321
BartSu wants to merge 1 commit into
alphaXiv:mainfrom
BartSu:feat/composer-voice-input

Conversation

@BartSu

@BartSu BartSu commented Sep 14, 2026

Copy link
Copy Markdown

What

Adds voice input to the chat composer. A mic button next to the attach button dictates into the draft using the browser's own Web Speech API — no API key, no backend endpoint, and no audio passing through orx. Only ui/ changes; the Rust side is untouched.

Interim words appear in the field as they are heard and are rewritten in place until the phrase settles.

How

ui/src/useDictation.ts (new) wraps SpeechRecognition:

  • Language follows the UI locale (enen-US, fafa-IR, zh-CN as-is); switching it restarts the recognizer in place.
  • Insertion is anchored where listening began, and re-anchors at the caret whenever the field no longer holds what we last wrote — so typing and speaking interleave without either clobbering the other. Results already shown become part of the user's text, and later revisions of them are dropped rather than painted twice.
  • Idempotence: each event rebuilds the text from the session's result list instead of appending to it, so an engine that re-delivers results it already settled (Safari) cannot double a phrase.
  • Continuity: Chrome ends a session on silence, so sessions restart automatically. A strike budget gives up on a recognizer that keeps ending the instant it starts, instead of spinning on start/end.
  • Spacing is decided in one place (joinHeard) for all three boundaries: existing draft ↔ dictation, settled ↔ interim, and across a session restart, where the first transcript does not lead with a space.

ui/src/components/ChatPanel.tsx:

  • Mic button with three states — idle, listening (red, pulsing), and unsupported (visible but disabled, with a tooltip naming the browser requirement, since there is nothing actionable on Firefox).
  • The button declines focus on mousedown, so a caret placed mid-message survives the click.
  • Errors the user can act on (mic blocked, no device, offline) surface above the composer; no-speech and aborted are normal and stay silent.
  • send() and runShell() stop listening.
  • Dictation writes through the same setDraft path as the keyboard, so skill chips, the slash menu, and send need no special case.

New message keys are added to all three catalogs (en, zh-CN, fa).

Privacy note

Chrome's Web Speech implementation sends audio to Google's recognition service; Safari sends it to Apple's. This is worth knowing given the project's local-first framing — nothing is routed through orx itself, but the audio does leave the machine via the browser. Happy to add a line to the settings UI saying so, or to swap in a local engine later, if you'd prefer.

Testing

  • node ui/scripts/check-i18n.mjs, node ui/scripts/check-styles.mjs, pnpm typecheck, pnpm build, and the 162 UI unit tests all pass.
  • ui/dist is regenerated and committed.
  • The recognition path was driven end-to-end in the running dashboard with a stubbed SpeechRecognition, covering: interim rewritten in place, phrases settling, new phrases appended, session restart continuity and its word boundary, re-delivery of settled results, typing before the first result, typing mid-phrase followed by that phrase settling, and stop/start.
  • The live path was also confirmed against the real recognizer in Chrome (start, focus, stop), though transcription accuracy itself was not measured.

Dictation through the browser's own Web Speech API — no key, no backend,
no audio through orx. A mic button next to the attach button starts and
stops it; interim words appear in the field as they are heard and are
rewritten in place until the phrase settles.

Recognition listens in the UI language. Insertion is anchored where
listening began and re-anchors at the caret when the value changes
underneath it, so speaking and typing interleave without either
clobbering the other. Each event rebuilds the text from the session's
result list rather than appending to it, which keeps an engine that
re-delivers settled results (Safari) from doubling a phrase. Chrome ends
a session on silence, so sessions restart automatically under a strike
budget that gives up on a recognizer that will not start.

The button stays visible but inert where the browser has no speech API,
and errors the user can act on — mic blocked, no device, offline —
surface above the composer. Sending or running a shell command stops
listening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBW3eGgo7p44Prbh7xwHyk
@BartSu BartSu changed the title Add voice input to the chat composer feat: voice dictation to the chat composer Sep 14, 2026
@BartSu

BartSu commented Sep 14, 2026

Copy link
Copy Markdown
Author

Could a maintainer please approve the GitHub Actions workflows for this PR? The required checks are currently waiting for approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant