feat: voice dictation to the chat composer - #321
Open
BartSu wants to merge 1 commit into
Open
Conversation
Dictation through the browser's own Web Speech API — no key, no backend, no audio through orx. A mic button next to the attach button starts and stops it; interim words appear in the field as they are heard and are rewritten in place until the phrase settles. Recognition listens in the UI language. Insertion is anchored where listening began and re-anchors at the caret when the value changes underneath it, so speaking and typing interleave without either clobbering the other. Each event rebuilds the text from the session's result list rather than appending to it, which keeps an engine that re-delivers settled results (Safari) from doubling a phrase. Chrome ends a session on silence, so sessions restart automatically under a strike budget that gives up on a recognizer that will not start. The button stays visible but inert where the browser has no speech API, and errors the user can act on — mic blocked, no device, offline — surface above the composer. Sending or running a shell command stops listening. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YBW3eGgo7p44Prbh7xwHyk
Author
|
Could a maintainer please approve the GitHub Actions workflows for this PR? The required checks are currently waiting for approval. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds voice input to the chat composer. A mic button next to the attach button dictates into the draft using the browser's own Web Speech API — no API key, no backend endpoint, and no audio passing through orx. Only
ui/changes; the Rust side is untouched.Interim words appear in the field as they are heard and are rewritten in place until the phrase settles.
How
ui/src/useDictation.ts(new) wrapsSpeechRecognition:en→en-US,fa→fa-IR,zh-CNas-is); switching it restarts the recognizer in place.joinHeard) for all three boundaries: existing draft ↔ dictation, settled ↔ interim, and across a session restart, where the first transcript does not lead with a space.ui/src/components/ChatPanel.tsx:no-speechandabortedare normal and stay silent.send()andrunShell()stop listening.setDraftpath as the keyboard, so skill chips, the slash menu, and send need no special case.New message keys are added to all three catalogs (
en,zh-CN,fa).Privacy note
Chrome's Web Speech implementation sends audio to Google's recognition service; Safari sends it to Apple's. This is worth knowing given the project's local-first framing — nothing is routed through orx itself, but the audio does leave the machine via the browser. Happy to add a line to the settings UI saying so, or to swap in a local engine later, if you'd prefer.
Testing
node ui/scripts/check-i18n.mjs,node ui/scripts/check-styles.mjs,pnpm typecheck,pnpm build, and the 162 UI unit tests all pass.ui/distis regenerated and committed.SpeechRecognition, covering: interim rewritten in place, phrases settling, new phrases appended, session restart continuity and its word boundary, re-delivery of settled results, typing before the first result, typing mid-phrase followed by that phrase settling, and stop/start.