feat: add Apple Speech (SpeechAnalyzer) model on macOS 26+ - #1680
feat: add Apple Speech (SpeechAnalyzer) model on macOS 26+#1680akurilin wants to merge 12 commits into
Conversation
The System section header and badge borrowed theme.options.system, which translators localized in the theme-selector context; models now use dedicated keys. Also register the model under onboarding.models so its name/description localize like every other model, and rename it to "Apple Speech" so the user-facing name doesn't track Apple's internal module names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SpeechTranscriber punctuates well but always emits lowercase sentence starts, and its API offers no casing option (DictationTranscriber has one, but its punctuation is pause-driven and drops most marks in continuous dictation — tried and reverted). Uppercase the first letter and each letter after sentence-final punctuation instead, engine-side so other engines' native casing is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The export emits a trailing space after union members; an editor trim removed one, so every cargo check left the file dirty again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ssets The asset download is owned by macOS, fires at most once per locale, and cannot be re-triggered for testing, yet reporting its percentage required an FFI callback with a polling task, fake byte totals impersonating the catalog download events, two extra events to suppress the download- completion side effects, and progress-bar changes that leaked into every model card. The existing "Switching…" loading state already covers the wait, so keep the blocking install and state that first-use downloads happen in the model description instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SpeechAnalyzer load path hands effective_language()'s result to the OS as a locale identifier, so the prefer-English fallback is the only thing keeping the default "auto" setting from reaching ffi::prepare — pin it. Extract speech_analyzer_languages() so the BCP-47 → picker-code collapse (dedupe, zh/yue split, selection flag) is testable, and cover the ellipsis case of capitalize_sentence_starts, the sentence-end character a matcher rewrite would most plausibly drop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Name the SpeechAnalyzer engine explicitly and say assets are managed by macOS, not just downloaded on first use — the OS can also evict them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thank you so much for this, and also thank you for adding screenshots. I will take a review and test of this very shortly, or hopefully, within the next few days. I think the only thing that I immediately see is that for the system models versus downloaded models, I actually think we can probably just keep this all within 'downloaded' for now. Unless this actually makes things more complex. The main reason is I do want to do some more testing with the model and possibly this would change recommendation orders and stuff like this Otherwise, I will try and take a look this week. Overall, it seems like a really good change for us to have and ultimately be able to use within handy. Regarding the pink, I believe it's for recommended models, but certainly on the onboarding, I think I need to clarify this, and I'll probably take it when I take a pass at this PR. More or less, it either should have a recommended badge or not have the pink color. I think the recommended badge though would be good for that |
|
I had a similiar feature I was testing, that I tagged this as reference to. Please disregard my mention here as I prefer this implementation |
|
Sounds fair to me, removed the separate section for system models. Here's what it looks like now on the main view: 
Onboarding with support: 
And here it is on a machine with no support 
|
|
Excellent thanks |
|
This will be merged soon just thinking about some UX stuff |
Has anything progressed on this? |
|
Be patient |
4 participants
Before Submitting This PR
Please confirm you have done the following:
If this is a feature or change that was previously closed/rejected:
Human Written Description
Adds an "Apple Speech" model backed by the SpeechAnalyzer API on macOS 26+ (Apple Silicon). It shows up in a separate "System Models" section since the OS owns the model assets. There's nothing for Handy to download, store, or delete, it's all managed by the OS. This deviates from the existing pathway for how models are downloaded and deleted on user's command, and adds a new type of model to the system in its own section. The change is inspired by this article.
The Swift bridge follows the same pattern as the existing Apple Intelligence bridge: compiled by build.rs, with a stub fallback when the SDK is older than macOS 26 (you need the Xcode 26 SDK to build the real thing).
One quirk worth knowing: SpeechTranscriber punctuates well but starts every sentence lowercase and has no API option to change that, so there's a small post-processing pass that capitalizes sentence starts (unit-tested).
Caveats: Apple Silicon + macOS 26 only, and building it requires the Xcode 26 SDK (older toolchains compile a stub and the model simply doesn't appear).
Related Issues/Discussions
Discussion: #1031 (Native Apple Speech)
Also related: #1244 (user asking for exactly this)
No existing PR implements this. I searched open/closed/merged PRs for SpeechAnalyzer, Apple Speech, native speech, dictation, etc. (the existing "apple" PRs are all about Apple Intelligence post-processing).
Community Feedback
Requested in #1031 and #1244. @cjpais invited this PR in #1031: "You can open a PR for this I'll pull it in."
Testing
Daily-driving it for real dictation on an M-series Mac (macOS 26): accuracy is good, punctuation lands where you'd expect.
Quick 5x benchmark against a 34s recording of my voice:
Unit tests cover the sentence-capitalization pass and the language/locale invariants the backend relies on.
Verified the model is correctly hidden on unsupported environments (tested in a macOS 26 VM, where
SpeechTranscriber.isAvailableis false).Screenshots/Videos (if applicable)
Onboarding with no access to the API:
Onboarding with access to the API:
Note: it would be neat to clarify what exactly the pink background means on that UI page, I wasn't sure if pink meant those were the suggested defaults, or if those models were already downloaded, or if that was a pre-selected bundle I needed to download all at once.
App's main screen after onboarding, when the API is available on the system:
AI Assistance
If AI was used:
Codex 5.6 Sol on Extra High and Claude Code (Fable 5) on High. Heavily assisted on the code; I reviewed, tested, and have been running it all day for my own usage. Some of the changes go beyond my expertise, so feedback is very welcome. Everything added already had an existing pattern in the codebase that the agent could piggyback off of without having to get creative.