Accurate or fully local?
You don’t have to choose.
够准和完全本地
不必二选一
Eight speech engines, nine runs, one recording. Whisper large-v3-turbo on a Mac ties the top cloud engines on developer jargon, and the audio never leaves the machine. 八个引擎，九路运行，同一段录音。跑在 Mac 上的 Whisper large-v3-turbo 在开发者行话上与最强的云端引擎打平，而音频从未离开这台电脑。
The recording 这段录音
Scott Chacon’s talk “So You Think You Know Git”, FOSDEM 2024, minutes 10 to 20. A real conference hall: accent, changing pace, room reverb. The talk has no official transcript. We count two things: how many times each engine caught git, blame and the rest, and whether each transcription is complete. Scott Chacon 的演讲 So You Think You Know Git，FOSDEM 2024，第 10 到 20 分钟。真实会场：有口音、有语速变化、有房间混响。这场演讲没有官方文字稿。我们数两样：git、blame 这些词每家听出了几次，转写完不完整。
| Engine引擎 | Runs on跑在哪 | Words词数 | Coverage覆盖率 | git | GitHub | blame | rebase | commit |
|---|---|---|---|---|---|---|---|---|
| Apple Speech (no vocabulary)（不带词表） | Your Mac你的 Mac | 2131 | 100% | 0 | 0 | 9 | 0 | 6 |
| Apple Speech (with vocabulary)（带词表） | Your Mac你的 Mac | 2128 | 100% | 0 | 0 | 9 | 1 | 6 |
| Whisper tiny | Your Mac你的 Mac | 2155 | 100% | 0 | 0 | 13 | 2 | 5 |
| Whisper large-v3-turbo | Your Mac你的 Mac | 2116 | 100% | 35 | 4 | 13 | 2 | 7 |
| Aliyun Tingwu | Cloud云端 | 2081 | n/a测不出 | 0 | 0 | 9 | 0 | 3 |
| OpenAI GPT-4o Transcribe | Cloud云端 | 2112 | 100% | 36 | 4 | 13 | 2 | 6 |
| Deepgram Nova-3 | Cloud云端 | 2171 | 100% | 35 | 4 | 13 | 2 | 7 |
| AssemblyAI Universal-3.5 Pro | Cloud云端 | 2095 | 100% | 36 | 4 | 13 | 2 | 6 |
| Speechmatics Melia 1 | Cloud云端 | 2141 | 100% | 18 | 4 | 10 | 3 | 6 |
The gap between 35 and 36 sits inside OpenAI’s and AssemblyAI’s own run-to-run spread of plus or minus one, so the top four rows are one tier, not a ranking. Tingwu’s coverage reads n/a because its transcript diverges too far for tail matching; its word count of 2081 shows it heard the whole file. 35 与 36 的差距落在 OpenAI 和 AssemblyAI 自身正负一的运行波动之内，所以前四行是同一档，不构成排名。听悟那格写测不出，是它的转写偏离太大、尾部对不上参照；2081 的词数说明它把音频听完了。
Clean audio, exact scores 干净音频，精确打分
A second corpus: 18.8 seconds of synthesized English with a fixed reference text, so word error rate is computable. Best of five runs. This is also where the local speed numbers come from. 第二份语料：18.8 秒合成英文，参考文本固定，所以能算词错率。五次取最快。本地速度数字也出自这里。
| Engine引擎 | Prose WER散文词错率 | Jargon WER行话词错率 | First word出第一个字 | Speed速度 |
|---|---|---|---|---|
| Apple Speech | 3.7% | 3.3%, 0.0% with vocabulary3.3%，带词表 0.0% | 292 ms | 14× real time14 倍速 |
| Whisper tiny | 0.0% | 0.0% | 349 ms | 48× real time48 倍速 |
| Whisper large-v3-turbo | 0.0% | 0.0% | 912 ms | 8× real time8 倍速 |
Apple’s 3.7% is a single word: the synthesized voice says “Write” in a way it hears as “Ride”. Punctuation does not count as an error; split words like “re-base” do, because that is exactly what a vocabulary is supposed to fix. Cloud speed depends on your provider and your network, so our number would not be yours; we left it out. Apple 那 3.7% 是一个词：合成音把 Write 读得像 Ride。标点不算错，re-base 这种拆词算错，因为那正是词表该治的东西。云端速度取决于你的厂商和你的网络，我们测出的数字不会是你的数字，所以没列。
Three things only running it can teach 只有跑过才知道的三件事
In ten minutes of a git talk it wrote git zero times, GitHub zero times, and “get blame” nine times. Feeding git into its custom vocabulary does not rescue it. This is a hard boundary of the on-device model. That is why Puffy defaults to Apple Speech for short commands, where its 292 ms first word wins, and to Whisper large for anything long or jargon-heavy. 十分钟的 git 演讲里，它写出 git 零次、GitHub 零次，get blame 九次。把 git 塞进它的自定义词表也救不回来。这是设备端模型的硬边界。所以 Puffy 的默认分工是：短口令走 Apple Speech，292 毫秒出字它赢；长内容和重行话走 Whisper large。
Apple’s contextual strings quietly fixed “Re-base” to “Rebase”, “PNPM” to “pnpm” and “pool request” to “pull request”, with zero change to ordinary prose. The same word list handed to Whisper as a decoding prompt wrecked it: a passage it had transcribed with zero errors came back with six words wrong out of 27. So Puffy ships vocabulary boosting for Apple Speech only. The settings you get are the ones the measurements chose. Apple 的情境词表悄悄把 Re-base 修成 Rebase、PNPM 修成 pnpm、pool request 修成 pull request，普通句子一个字不动。同一份词表喂给 Whisper 当解码提示就是灾难：一段原本零错的文字，27 个词错了 6 个。所以 Puffy 只给 Apple Speech 开词表增强。你拿到的默认设置，是测量替你选的。
Three minutes came back complete. Ten minutes stopped at 83.5%, mid-sentence, while reporting success. Forty-seven minutes returned an empty error. Puffy’s fix is to slice cloud transcription into four-minute windows; after it, the same audio comes back 99.5% complete, within half a percent of local Whisper. This quirk also explains the catastrophic long-audio scores GPT-4o gets in some public benchmarks. 三分钟完整返回。十分钟停在 83.5%，从句子中间断掉，还报告成功。四十七分钟直接返回空错误。Puffy 的修法是把云端转写切成四分钟窗口，修后同一段音频回来 99.5% 完整，与本地 Whisper 相差半个百分点。这个毛病也解释了某些公开基准里 GPT-4o 长音频得分崩掉的原因。
Run it yourself 你自己也能跑
All the engines above are the ones Puffy ships. The talk is public. Pull the same ten minutes, transcribe them with the local engines for free, plug in your own keys for the cloud lanes, and count the same words in your own output. Cloud runs bill to your own account at the vendor’s prices. 上面所有引擎就是 Puffy 内置的那些。演讲是公开的。抓同样的十分钟，本地引擎免费转写，云端通道插你自己的 key，然后在你自己的结果里数同样的词。云端用量按厂商价格记在你自己的账上。