There's something quietly magical about speaking to your computer and watching your words appear on screen — no cloud required, no subscription, no data leaving your desk. Local speech-to-text has crossed the threshold from "technically possible" to genuinely useful for daily work. Not perfect, but useful.
Two projects dominate the local Whisper landscape right now: Faster-Whisper and whisper.cpp. Both run OpenAI's Whisper models entirely on your own hardware. Both are free. But they make very different trade-offs, and picking the wrong one for your use case means the difference between a tool you rely on and one you abandon after two frustrating afternoons.
This article walks through what each project is good at, how they compare on dictation and subtitle generation, and what a working setup actually looks like.
Why Run Whisper Locally?
The cloud-based speech-to-text APIs — Whisper's own, Google's, Deepgram's — are excellent. They're also metered, require an internet connection, and send your audio to someone else's server.
For dictation, that third point stings. A dictated journal entry, a brainstorming session for a project under NDA, a private meeting transcript — these are exactly the kinds of audio you might prefer to keep off someone else's disk. Even setting privacy aside, offline capability means working on a plane, in a workshop with spotty Wi-Fi, or anywhere connectivity is unreliable.
The local Whisper ecosystem has matured enough that the accuracy gap with cloud APIs is narrow for English. Both Faster-Whisper and whisper.cpp use the same underlying models — you're not trading away recognition quality for privacy. What you are trading is convenience and speed, and that's where the two projects diverge.
The Contenders at a Glance
Here are the two approaches laid out side by side.
| Aspect | Faster-Whisper | whisper.cpp |
|---|---|---|
| Language | Python (CTranslate2 backend) | C/C++ (no runtime dependencies) |
| GPU support | CUDA (NVIDIA), limited CPU fallback | CUDA, Metal (Apple Silicon), Vulkan, OpenCL, CPU |
| Model format | Converts Whisper models to CTranslate2 format | Converts to custom GGML format |
| Memory use | Higher, but with intelligent caching | Extremely low, runs on a Raspberry Pi |
| Latency (real-time) | Good with VAD, but not its primary design | Excellent with streaming support |
| Ease of setup (Python user) | Very easy — pip install faster-whisper |
Moderate — compilation step or prebuilt binary |
| Ease of setup (non-Python user) | Requires Python environment | Single binary, zero dependencies |
| Batch transcription speed | Very fast on GPU, good on CPU | Competitive on GPU, sometimes faster on CPU |
| Community and docs | Smaller, Python-focused | Large, many front-ends and bindings available |
Neither is universally better. The right choice depends entirely on your hardware and what you're trying to do.
Setup and Installation
The setup experience alone often determines which tool someone sticks with. Let's walk through both.
Faster-Whisper
If Python is already part of your workflow, Faster-Whisper is trivial to install. Create a virtual environment, install the package, and you're done.
python -m venv whisper-env
source whisper-env/bin/activate
pip install faster-whisper
That's the happy path. The NVIDIA side requires CUDA libraries (cuBLAS, cuDNN) to be present on the system. The pip package bundles the CTranslate2 binary, but it links against whatever CUDA version you have. If the versions mismatch, you'll get a cryptic error at runtime instead of install time — a known frustration.
On CPU-only machines, Faster-Whisper still works, but performance drops hard. An Intel Mac or a Linux server without a GPU will transcribe at roughly 0.3–0.5x real-time for the medium model, which is enough for batch work but not for interactive dictation.
whisper.cpp
whisper.cpp ships as C source. You compile it, and you get a single static binary. No Python, no package manager, no virtual environments.
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make
That builds the CPU version. For GPU acceleration, you append a flag: WHISPER_CUDA=1 make for NVIDIA, or simply make on an Apple Silicon Mac to get Metal support automatically (it detects the platform).
The extra step is downloading a model. whisper.cpp uses GGML-format model files, which you download separately. The project provides a shell script for this:
bash ./models/download-ggml-model.sh medium.en
After compilation and model download, transcription is a one-liner:
./main -m models/ggml-medium.en.bin -f audio.wav
The binary approach feels refreshingly Unix-like. No environment to activate, no pip dependency resolver churning for minutes. It's also what makes whisper.cpp the clear winner for embedding in other applications — dozens of GUI wrappers and plugin integrations exist because they just need to bundle a binary and a model file.
Dictation Showdown: Real-Time Responsiveness
Dictation is the hardest test. The user speaks, pauses, and expects text to appear within a second or two. Any longer and the experience breaks — you second-guess what you said, lose your train of thought, or start watching the spinner.
How whisper.cpp handles streaming
whisper.cpp has a dedicated streaming mode built around Voice Activity Detection (VAD). The stream example application listens to your microphone, detects when you start and stop speaking, and transcribes each utterance as a chunk.
The key advantage is architecture-level: whisper.cpp keeps the model loaded in memory between chunks. There's no teardown and reload per utterance. On Apple Silicon with the small or medium model, transcription latency can dip below 500ms from end-of-speech to text output. That's fast enough to feel interactive.
A practical command:
./stream -m models/ggml-medium.en.bin -t 6 --step 500 --length 5000
The --step and --length flags control the sliding window. Shorter windows mean lower latency but more context cuts, which can hurt accuracy on long sentences.
How Faster-Whisper handles streaming
Faster-Whisper doesn't have a built-in streaming mode. Its design optimizes for full-file batch transcription using CTranslate2's efficient inference engine. You feed it a complete audio file and get a complete transcript back.
For dictation, you need an external VAD layer. A common pattern pairs Faster-Whisper with Silero VAD: the VAD detects speech segments in the audio stream, cuts them into small WAV files, and hands each to Faster-Whisper for transcription. The tool Buzz does exactly this, providing a desktop GUI for real-time dictation using Faster-Whisper as the backend.
This works, but the model reload penalty stings. Each new audio chunk means a fresh call to the Faster-Whisper API, and while CTranslate2 caches aggressively, there's overhead the whisper.cpp streaming path avoids entirely. For sustained dictation sessions, whisper.cpp is the smoother experience.
The VAD factor
Both approaches depend heavily on VAD quality. A VAD that cuts too early fragments your sentences. One that cuts too late makes you wait. Silero VAD is the go-to for Python pipelines — it's fast, accurate, and picks up speech boundaries well even in mildly noisy rooms. whisper.cpp bundles its own VAD (silero-vad integrated directly), which is one less integration point to debug.
My advice: if dictation is your primary use case, start with whisper.cpp's stream example. Get a feel for the latency on your hardware. If it's snappy enough with the small or medium English-only model, you're done. If not, the bottleneck is likely the model size, not the software.
Subtitles and Batch Transcription: Processing Long Audio
Flip the use case: you have an hour-long podcast recording, a meeting audio file, or a video that needs SRT subtitles. Now you care about throughput more than latency. You want the file processed as fast as possible.
Throughput comparison
On GPU-equipped machines, both projects are fast. Faster-Whisper's CTranslate2 backend applies kernel optimizations (operator fusion, memory layout optimizations) that can push it ahead on NVIDIA hardware. whisper.cpp's CUDA support is solid but generally a touch behind in raw tokens-per-second on the same GPU and model.
On CPU, the story can invert. whisper.cpp's integer quantization and platform-specific SIMD optimizations (AVX2 on x86, NEON on ARM) make it surprisingly quick even without a GPU. A modern x86 desktop running whisper.cpp with the medium model often transcribes faster than real-time — around 1.2–1.5x on a Ryzen 7000 series, for example.
Faster-Whisper on CPU uses CTranslate2's CPU backend, which is efficient but doesn't match whisper.cpp's hand-tuned kernels. Expect roughly 30–50% lower throughput on the same hardware compared to whisper.cpp.
Word-level timestamps and SRT output
Subtitles need timestamps. Both projects support word-level timestamps, but the implementation quality differs.
whisper.cpp outputs timestamps at the token level when you pass --print-special and use a timestamp-capable model. The -osrt flag dumps directly to SRT format:
./main -m models/ggml-medium.en.bin -f podcast.wav -osrt
Faster-Whisper returns word-level timestamps through its Python API natively — you iterate over segments and access each word's start and end time. Generating SRT from that is straightforward with a small script, but it's not a built-in flag.
For batch subtitle work, whisper.cpp's one-command SRT output is hard to beat. For programmatic integration where Python is already in the mix, Faster-Whisper's native word timing API is cleaner than parsing whisper.cpp's text output.
Model Selection: Size Matters More Than Software
Both tools support the full range of Whisper models: tiny, base, small, medium, and large (v1, v2, and v3). The software differences pale next to the model size impact.
| Model | Size (VRAM/RAM) | English Accuracy | Speed (relative) | Good for |
|---|---|---|---|---|
| tiny.en | ~75 MB | Okay for clean speech | Very fast | Real-time dictation on weak hardware |
| base.en | ~145 MB | Decent, struggles with accents | Fast | Quick transcripts, low-resource devices |
| small.en | ~490 MB | Good, handles most accents | Moderate | The sweet spot for dictation |
| medium.en | ~1.5 GB | Very good, rare mistakes | Slower | Subtitle production, meeting transcripts |
| large-v3 | ~3.1 GB | Best available, nuanced | Slowest | Professional transcription, heavy accents |
The English-only models (.en suffix) are consistently more accurate for English than their multilingual counterparts at the same size. They're also faster and use less memory. Use them unless you genuinely need multi-language support.
Hardware Considerations
Your hardware dictates which model you can run comfortably.
Apple Silicon (M1/M2/M3): whisper.cpp with Metal acceleration is the clear winner. The unified memory architecture means even the large model fits comfortably on a 16GB machine, and the ANE (Neural Engine) delivers impressive throughput. Faster-Whisper has no Metal backend — it runs on CPU only on Mac, so it's not competitive here.
NVIDIA GPU (8GB+ VRAM): Both projects perform well. Faster-Whisper with CUDA offers slightly higher throughput. The large model fits in 8GB with some headroom. Real-time dictation is comfortable with medium.
AMD GPU: whisper.cpp's Vulkan backend works. Performance is less polished than CUDA or Metal, but it functions. Faster-Whisper has no AMD GPU support.
CPU-only (modern x86 or ARM): whisper.cpp wins by a significant margin. Its quantization and SIMD work pay off. A Raspberry Pi 4 running whisper.cpp with the tiny model can transcribe roughly in real-time — limited but functional. Faster-Whisper on the same hardware would be painfully slow.
Raspberry Pi / low-power ARM: whisper.cpp is the only realistic choice. The tiny and base models run acceptably. Expect 0.8–1.2x real-time for tiny on a Pi 4, slower on a Pi 3.
Choosing Your Path
For dictation — interactive, real-time speech-to-text — whisper.cpp is the stronger option. Its streaming architecture, low memory footprint, and broad hardware support make it the better foundation for a responsive dictation tool. Pair it with the small or medium English-only model for a balance of speed and accuracy.
For batch transcription and subtitle generation, the choice narrows to workflow fit. If Python is already your glue language and you have an NVIDIA GPU, Faster-Whisper integrates cleanly and performs excellently. If you want a single binary that outputs SRT with zero scripting, whisper.cpp is the pragmatic pick.
Both projects improve quickly. The gap in streaming support might close, and the throughput differences shift with each release. The safe bet is to have both installed — they're small investments, and the right tool for a given file can change depending on length, noise level, and how fast you need the result. Start with the model that fits your RAM, pick the engine your hardware likes, and test with real audio from your actual environment. The spec sheets are useful guides, but your own voice, your own microphone, and your own tolerance for latency are the only benchmarks that matter.
