How to Install Whisper.cpp on Ubuntu: Self-Hosted Setup Guide

I spent three months running Whisper’s Python implementation on a modest server before I realized I was burning CPU cycles and waiting 8–12 seconds to transcribe a 10-second audio clip. The moment I switched to Whisper.cpp, transcription time dropped to 2–3 seconds on the same hardware. That’s the problem this guide solves: how to install Whisper.cpp on Ubuntu and actually use it without pretending that waiting for inference is acceptable.

Why Whisper.cpp Instead of the Python Version

OpenAI’s Whisper is accurate and flexible, but the Python implementation trades performance for ease of use. Whisper.cpp strips that away. It’s a C++ port that typically runs 4–8 times faster than the official version, uses less RAM, and handles GPU acceleration cleanly if you have a compatible NVIDIA card.

I’m not saying the Python version is bad. It’s fine for batch jobs where you queue up 100 audio files and let them run overnight. But if you’re building a real-time transcription service, integrating voice input into a home assistant, or just tired of your CPU getting hammered, Whisper.cpp changes the equation. You get nearly identical accuracy without the resource tax.

The trade-off is simplicity. The Python version is literally one pip install away. Whisper.cpp requires a build step, some configuration, and basic familiarity with compiled binaries. This guide gets you there.

Prerequisites and Hardware Specs

You’ll need an Ubuntu 20.04 LTS box or newer. This works on bare metal, VMs, or as a containerized service. Here’s what I’m assuming:

• Ubuntu 20.04 LTS or 22.04 LTS (or any Debian-based distro, really)
• At least 4GB RAM (8GB recommended if you want headroom)
• 2+ CPU cores
• 4GB free disk space for the base model; larger models need 6–15GB
• Git and basic build tools installed

If you have an NVIDIA GPU (RTX, Tesla, GTX with compute capability 5.0 or higher), Whisper.cpp can use CUDA for inference. That’s a nice bonus but not required. The CPU version is perfectly usable.

For Raspberry Pi: technically possible with smaller models, but honestly, don’t. Even with a Pi 5 and a quantized tiny model, you’re looking at 15–20 second transcriptions. This guide assumes x86-64.

Installing Whisper.cpp from Source

Start by grabbing the repository and building the binary.

cd /opt
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make

That’s it. The build takes about 2–3 minutes on a modern CPU. If you hit errors about missing dependencies (usually gcc or make), run:

sudo apt update
sudo apt install build-essential git

then try make again.

By default, this builds a CPU-only binary. If you want GPU acceleration, you need CUDA installed first. I’ll skip the full CUDA walkthrough (it’s not trivial), but once CUDA 11.8 or newer is on your system, build with:

make clean
CUDA_DOCKER_ARCH=compute_86 make

Swap compute_86 for your GPU’s compute capability if needed. NVIDIA’s documentation covers that if you’re not sure.

After make finishes, you’ll have a main binary at ./main. Test it:

./main --help

You should see a wall of options. That’s a good sign.

Downloading a Model and First Transcription

Whisper.cpp ships with a model downloader. Models come in five sizes: tiny, base, small, medium, large. Start with base if you want a balance of speed and accuracy. Tiny is for speed; large is for fidelity.

cd /opt/whisper.cpp
bash ./models/download-ggml-model.sh base

The script downloads about 140MB and converts it to the quantized GGML format that Whisper.cpp uses. This takes a minute or two. Once done, you’ll have a file like models/ggml-base.bin.

Now grab a test audio file. I’ll use a sample from the web:

wget https://upload.wikimedia.org/wikipedia/commons/e/ea/En.01.ogg -O test.ogg

Convert it to WAV (Whisper.cpp prefers WAV):

ffmpeg -i test.ogg -ar 16000 -ac 1 test.wav

Run the transcription:

./main -m models/ggml-base.bin -f test.wav

On a CPU, this takes 10–15 seconds for a 10-second audio clip. Output is printed to stdout. You should see the transcribed text at the end, preceded by some processing logs.

If it works, congratulations. You’ve got Whisper.cpp running. Now the real work begins: integrating it somewhere useful.

Setting Up a Persistent Service

Running ./main from the command line is fine for testing. For actual use, you want a systemd service or a containerized setup that starts on boot and stays running.

I prefer Docker, so here’s a docker-compose.yml that wraps Whisper.cpp and exposes it via a simple HTTP API:

version: '3.8'

services:
  whisper:
    image: ghcr.io/ggerganov/whisper.cpp:latest
    container_name: whisper-cpp
    ports:
      - "8000:8000"
    volumes:
      - ./models:/app/models
      - ./uploads:/app/uploads
    environment:
      - MODELS_PATH=/app/models
    command: >
      /app/main
      -m /app/models/ggml-base.bin
      --server
      --host 0.0.0.0
      --port 8000
    restart: unless-stopped

Save that as docker-compose.yml in a working directory. Adjust the model path if you’re using a different size. Then:

docker compose up -d

The image will download (it’s about 1.5GB), and once running, Whisper.cpp will start a web server on port 8000. Test it with curl:

curl -X POST -F "[email protected]" http://localhost:8000/inference

You’ll get back JSON with the transcribed text. Nice.

One thing that surprised me: the server mode in Whisper.cpp is functional but minimal. There’s no built-in request queue, no batching, no auth. For a homelab that’s fine. If you need something production-grade, you’ll want to layer a reverse proxy like nginx in front and maybe write a thin wrapper to handle concurrent requests properly.

Common Issues and How to Fix Them

Slow inference (30+ seconds for short audio): You’re probably on a single-core or using a large model on weak hardware. Switch to the tiny or base model and make sure you’re not sharing CPU with other workloads. If you have a GPU and it’s not being used, your build didn’t enable CUDA. Rebuild with CUDA flags.

Out of memory errors: The model size determines RAM usage. The large model needs about 6GB alone. The base model is around 500MB. If you’re tight on RAM, quantize the model further (Whisper.cpp supports 4-bit and 8-bit quantization) or use a smaller variant. The trade-off is slightly lower accuracy, but honestly, it’s minimal on the base model.

“Model not found” when using Docker: Your volume mount is wrong. The container expects models at /app/models. Make sure your docker-compose volume points to the right directory on the host. Docker relative paths are relative to where you run docker compose, not your working directory.

Audio format errors: Whisper.cpp prefers 16-bit PCM WAV at 16kHz mono. If you’re throwing MP3 or stereo audio at it, convert first with ffmpeg. The error messages aren’t always clear on this.

Next Steps: Integration and Optimization

Once Whisper.cpp is running, you can wire it into other services. Common patterns in my homelab:

• Home Assistant: Use the OpenAI Whisper integration pointed at your local server. It works if you expose Whisper.cpp behind a proper HTTP interface (not just the basic server mode).
• Immich (photo library): If you’re self-hosting photo management, you can transcribe voice memos by calling the Whisper endpoint via a webhook.
• n8n (automation): Build a workflow that listens for audio files, calls Whisper.cpp, and routes transcriptions to a database or notification service.

For performance, experiment with the quantized model variants. Whisper.cpp includes Q5_0 and Q8_0 versions that shave 20–30% off inference time with almost no accuracy loss. If you’re batch-processing (say, transcribing hours of video), the –threads flag lets you tune CPU utilization. I usually set it to one less than the core count to avoid starving the system.

One more thing: keep an eye on the GitHub releases. Whisper.cpp is actively maintained, and new optimizations land fairly often. I’ve had good luck just pulling the latest and rebuilding every few months.

FAQ

Can Whisper.cpp run on a Raspberry Pi?

Yes, but don’t expect real-time performance. A Pi 5 with the tiny model gets about 15–20 seconds per 10 seconds of audio. It works for non-critical use cases, but for anything time-sensitive, x86 hardware is worth the upgrade.

How much disk space does Whisper.cpp need?

The tiny model is 75MB, base is 140MB, small is 466MB, medium is 1.5GB, and large is 2.9GB. Add another 1–2GB for the Docker image if you’re containerizing. Most setups use base or small, so 500MB–1GB is typical.

Does Whisper.cpp support GPU acceleration on AMD or Intel Arc cards?

Not directly. The official build supports NVIDIA CUDA only. AMD GPU support via HIP exists but requires manual compilation and is not widely tested in the community. Intel Arc is even less supported. If you have non-NVIDIA hardware, you’re better off sticking with optimized CPU inference.

How accurate is Whisper.cpp compared to the Python version?

Identical. It’s the same model and inference logic, just rewritten in C++ for speed. Accuracy differences you see are usually due to audio quality or model size choice, not the implementation.

What’s the difference between the model sizes?

Tiny is fastest but makes more mistakes (96% accuracy on English). Base balances speed and accuracy (97.5%). Medium is slower but more accurate (98.5%). Large is slowest but most accurate (99%+). For English speech, base is usually the sweet spot. Other languages may favor medium or large.

Explore Whisper.cpp in our AI Homelab Toolkit.