Whisper Docker Compose Setup: My Homelab Config with Explanations

I’ve been running Whisper on my homelab for about eight months now, mostly for transcribing meeting recordings and the occasional voice memo. It works well enough that I barely think about it, which is the goal with any self-hosted thing. The model is impressively accurate—better than I expected for something I’m not paying per-minute for. The catch is that getting it running properly, with the right hardware allocation and a sensible reverse proxy setup, took more iteration than the README suggests.

Why Run Whisper Locally in Docker

Whisper is OpenAI’s open-source speech recognition model. It handles 99 languages, translates between them, and runs entirely on your own hardware. No API calls, no usage limits, no transcription fees adding up at the end of the month. For a homelab, the appeal is straightforward: you get reliable transcription without external dependencies.

The downside: it’s compute-intensive. The base model runs fine on modern consumer CPUs, but a GPU accelerates it dramatically. I’m using an RTX 3060 in my server and seeing 2-3 minute transcriptions complete in under 30 seconds. Without it, you’re looking at real-time or slower performance depending on audio length.

Whisper also integrates with Home Assistant if you want voice commands without waking a cloud service or relying on local wake-word detection. I haven’t fully wired that up, but I know people running it that way.

Prerequisites and Hardware Considerations

You need Docker and Docker Compose running. I’m assuming a Linux host; I haven’t tested this on Windows or macOS Docker Desktop, though it should work.

For hardware: CPU-only works fine if you’re patient. A quad-core modern processor will do 5-10 minute audio in real time on the base model. If you have a GPU—NVIDIA with CUDA support, or AMD with ROCm—Whisper will use it automatically if you pass it through to the container. I’m not running this on a Pi. You could, technically, but you’d wait 10-15 minutes for a 3-minute clip.

RAM: 4GB is adequate for the base model. The ‘small’ model needs about 1.5GB, ‘medium’ needs 3-4GB, ‘large’ is pushing 6GB. I’m using ‘base’ most of the time because the accuracy difference isn’t worth the extra processing time for my use case.

The Docker Compose Configuration

I’m using openai/whisper containerized through a project called faster-whisper, which is optimized for inference speed. There are a few ways to wrap this—some people use FastAPI endpoints, others use simpler container setups. I went with the FastAPI approach because it gives you a REST API that integrates cleanly with reverse proxies and Home Assistant.

version: '3.8'

services:
  whisper:
    image: ghcr.io/aarnq/faster-whisper:latest-gpu
    container_name: whisper
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      - MODEL_SIZE=base
      - DEVICE=cuda
      - COMPUTE_TYPE=float16
      - LOG_LEVEL=info
    volumes:
      - ./whisper_models:/root/.cache/whisper
      - ./uploads:/app/uploads
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

networks:
  default:
    name: homelab

A few decisions here worth explaining:

Image choice: I’m using aarnq/faster-whisper because it’s actively maintained and includes the FastAPI server out of the box. The base openai/whisper is just the CLI tool, which works fine if you want to trigger transcription via shell scripts, but an HTTP API is cleaner for integration.

Model size: base is 140MB and hits around 95% accuracy on English. small is better but slower. medium is noticeably better on accents and background noise, but I found the time-per-quality tradeoff not worth it for my personal recordings. If you’re transcribing podcasts or low-quality audio, jump to small or medium.

Device and compute type: If you don’t have an NVIDIA GPU, change DEVICE=cuda to DEVICE=cpu and remove the entire deploy.resources section. If you’re on AMD with ROCm, use DEVICE=rocm. float16 is faster than float32 and uses less VRAM. The quality loss is negligible.

Port binding: I’m binding to 127.0.0.1:8000, not 0.0.0.0. This container stays internal to my homelab network. A reverse proxy sitting in front handles external access.

Volumes: The models cache to disk so you don’t re-download them. The uploads folder is where temp audio files sit. Make sure both directories exist and have adequate space. Models are typically 140MB to 3GB depending on size.

Healthcheck: This one matters. The FastAPI app needs 30-40 seconds to fully initialize on first start, and if your orchestrator (or Home Assistant, if you’re using it) tries to connect before that, you’ll see spurious errors. The 40-second start_period prevents that.

Reverse Proxy Configuration (Nginx)

I’m running Nginx in front of Whisper because I have other services there already, and it gives me authentication, rate limiting, and cleaner URLs.

upstream whisper_backend {
    server 127.0.0.1:8000;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name whisper.example.com;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    # Increase timeouts for long audio files
    proxy_connect_timeout 300s;
    proxy_send_timeout 300s;
    proxy_read_timeout 300s;

    location / {
        auth_basic "Whisper";
        auth_basic_user_file /etc/nginx/.htpasswd;

        proxy_pass http://whisper_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Buffer settings for file uploads
        client_max_body_size 100m;
        proxy_buffering off;
    }

    location /health {
        access_log off;
        proxy_pass http://whisper_backend;
    }
}

server {
    listen 80;
    server_name whisper.example.com;
    return 301 https://$server_name$request_uri;
}

Key points: The timeouts are generous because transcribing large files takes time. A 30-minute audio file might take 5-10 minutes to process on my setup, so I don’t want Nginx timing out the connection.

client_max_body_size is set to 100m, which covers most use cases. If you’re uploading long recordings, you might need 200m or more.

The /health endpoint is excluded from auth logging because the container health check hits it every 30 seconds, and I don’t want log noise.

First Run and Model Download

When you start the container for the first time, it downloads the model. The ‘base’ model is about 140MB. Don’t interrupt this—if you do, the cache gets corrupted and you’ll need to delete the volume and re-download.

docker-compose up -d whisper
docker-compose logs -f whisper

Wait for the line Uvicorn running on http://0.0.0.0:8000. Once you see that, the API is live. Test it:

curl -F "file=@test_audio.mp3" http://localhost:8000/asr

You should get back JSON with the transcription. If the container crashes with an out-of-memory error, you don’t have enough RAM allocated to Docker. Either lower the model size or increase Docker’s memory limit.

Real-World Issues and Tuning

First surprise: GPU memory. Even with float16, the ‘large’ model eats about 5GB of VRAM on my setup. I thought it’d be tighter. Switched to ‘base’ and haven’t regretted it.

Second: concurrent requests will queue. This isn’t bad—it’s just how it is. The server processes one transcription at a time. If you send it three files, the third one waits until the first two finish. I’m comfortable with that for a homelab. If you need parallelism, you’d run multiple containers and load-balance between them, but that’s complexity I don’t need.

Third: silence detection isn’t magic. If you send a file with 5 minutes of silence at the end, it still processes the entire thing. The API respects whatever audio you give it. Trim your files first if length is a problem.

For performance tuning, I’ve found that float16 on NVIDIA gives me the best time-to-quality ratio. If you’re CPU-only, the ‘small’ model is a reasonable middle ground—better accuracy than ‘base’, not glacially slow.

One thing I didn’t expect: Whisper handles code snippets in speech surprisingly well. If you’re dictating Python or SQL, it tends to format it correctly. Not perfect, but better than I’d have guessed.

Integration with Home Assistant (Optional)

If you want voice commands in Home Assistant without waking a cloud service, you can connect Home Assistant to your Whisper instance. Set up a REST integration pointing to your reverse proxy URL, and then create an automation that sends audio from a wake-word detector (like Wyoming Piper’s included tools) to Whisper for transcription. I’m not running this myself because my wake-word setup is stable enough, and adding another layer feels like overkill. But the plumbing exists and works.

FAQ

Can Whisper run on a Raspberry Pi?

Technically yes, but it’ll be slow. CPU transcription on a Pi4 takes 20-30x real-time for the ‘base’ model. If you have a Pi5 with 8GB RAM, it’s workable for short clips. I wouldn’t do it for regular use.

How much GPU memory does Whisper need?

The ‘base’ model needs about 1.5GB, ‘small’ needs 2-3GB, ‘medium’ needs 5GB, ‘large’ needs 8-10GB. float16 reduces this by half compared to float32. An RTX 3060 (12GB) handles everything comfortably.

Does Whisper need internet to run?

No. Once the model downloads (which requires internet), everything runs locally. No API calls, no tracking. Fully air-gappable if you need it.

How accurate is Whisper compared to commercial services?

On clear English audio, it’s within 1-2% of Google or Azure. On heavily accented speech or noisy audio, it degrades faster. The ‘large’ model is more robust, but most people find ‘base’ or ‘small’ sufficient for personal use.

Can I transcribe in languages other than English?

Yes. Whisper handles 99 languages. It auto-detects the language by default. You can also force a specific language by passing it in the API request. Translation to English is also supported if you want to transcribe, say, Spanish audio and get English text.

Explore Whisper in our AI Homelab Toolkit.