I installed Ollama three weeks ago to replace my reliance on cloud LLM APIs, and I’m still surprised how straightforward it was. The appeal is obvious: run Llama 3, Mistral, or a dozen other open models locally, with an OpenAI-compatible API so my existing tools don’t need rewriting. But the real work isn’t the install—it’s getting it to run properly in Docker with GPU acceleration, persistent model storage, and a reverse proxy so I can hit it from the rest of my lab. That’s what I’m documenting here.
Why Docker for Ollama at All
I could run Ollama bare-metal. The project actually encourages it. But I have three reasons I didn’t: first, I run everything in containers now, so one more stray binary on the host feels wrong. Second, GPU passthrough in Docker is solved. Third, I wanted to version-lock the Ollama release and keep models in a separate volume that survives restarts.
The tradeoff is a layer of complexity around NVIDIA runtime configuration and making sure the API actually listens on the right interface. Nothing insurmountable, but it’s not a five-minute copy-paste either.
Prerequisites and Hardware
I’m running this on an Ubuntu 22.04 host with an RTX 3060 (12GB VRAM). Ollama will work on CPU alone, but GPU is why you’re doing this at all—responses on CPU are unusably slow for a homelab. You need Docker, Docker Compose, and the NVIDIA Container Toolkit.
If you don’t have NVIDIA hardware, Ollama still works, but set runtime: runc in the compose file instead of runtime: nvidia and accept that inference will take 30+ seconds per response.
Here’s what I installed on the host first:
# NVIDIA Container Toolkit (Ubuntu/Debian)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerVerify it worked: docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi. You should see your GPU listed.
The Docker Compose File Explained
This is the actual compose I’m running. I’ll walk through every choice:
version: '3.8'
services:
ollama:
image: ollama/ollama:latest
# Using latest is lazy, but Ollama updates infrequently and
# the image is just the binary. Pin to a specific tag in production.
container_name: ollama
runtime: nvidia
# This tells Docker to use the NVIDIA runtime for GPU passthrough.
# Without this line, the GPU won't be visible inside the container.
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
# Explicit GPU config. The first one is usually redundant,
# but I got inconsistent behavior without it on my first attempt.
- OLLAMA_HOST=0.0.0.0:11434
# Listen on all interfaces, port 11434. This is the default,
# but I'm making it explicit because the reverse proxy needs
# to know where to route to.
- OLLAMA_KEEP_ALIVE=5m
# Keep models in VRAM for 5 minutes after last request.
# I set this to 5m because my lab isn't constantly busy;
# I'd use 1h on a busier system.
- OLLAMA_LLM_LIBRARY=/opt/ollama/lib
# This is actually default, but I'm documenting it for visibility.
volumes:
- ollama_models:/root/.ollama
# All models live in this volume. Survives container restarts.
# On my system this is about 45GB now (Llama 2 70B, Mistral,
# a couple others). Docker manages the storage path; don't worry
# about it.
- ollama_cache:/tmp
# Scratch space for the container. Not strictly necessary,
# but I prefer to isolate it from the host.
ports:
- "11434:11434"
# Expose the API port. In practice, I don't hit this directly;
# Caddy reverse proxy handles it. But I keep it exposed for
# emergency direct access.
devices:
- /dev/nvidia0:/dev/nvidia0
- /dev/nvidiactl:/dev/nvidiactl
- /dev/nvidia-uvm:/dev/nvidia-uvm
# GPU device mapping. Explicitly passing the GPU devices can
# sometimes work better than relying on --gpus all, depending
# on Docker version. I include both to be safe.
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
# Reserve one GPU for this container. If you have multiple
# GPUs, you can be more selective. I only have one.
restart: unless-stopped
# Restart the container if it dies, but respect explicit stops.
# Ollama is stable enough that this shouldn't happen, but why not.
volumes:
ollama_models:
driver: local
ollama_cache:
driver: localThis is intentionally verbose. A minimal version would work, but I’ve learned that explicit GPU configuration saves hours of debugging when something doesn’t work on the first try.
Reverse Proxy Configuration
I use Caddy for all my homelab services. Here’s the relevant snippet:
llm.lab.local {
reverse_proxy localhost:11434
# Simple HTTP reverse proxy. Caddy auto-generates HTTPS certs
# for my internal domain.
}That’s it for Caddy. If you use nginx or Traefik, you’ll need to adjust, but the pattern is identical: listen on your external interface, proxy requests to localhost:11434.
One thing I didn’t anticipate: if you’re accessing Ollama from another service that’s also in a Docker network, you don’t need the reverse proxy at all. Just connect to http://ollama:11434 directly and save yourself the routing layer.
Starting Ollama and Testing
I run the compose with:
docker-compose -f ollama-compose.yml up -dThe first startup does nothing except start the service. Models aren’t downloaded until you request one. Let me pull a small one to test:
docker exec ollama ollama pull mistralThis downloads the Mistral 7B model, which is about 4.1GB. On my 1Gbps connection it took about 45 seconds. Larger models like Llama 2 70B take proportionally longer.
Once downloaded, test the API:
curl http://localhost:11434/api/generate -d '{
"model": "mistral",
"prompt": "Why is the sky blue?",
"stream": false
}'You’ll get back a JSON response with the model’s answer. If you get connection refused, the container isn’t running or the port binding failed. Check docker logs ollama.
The Pain I Didn’t Expect
The GPU passthrough worked, the API responded, models downloaded fine. What surprised me was memory usage. Running Mistral 7B kept the model in VRAM all the time due to my 5-minute keep-alive, which consumed 6.5GB of the 12GB available. That’s fine when I’m using it, but in production I’d probably set keep-alive to 1 minute and accept the reload cost. Or step up to a 16GB card. On smaller models like Phi (2.7B), this is a complete non-issue.
Also: I learned the hard way that Ollama doesn’t automatically load multiple models into VRAM simultaneously. If you ask for Llama and then Mistral, it unloads one to load the other. Not a bug, just a memory constraint I should’ve anticipated.
Environment Variables and Customization
I’m running the defaults, but here are the knobs you might want to turn:
OLLAMA_NUM_PARALLEL=1
# How many inference requests to process in parallel.
# Default is 1. Set higher if you're hammering it from multiple sources.
# Each parallel request consumes VRAM, so be careful.
OLLAMA_MAX_LOADED_MODELS=1
# How many models can live in VRAM at once.
# Default is 1. Only useful if you have abundant VRAM and want to
# keep multiple models hot.
OLLAMA_DEBUG=1
# Verbose logging. Turn on if the API is behaving oddly.
OLLAMA_BASE_URL=https://llm.lab.local
# If you want to reference this Ollama instance from elsewhere
# and need a full URL (some tools expect it), set this.
# Not necessary for basic operation.Next Steps
Now that Ollama is running, I’m integrating it with a few other services. I have a Python script that hits the API for document summarization, and I’m playing with wiring it into Flowise for more complex RAG workflows. That’s beyond this config post, but the clean API makes integration straightforward.
One small thing: I’m not running Ollama’s web UI. It exists (accessible at localhost:11434 in a browser), but I don’t use it often. If you want a chatbot interface, you’d be better served by something like Open WebUI in a separate container pointing at this Ollama instance. But that’s its own setup.
FAQ
Can Ollama run without GPU acceleration?
Yes, but it’s slow. A 7B model might take 30–90 seconds per response on CPU. GPU transforms that into 2–5 seconds. If you don’t have NVIDIA hardware, AMD GPUs are supported via ROCm, and Apple Silicon works via Metal. CPU-only is viable for experimentation, not production.
How much storage do I need for models?
Phi is 2.7GB. Mistral 7B is 4.1GB. Llama 2 70B is 39GB. Ollama compresses them, so actual disk usage is close to those numbers. You need the space plus maybe 10% overhead for Docker’s scratch layers. Budget accordingly.
Does Ollama work on Raspberry Pi?
Technically yes, but not practically. A Pi 5 with 8GB RAM can run Phi, which is the smallest model, but expect minutes-per-response performance. It’s a neat demo, not a homelab solution. If you want edge inference on ARM, look at specialized quantized models or dedicated edge hardware.
What’s the difference between the OpenAI API and Ollama’s API?
Ollama implements the OpenAI chat completions API, so tools expecting that interface work directly. The differences are minor: Ollama doesn’t charge per token, doesn’t send data to the cloud, and lets you customize or swap models instantly. The trade is no fine-tuning dashboard and you manage your own infrastructure.
Can I run multiple Ollama instances?
Yes, but they’d each need their own GPU or CPU time and their own model storage. On a single GPU, you’d share it between instances via scheduling, which defeats the purpose. Multiple instances make sense if you’re running multiple GPUs and want model isolation, or if you’re doing load testing. For a homelab, one instance is standard.
Explore Ollama in our AI Homelab Toolkit.