I’ve been running Ollama on my homelab for about eight months. It works. Pull a model, run it, hit the API. Simple. But around model four or five, I started wanting something on top of it—document storage, user isolation, a UI that wasn’t just curl requests. I looked at Open WebUI first. Then I built a rough LangChain wrapper. Then I found AnythingLLM and realized I’d been working around a tool that already existed.
This is the article I wish I’d read before switching.
What I Was Running Before
Ollama on Docker. Nothing fancy. A single user (me). Models stored in a volume. I’d pull llama2 or mistral depending on what I needed, fire up a container, and hit localhost:11434 from wherever. Fast enough for my use. I wrote a small Python wrapper that let me ask questions about local documents by chunking them and feeding them into the model manually.
It worked until it didn’t. The wrapper was fragile. It didn’t persist conversations well. Adding a second user meant sharing API access, which is bad practice and I knew it. I wanted multi-user support, better document handling, and a real interface. Ollama doesn’t do that. It’s a runtime, not a framework.
Why I Switched to AnythingLLM
AnythingLLM is self-hosted, full-stack, and it runs Ollama in the background if you want it to. That last part is key. I wasn’t abandoning Ollama—I was building on top of it with something that actually knows how to handle documents, workspaces, and users.
The selling point for me was threefold: RAG support without writing it myself, workspace isolation for different projects, and a UI my partner could actually use instead of me explaining API flags. Docker deployment meant I could move it from my main rig to a dedicated box without much friction.
I also liked that it connects to multiple model providers. Ollama is local-only by design. AnythingLLM lets you mix—Ollama for one workspace, OpenAI for another if you need to. That flexibility felt right for a homelab that grows unevenly.
The Migration: What Moved, What Didn’t
AnythingLLM doesn’t import Ollama conversations or weights directly. The models stay in Ollama. AnythingLLM just talks to it. So my first step was getting AnythingLLM running alongside Ollama, not replacing it.
Docker Compose made this straightforward:
version: '3.8'
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
environment:
- OLLAMA_HOST=0.0.0.0:11434
anythingllm:
image: mintplex/anythingllm:latest
container_name: anythingllm
ports:
- "3001:3001"
volumes:
- anythingllm_data:/app/server/storage
environment:
- LLM_PROVIDER=ollama
- OLLAMA_BASE_URL=http://ollama:11434
depends_on:
- ollama
volumes:
ollama_data:
anythingllm_data:I ran this on a spare box—an old i5 NUC with 16GB RAM. Start to full operation took about five minutes. AnythingLLM found Ollama automatically and listed available models in the UI.
Old documents I’d been using for testing? I uploaded them into AnythingLLM workspaces and indexed them. It handles PDFs, text, markdown. The UI ingestion is clean—drop files, pick a workspace, hit index. I didn’t have to rewrite anything. The documents just worked.
What I Gained vs. What I Lost
Gained: Real multi-user support. My partner can now have her own workspace and conversations. They don’t bleed into mine. The API tokens are managed per-user, which feels right. I gained document management that doesn’t rely on my janky Python scripts. I gained agents—basic automation of tasks like searching documents, summarizing, or calling external APIs. The web UI is functional enough that I stopped needing to write wrapper code for basic tasks.
Gained: Workspace isolation. One workspace for tech notes, one for recipes, one for testing. Each has its own document store and model configuration. This level of organization wasn’t possible with raw Ollama.
Lost: Raw speed. There’s a small overhead from AnythingLLM’s request handling, vectorization, and web layer. A query that took 800ms in Ollama takes 1.2–1.5 seconds in AnythingLLM. It’s not terrible, but it’s noticeable. This surprised me. I expected the difference to be negligible on local hardware.
Lost: Model switching simplicity. With Ollama, I’d just run ollama run mistral and switch instantly. AnythingLLM wants you to configure models per-workspace or globally, which is more deliberate but less spontaneous. I can’t just fire up a different model on a whim anymore.
Lost: Direct API control. I can’t hit the AnythingLLM API the same way I hit Ollama. The endpoints exist but they’re higher-level and require more ceremony—auth tokens, workspace IDs, etc. That’s a feature for security. But when I want to write a quick one-off script, it’s friction I didn’t have before.
Hardware and Resource Tradeoffs
Ollama on my NUC uses about 4GB RAM at idle, 10–12GB with a large model loaded. AnythingLLM adds another 1.5–2GB for its runtime, database, and vectorization layer. The NUC has 16GB total. Tight, but workable. If I had 8GB or less, I’d be running into swap or OOM kills regularly.
CPU-wise, there’s more load during document indexing. AnythingLLM chunks, embeds, and stores documents in a vector database. For a 50-page PDF, indexing takes 20–30 seconds on the i5. That’s acceptable. For large batches of documents, I’d probably want more cores.
Storage hasn’t been an issue. The combined image and data volumes run about 15GB. I allocated 50GB to be safe.
What Surprised Me (in a Bad Way)
The web interface occasionally loses sync with the backend. I’ll submit a query and the UI hangs for a few seconds before returning results. Other times it works instantly. I never figured out if this is a networking thing, a resource thing, or just how it behaves. It’s infrequent enough that I stopped worrying about it, but it’s there.
The vector database—AnythingLLM defaults to using whatever embedding model you configure—sometimes returns irrelevant chunks from documents. I thought RAG would be a solve-all for accuracy. It isn’t. The quality of retrieved context depends on your embedding model and how well your documents are structured. Garbage in, garbage out still applies. I had to be more careful about document preparation than I expected.
Configuration is scattered across environment variables, the UI settings panel, and a JSON config file. I had to dig into three places to get everything working as I wanted. There’s no single source of truth. If something’s wrong, you’re hunting.
Would I Do It Again?
Yes, but I’d make a couple of decisions differently. I’d start AnythingLLM on a dedicated machine from the beginning instead of running it on the same box as Ollama. I’d spend more time upfront tuning the embedding model and document chunking strategy. I’d probably set up a proper database backend (Postgres) instead of relying on the default, since the local database occasionally gets sluggish when you accumulate conversations.
The switch wasn’t necessary for what I was doing alone. I could have lived with Ollama plus a better wrapper. But now that I have multiple people using the system and multiple projects running in parallel, AnythingLLM earns its place. It’s doing real work.
I’m not back to raw Ollama anymore. I’m also not planning to write a replacement. For now, this is the right layer between the runtime and the users.
FAQ
Can I run AnythingLLM with Ollama on the same machine?
Yes. Docker Compose makes it easy. But you’ll want at least 16GB RAM. With 8GB or less, you’ll run into resource contention during model inference and document indexing. I tested it and both services slow to a crawl.
Do I need to export or convert my Ollama models?
No. AnythingLLM talks to Ollama over its API. The models stay where they are. Just point AnythingLLM to your Ollama instance and it lists available models in the UI.
Is AnythingLLM’s RAG better than doing it manually with LangChain?
It’s faster to set up and manage, but not fundamentally different. The quality depends on your embedding model, chunking strategy, and document structure. Manual control with LangChain gives you more tuning options. AnythingLLM gives you usable defaults.
How much disk space does AnythingLLM need?
The container and base installation need about 2GB. Stored documents, vector embeddings, and conversation history grow from there. A few hundred documents might use 5–10GB depending on size and embedding model. Add 20–30GB if you’re indexing large PDF libraries.
Can I use AnythingLLM with only local models, no cloud providers?
Yes. Run it with Ollama or LocalAI or similar. You never have to touch OpenAI, Anthropic, or anyone else. Configure the LLM_PROVIDER environment variable and you’re set. Everything stays on your hardware.
Explore AnythingLLM in our AI Homelab Toolkit.
Recommended Hardware & Hosting
Build your homelab with hardware tested and used by our team.
Affiliate links — we may earn a small commission at no extra cost to you.