Grafana + AI Docker Setup: 5 Errors I Fixed and Why

I spent three evenings getting Grafana + AI working in Docker. Not because it’s inherently broken—it’s not—but because I made the mistakes you’re about to avoid. Natural language queries against Prometheus data, anomaly detection on my lab metrics, root cause analysis that actually tells you something useful. The features are solid. The setup just has teeth if you’re not careful about a few specific things.

Grafana + AI screenshot — Grafana + AI u2014 from the official site

Why Grafana + AI Docker Setup Fails (And How to Debug It)

Most of the failures I hit weren’t in Grafana’s code. They were in my assumptions: bad networking, missing environment variables, insufficient memory allocations, and one brutal typo that took an hour to find. If you’re running Grafana + AI in Docker and it’s not working, it’s probably one of these five things. The errors look different, but the patterns are predictable once you know where to look.

Error 1: “Cannot connect to Prometheus” (Port Binding Issues)

The error shows up in the Grafana UI when you try to add your Prometheus data source:

error reading Prometheus: failed to connect to http://prometheus:9090
Context deadline exceeded

I had Prometheus running in one container and Grafana in another. Both were on the same Docker Compose stack. Should work. Didn’t.

The problem was my networking assumption. In Docker Compose, containers communicate by service name—http://prometheus:9090 is correct. But you also need them on the same network. I’d created Prometheus first, let it sit, then added Grafana weeks later in a different compose file. Different networks. No connection.

The fix: explicit network in your Compose file.

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana-ai
    ports:
      - "3000:3000"
    environment:
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    networks:
      - monitoring
    depends_on:
      - prometheus

networks:
  monitoring:
    driver: bridge

The depends_on clause isn’t strictly necessary for networking, but it ensures Prometheus starts first. More importantly, both services explicitly declare they’re on the monitoring network. Test it by exec’ing into Grafana and pinging Prometheus:

docker exec grafana-ai ping prometheus

If you get ping responses, your network is correct. If you get “unknown host,” you have a network issue, not a Prometheus issue.

Error 2: “NLP Not Available” (Missing Plugin or Version Mismatch)

You add a panel, go to try natural language queries, and there’s no option. Or it’s greyed out. The UI suggests you need to install something, but nothing’s clear.

Grafana’s AI features—natural language queries, anomaly detection, the Sift root-cause stuff—require specific versions and occasionally plugins. I was running Grafana 9.4 in a container. The AI features started in 10.0. So I didn’t have them.

Check your current version first:

curl http://localhost:3000/api/health | jq '.version'

If you’re below 10.0, upgrade. If you’re at 10.0 or higher and the features still don’t show, check that you’re actually trying to use them in the right place. Natural language queries are in the explore interface, not in basic panels. Go to Explore (left sidebar), pick your data source, and look for a “Natural language” button or input field.

There’s also a config flag you can verify in your environment:

environment:
  - GF_FEATURE_TOGGLES_ENABLE=mlAnomalyDetection,nlQueryEnabled

Add those if they’re missing. Without them, the UI won’t show the buttons even if the version is right.

Error 3: “OutOfMemory: Container Killed” (Resource Limits Too Tight)

This one’s subtle. Your container starts fine, works for ten minutes, then dies. You check the logs and see nothing useful. The actual error lives in Docker’s daemon logs or the system journal.

The AI features—especially if you’re running anomaly detection on large metric sets—need real memory. Grafana itself is fine on 512 MB. With ML models in the same container, you’re looking at 1.5 to 2 GB if you’re doing anything real.

I had:

resources:
  limits:
    memory: 512M

That’s not enough once you start querying AI features. Check what you’re actually using:

docker stats grafana-ai

If you see the memory number climb toward your limit and then the container restarts, raise it:

resources:
  limits:
    memory: 2G
  requests:
    memory: 1G

The requests value tells your orchestrator how much to reserve. The limits value kills the container if it exceeds it. Be generous with AI workloads. Tight memory allocation will make everything feel flaky.

Error 4: “Unauthorized: Invalid API Key” (Auth and Token Issues)

You’ve set up your data source, clicked “test,” and hit a 403 or 401. Or you’re trying to use the API directly and getting rejected.

Grafana needs authentication tokens to talk to some services, especially if you’re using cloud features or if you’ve locked down your backend. I had set GF_SECURITY_ADMIN_PASSWORD but wasn’t using API tokens for my Prometheus connection properly.

Create an API token in Grafana. Go to Administration (gear icon) > API Keys > Add API key. Give it a name, pick the role (Viewer is fine for read-only data), set an expiration, and generate it.

If you’re connecting to Prometheus over authentication, you’ll put that token in your Prometheus data source config in Grafana. But for most homelab setups, Prometheus doesn’t require auth. The real issue is usually Grafana’s own API access.

If you’re hitting the API directly (e.g., scripting alerts or queries), pass your token as a header:

curl -H "Authorization: Bearer YOUR_API_TOKEN" http://localhost:3000/api/datasources

Check that the token hasn’t expired and that it has the right permissions for what you’re trying to do. Viewer tokens can’t create dashboards. Editor tokens can’t delete users. Read the scope carefully.

Error 5: “GPU/ML Services Unavailable” (Feature Dependencies)

You’ve enabled all the AI features, but anomaly detection doesn’t work. You get warnings in the logs about TensorFlow or ONNX not being available. Sift (Grafana’s automated root cause analysis) shows “service unavailable.”

The Grafana container image doesn’t include ML libraries by default. Some AI features are shipped as separate services or sidecars. You can run Grafana with basic NLP (question-answering against your metrics), but deeper ML work requires additional setup.

If you want real anomaly detection, you’re running a separate Grafana ML container. That’s grafana/grafana-oss-mlservices or similar. Your compose file needs both:

grafana-ml:
  image: grafana/grafana-oss-mlservices:latest
  container_name: grafana-ml
  ports:
    - "8080:8080"
  networks:
    - monitoring
  environment:
    - LOG_LEVEL=info

Then in your main Grafana container, set:

environment:
  - GF_ML_ENABLED=true
  - GF_ML_URL=http://grafana-ml:8080

The ML service talks to Grafana on that URL. Verify it’s working by hitting the health endpoint from inside Grafana:

docker exec grafana-ai curl http://grafana-ml:8080/health

If that fails, check network connectivity again (back to Error 1). If it works but anomaly detection still doesn’t show up, verify you’re on Grafana 10.2 or later—some ML features rolled out in stages.

Testing and Validation

Once all five errors are cleared, your setup should actually work. I spent a lot of time chasing symptoms instead of causes, so here’s what I do now:

Check version with curl http://localhost:3000/api/health
Verify data source connectivity by adding the source and hitting “Save & test”
Look at actual container logs: docker logs grafana-ai
Check memory and CPU with docker stats to catch resource issues early
Test API access with a curl command, not just the UI

Grafana + AI in Docker works fine if you’re methodical. It fails loudly and specifically if you’re not. Most of what I fixed wasn’t Grafana’s fault—it was networking assumptions, memory budgets I hadn’t thought through, and one typo in a Compose file that took longer to find than it should have.

FAQ

Can Grafana + AI run on a Raspberry Pi?

Technically yes, but not really. Grafana runs fine. The AI features (anomaly detection, ML-based insights) are resource-heavy. A Pi with 4 GB RAM will run the UI, but any serious ML workload will either be slow or crash. Stick with x86 or ARM64 with at least 8 GB for the full feature set.

How much RAM does Grafana + AI actually need?

Base Grafana: 512 MB. With AI features enabled and running anomaly detection or NLP queries regularly: 1.5–2 GB minimum. If you’re querying large datasets, go to 2–4 GB. I settled on 2 GB after my first OOM kill.

Do I need a GPU for Grafana + AI anomaly detection?

No. CPU-based ML is fine for a homelab. GPU acceleration helps if you’re running inference on massive datasets or real-time streaming, but that’s not typical homelab monitoring. CPU works.

What data sources work with Grafana + AI natural language queries?

Prometheus, InfluxDB, Loki, and most time-series databases work. Grafana’s NLP layer understands time-series concepts like “CPU spike last hour” or “error rate by service.” SQL databases and document stores don’t have the same NLP support out of the box.

Is Grafana + AI self-hostable without internet?

Yes. The NLP and anomaly detection run locally in your Docker containers. You don’t send data to Grafana Labs’ servers unless you explicitly configure cloud integrations. Everything stays on your hardware.

Explore Grafana + AI in our AI Homelab Toolkit.