How Continue Works: Local LLM Architecture Explained

The moment I realized I could run AI code completion entirely on my own hardware without sending anything to Anthropic or OpenAI was the moment I started looking at Continue. Not because I’m paranoid, but because I was tired of context limits, rate throttling, and wondering what happened to my code snippets. I needed to understand what was actually happening when I hit a keystroke in VS Code and got back a suggestion. This article is that breakdown.

The Core Problem: Why Local AI Coding Assistants Matter

Most developers use GitHub Copilot or Claude’s VS Code extension. They work well. They also send your code to a remote API. That’s not inherently bad—it’s the trade-off you accept for convenience. But if you’re running a homelab, you’ve already decided you want to own your infrastructure. You’ve probably got spare GPU capacity sitting there. Why not use it?

The friction point is integration. You can run Ollama locally and talk to it from the command line, sure. But getting a polished IDE experience—real-time completions, context-aware chat, inline edits—that requires an extension that knows how to talk to your local models. That’s where Continue comes in. It’s purpose-built for this.

The architecture is simpler than you’d expect, but there are enough moving parts that it’s worth understanding what’s really happening under the hood.

Understanding Continue’s Extension Architecture

Continue itself is not an LLM. It’s a broker. Think of it as a very smart wrapper around whatever model you point it at.

The extension runs in two processes: the VS Code extension process (which handles UI, keybindings, and editor state) and a separate Node.js background process that does the actual work. This separation matters because it keeps the IDE responsive even when the LLM is thinking.

When you install Continue via the VS Code Marketplace, you get:

• A sidebar panel for chat and documentation
• Inline completions triggered by typing or a keyboard shortcut
• An edit command that rewrites selected code
• Context awareness that reads your current file, related files, and sometimes your git history

All of this is defined in a config file—by default ~/.continue/config.json. This is where you tell Continue where your LLM lives and what it should do.

How the Request Flow Works: From Keystroke to Model Output

Here’s what happens when you trigger a completion:

1. The editor captures context. Continue reads the current file, your cursor position, and optionally nearby files or git diffs. It constructs what’s called a “context window”—the information it’ll send to the model.
2. It builds a prompt. The extension doesn’t send raw code. It wraps it in a system prompt that tells the model what task to perform (complete this line, explain this function, generate a test, etc.). This prompt is configurable.
3. It connects to your model. Here’s where the magic happens. Continue doesn’t care if your model is running on localhost:11434 (Ollama default), in a Docker container, or even on another machine. It speaks multiple protocols: OpenAI-compatible API, Ollama native, Anthropic, custom HTTP endpoints.
4. The model streams tokens back. This is crucial. Most LLMs can stream their output token-by-token instead of waiting for the full response. Continue renders these tokens in the editor in real-time, so you see suggestions appear as they’re being generated, not in one big dump at the end.
5. The editor displays the result. Completions appear as grey ghost text. Chat responses appear in the sidebar. Edits are shown as diffs.

This flow takes milliseconds from keystroke to first token appearing on screen. If it takes longer, usually something in your setup is slow—your network to the LLM, your model’s inference, or your hardware.

The Config File: Where You Tell Continue What to Do

This is where the system gets flexible. A basic config looks like this:

{
  "models": [
    {
      "title": "Mistral 7B",
      "provider": "ollama",
      "model": "mistral"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Mistral 7B",
    "provider": "ollama",
    "model": "mistral"
  },
  "completionOptions": {
    "maxTokens": 64,
    "temperature": 0.5
  }
}

The models array defines which LLMs are available for chat. The tabAutocompleteModel is specifically for inline completions (the ones that appear as you type). You can use different models for different tasks—a smaller, faster model for completions, a larger one for reasoning in chat.

The completionOptions control temperature (creativity), max tokens (length), and other parameters. Lower temperature gives more predictable, coherent completions. Higher temperature gives more variety but less consistency.

What I found useful is that you can also customize the system prompts. Continue has reasonable defaults, but you can override them in the config to change how the model behaves. For example, you can tell it your project uses a specific coding style, or that it should only respond in Python, or that it should always include type hints.

Supporting Multiple LLM Providers

One strength of Continue is that it abstracts away the differences between LLM providers. The extension handles:

• Ollama: Talks directly to the Ollama HTTP API on port 11434 by default.
• OpenAI-compatible APIs: LM Studio, Text Generation WebUI, LocalAI, vLLM—anything that mimics the OpenAI API. You point it at a custom endpoint.
• Cloud models: If you want, you can configure GPT-4, Claude, Gemini. Continue doesn’t force local-only, though that’s the point of this setup.
• Custom HTTP providers: If your model serves a non-standard API, you can write a custom provider.

Internally, Continue normalizes these into a common interface. Regardless of where your model lives, the extension sends requests and receives responses in a consistent way.

One thing that caught me off-guard: the extension creates a cache of recent completions and responses in ~/.continue. This helps with performance and also lets it remember context across sessions. It’s small—a few megabytes—but it’s worth knowing about if you’re debugging unexpected behavior.

Context Windows and Token Limits

Here’s where real constraints show up. Your LLM has a maximum context window—usually 4K, 8K, 32K, or 128K tokens depending on the model. Continue has to fit your code, your prompt, and the model’s instructions all within that limit.

When you ask for a completion on a huge file, Continue doesn’t send the entire file. It uses a strategy called “semantic context retrieval.” It reads your current file and nearby imports, and it can optionally scan your codebase with language-aware parsers (LSP, Language Server Protocol) to find relevant functions and classes. Only the most relevant parts go into the context window.

You can configure this behavior. In the config, you can set contextLength to control how many tokens of context to include, and you can enable or disable features like “include open tabs” or “include recent files.” On a slow machine or with a very large codebase, being aggressive about context pruning helps.

There’s a tradeoff here I’ve felt in practice. More context means better completions, but also slower responses. A 7B model with 8K context is faster than a 34B model with 32K context, but will miss more domain-specific patterns. You have to experiment and find what works for your hardware and patience threshold.

Why Streaming Matters (and Why It Sometimes Breaks)

Continue’s default behavior is to stream responses token-by-token. This is not required—you could wait for the full response before showing anything. But streaming changes the experience. You see the LLM thinking in real-time. If it’s going off the rails, you can hit Escape and cancel. If it’s right on track, you can accept it mid-completion.

The downside: streaming adds complexity. If the connection drops mid-stream, you get a partial response. If your model is slow to produce the first token (cold start), you stare at a blank screen for a few seconds. If your proxy or firewall drops long-lived HTTP connections, streaming breaks.

Most of this just works with Ollama running locally. Over a network, I’ve had issues with timeouts and dropped connections. Ollama has some built-in retry logic, but it’s something to be aware of if you’re running your model on a separate machine.

The Indexing and Embeddings Question

Here’s where I expected Continue to do something it doesn’t (yet). Some code assistants like Codebase AI or Cursor use embeddings—vector representations of your code—to do semantic search across your entire project. This would let Continue find relevant code without needing the LSP parser to tell it what’s related.

Continue currently doesn’t do this. It relies on simpler heuristics: file proximity, imports, and what’s currently open. For a small to medium project, this is fine. For a 50K-line codebase, you might notice that Continue sometimes misses context it could have found with embeddings.

This isn’t a dealbreaker. Most day-to-day coding doesn’t benefit from this. But it’s worth knowing if you’re evaluating Continue against other tools.

Security and Privacy Properties

Since you’re running this locally (or on your own network), your code never leaves your network unless you explicitly configure a cloud model. Continue doesn’t log your requests to any external service. It doesn’t phone home.

That said, the code you’re using as context is visible to whatever process is running the LLM. If Ollama is running as your user account, it can read it. If you’re running the model in a container, the container can read it. If you’re paranoid about this, you’d need to sandbox your LLM process further—use seccomp or AppArmor to limit what it can access. Most people don’t go that far.

The extension itself is open source, so you can audit it if you want. The repository is on GitHub. Unlike proprietary tools, there’s no code that’s closed-box.

What I Wish Was Different

After several months running Continue on my homelab, a few things stand out as friction points.

First: the initial setup is still manual. You have to edit a JSON config file by hand. It’s not hard, but it’s not graphical. For someone used to Copilot, this is a step backward. There’s a UI for some settings in newer versions, but it’s incomplete.

Second: completions on very large files can stall. If your file is 5000 lines and you’re at the bottom, Continue has to read and parse a lot of context. I’ve had 10-15 second delays waiting for the first token. Smaller models (3B-7B) help here, but large models struggle.

Third: there’s no built-in way to fine-tune behavior per-project. You could manage multiple config files and switch between them, but it’s clunky. I’d like to see a `.continue/config.json` in each project that overrides the user-level config.

These aren’t showstoppers. They’re just things I’ve bumped into and worked around.

FAQ

Can Continue run on a Raspberry Pi?

Continue itself runs fine on low-power hardware—it’s just a Node.js extension. The bottleneck is the LLM. Running anything beyond a 3B quantized model on a Pi would be extremely slow. If you have a Pi with 8GB RAM and want to use it as a client talking to an LLM running on a more powerful machine elsewhere, that works well.

Does Continue work offline?

Yes, if your LLM is running locally and you don’t have any cloud providers configured. Once models are cached in Ollama, you can unplug from the internet and Continue will still work. Network latency is only a factor if your LLM is on a different machine.

How much RAM does Continue need to run?

The extension itself uses 100-300MB. The LLM is separate—a 7B model quantized to 4-bit needs roughly 4-6GB, a 13B model needs 8-10GB. Total system RAM should be at least 16GB if you want the extension and model running comfortably alongside your editor and other tools.

Can I use Continue with GitHub Copilot at the same time?

You can install both, but they’ll both try to provide completions. You’ll get duplicate suggestions unless you disable one. Most people pick one. Continue is better if you want full control and privacy; Copilot is better if you want the model to be as smart as possible.

What’s the latency difference between Continue and cloud-based Copilot?

A local 7B model typically produces the first token in 200-800ms depending on your hardware. Copilot over the internet is often faster (50-200ms) because the servers are optimized for speed. Larger local models can take 2-5 seconds for first-token latency. The tradeoff is privacy and no API limits.

Explore Continue in our AI Homelab Toolkit.