How to Install Hugging Face on Ubuntu: Self-Hosted Setup

If you’re working with machine learning models in your homelab, you’ll eventually need to access Hugging Face — either through their web API or by pulling models down to run locally. The web interface is convenient for exploring, but self-hosting Hugging Face gives you control over model selection, inference speed, and privacy. This guide covers installing Hugging Face on Ubuntu Server, configuring the Transformers library, and getting your first model running.

Why Self-Host Hugging Face

Hugging Face is essentially a repository of pre-trained machine learning models, datasets, and tools. The Transformers library is what lets you actually use those models in Python. You have three ways to interact with Hugging Face: browse the hub online, use their Inference API (paid, rate-limited), or download models and run inference locally.

Self-hosting means you own the compute. You avoid API calls, rate limits, and monthly bills. You also get predictable latency — important if you’re integrating models into other services on your network. The tradeoff is that you need enough disk space for models (they range from 350MB to 40GB+) and enough RAM or GPU memory to load them.

Most of my use has been pulling text models like Mistral and Llama for local LLM work, but the same process applies to image models, audio models, and classifiers.

Prerequisites and Hardware

You’ll need an Ubuntu 20.04+ system with Python 3.8 or higher. I’ve tested this on a dedicated Ubuntu Server machine with 16GB RAM and a GTX 1080 Ti — though you can run smaller models on CPU-only systems if you’re patient.

Minimum specs for text inference:

• 8GB RAM (16GB recommended)
• 20GB free disk space (more if you plan multiple models)
• Python 3.8+
• pip or conda for package management

Recommended for faster inference:

• NVIDIA GPU with 6GB+ VRAM (RTX 3060 or better)
• CUDA 11.8+ if using GPU
• cuDNN 8.x for GPU acceleration

If you’re on a system without a GPU, CPU inference will work but will be slower. A 7B parameter model on CPU takes roughly 30-60 seconds per generation. With a decent GPU, you’re looking at 2-5 seconds.

Step 1: Install Python and Required Dependencies

First, make sure your system is up to date:

sudo apt update
sudo apt upgrade -y

Install Python development headers and build tools:

sudo apt install -y python3-dev python3-pip python3-venv build-essential

Create a virtual environment for your Hugging Face work. This keeps dependencies isolated from your system Python:

python3 -m venv ~/hf-env
source ~/hf-env/bin/activate

Upgrade pip inside the virtual environment:

pip install --upgrade pip setuptools wheel

If you have an NVIDIA GPU and want to use it, install the NVIDIA CUDA toolkit. Check your CUDA version compatibility first:

nvidia-smi

This command tells you your GPU, driver version, and CUDA capability. If you don’t have CUDA installed yet, you can grab it from NVIDIA’s site, but honestly, I usually install torch with CUDA bundled, which is simpler.

Step 2: Install Transformers and PyTorch

With your virtual environment active, install the Transformers library and PyTorch. For GPU support, specify the CUDA version:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

That installs PyTorch with CUDA 11.8 support. If you’re CPU-only, use:

pip install torch torchvision torchaudio

Now install Transformers and some useful companions:

pip install transformers datasets accelerate huggingface-hub

The accelerate library helps distribute inference across multiple GPUs or optimize for single GPUs. The huggingface-hub package gives you utilities to download and manage models programmatically.

Test the installation:

python3 -c "import transformers; print(transformers.__version__)"

You should see a version number, something like 4.35.2. If it errors, go back and check that pip installed into your virtual environment.

Step 3: First-Run Configuration and Model Download

Hugging Face stores models in a cache directory. By default, that’s ~/.cache/huggingface/. You can change this if you want models on a different disk or partition:

export HF_HOME=/path/to/your/large/disk/huggingface

Add that to your ~/.bashrc or ~/.bash_profile if you want it persistent across sessions.

Create a simple Python script to download and test a model. Start small — a distilled model like DistilBERT runs fast and uses minimal memory:

#!/usr/bin/env python3
from transformers import pipeline

# Download and cache the model
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

# Test it
result = classifier("I really enjoyed that meal.")
print(result)

Save that as test_hf.py and run it:

python3 test_hf.py

On first run, the script downloads the model (about 268MB for DistilBERT), caches it, then runs inference. You should see output like:

[{'label': 'POSITIVE', 'score': 0.9998}]

That means it’s working. The model was downloaded to ~/.cache/huggingface/hub/. Future runs will use the cached version and skip the download.

Now try a larger model if your hardware supports it. A 7B parameter LLM needs about 14GB RAM (or VRAM) for float16 precision:

#!/usr/bin/env python3
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "mistralai/Mistral-7B-Instruct-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = "Tell me a short joke:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This downloads Mistral-7B (about 14GB), loads it in half precision to save VRAM, and generates text. The first run takes a while because of the download. Subsequent runs are much faster.

Common Installation Gotchas

Out of memory during model loading: If you get a CUDA out-of-memory error, try loading the model in 8-bit quantization instead of float16:

pip install bitsandbytes

Then in your script:

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_8bit=True,
    device_map="auto"
)

This roughly halves VRAM usage at a small cost to quality. A 7B model in 8-bit needs about 7GB VRAM instead of 14GB.

Slow downloads: If Hugging Face’s CDN is slow from your location, you can configure a mirror. Some regions use different mirrors. This is rare but worth knowing if you’re downloading large models repeatedly.

Models downloading to the wrong location: If you set HF_HOME but models still go elsewhere, check that you exported it before running Python. Environment variables set in the shell don’t affect processes already running. Restart your terminal or explicitly pass the cache dir in code:

from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="model/name", cache_dir="/path/to/cache")

Tokenizer warnings about trust_remote_code: Some models require executing code from the hub. Hugging Face issues a warning for security reasons. If you trust the model author, you can suppress it with:

model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)

Be deliberate about this — don’t blindly enable it for untrusted models.

What to Do Next

Once you have Hugging Face running, you have a few natural next steps. Most people either wrap their models in a web service (using Flask or FastAPI) to integrate with other tools, or they set up a notebook environment like Jupyter for interactive exploration.

If you want to serve models over HTTP, something like vLLM or Text Generation WebUI is more convenient than writing your own server. If you want to automate batch inference — say, classifying a pile of text files — you can write a simple Python script and schedule it with cron.

The Hugging Face Hub itself is worth browsing. Models are tagged by task (sentiment-analysis, text-generation, image-classification, etc.), and you can filter by size, framework, and license. Most open-source models include a model card with performance benchmarks and known limitations. Reading those before you commit to running something locally saves time.

I’ve spent more time than I’d like troubleshooting VRAM exhaustion and managing multiple large models on limited disk space. You’ll hit those problems too eventually. When you do, quantization and careful model selection become your friends. But that’s a tuning problem, not an installation problem — and it means you’re already up and running.

FAQ

How much RAM do I need to run Hugging Face models locally?

Minimum 8GB system RAM for smaller models. Larger models like Mistral 7B need 14-16GB RAM for float16 inference, or 7-8GB with 8-bit quantization. GPU VRAM requirements are similar. CPU-only inference is possible on less RAM but much slower.

Can I run Hugging Face on a Raspberry Pi?

Technically yes, but not practically. A Raspberry Pi 4 has 8GB RAM maximum and an ARM processor, so only tiny models (under 500MB) run without extreme patience. A small x86 server or older laptop is a better option.

Do I need an NVIDIA GPU for Hugging Face?

No. CPU inference works fine, just much slower. A GPU (NVIDIA, AMD, or Apple Silicon) makes inference 10-50x faster depending on the model. For experimentation and small batches, CPU is adequate.

What’s the difference between downloading models and using the Hugging Face Inference API?

The Inference API is hosted by Hugging Face and billed per request — simpler but has rate limits and costs money. Self-hosting downloads models to your machine and runs inference locally — one-time setup, no ongoing costs, and full control over latency and privacy.

How do I update models after downloading them?

Models are cached by commit hash. To get a newer version, delete the cached folder (in ~/.cache/huggingface/hub/) or set revision="main" when loading to always fetch the latest. Usually you don’t need to update unless you specifically want a new version.

Explore Hugging Face in our AI Homelab Toolkit.