Black Friday mini PC deals are live, checked daily through Cyber Monday. See the deals guide
← All Guides
intermediate

Self-Host an LLM at Home With Ollama

Run a private, offline ChatGPT alternative on your own hardware with no ongoing subscription cost. No GPU required. Setup from hardware to first conversation.

Budget Homelab ·
dockeraimini-pchow-to

This post contains affiliate links. If you buy hardware through these links, I earn a small commission at no extra cost to you.

Prices checked October 2026. The ranges on this page are rough and rounded. Mini PC and RAM prices moved a lot in 2026, so check the current price before you buy.

ChatGPT Plus costs $20/month. Claude Pro costs $20/month. Both send your prompts to someone else’s servers.

A decent mini PC runs 7B models locally, and draws about as much power as a dim light bulb. The catch this year is the price. A new Ryzen mini PC that runs 7B models comfortably starts around $400 and goes up fast, so check the current price (checked October 2026). The break-even math against a subscription is longer than it used to be, roughly two years or more, so run it with your own numbers in the homelab vs cloud cost breakdown. Every conversation stays on your network, which is the part a subscription cannot give you.

This guide covers the whole setup: picking the right hardware, installing Ollama, choosing models that fit your specs, and adding Open WebUI for a proper chat interface. If you’ve never run a local LLM before, this is the place to start.

For hardware context outside the LLM-specific picks below, the best mini PCs for a homelab in 2026 covers the broader landscape. If you’re deciding what else to self-host, see replacing Google One and other subscriptions with self-hosted tools.

Hardware You Need

Ollama runs on CPU without any GPU. What determines which models you can run is RAM. Models load entirely into RAM during inference.

RAMWhat you can runApprox. response speed
8GB3B models (Phi-3 Mini, Llama 3.2 3B)Fast on modern CPU
16GB7B-8B models (Llama 3.1 8B, Mistral 7B)3-8 tokens/sec
32GB13B models, or 7B with room for other servicesSame speed, bigger models
32GB + GPU (12GB VRAM)13B fully GPU-loaded40-60+ tokens/sec

“3-8 tokens per second” means roughly one word every 0.3-0.6 seconds. Slow for real-time conversation, workable for “generate this code block” or “summarize this document.” Most people find it usable and consider the privacy tradeoff worth it.

Mini PC Picks

For CPU inference, you want a fast multi-core processor and as much RAM as the board supports. AMD Ryzen 7000/8000 series mini PCs are the current sweet spot.

Budget tier (new N-series box):

The Beelink EQ14 has an Intel N150 and, in the configuration on Beelink’s page, 16GB of RAM. I’d treat that as the entry tier for 3B models. The N-series has efficient cores, but single-core throughput limits it with 7B models. Fine for light use. A new 16GB box in this tier costs about $300-450 (checked October 2026). If you already own a machine with 16GB, you do not need to buy anything to try this.

Mid-tier (Ryzen, about $400 and up, check current price):

The MinisForum UM790 Pro (Ryzen 9 7940HS) and the Beelink SER8 (Ryzen 9 8945HS) are the machines I’d look at for anyone serious about local LLMs. The Ryzen cores are meaningfully faster for inference than Intel N-series machines. Both carry AMD Radeon 780M integrated graphics, which supports partial GPU offloading. It doesn’t transform CPU performance, but it helps with smaller models. The UM790 Pro listing ships with 32GB DDR5 and a 1TB SSD, and it costs far more than it did a year ago, so check the current price before you plan around it.

Buy the RAM you need already installed. A 32GB DDR5 SO-DIMM costs about $400 or more right now (checked October 2026), which is why a configured 32GB box usually beats a 16GB box plus an upgrade. Confirm the slot count and the real supported maximum for your model first; mini PC RAM upgrades for a homelab goes through what fits and what gets returned.

Storage: Models eat disk space. Llama 3.1 8B alone is about 5GB. If you want several models installed, plan for at least a 1TB NVMe drive, which costs about $150-180 right now. Most mini PCs have a spare M.2 slot so you can add storage without replacing the built-in drive.

GPU upgrade path (check current price): A used NVIDIA RTX 3060 12GB changes the experience significantly: 13B models run fully GPU-accelerated at 40-60+ tokens per second. A used RTX 3090 24GB handles 70B quantized models, which puts you at the same capability tier as top commercial offerings. This path requires a desktop, not a mini PC, but it’s worth knowing the ceiling.

Install Ollama

Ollama is the engine. It loads models, serves them over a local API, and runs as a background service.

Linux (one-line install):

curl -fsSL https://ollama.com/install.sh | sh

This installs Ollama as a systemd service, starts it immediately, and sets it to start on boot. It listens on port 11434.

Verify it’s running:

systemctl status ollama

You should see active (running).

macOS: Download the native app from ollama.com. It runs in the menu bar and sets up the same local API.

Pull Your First Model

Models are separate downloads from Ollama itself. Start with:

ollama pull llama3.1:8b

About 4.7GB download. On a typical home connection, 5-10 minutes.

Test it immediately:

ollama run llama3.1:8b "What's the capital of France?"

If you get a coherent response, Ollama is working. Type /bye to exit interactive mode.

Other models worth trying once the basics are confirmed:

ollama pull mistral:7b          # Strong at following instructions
ollama pull qwen2.5-coder:7b    # Best code model in this size class
ollama pull gemma2:9b           # Google Gemma 2 (good reasoning)
ollama pull phi3:mini           # Microsoft Phi-3 Mini (very fast, capable for its size)

Model size reference

ModelDownload sizeRAM neededBest for
phi3:mini2.2GB4GBFast responses, simple tasks
llama3.2:3b2.0GB4GBQuick answers, low-RAM systems
llama3.1:8b4.7GB8GBGeneral purpose, strong starting point
mistral:7b4.1GB8GBInstruction following, Q&A
qwen2.5-coder:7b4.7GB8GBCode generation and review
gemma2:9b5.5GB10GBReasoning tasks

For a 16GB mini PC running other homelab services alongside it, llama3.1:8b is the practical ceiling. With 32GB, you can load a 13B model and still have RAM left over.

Set Up Open WebUI

The terminal interface works but isn’t what most people want for daily use. Open WebUI provides a ChatGPT-like interface with conversation history, model switching, and document uploads.

It runs in Docker and connects to your Ollama instance. If you don’t have Docker installed, the Docker Compose basics guide covers installation.

Create a directory:

mkdir -p /opt/open-webui

Create /opt/open-webui/docker-compose.yml:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    volumes:
      - /opt/open-webui/data:/app/backend/data
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://YOUR_SERVER_IP:11434
    restart: unless-stopped

Replace YOUR_SERVER_IP with your server’s LAN IP (e.g., 192.168.1.50). If Open WebUI and Ollama are on the same machine, you can use the Docker bridge IP 172.17.0.1 instead.

Start it:

cd /opt/open-webui
docker compose up -d

Navigate to http://YOUR_SERVER_IP:3000. Create an account: the first account created gets admin privileges. Select your model from the top dropdown and start a conversation.

What You Can Actually Do With This

Code help: Paste a function and ask what it does, how to improve it, or to write a test. CPU inference speed is fine here because the interaction is write-once, wait-once rather than back-and-forth conversation.

Summarization: Paste a long article, meeting notes, or documentation section. Ask for a bullet summary. 7B models handle this well.

Document Q&A: Open WebUI supports file uploads. Upload a PDF and ask questions about its contents.

Private use cases: A local LLM you control is valuable for anything you’d never type into a commercial service: personal decisions, health questions, sensitive work topics. It’s not connected to anything outside your network.

API access for other apps: Ollama exposes an OpenAI-compatible API at http://YOUR_IP:11434/v1. Applications that support a custom OpenAI endpoint can often point at your local Ollama instead of the real OpenAI API.

GPU Acceleration

CPU inference is usable. GPU inference is a different tier of experience.

For NVIDIA GPU passthrough in Docker, install the NVIDIA Container Toolkit on the host first, then update your Ollama deployment:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - /opt/ollama/models:/root/.ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

Ollama detects and uses the GPU automatically after restart. A 7B model that generates text at 5 tokens/sec on CPU will run at 50+ tokens/sec on a GPU, fast enough to feel like a real conversation.

VRAM is the hard limit for GPU inference. Models must fit in VRAM to run GPU-accelerated; overflow spills to CPU and slows everything down.

GPU VRAMModels that fit fullyTypical speed
8GBUp to 7B models30-50 tokens/sec
12GB (RTX 3060)Up to 13B models40-60 tokens/sec
24GB (RTX 3090)34B models; 70B Q4 quantized60-80 tokens/sec

If you’re already running a homelab desktop with a capable GPU sitting idle, enabling GPU passthrough for Ollama costs nothing. For help picking the right GPU, the GPU buyer’s guide for self-hosted AI compares the used 3060, 3090, P40, and Intel Arc on real VRAM, power draw, and price-per-token.

Remote Access

Open WebUI is accessible from any device on your local network at http://YOUR_IP:3000. For access from outside your home, Tailscale is the right path.

If Tailscale isn’t already on your server, the Tailscale homelab setup guide walks through it. Once installed, Open WebUI is accessible at http://YOUR_TAILSCALE_IP:3000 from any device on your tailnet. No port forwarding, no public exposure, no SSL required for this use case.

Maintenance

Models don’t auto-update. When a new version releases, pull it manually:

ollama pull llama3.1:8b    # pulls the latest version
ollama list                # see what's installed + storage used
ollama rm gemma2:9b        # remove a model you're not using

Update Open WebUI when new versions drop:

cd /opt/open-webui
docker compose pull
docker compose up -d

Conversation history is stored in the data volume and persists through updates.


A self-hosted 7B or 13B model won’t replace frontier AI for complex reasoning tasks. But for code help, summarization, private Q&A, and most everyday AI assistance, a mini PC running Ollama competes well with a $20/month subscription on privacy and control, and everything stays on your hardware. The cost side of that comparison is a range now, so run the numbers with the break-even math before you buy.