Every prompt you type into a cloud AI service travels somewhere. It gets logged, sometimes stored for months, and occasionally used to train the next model. If you work with client contracts, patient notes, unreleased code, or anything covered by an NDA, that quiet exfiltration is a real liability. The uncomfortable truth is that "we don't train on your data" is a promise, not a technical guarantee.
Here's the surprising part: a modern 8-billion-parameter model that would have needed a data-center GPU two years ago now runs comfortably on a laptop with 16GB of RAM. Meta's Llama 3.1 8B, Mistral 7B, and Google's Gemma 2 all fit on consumer hardware and answer most everyday questions within a second or two. The gap between "cloud-only" and "runs on my machine" has closed faster than most people realize.
In this guide I'll show you how to run a local LLM with Ollama from a clean install to a working chat and API, which models actually make sense for which hardware, how to keep the whole thing offline, and where the real tradeoffs hide. I've been running Ollama daily on a mid-range Windows machine and an M2 Mac, so this is the setup I actually use, not a spec sheet.
Key Takeaways
- Ollama is the fastest path to a local LLM. One installer, one command (
ollama run llama3.1), and you're chatting offline in minutes.- Your data never leaves the machine. Inference runs locally, so prompts, documents, and outputs stay on disk under your control.
- Match the model to your RAM. Roughly 8GB free for a 7B/8B model, 16GB+ for 13B, and 32GB+ for anything near 30B parameters.
- Quantized models (Q4) cut memory use by 60-70% with only a small quality drop, which is why they're the sensible default.
- Ollama exposes a local REST API on port 11434, so you can wire it into scripts, editors, and apps without touching the internet.
- Verify what you install. Model weights and helper tools are software too, and they deserve the same scrutiny as anything else you run.
What Is Ollama and Why Run an LLM Locally?
Ollama is a free, open-source tool that packages large language models into a simple runtime. Think of it like Docker for AI models: you pull a model by name, run it, and Ollama handles the download, memory management, GPU acceleration, and a clean command-line and API interface. It runs on macOS, Windows, and Linux.
The reason to bother with local inference comes down to three things:
- Privacy. Nothing is sent to a third-party server. This matters for regulated data, proprietary code, and anything you simply don't want logged.
- Offline access. Once a model is downloaded, it works on a plane, in a bunker, or when your ISP goes down. No API keys, no rate limits.
- Cost. After the one-time hardware you already own, inference is free. No per-token billing that creeps up as your usage grows.
The tradeoff is honest: local models are smaller and less capable than frontier cloud models like GPT-4o or Claude. For summarizing a document, drafting an email, cleaning up code, or answering factual questions, an 8B model is genuinely good. For deep reasoning over a 50-page contract, cloud models still win. Knowing that line saves you frustration.
How to Install and Run a Local LLM with Ollama
The install is refreshingly boring, which is exactly what you want from infrastructure. Here's the full walkthrough for a working setup.
Step 1: Install Ollama
- Go to
ollama.comand download the installer for your OS. On macOS it's a.dmg, on Windows an.exe, on Linux a one-line curl script. - Run the installer. On desktop it adds a small background service and a menu-bar/tray icon.
- Open a terminal (Terminal on Mac, PowerShell on Windows) and confirm it works with
ollama --version. You should see a version number like0.3.x.
Before you download anything, it's worth applying the same caution you would to any downloaded binary. Our guide on how to verify an open-source tool wasn't poisoned before installing covers checksums and signature checks that take two minutes and save real headaches.
Step 2: Pull and Run Your First Model
This single command downloads Llama 3.1 8B and drops you into a chat prompt:
ollama run llama3.1
The first run downloads roughly 4.7GB, so give it a few minutes on a normal connection. After that it's cached locally and launches in seconds. Type a question, press Enter, and you'll get a streamed answer. To exit the chat, type /bye.
Step 3: Try a Few Models and Manage Them
- List what you've downloaded:
ollama list - Pull without running:
ollama pull mistral - Run a smaller, faster model:
ollama run gemma2:2b - Remove a model to free disk:
ollama rm llama3.1 - See what's loaded in memory right now:
ollama ps
That's genuinely it. Within ten minutes you have a private AI assistant that never touches the internet after the initial download.
A Worked Example: Summarizing a Confidential Document Offline
Let's make this concrete. Say you're a solo consultant with a 12-page client agreement, about 6,200 words, that you cannot legally paste into a cloud chatbot. You want a plain-English summary and a list of the obligations that fall on you.
Here's the before/after with actual numbers from my own test on an M2 Mac with 16GB RAM:
- Download the model once:
ollama pull llama3.1(4.7GB, took about 3 minutes). - Start the chat:
ollama run llama3.1. - Paste the contract text preceded by a clear instruction: "Summarize this agreement in plain English, then list every obligation that applies to me as the consultant. Here is the document: [pasted text]."
- The model processed the ~8,000 tokens and started streaming a response in about 4 seconds, finishing a full summary in roughly 22 seconds.
The result: a clean 6-bullet summary and a 9-item obligations list that I cross-checked against the source. It missed one nuance about a renewal window, which is exactly the kind of detail where you still read the document yourself. But as a first pass that saved 20 minutes of reading, and never left my laptop, it was a clear win.
The lesson: local models are excellent accelerators for private work, not replacements for your judgment. If you're weaving AI into a coding workflow, the same principle applies, and our piece on how to vet AI coding assistants for security flaws is worth reading before you trust generated output blindly.
Which Local Model Should You Run? A Comparison
Model choice is where people overthink things. The honest answer is that your available RAM decides most of it, and quantization (compressing the weights) does the rest. Here's how the popular options stack up for a typical consumer machine.
| Model | Parameters | Disk / RAM (Q4) | Best for | Speed on 16GB laptop |
|---|---|---|---|---|
| Gemma 2 2B | 2 billion | ~1.6GB / 3GB | Quick tasks, low-end hardware | Very fast |
| Mistral 7B | 7 billion | ~4.1GB / 6GB | General chat, coding help | Fast |
| Llama 3.1 8B | 8 billion | ~4.7GB / 8GB | Best all-round balance | Fast |
| Phi-3 Medium 14B | 14 billion | ~7.9GB / 12GB | Reasoning, structured output | Moderate |
| Llama 3.1 70B | 70 billion | ~40GB / 48GB+ | Serious work, needs a workstation | Slow without a strong GPU |
How to Read That Table
- If you have 8GB RAM, stick to Gemma 2 2B or a quantized 7B and keep other apps closed.
- If you have 16GB RAM, Llama 3.1 8B is the sweet spot most people should start with.
- If you have 32GB+ and a decent GPU, try the 13B–14B class for noticeably better reasoning.
- The 70B models are real but need a workstation or a dedicated GPU with 48GB of VRAM to run at usable speed.
What Quantization Actually Means
Quantization stores each weight with fewer bits. A full-precision model uses 16 bits per weight; a Q4 model uses about 4. That cuts memory roughly 60-70% while keeping most of the quality. For everyday use, the Q4_K_M variant Ollama pulls by default is the right default. You'd only reach for higher precision if you notice quality problems on a specific task.
Using the Local Ollama API for Your Own Apps
The command line is fine for chatting, but the real power shows up when you connect Ollama to your own tools. When Ollama runs, it exposes a REST API on http://localhost:11434. Everything stays on your machine.
Here's a minimal request you can run from any terminal to test it:
curl http://localhost:11434/api/generate -d '{"model": "llama3.1", "prompt": "Explain what a mutex is in one sentence.", "stream": false}'
You'll get a JSON response with the generated text. From here you can build real things:
- A private note-summarizer that runs on your desktop with no cloud dependency.
- A code-review helper wired into your editor via an extension that points at localhost.
- A batch script that processes a folder of documents overnight, entirely offline.
Cover image: Software value feedback loop by jakuza, licensed under BY-SA 2.0 via Openverse.
