Every prompt you type into a cloud AI service travels somewhere. It hits a server you don't control, gets logged for "quality and safety," and in many cases becomes training data unless you dug through the settings and opted out. For a casual question about pasta recipes, fine. But if you're pasting client contracts, proprietary code, patient notes, or unreleased financials into a chatbot, you've just handed your most sensitive material to a third party you'll never audit.
Here's a fact that surprises most people: a capable open-weight model like Llama 3.1 8B or Mistral 7B runs comfortably on a laptop with 16 GB of RAM, no GPU required, and produces answers good enough for the vast majority of everyday work. You don't need a data center. You need a free tool called Ollama and about fifteen minutes.
In this guide I'll walk you through exactly how to run Ollama locally, which models to pick for your hardware, how to verify your data never leaves the machine, and where local inference genuinely beats the cloud (and where it doesn't). I've been running models on my own gear daily for months, so this is the honest version, tradeoffs included.
Key Takeaways
- Ollama is a free, local runtime that downloads and runs open-weight LLMs entirely on your own computer, no account or internet connection required after setup.
- Privacy is the headline benefit: prompts and responses never touch an external server, which matters for regulated industries, NDAs, and internal code.
- You need roughly 8 GB of RAM for a 7B model and 16 GB+ for smoother multitasking. A GPU speeds things up but is optional.
- Start with Llama 3.1 8B or Phi-3 for general use, then graduate to larger models only if your hardware and needs justify it.
- Verify isolation by monitoring network traffic and, for maximum safety, running inference on an air-gapped machine.
- Local isn't always better: cloud models still win on raw capability for hard reasoning tasks. Use the right tool for the job.
What Ollama Actually Is (and Why Local Matters)
Ollama is an open-source command-line tool and background service that manages large language models on your own device. Think of it as a package manager for AI models. You type ollama run llama3.1, it downloads the weights once, and from then on the model runs offline against your local hardware.
The critical distinction from ChatGPT, Claude, or Gemini is where the computation happens. Cloud services process your input on remote servers. Ollama processes it on the same machine you're typing on. Nothing is transmitted, logged externally, or retained by a vendor.
Who benefits most from local inference
- Lawyers and accountants bound by client confidentiality who can't paste documents into third-party tools.
- Developers working with proprietary code that violates NDAs the moment it hits a cloud API.
- Healthcare and finance teams under HIPAA, GDPR, or similar rules where data residency is non-negotiable.
- Privacy-minded individuals who simply don't want a running transcript of their thoughts stored by a corporation.
- Anyone on unreliable internet who wants an AI assistant that works on a plane or in a basement office.
If you're the sort of person who already thinks carefully about which tools you install, you'll appreciate the same discipline we outline in our guide on how to verify open-source software before adding it to your stack. Ollama earns trust the honest way: its code is public and its behavior is inspectable.
Checking Your Hardware Before You Start
The single biggest predictor of a good experience is available RAM. Models are memory-hungry, and the "parameter count" (the B in 7B, 8B, 70B) roughly tracks how much you'll need.
Here's a realistic before/after scenario. Say you have a 2020-era laptop with 8 GB of RAM and integrated graphics. Running a 7B model quantized to 4-bit uses about 4.5 GB, leaving enough headroom for your browser and editor. Responses come at maybe 8 to 12 tokens per second, which reads like a person typing quickly. Now upgrade that mental picture to a 32 GB machine with a modern discrete GPU: the same model streams at 40 to 60 tokens per second and you can load a 14B model without breaking a sweat.
Minimum and recommended specs
- Bare minimum: 8 GB RAM, any 64-bit CPU from the last decade. You'll run 3B to 7B models.
- Comfortable: 16 GB RAM. Run 7B to 8B models with room to work alongside them.
- Enthusiast: 32 GB RAM plus an 8 GB+ GPU (NVIDIA RTX or Apple Silicon). Run 13B to 14B models fluidly.
- Serious local rig: 64 GB+ and a 24 GB GPU for 70B models, though these still feel slow.
Apple Silicon Macs (M1 through M4) deserve a special mention. Their unified memory architecture lets the GPU tap the full RAM pool, so a 16 GB MacBook Air punches well above its weight for local AI. It's genuinely one of the best value platforms for this today.
How to Install and Run Ollama Locally, Step by Step
This is the core walkthrough. Follow it top to bottom and you'll have a working local AI assistant in under fifteen minutes.
- Download Ollama. Go to ollama.com and grab the installer for your operating system: macOS, Windows, or Linux. On Linux you can also run the one-line install script
curl -fsSL https://ollama.com/install.sh | sh. - Install it. Run the downloaded file. On macOS and Windows it installs a small background service and a menu-bar or system-tray icon. No account creation, no email, nothing.
- Open a terminal. On Windows use PowerShell or Command Prompt; on macOS use Terminal; on Linux your usual shell.
- Verify the install. Type
ollama --version. You should see a version number, confirming the service is running. - Pull and run your first model. Type
ollama run llama3.1. The first time, Ollama downloads roughly 4.7 GB of model weights. Subsequent runs are instant because the weights are cached locally. - Start chatting. Once the download finishes you get a prompt. Type a question, press Enter, and watch the response stream in. Type
/byeto exit the session. - List your models. Run
ollama listanytime to see what you've downloaded and how much disk each model uses. - Remove a model you don't need with
ollama rm modelnameto reclaim disk space.
That's the whole loop. From here you can pull other models the same way, for example ollama run phi3 or ollama run mistral, and switch between them freely.
Adding a friendly interface
The command line is fine for quick questions, but most people want a proper chat window. Ollama exposes a local API at http://localhost:11434, and several free front-ends plug straight into it:
- Open WebUI gives you a ChatGPT-style interface running entirely on your machine.
- Continue and other IDE extensions wire Ollama into VS Code for local code completion.
- Custom scripts can hit the API directly, which is how you'd build local tooling.
If you like the idea of local-first productivity tools, you might also enjoy our breakdown of how to pair AI assistants with spreadsheets to replace productivity apps, which pairs nicely with a private model backend.
Choosing the Right Model for Your Machine
Not all models are equal, and bigger isn't automatically better for your use case. Here's how the popular options stack up on the criteria that actually matter day to day.
| Model | Size (4-bit) | RAM needed | Best for | Speed on CPU |
|---|---|---|---|---|
| Phi-3 Mini (3.8B) | ~2.3 GB | 8 GB | Fast general chat, weak laptops | Fast |
| Llama 3.1 (8B) | ~4.7 GB | 16 GB | Balanced all-rounder | Moderate |
| Mistral (7B) | ~4.1 GB | 16 GB | Concise reasoning, coding | Moderate |
| Gemma 2 (9B) | ~5.4 GB | 16 GB | Strong writing quality | Moderate |
| Llama 3.1 (70B) | ~40 GB | 64 GB + GPU | Complex reasoning | Slow |
My honest recommendation: start with Llama 3.1 8B. It's the sweet spot of quality and hardware demand, and it handles summarizing, drafting, coding help, and Q&A well. If it stutters, drop to Phi-3. If you have serious hardware and need deeper reasoning, step up to a larger model, but expect slower responses.
Quantization in plain English
You'll see tags like q4_0 or q8_0. Quantization compresses the model by storing its numbers with less precision. A 4-bit version is roughly half the size of an 8-bit one and runs faster, with a small, usually unnoticeable drop in quality. For local use, 4-bit is the practical default.
Proving Your Data Stays Private
Running locally should mean nothing leaves your machine, but "should" isn't proof. Here's how to verify it, because trust in security tools comes from checking, not hoping.
- Watch network activity while you prompt. On macOS, open Activity Monitor and select the Network tab. On Windows, use Resource Monitor. Type a long prompt into Ollama and watch: after the initial model download, you should see
Cover image: Software value feedback loop by jakuza, licensed under BY-SA 2.0 via Openverse.
