How to Safely Run a Local AI Model with Ollama

The pitch for local AI is intoxicating: a genuinely capable language model running on your own machine, no API keys, no per-token billing, no data leaving your desk. And thanks to Ollama, that pitch has stopped being a weekend hacker fantasy. In about ten minutes you can pull down a 7-billion-parameter model and start chatting with it offline. I did exactly this on a mid-range laptop and had a working assistant before my coffee went cold.

But here is the part nobody puts in the launch tweet: a local model is still software that opens network ports, downloads binaries, and executes code with your user privileges. In late 2024, security researchers disclosed CVE-2024-37032 (nicknamed "Probllama"), a path-traversal flaw in Ollama that allowed remote code execution on exposed instances. Thousands of servers were sitting on the public internet with port 11434 wide open. Running AI locally is safer than shipping your data to a third party, but "local" and "safe" are not synonyms.

This guide walks through how to run a local AI model with Ollama the right way: installing it cleanly, choosing a model that fits your hardware, locking down the network surface, and verifying that nothing is quietly phoning home. By the end you will have a private assistant you actually trust.

Key Takeaways

Ollama runs models locally but exposes an HTTP API on 127.0.0.1:11434 by default. Keep it bound to localhost unless you deliberately need remote access.

Match the model to your RAM and VRAM. A rough rule: a 7B model in 4-bit quantization needs about 5–6 GB of memory; a 13B needs 9–10 GB.

Never expose port 11434 to the internet without authentication in front of it. Probllama (CVE-2024-37032) proved why.

Verify model provenance. Pull from the official library or a known publisher, and check digests. A model file is executable data.

Monitor outbound traffic the first time you run any new tool. Ollama should only talk to its registry during pulls, nothing else.

Budget for updates. Local software you install once and forget becomes tomorrow's unpatched CVE.

What Ollama Actually Is (and What It Is Not)

Ollama is a lightweight runtime that packages open-weight language models, their configuration, and a serving layer into a single command-line tool. Think of it as Docker for LLMs: you pull a model, you run it, and it manages the rest. Under the hood it uses llama.cpp to run inference efficiently on CPUs and GPUs alike.

What Ollama is not: it is not a cloud service, it is not a chatbot product, and it is not inherently sandboxed. When you install it, it runs as a background service on your machine with the ability to download files, listen on a port, and load model weights into memory. That is exactly what makes it useful, and exactly why you should install it with the same care you would give any other privileged desktop utility.

If you are the kind of person who audits what you install before you trust it, you will feel at home. That same instinct is worth applying to everything on your system, which is why we have written before about how to vet vibe-coded apps before you buy or ship them.

Choosing a Model That Fits Your Hardware

The single most common mistake beginners make is pulling a model that is too big for their machine, watching it swap to disk, and concluding that "local AI is slow." It is not slow. You just fed a 33B model to a laptop with 16 GB of RAM.

Models come in parameter sizes (7B, 13B, 70B) and quantization levels (roughly, how many bits per weight). Lower quantization means smaller files and lower memory use, at a modest quality cost. For most desktops, a 4-bit quantized 7B or 8B model is the sweet spot.

A worked example: sizing a model to a real laptop

Say you have a laptop with 16 GB of RAM and no dedicated GPU, which describes a huge slice of business machines. Here is how the math plays out:

Llama 3.1 8B (Q4): about 4.9 GB on disk, roughly 6 GB in memory during use. Leaves plenty of headroom. Runs at a comfortable reading pace on CPU.
Mistral 7B (Q4): about 4.1 GB on disk. Fast, punchy, great for summarization and drafting.
Llama 3.1 70B (Q4): about 40 GB. This will not run on your machine. Do not even pull it.

On this laptop I would run the 8B for general work and keep Mistral 7B around for quick tasks. If you later add a GPU with 12 GB of VRAM, a 13B model becomes comfortable and noticeably sharper.

Ollama vs. the Alternatives

Ollama is not the only way to run models locally. Here is an honest comparison of the leading options across the criteria that actually matter for a desktop user.

Tool	Ease of setup	GUI included	API server	Best for	Main tradeoff
Ollama	Very high (one command)	No (CLI-first)	Yes, OpenAI-compatible	Developers, scripting, integration	Terminal comfort required
LM Studio	High	Yes, polished	Yes	Non-technical users, browsing models	Closed source, heavier app
GPT4All	High	Yes	Limited	Simple offline chat	Smaller model catalog
llama.cpp (raw)	Low	No	Yes (manual)	Maximum control, tinkering	Steep learning curve

My take after using all four: Ollama wins for anyone who will eventually wire the model into scripts, editors, or apps, precisely because of its clean, OpenAI-compatible API. If you never want to touch a terminal, LM Studio is the friendlier front door. For a broader look at evaluating tools like these, our AI Tools category and the guide on measuring real AI coding assistant ROI beyond speed are both worth a read.

How to Install and Run a Local AI Model with Ollama, Step by Step

Here is the full walkthrough. I will assume Windows or macOS, though Linux is nearly identical.

Download from the official source. Go to ollama.com/download and grab the installer for your OS. Do not download Ollama from a mirror, a torrent, or a "cracked" bundle. The whole point of local AI is trust, and it starts here.
Verify the installer before running it. On macOS the app is notarized by Apple; on Windows check that the publisher shows in the signature dialog. This 20-second check is the same discipline we recommend when you audit software vendor ownership before you buy.
Install and confirm the service is running. After installation, open a terminal and run ollama --version. You should see a version number. Ollama also starts a background service that listens on 127.0.0.1:11434.
Pull your first model. Run ollama pull llama3.1:8b. You will see a progress bar as it downloads several gigabytes. When it finishes, Ollama prints the model's digest, a SHA-256 hash you can use to confirm you got the exact file the registry published.
Run it. Type ollama run llama3.1:8b. After a few seconds you get an interactive prompt. Ask it something. On a 16 GB CPU-only laptop, expect the first token in 2–4 seconds and a steady stream after.
Test the API. In another terminal, run curl http://localhost:11434/api/tags. This lists your installed models and confirms the local API is reachable only from your machine.
Exit cleanly. Type /bye in the chat to end the session. The model stays cached, so the next run is instant.

That is the entire happy path. The difference between a hobbyist and a professional is what happens next: the security hardening most tutorials skip.

Locking Down Ollama: The Security Steps Most Guides Skip

By default Ollama binds to localhost, which is good. Trouble starts when people follow a blog post that says "set OLLAMA_HOST=0.0.0.0 so you can access it from your other devices" and never think about it again. That single change exposes an unauthenticated code-execution surface to your entire network, and if your firewall is loose, to the internet.

1. Keep the bind address on localhost

Unless you have a specific, deliberate reason, leave OLLAMA_HOST at its default of 127.0.0.1. Verify with netstat -an | grep 11434 (or Get-NetTCPConnection on Windows). You want to see it bound to 127.0.0.1, not 0.0.0.0.

2. If you must go remote, put a gateway in front

If you genuinely need to reach your model from a phone or another laptop, do not expose Ollama directly. Instead:

Bind Ollama to localhost still.
Run a reverse proxy (Caddy, nginx) that adds authentication and TLS.
Restrict access by IP or, better, put it behind a VPN or a tunnel like Tailscale.

This mirrors the same logic behind hardening any web-facing service. If you run WordPress or e-commerce alongside your AI experiments, tools like SiteGuard Pro and the WordPress IP Blocker Pro exist precisely because an open port is an open door.