Ollama: run open models on your own machine — and when that's the right call

Ollama is the simplest way to run open-weight language models — Llama, Mistral, Gemma, Qwen and friends — on your own hardware. Pull a model, run it, and you've got a local LLM with an OpenAI-compatible API, no account and no data leaving your machine.

What it is

A single lightweight tool with a model registry behind it. You pull a model by name, and Ollama handles the weights, quantisation, and GPU/CPU execution. It exposes a local HTTP API that speaks the OpenAI format, so most SDKs and frameworks point at it with a one-line change.

ollama run llama3          # pull + chat in your terminal

# ...or hit it from code, OpenAI-compatible:
curl http://localhost:11434/v1/chat/completions \
  -d '{ "model": "llama3", "messages": [{"role":"user","content":"hi"}] }'

When local is the right call

• Data privacy & compliance — sensitive data that legally or contractually can't leave your environment.
• Cost at volume — high-throughput, non-frontier tasks where per-token API pricing adds up fast.
• Offline or edge — on-prem, air-gapped, or on-device scenarios with no reliable internet.
• Prototyping — iterate freely without metering every experiment against an API bill.

When it isn't

• You need frontier-level quality — the best open models are excellent, but for the hardest tasks the top proprietary models still lead.
• You need high concurrency — serving many simultaneous users well means real GPU infrastructure, at which point managed inference may be cheaper and simpler.
• You don't have the hardware — a laptop runs small models fine; large ones need serious memory and a capable GPU.

Ollama makes local models a one-liner. The judgement call is whether local is the right place for that particular workload.