Running local AI models with Ollama: what hardware you actually need

What Ollama is, as documented

Ollama is a free, open-source tool for running large language models on your own machine. Its GitHub README describes the goal as: "Start building with open models. Access the latest releases locally or in the cloud with complete privacy." In practice it is three things at once: a command-line tool (ollama run llama3.2 downloads a model and opens a chat), a local HTTP server exposing a REST API on http://localhost:11434 with official Python and JavaScript libraries, and a model library at ollama.com listing the downloadable models. Inference runs on the llama.cpp backend named in the README.

The privacy boundary is explicit in the FAQ: "Ollama runs locally. We don't see your prompts or data when you run locally." Cloud features exist but are optional, and the FAQ documents how to switch them off entirely — set OLLAMA_NO_CLOUD=1 or disable_ollama_cloud in ~/.ollama/server.json — at the cost of losing cloud models and web search. Downloaded models live in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models for the standard Linux service install, and C:\Users\<username>\.ollama\models on Windows; the OLLAMA_MODELS environment variable relocates them.

Official system requirements

Ollama publishes platform requirements — but, notably, no minimum RAM figure:

• Windows: Windows 10 22H2 or newer (Home or Pro); NVIDIA driver 551.61 or newer if you have an NVIDIA card; for AMD, a ROCm v7 / HIP7-capable driver stack or a Vulkan-capable Radeon driver. The install needs at least 4GB of disk space, and the docs warn models themselves run "tens to hundreds of GB".
• macOS: macOS Sonoma (v14) or newer. Apple M-series chips get CPU and GPU support; Intel x86 Macs are CPU-only.
• Linux: a one-line install script (curl -fsSL https://ollama.com/install.sh | sh); AMD GPU users add a separate ROCm package.

That missing RAM number is the crux of this guide. Ollama does not publish a flat "you need X GB" table. What it does publish is per-model download sizes in the library, VRAM-based context defaults, and one concrete sizing note in the quickstart. Put together, those let you derive the hardware answer model by model — which is what the rest of this article does.

Which GPUs Ollama supports

The hardware-support page lists four acceleration paths.

NVIDIA: GPUs with compute capability 5.0 or higher and driver 550 or newer; cards in the compute-capability 5.0–6.2 range need driver 570 or newer. The compatibility table runs from GTX 750 Ti and the GTX 900/1000 series through RTX 30xx, 40xx and 50xx, professional RTX cards, A100, H100/H200, and the GB10 in the DGX Spark.

AMD (ROCm): Linux requires the ROCm v7 driver; the supported list covers Radeon RX 6000 through 9000 series, Radeon PRO W-series, Ryzen AI chips including the Ryzen AI Max+ 395, and Instinct MI100 through MI350X. Windows support is narrower — Radeon RX 7600 up to 7900 XTX and Radeon PRO W7500–W7900. ROCm does not cover every AMD GPU; for unsupported cards the docs describe forcing the closest LLVM target with HSA_OVERRIDE_GFX_VERSION.

Apple: GPU acceleration on Apple devices via the Metal API. Vulkan: additional GPU support on Windows and Linux, enabled by default when the backend is installed — the catch-all for hardware CUDA and ROCm miss.

One Docker caveat from the FAQ: GPU acceleration in the official container works on Linux and Windows/WSL2 with the nvidia-container-toolkit, and is not available for Docker Desktop on macOS due to the lack of GPU passthrough.

CPU vs GPU inference

Every model can run on the CPU alone; a GPU is a speed and capacity upgrade, not a licence to run. The docs describe the loading behaviour precisely:

• ollama ps shows where a loaded model lives. Its Processor column reads 100% GPU, 100% CPU (system memory), or a split such as 48%/52% CPU/GPU when part of the model is offloaded to RAM.
• The quickstart's published sizing example: its recommended local model is a roughly 7.2GB download, and Ollama advises "8 GB of available VRAM, or unified memory on a Mac. Larger context windows need more memory. With less VRAM, Ollama can use system RAM, but responses may be slower."
• With multiple GPUs, Ollama loads a model onto a single GPU if it entirely fits — per the FAQ, "this typically provides the best performance as it reduces the amount of data transferring across the PCI bus" — otherwise it spreads the model across all available GPUs.
• Concurrent model loads have a hard rule: "When using GPU inference new models must be able to completely fit in VRAM to allow concurrent model loads."
• You can force CPU-only operation with an invalid GPU id, e.g. CUDA_VISIBLE_DEVICES=-1 or ROCR_VISIBLE_DEVICES=-1.

What the docs do not publish is speed. There are no tokens-per-second figures anywhere in Ollama's documentation, so any "model X runs at Y tokens/s on card Z" claim comes from third-party benchmarks, not from Ollama, and is not verified here.

Model sizes from the official library

Each library page lists a default tag plus size variants with a published download size and maximum context window; parameter counts come from the tag names and model cards. A representative selection, as listed on 19 October 2026:

* The llama3.1 card lists the family as 8B / 70B / 405B; with 70b and 405b carrying explicit tags, the 4.9GB default corresponds to the 8B model — an inference from the card's listing.

The pattern across the library: at their shipped quantization, models of 3B parameters and up cost roughly 0.6–0.7GB of download per billion parameters (2.0GB for 3B, 4.4GB for 7B, 9.1GB for 14B, 43GB for 70B). That ratio is arithmetic on the published sizes — a DealParse derivation, not an Ollama claim — but it lets you estimate any model outside the table.

How much RAM/VRAM you actually need

Two documented facts anchor the decision. First, the quickstart pattern: a ~7.2GB model file with a recommendation of 8GB VRAM, with system RAM as the slower fallback. Budget VRAM of at least the download size, plus room for context. Second, Ollama's published context-length defaults by VRAM: under 24 GiB VRAM you get a 4K-token context by default; 24–48 GiB gets 32K; 48 GiB and above gets 256K. The FAQ separately notes the plain default context window is 4096 tokens, overridable with OLLAMA_CONTEXT_LENGTH, /set parameter num_ctx, or the API's num_ctx. Raising context costs memory — the docs say so explicitly — and parallel requests multiply it: four parallel requests at a 2K context allocate 8K.

Combining the published file sizes with those tiers gives the guidance below. The middle column is derived by DealParse from the two documented facts above; it is not an official Ollama requirements table, because Ollama does not publish one.

Quantization: why 70B fits in 43GB

Every library model ships quantized — weights compressed to fewer bits — which is what makes 7B–70B models runnable on consumer hardware at all. Ollama's published material shows the trade-offs rather than hiding them:

• The Gemma 3 card offers quantization-aware-trained (QAT) variants that, it says, "preserve similar quality as half precision models (BF16) while maintaining a lower memory footprint (3x less compared to non-quantized models)".
• The context (K/V) cache can be quantized separately via OLLAMA_KV_CACHE_TYPE: f16 (default), q8_0 (about half the memory, "very small loss in precision" that "usually has no noticeable impact"), and q4_0 (about a quarter the memory, with a "small-medium loss" that "may be more noticeable at higher context sizes").
• Flash Attention is enabled automatically when the backend and devices support it and, per the FAQ, "can significantly reduce memory usage as the context size grows"; OLLAMA_FLASH_ATTENTION=1/0 forces it on or off.

The honest reading: quantization is the reason this works on consumer hardware, and the docs acknowledge the quality cost grows the harder you compress. Ollama's own guidance is to experiment: "you may need to experiment with different quantization types to find the best balance between memory usage and quality."

Realistic limits and expectations

What the documentation supports, stated plainly:

• Default context is small. Under 24 GiB VRAM you default to about 4K tokens, while the context docs recommend at least 64,000 tokens for agentic workloads — coding tools, web search, agents — a setting that "will increase the amount of memory required to run a model". Big models and big contexts do not fit on most consumer GPUs at once.
• Models unload. By default a loaded model stays in memory for five minutes before being unloaded; the first request after idle pays the reload cost. keep_alive / OLLAMA_KEEP_ALIVE change that.
• Concurrency eats memory. OLLAMA_NUM_PARALLEL defaults to 1, and required RAM scales by parallel requests × context length. The server queues up to 512 requests (then returns 503).
• One documented Windows quirk: Radeon GPUs currently default to a single loaded model because of ROCm v5.7 VRAM-reporting limits.
• Disk adds up. "Tens to hundreds of GB" per the platform docs — the deepseek-r1:671b tag alone is a 404GB download.
• No published speed numbers. Throughput claims for specific CPUs or GPUs come from third-party benchmarks, not Ollama's docs. Not verified

Local LLMs with Ollama are less a hardware question than a model-selection question. The documentation gives you every number needed to answer it: pick a model tag, read its published download size, compare it against your VRAM (or system RAM for CPU-only), and remember you default to a 4K context below 24 GiB. Everything else — context length, K/V-cache quantization, keep-alive — is tuning, and the docs cover each knob.