ModelRefs / Ollama — AI Glossary
Ollama — AI Glossary
An open-source tool for downloading, managing, and serving open-weights LLMs locally on macOS, Linux, and Windows.
Overview
Ollama wraps llama.cpp and provides a Docker-like CLI (`ollama pull`, `ollama run`) and an OpenAI-compatible REST API. It is the fastest path to local LLM inference for developers.
Reference details
| Topic | infrastructure |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: Check the configured context, not the model's maximum
A model card advertising a very long context does not mean your local server is using it. The runtime has its own context setting, and if it is lower than the model's maximum, prompts are quietly truncated to fit — no error, no warning, just a model that appears to forget the start of a long document. This is the most common local-inference confusion, and it looks like a quality problem rather than a configuration one. Before concluding a local model is weak at long inputs, print the context length the server actually loaded it with.
Commonly confused with
Ollama is a local runtime and model manager, not a model and not a serving stack for production. It wraps a CPU/GPU inference engine and exposes a convenient CLI and an OpenAI-shaped API. For concurrent multi-user serving, engines built for batching and paged attention are the appropriate comparison; Ollama optimises for a single developer's machine.
When to use it
Reach for it when:
- Local development and prototyping against open-weights models without an API key
- Data that must not leave the machine, where any hosted call is disqualifying
- Quickly comparing quantizations of a model on your own hardware
Reach for something else when:
- Production multi-user serving, where a batching inference server is the right tool
- Benchmarking throughput and citing it as the model's performance
- Assuming default context, quantization and sampling settings match the hosted version of the same model
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Ollama — AI Glossary.
Frequently asked questions
What is Ollama?
An open-source tool for downloading, managing, and serving open-weights LLMs locally on macOS, Linux, and Windows.
What concepts are related to Ollama?
Closely related concepts include gguf, open weights, inference endpoint.