ModelRefs / Inference Cost — AI Glossary

Inference Cost — AI Glossary

The marginal cost of generating output, usually expressed per million input or output tokens. Token price is not cost per task.

Overview

Inference cost = input price × input tokens + output price × output tokens. Output tokens are typically 3–5× more expensive than input tokens because they are generated sequentially (each token requires a full forward pass over the KV cache).

Reference details

Topic	inference
Last reviewed	2026-06-24

Example: Where the bill actually comes from

A summarisation call sending 8,000 tokens and returning 500, at $3 per million input and $15 per million output: input costs 8000 × 3 / 1e6 = $0.024, output costs 500 × 15 / 1e6 = $0.0075. Input dominates here. Flip it — a short prompt generating 4,000 tokens — and output dominates. Which side to optimise depends entirely on the shape of your traffic.

input : 8,000 x $3  / 1e6 = $0.0240
output:   500 x $15 / 1e6 = $0.0075
total                      = $0.0315 per call

Commonly confused with

Token price is not cost per task. A cheaper model that needs three attempts, longer prompts or a retry on malformed output can cost more per completed task than a pricier one that succeeds first time. Compare cost per successful outcome, not cost per million tokens.

When to use it

Reach for it when:

Sizing unit economics before committing to an architecture
Deciding between routing tiers, caching or a smaller model
Reasoning models are in play, where hidden reasoning tokens are billed

Reach for something else when:

As the only axis — latency and quality usually matter more to the product
Estimated from list price alone, ignoring retries, failures and caching

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Inference Cost — AI Glossary.

Frequently asked questions

What is Inference Cost?

The marginal cost of generating output, usually expressed per million input or output tokens.

What concepts are related to Inference Cost?

Closely related concepts include token, latency, throughput.