ModelRefs / Rate Limiting — AI Glossary
Rate Limiting — AI Glossary
Restricting the number of API requests or tokens a client can consume per unit time to protect infrastructure and control costs. They need separate handling.
Overview
LLM API providers enforce rate limits in RPM (requests per minute) and TPM (tokens per minute). Exceeded limits return 429 errors. Production systems handle this with exponential backoff, request queuing, and multiple API keys.
Reference details
| Topic | infrastructure |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: Work out which limit actually binds
Say an account is capped at 500 requests per minute and 200,000 tokens per minute. If an average request carries 2,000 tokens of prompt and response, the token cap allows 200,000 / 2,000 = 100 requests per minute — one fifth of the request cap, which you will therefore never reach. Teams size concurrency against the RPM figure, see 429s at a fifth of it, and conclude the limit is wrong. Compute both, take the smaller, and remember that the binding constraint moves: shortening prompts raises your effective request ceiling without any change to the quota.
Commonly confused with
Provider rate limiting protects the provider; your own rate limiting protects you — from runaway spend, from a retry storm you generated, and from one tenant consuming another's capacity. They need separate handling. A 429 is also not an error in the usual sense: it is a scheduling signal, and the correct response is to wait and retry, not to fail the request.
When to use it
Reach for it when:
- Any production integration — assume limits and design the queue before you hit them
- Multi-tenant systems, where per-tenant budgets prevent one caller starving the rest
- Alongside backoff with jitter, so retries do not synchronise into a second spike
Reach for something else when:
- Treating 429 as a failure to surface to users instead of a wait-and-retry
- Sizing against the request cap alone when the token cap is the one that binds
- Spreading load across multiple keys to evade limits, which usually violates the terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Rate Limiting — AI Glossary.
Frequently asked questions
What is Rate Limiting?
Restricting the number of API requests or tokens a client can consume per unit time to protect infrastructure and control costs.
What concepts are related to Rate Limiting?
Closely related concepts include model gateway, inference cost.