ModelRefs / Ray Serve — AI Glossary
Ray Serve — AI Glossary
A scalable model serving library built on Ray that handles request batching, model multiplexing, and multi-node deployment.
Overview
Ray Serve (Anyscale) provides composable deployment graphs for LLMs: HTTP endpoints, automatic batching (continuous batching for LLMs), fractional GPU allocation, and multi-model routing (A/B tests, canary). Used by Anyscale endpoints and as the serving backend for vLLM production deployments.
Reference details
| Topic | infrastructure |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Ray Serve — AI Glossary.
Frequently asked questions
What is Ray Serve?
A scalable model serving library built on Ray that handles request batching, model multiplexing, and multi-node deployment.
What concepts are related to Ray Serve?
Closely related concepts include vllm, bentoml, model routing.