ModelRefs / Model Deployment with FastAPI — Tutorial

Model Deployment with FastAPI — Tutorial

Wrap any ML model in a production-ready REST API — health checks, async inference, and versioning. Covers Why FastAPI for ML serving.

Overview

Wrap any ML model in a production-ready REST API — health checks, async inference, and versioning

Level: Advanced. Estimated reading time: 40 minutes.

Why FastAPI for ML serving

Flask is synchronous and single-threaded — fine for experiments, wrong for ML serving. Under concurrent load, a blocking Flask route stalls all other requests while the GPU runs inference.

FastAPI is built on Starlette (async) and Pydantic (schema validation). Key advantages for ML:

Async-first: define endpoints with async def to handle many concurrent requests without blocking. CPU-bound inference should be offloaded to a thread pool with run_in_executor.

Auto-generated OpenAPI docs at /docs — engineers can test your model directly from the browser.

Pydantic input/output schemas: define the request and response shape as Python dataclasses. FastAPI validates inputs automatically and returns 422 with human-readable errors on bad input.

Type-checked at the boundary: ML models have specific input shapes. Pydantic catches "text is missing" or "batch is too large" before the inference call.

Model loading, lifespan, and thread pools

Never load the model inside the endpoint handler. On each request that would re-load weights from disk — 10–30 seconds for a large model.

The correct pattern: load the model once at startup using FastAPI's lifespan context manager (FastAPI 0.93+). The model is stored in app.state and shared across all requests.

CPU-bound inference blocks the event loop. Use asyncio's run_in_executor to offload to a ThreadPoolExecutor:

result = await loop.run_in_executor(executor, model.predict, input_data)

For GPU inference: one GPU forward pass at a time is usually fine since the GPU itself is the bottleneck. Use a semaphore (asyncio.Semaphore(1)) to serialise GPU calls and prevent OOM from concurrent requests.

Batching: collect multiple requests over a short window (5–20ms) and run them as a single batched inference call. This is called dynamic batching and dramatically increases GPU utilisation. Libraries like NVIDIA Triton handle this automatically.

Health checks, versioning, and error handling

Health checks are required by Kubernetes, load balancers, and CI/CD pipelines. Three endpoints:

GET /health — liveness probe. Returns 200 if the process is alive. Should never fail (even if the model isn't loaded yet).

GET /ready — readiness probe. Returns 200 only if the model is loaded and ready to serve. Returns 503 during startup or after a model reload.

GET /metrics — (optional) Prometheus-format metrics: request count, latency percentiles, error rate, GPU memory usage.

API versioning: prefix routes with /v1/ so you can release /v2/ without breaking existing clients. Use a router: app.include_router(v1_router, prefix="/v1").

Error handling: catch model-specific exceptions (shape mismatch, NaN in output, timeout) and return structured JSON errors: {"error": "inference_timeout", "message": "...", "request_id": "..."}. Never return a 500 with a Python traceback — it leaks implementation details.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Deployment with FastAPI — Tutorial.