ModelRefs / Kubernetes for ML — Tutorial
Kubernetes for ML — Tutorial
Deployments, HPA, GPU node pools, and rolling updates for production ML inference at scale. Covers Why Kubernetes for ML inference.
Overview
Deployments, HPA, GPU node pools, and rolling updates for production ML inference at scale
Level: Expert. Estimated reading time: 45 minutes.
Why Kubernetes for ML inference
Docker Compose is fine for one machine. When you need more than one server — for availability, scale, or cost efficiency — you need an orchestrator.
Kubernetes (K8s) manages a cluster of machines as a single resource pool. Key capabilities for ML:
Horizontal Pod Autoscaling (HPA): automatically add or remove inference pod replicas based on CPU, GPU, or custom metrics (latency, queue depth). Scale to zero when idle, scale to 100 under load.
GPU scheduling: K8s schedules pods that request GPU resources onto GPU nodes. Multiple pods can share a node's non-GPU resources; only GPU pods land on GPU nodes.
Rolling updates: deploy a new model version with zero downtime — K8s gradually replaces old pods with new ones and rolls back automatically if health checks fail.
Self-healing: if an inference pod crashes, K8s restarts it automatically (up to the backoff limit).
Managed offerings: GKE (Google), EKS (AWS), AKS (Azure) handle cluster management. For pure inference, hosted platforms like Vertex AI, SageMaker, and Azure ML manage K8s complexity for you.
Core Kubernetes objects for ML
Deployment: declares the desired state — N replicas of your inference container, resource requests/limits, health checks. K8s continuously reconciles actual state with desired state.
Service: exposes your Deployment as a network endpoint. ClusterIP (internal only), NodePort (external via node IP), LoadBalancer (cloud LB). For production, use LoadBalancer or pair with an Ingress controller.
ConfigMap and Secret: separate configuration from the image. Model path, API keys, batch size — inject as environment variables or mounted files.
HorizontalPodAutoscaler: scale replicas based on observed CPU (or custom metrics from Prometheus). Scale up when average CPU > 70%, scale down when < 30%.
ResourceQuota: limits CPU/GPU/memory used by a namespace — prevents one team from consuming the entire cluster.
PodDisruptionBudget: ensures at least N replicas stay available during voluntary disruptions (node drains, cluster upgrades).
GPU scheduling and resource requests
Request GPUs in the pod spec:
resources: requests: {nvidia.com/gpu: "1"} limits: {nvidia.com/gpu: "1"}
K8s will only schedule this pod on a node with an available NVIDIA GPU. The cluster needs the NVIDIA device plugin DaemonSet to expose nvidia.com/gpu as a schedulable resource.
Multi-instance GPU (MIG): NVIDIA A100/H100 support MIG — splitting one physical GPU into multiple isolated instances (e.g. 7× 1g.10gb slices). Each slice appears as a separate nvidia.com/gpu resource to K8s, letting 7 pods share one A100 safely.
Node selectors / node affinity: label GPU nodes (gpu=true, gpu-type=a100) and use nodeSelector or affinity rules to ensure GPU pods land on GPU nodes and CPU pods don't waste GPU resources.
Vertical Pod Autoscaler (VPA): recommends or auto-applies CPU/memory request changes based on observed usage — useful for right-sizing inference pods.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Kubernetes for ML — Tutorial.