LLM Inference on Kubernetes: Pay for Tokens, Not Idle GPUs

In 2026 most AI GPU spend goes to inference, not training, so serving a model stopped being a research problem and became a platform and FinOps one. Kubernetes is where that serving lives now, but a language model breaks nearly every assumption baked into the stateless-web tooling you already run. Get the mismatch wrong and you either time out under load or pay for H100s that sit at 20% utilisation.

The fix is not heroics. It is a small stack of inference-aware components that all matured this year, and a shift in what signal you scale on.

Why the web playbook fails

A web pod is stateless, cheap, and answers in milliseconds. An LLM replica is none of those things, and three properties explain why the usual defaults misfire.

• It is GPU-bound. A vLLM pod can sit at 5-8% CPU while its GPU is pinned at 95% and its queue is backing up. HPA on CPU never fires, so it never scales until users are already timing out.
• It holds a growing KV cache. Attention caches the key/value tensors for every token seen so far, in GPU memory. That cache, not CPU or RAM, is the real capacity limit of a replica, and routing, batching and autoscaling all revolve around its pressure.
• Prefill and decode fight for the same GPU. Reading the prompt is one compute-heavy pass (FLOPS-bound); generating each token re-reads the whole KV cache (memory-bandwidth-bound). One pod doing both runs neither phase at its best, which is the whole motivation for disaggregating them.

None of that is visible to a CPU-based HPA or a round-robin load balancer. They are measuring and balancing the wrong things.

Autoscale on the right signal

Press Send traffic. An inference gateway routes each request to the replica most likely to have a warm cache, not just the next one in line. As load ramps, queue depth, not CPU, crosses the threshold, KEDA scales a new GPU replica, and once it survives its cold start the queues drain and utilisation climbs.

The 2026 serving stack

The pieces are settled enough to standardise on, and the rule is to reach for the smallest one that fits before adding distribution.

• vLLM, the default engine. PagedAttention manages the KV cache in non-contiguous pages, and continuous batching swaps finished sequences out and queued ones in at every decode step, dragging utilisation from a naive 20-30% up toward 70-90%.
• KServe or Ray Serve for orchestration. KServe's newer LLMInferenceService CRD hides GPU scheduling, routing and even disaggregated prefill/decode behind one declarative object; Ray Serve shines for multi-node models and deployment graphs.
• An inference gateway, the Gateway API Inference Extension (InferencePool plus an Endpoint Picker, now developed in llm-d). It routes on queue depth, KV-cache utilisation and prefix locality instead of round-robin.
• KEDA for autoscaling on GPU-native signals, and the NVIDIA GPU Operator with DRA and MIG for sharing a card instead of pinning a whole 80GB GPU to a small model.

# Scale vLLM on real inference pressure, not CPU
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama-70b
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-llama-70b          # the Deployment serving the model
  minReplicaCount: 1             # set 0 with the KEDA HTTP add-on for scale-to-zero
  maxReplicaCount: 8
  cooldownPeriod: 300            # long: a cold replica costs minutes to warm
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        # queued requests: vLLM exports this natively
        query: max(vllm:num_requests_waiting{service="vllm-llama-70b"})
        threshold: "5"            # more than 5 waiting -> add a replica
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        # KV-cache fill: fires before memory pressure evicts sequences
        query: max(vllm:kv_cache_usage_perc{service="vllm-llama-70b"})
        threshold: "0.9"

Where the money actually goes

Cost per token is essentially the GPU-hour price divided by the tokens you actually produce that hour, so utilisation is the whole story. Leaving replicas always-on at 20-30% utilisation can make self-hosting more expensive than a managed API even at real volume. Four levers move the number more than buying a bigger node:

• Continuous batching and right-sizing, the single biggest utilisation lever; measure tokens produced per GPU-hour, not GPU count.
• KV-cache-aware routing, because avoiding one cold prefill is worth more than perfectly even request counts. On real workloads it has reported multiple-x throughput and time-to-first-token gains over round-robin.
• MIG for production co-tenancy, so a small model does not pin a whole card, with time-slicing reserved for dev where isolation does not matter.
• Spot GPUs for interruptible batch, kept well away from the latency-critical interactive path that cannot survive a mid-generation reclaim.