# Making GPU Failure Invisible: A Zero-Cost Fallback for LLM Inference on Kubernetes

> I will be giving a formal talk on this implementation soon. More to come in a highly deep dive technical blog followed with the talk

Most GPU-aware routing demos assume you have GPUs. I wanted to know what happens when you don't, on purpose, and whether a cluster could degrade from "canary 90% GPU, 10% CPU" down to "100% CPU" without a single line of failover code.

It can. The trick is refusing to write any failover code at all, and letting three things Kubernetes already does for free do the work instead.

## The Setup

The routing layer is the Kubernetes Gateway API Inference Extension running behind NGINX Gateway Fabric: a `Gateway`, an `HTTPRoute` with a weighted canary split, and two `InferencePool`s, one GPU-backed, one CPU-backed, each fronting an Ollama deployment serving Llama 3.

![](https://cdn.hashnode.com/uploads/covers/69532a0d5aa6a75b65a6241f/d3aad417-bef1-470c-be38-3dde3737a6a9.png align="center")

Nothing unusual yet. The interesting part is what happens when the GPU side has nowhere to run.

## Layer one: scheduling does the failing for you

The GPU deployment requests `nvidia.com/gpu: 1`. On a cluster with no GPU nodes and no device plugin, that pod never leaves `Pending`. No crash loop, no error to catch, nothing to alert on. It just never becomes Ready, which means the GPU `InferencePool` has zero healthy endpoints. The Gateway routes 90% of traffic at a pool with nothing behind it, and every one of those requests would 502.

That's the gap layer two closes.

## Layer two: a CronJob that reads reality and writes weights

A small CronJob polls allocatable `nvidia.com/gpu` across all nodes every two minutes and patches the `HTTPRoute` accordingly:

```shell
GPU_CAPACITY=$(kubectl get nodes -o json \
  | jq '[.items[].status.allocatable["nvidia.com/gpu"] // "0" | tonumber] | add')
 
if [ "${GPU_CAPACITY}" -gt 0 ]; then
  WEIGHTS='{"gpu": 90, "cpu": 10}'
else
  WEIGHTS='{"gpu": 0, "cpu": 100}'
fi
# ...JSON patch the HTTPRoute backendRefs with $WEIGHTS
```

No GPU capacity, no traffic sent to it. The RBAC for this is deliberately narrow: read-only on nodes, patch on exactly one `HTTPRoute`. If a GPU node group shows up later (say, a Karpenter `NodePool` reacting to those `Pending` pods), the same CronJob notices on its next run and shifts weight back without anyone touching a manifest.

Two primitives, scheduling and a five-line reconciliation loop, and the failure mode disappears entirely.

## Layer three: where it actually got hard

Building the demo took an afternoon. Making it survive contact with a real, deliberately locked-down VPC took considerably longer, and that's where most of the actual engineering happened.

The target EKS cluster runs an IPv6-only Service CIDR with no NAT gateway for IPv4 egress by default, just an egress-only path. That single constraint cascades:

1.  **Image pulls** needed rethinking, since neither [`ghcr.io`](http://ghcr.io) nor Docker Hub are reachable from a node with no IPv4 route out and no AAAA record on the registry. Private ECR, reached over existing VPC endpoints, became the only viable pull path.
    
2.  **The Gateway's own load balancer** turned out to be IPv6-only by inheritance from the cluster's Service CIDR, which AWS's load balancer controller doesn't handle with its defaults; an NLB provisioned the normal way simply refuses to attach. Getting a healthy, publicly reachable target group meant working through target-registration mode, address-type, and scheme, in that order, each one surfacing a different opaque error only after the previous one was fixed.
    
3.  **A few of AWS's own background health checks** (unrelated to the actual routing path) turned out to be silently blocking every reconcile for minutes at a time in an environment where their upstream APIs weren't reachable. No error, no timeout message, just nothing happening.
    

None of that shows up in a diagram. All of it is the difference between "works in a demo cluster" and "works."

## **What this actually demonstrates**

The interesting engineering here isn't the YAML, it's the constraint-driven debugging such as reading past a generic AWS error string to the actual API call failing underneath it, working out which of several plausible IPv6/dual-stack interactions was the real blocker, and doing it without shell access being the bottleneck, since most of this had to be diagnosed from controller logs and AWS console state alone.

I'm holding the full repo and the complete list of fixes for a talk I'm giving on this soon, diagrams, root causes, and all. If distributed inference routing or debugging Kubernetes on non-standard networking is something you deal with, I'd love to compare notes in the meantime.