Troubleshooting & Rollback
Error Handling
| Error / Symptom | Likely Cause | Remediation |
|---|---|---|
| No kubeconfig context | Not connected to a cluster | Run az aks get-credentials or equivalent |
make: *** No rule to make target | Not in AI Runway repo root | cd to repo root and retry |
| Controller in CrashLoopBackOff | Config or RBAC issue | kubectl logs -n airunway-system -l control-plane=controller-manager --previous |
| Provider not ready | Image pull or RBAC issue | kubectl describe pod for the provider pod |
| ModelDeployment stuck in Pending | GPU scheduling failure or provider not ready | kubectl describe modeldeployment events |
| Pod shows ImagePullBackOff | Wrong image reference or missing pull secret | kubectl describe pod for the model pod |
| 401 from HuggingFace at model load | Gated model, token secret not wired into CR | Ensure huggingFaceTokenSecretRef is set in the CR |
bfloat16 errors at inference | T4 or V100 lacks bfloat16 support | Add --dtype float16 to serving args |
Rollback
If a step fails and you need to undo a partial setup, work backwards through the steps:
| What to undo | Command |
|---|