Disaster Recovery

Emergency: disable all Autopilot operations

These procedures stop Autopilot from making any further changes. CRDs and their resources are preserved — the controller simply stops reconciling them.

Option 1 — Scale to zero (fastest):

kubectl -n ops-ai-system scale deploy/ops-ai-controller --replicas=0
kubectl -n ops-ai-system scale deploy/ops-ai-autoscaler --replicas=0

Option 2 — Enable global dry-run:

helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.dryRun=true

helm upgrade ops-ai-autoscaler deploy/helm/ops-ai-autoscaler \
  --namespace ops-ai-system --reuse-values --set autoscaler.dryRun=true

Option 3 — Delete the Helm releases entirely:

helm uninstall ops-ai-controller -n ops-ai-system
helm uninstall ops-ai-autoscaler -n ops-ai-system

CRDs and their resources are preserved after a Helm uninstall.

Manual override: force node count

If the autoscaler has scaled to an unwanted number of nodes:

Disable the autoscaler first (see above).

Manually resize the node pool via gcloud:

gcloud container clusters resize CLUSTER_NAME \
  --node-pool POOL_NAME \
  --num-nodes NUM \
  --zone ZONE

Verify nodes are in the desired state.

Re-enable the autoscaler with corrected thresholds.

Get Started

Platform

Operations

Migrating from CAST AI

Emergency: disable all Autopilot operations

Manual override: force node count

Manual override: stop rebalancing

​Emergency: disable all Autopilot operations

​Manual override: force node count

​Manual override: stop rebalancing

Emergency: disable all Autopilot operations

Manual override: force node count

Manual override: stop rebalancing