Operations Runbook

This runbook covers deployment, monitoring, and day-to-day operation of the Autopilot platform. It applies to both the controller (CRD-driven, runs as a Kubernetes controller-manager) and the autoscaler (standalone GKE-focused autoscaling loop).

For scale-down behavior see Node Scale-Down; for problem diagnosis see Troubleshooting; for emergency procedures see Disaster Recovery.

Deployment

Controller (Helm)

helm install ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system \
  --create-namespace \
  --set prometheus.url=http://prometheus.monitoring:9090 \
  --set controller.leaderElect=true

Common value overrides:

# GKE integration
--set gke.project=my-project \
--set gke.location=us-central1 \
--set gke.cluster=production

# Start in dry-run mode (safe for initial rollout)
--set controller.dryRun=true

# Tune intervals
--set controller.rebalanceInterval=10m \
--set controller.scaleCooldown=15m

# Custom image
--set image.repository=ghcr.io/myorg/ops-ai-controller \
--set image.tag=v0.2.0

# Resource limits
--set resources.requests.cpu=200m \
--set resources.requests.memory=512Mi \
--set resources.limits.cpu=1 \
--set resources.limits.memory=1Gi

Autoscaler (Helm)

helm install ops-ai-autoscaler deploy/helm/ops-ai-autoscaler \
  --namespace ops-ai-system \
  --create-namespace \
  --set gke.project=my-project \
  --set gke.location=us-central1 \
  --set gke.cluster=production

Common value overrides:

--set autoscaler.scaleUpThreshold=0.8 \
--set autoscaler.scaleDownThreshold=0.7 \
--set autoscaler.minNodes=3 \
--set autoscaler.maxNodes=50 \
--set autoscaler.scaleUpCooldown=3m \
--set autoscaler.scaleDownCooldown=10m \
--set autoscaler.preferSpot=true \
--set autoscaler.dryRun=true \
--set autoscaler.scanInterval=60s

CRD installation

CRDs are installed automatically when crds.install=true (default) in the controller chart. To install manually:

kubectl apply -f deploy/helm/ops-ai-controller/crds/
kubectl get crd | grep ops-ai.io

Expected output:

nodescalingpolicies.ops-ai.io    ...
podresizepolicies.ops-ai.io      ...
rebalancepolicies.ops-ai.io      ...

GKE Workload Identity

If using Workload Identity instead of node-level service account keys:

--set gke.workloadIdentityServiceAccount=ops-ai@my-project.iam.gserviceaccount.com
--set serviceAccount.annotations."iam\.gke\.io/gcp-service-account"=ops-ai@my-project.iam.gserviceaccount.com

The GCP service account needs these roles:

roles/container.clusterAdmin (for node pool management)
roles/compute.instanceAdmin.v1 (for node operations)

Health checks

Controller

The controller exposes health endpoints on port 8081 (configurable via controller.probeAddr):

kubectl -n ops-ai-system port-forward deploy/ops-ai-controller 8081:8081
curl http://localhost:8081/healthz   # Liveness — is the process alive?
curl http://localhost:8081/readyz    # Readiness — ready to serve?

Autoscaler

The autoscaler exposes health and metrics on port 8080:

kubectl -n ops-ai-system port-forward deploy/ops-ai-autoscaler 8080:8080
curl http://localhost:8080/healthz
curl http://localhost:8080/readyz

Kubernetes probes

Both Helm charts configure probes automatically:

Liveness: GET /healthz, initial delay 15s, period 20s
Readiness: GET /readyz, initial delay 5s, period 10s

Metrics and monitoring

Prometheus metrics

The controller and autoscaler expose Prometheus metrics on port 8080:

kubectl -n ops-ai-system port-forward svc/ops-ai-controller 8080:8080
curl http://localhost:8080/metrics

ServiceMonitor setup

serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s
  labels:
    release: prometheus   # match your Prometheus operator selector

Key metrics to watch

Set up alerts for these conditions:

Controller pod not ready for more than 5 minutes
Leader election lost (check logs for “leader election lost”)
High error rate in reconciliation loops
Scaling operations consistently failing
Rebalance plans generated but never executed (stuck in dry-run when not intended)

Common operational tasks

Enable/disable dry-run mode

Dry-run prevents the controller and autoscaler from making any changes. All analysis and planning still runs; only execution is skipped.

helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.dryRun=true

helm upgrade ops-ai-autoscaler deploy/helm/ops-ai-autoscaler \
  --namespace ops-ai-system --reuse-values --set autoscaler.dryRun=true

Individual CRD resources also have a spec.dryRun field. Setting it true on a specific RebalancePolicy disables execution for that policy only, even if the global dry-run is false.

Adjust rebalance thresholds

Via Helm (global):

helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.rebalanceInterval=10m

Via CRD (per-policy):

apiVersion: ops-ai.io/v1alpha1
kind: RebalancePolicy
metadata:
  name: aggressive
  namespace: production
spec:
  schedule: "*/2 * * * *"
  thresholds:
    imbalanceScore: 0.15
    minUtilization: 0.3
    maxUtilization: 0.75
  maxMovements: 20
  maxParallel: 5
  respectPDB: true
  dryRun: false

Adjust scale cooldown

helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.scaleCooldown=15m

helm upgrade ops-ai-autoscaler deploy/helm/ops-ai-autoscaler \
  --namespace ops-ai-system --reuse-values \
  --set autoscaler.scaleUpCooldown=5m --set autoscaler.scaleDownCooldown=15m

Change log level at runtime

helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.logLevel=debug

Valid levels: debug, info, warn, error.

Exclude namespaces from rebalancing

spec:
  excludeNamespaces:
    - kube-system
    - monitoring
    - istio-system
    - ops-ai-system

Controller pool node management

Infrastructure pinning

Critical monitoring infrastructure is pinned to the Autopilot controller pool node so it survives managed node scale-downs and rebalances. GKE-managed kube-system components (kube-dns, metrics-server, konnectivity) cannot be pinned — GKE’s control plane reconciler resets their Deployment specs within seconds. These are protected by PDBs instead. Re-apply after GKE upgrades (GKE may delete PDBs or reset monitoring Deployments):

scripts/pin-infra-to-pool.sh --context=xano-stage
# Or all clusters at once:
scripts/pin-infra-to-pool.sh --context=xano-dev,xano-dev-d2,xano-stage,xano-stage-s2

Pinned to the pool node:

Component	Method	Why
Prometheus	Helm values	Metrics — resizer/scaler data source
Alertmanager	Helm values	Alert delivery
kube-state-metrics	pin-infra-to-pool.sh	KSM metrics for Prometheus
Prometheus Operator	pin-infra-to-pool.sh	Manages Prometheus config
Autopilot controller	Helm chart	Controller must survive drains

Protected by PDB (GKE-managed, can’t pin):

Component	PDB	Protection
kube-dns	ops-ai-kube-dns	minAvailable: 1 — at least 1 DNS pod survives drains
konnectivity-agent	ops-ai-konnectivity	minAvailable: 1 — at least 1 agent survives

Verify:

POOL_NODE=$(kubectl get nodes -l ops-ai.io/role=controller -o name --context=xano-stage | head -1)
kubectl get pods -A -o wide --field-selector spec.nodeName=${POOL_NODE#node/} --context=xano-stage
kubectl get pdb -n kube-system --context=xano-stage

CRD reference

For the full CRD schemas (RebalancePolicy, NodeScalingPolicy, PodResizePolicy) with all fields and examples, see CRD Reference.

Get Started

Platform

Operations

Migrating from CAST AI

Deployment

Controller (Helm)

Autoscaler (Helm)

CRD installation

GKE Workload Identity

Health checks

Controller

Autoscaler

Kubernetes probes

Metrics and monitoring

Prometheus metrics

ServiceMonitor setup

Key metrics to watch

Common operational tasks

Enable/disable dry-run mode

Adjust rebalance thresholds

Adjust scale cooldown

Change log level at runtime

Exclude namespaces from rebalancing

Controller pool node management

Infrastructure pinning

CRD reference

​Deployment

​Controller (Helm)

​Autoscaler (Helm)

​CRD installation

​GKE Workload Identity

​Health checks

​Controller

​Autoscaler

​Kubernetes probes

​Metrics and monitoring

​Prometheus metrics

​ServiceMonitor setup

​Key metrics to watch

​Common operational tasks

​Enable/disable dry-run mode

​Adjust rebalance thresholds

​Adjust scale cooldown

​Change log level at runtime

​Exclude namespaces from rebalancing

​Controller pool node management

​Infrastructure pinning

​CRD reference

Deployment

Controller (Helm)

Autoscaler (Helm)

CRD installation

GKE Workload Identity

Health checks

Controller

Autoscaler

Kubernetes probes

Metrics and monitoring

Prometheus metrics

ServiceMonitor setup

Key metrics to watch

Common operational tasks

Enable/disable dry-run mode

Adjust rebalance thresholds

Adjust scale cooldown

Change log level at runtime

Exclude namespaces from rebalancing

Controller pool node management

Infrastructure pinning

CRD reference