> ## Documentation Index
> Fetch the complete documentation index at: https://autopilot.docs.xano.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations Runbook

> Deployment, health checks, monitoring, and day-to-day operational tasks for the Autopilot controller and autoscaler.

This runbook covers deployment, monitoring, and day-to-day operation of the Autopilot
platform. It applies to both the **controller** (CRD-driven, runs as a Kubernetes
controller-manager) and the **autoscaler** (standalone GKE-focused autoscaling loop).

<Note>
  For scale-down behavior see [Node Scale-Down](/operations/node-scale-down); for problem
  diagnosis see [Troubleshooting](/operations/troubleshooting); for emergency procedures see
  [Disaster Recovery](/operations/disaster-recovery).
</Note>

## Deployment

### Controller (Helm)

```bash theme={null}
helm install ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system \
  --create-namespace \
  --set prometheus.url=http://prometheus.monitoring:9090 \
  --set controller.leaderElect=true
```

Common value overrides:

```bash theme={null}
# GKE integration
--set gke.project=my-project \
--set gke.location=us-central1 \
--set gke.cluster=production

# Start in dry-run mode (safe for initial rollout)
--set controller.dryRun=true

# Tune intervals
--set controller.rebalanceInterval=10m \
--set controller.scaleCooldown=15m

# Custom image
--set image.repository=ghcr.io/myorg/ops-ai-controller \
--set image.tag=v0.2.0

# Resource limits
--set resources.requests.cpu=200m \
--set resources.requests.memory=512Mi \
--set resources.limits.cpu=1 \
--set resources.limits.memory=1Gi
```

### Autoscaler (Helm)

```bash theme={null}
helm install ops-ai-autoscaler deploy/helm/ops-ai-autoscaler \
  --namespace ops-ai-system \
  --create-namespace \
  --set gke.project=my-project \
  --set gke.location=us-central1 \
  --set gke.cluster=production
```

Common value overrides:

```bash theme={null}
--set autoscaler.scaleUpThreshold=0.8 \
--set autoscaler.scaleDownThreshold=0.7 \
--set autoscaler.minNodes=3 \
--set autoscaler.maxNodes=50 \
--set autoscaler.scaleUpCooldown=3m \
--set autoscaler.scaleDownCooldown=10m \
--set autoscaler.preferSpot=true \
--set autoscaler.dryRun=true \
--set autoscaler.scanInterval=60s
```

### CRD installation

CRDs are installed automatically when `crds.install=true` (default) in the controller chart.
To install manually:

```bash theme={null}
kubectl apply -f deploy/helm/ops-ai-controller/crds/
kubectl get crd | grep ops-ai.io
```

Expected output:

```
nodescalingpolicies.ops-ai.io    ...
podresizepolicies.ops-ai.io      ...
rebalancepolicies.ops-ai.io      ...
```

### GKE Workload Identity

If using Workload Identity instead of node-level service account keys:

```bash theme={null}
--set gke.workloadIdentityServiceAccount=ops-ai@my-project.iam.gserviceaccount.com
--set serviceAccount.annotations."iam\.gke\.io/gcp-service-account"=ops-ai@my-project.iam.gserviceaccount.com
```

The GCP service account needs these roles:

* `roles/container.clusterAdmin` (for node pool management)
* `roles/compute.instanceAdmin.v1` (for node operations)

## Health checks

### Controller

The controller exposes health endpoints on port 8081 (configurable via `controller.probeAddr`):

```bash theme={null}
kubectl -n ops-ai-system port-forward deploy/ops-ai-controller 8081:8081
curl http://localhost:8081/healthz   # Liveness — is the process alive?
curl http://localhost:8081/readyz    # Readiness — ready to serve?
```

### Autoscaler

The autoscaler exposes health and metrics on port 8080:

```bash theme={null}
kubectl -n ops-ai-system port-forward deploy/ops-ai-autoscaler 8080:8080
curl http://localhost:8080/healthz
curl http://localhost:8080/readyz
```

### Kubernetes probes

Both Helm charts configure probes automatically:

* **Liveness**: `GET /healthz`, initial delay 15s, period 20s
* **Readiness**: `GET /readyz`, initial delay 5s, period 10s

## Metrics and monitoring

### Prometheus metrics

The controller and autoscaler expose Prometheus metrics on port 8080:

```bash theme={null}
kubectl -n ops-ai-system port-forward svc/ops-ai-controller 8080:8080
curl http://localhost:8080/metrics
```

### ServiceMonitor setup

```yaml theme={null}
serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s
  labels:
    release: prometheus   # match your Prometheus operator selector
```

### Key metrics to watch

Set up alerts for these conditions:

* Controller pod not ready for more than 5 minutes
* Leader election lost (check logs for "leader election lost")
* High error rate in reconciliation loops
* Scaling operations consistently failing
* Rebalance plans generated but never executed (stuck in dry-run when not intended)

## Common operational tasks

### Enable/disable dry-run mode

Dry-run prevents the controller and autoscaler from making any changes. All analysis and
planning still runs; only execution is skipped.

```bash theme={null}
helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.dryRun=true

helm upgrade ops-ai-autoscaler deploy/helm/ops-ai-autoscaler \
  --namespace ops-ai-system --reuse-values --set autoscaler.dryRun=true
```

Individual CRD resources also have a `spec.dryRun` field. Setting it `true` on a specific
RebalancePolicy disables execution for that policy only, even if the global dry-run is
`false`.

### Adjust rebalance thresholds

Via Helm (global):

```bash theme={null}
helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.rebalanceInterval=10m
```

Via CRD (per-policy):

```yaml theme={null}
apiVersion: ops-ai.io/v1alpha1
kind: RebalancePolicy
metadata:
  name: aggressive
  namespace: production
spec:
  schedule: "*/2 * * * *"
  thresholds:
    imbalanceScore: 0.15
    minUtilization: 0.3
    maxUtilization: 0.75
  maxMovements: 20
  maxParallel: 5
  respectPDB: true
  dryRun: false
```

### Adjust scale cooldown

```bash theme={null}
helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.scaleCooldown=15m

helm upgrade ops-ai-autoscaler deploy/helm/ops-ai-autoscaler \
  --namespace ops-ai-system --reuse-values \
  --set autoscaler.scaleUpCooldown=5m --set autoscaler.scaleDownCooldown=15m
```

### Change log level at runtime

```bash theme={null}
helm upgrade ops-ai-controller deploy/helm/ops-ai-controller \
  --namespace ops-ai-system --reuse-values --set controller.logLevel=debug
```

Valid levels: `debug`, `info`, `warn`, `error`.

### Exclude namespaces from rebalancing

```yaml theme={null}
spec:
  excludeNamespaces:
    - kube-system
    - monitoring
    - istio-system
    - ops-ai-system
```

## Controller pool node management

### Infrastructure pinning

Critical monitoring infrastructure is pinned to the Autopilot controller pool node so it
survives managed node scale-downs and rebalances. GKE-managed kube-system components
(kube-dns, metrics-server, konnectivity) cannot be pinned — GKE's control plane reconciler
resets their Deployment specs within seconds. These are protected by PDBs instead.

**Re-apply after GKE upgrades** (GKE may delete PDBs or reset monitoring Deployments):

```bash theme={null}
scripts/pin-infra-to-pool.sh --context=xano-stage
# Or all clusters at once:
scripts/pin-infra-to-pool.sh --context=xano-dev,xano-dev-d2,xano-stage,xano-stage-s2
```

**Pinned to the pool node:**

| Component            | Method               | Why                                  |
| -------------------- | -------------------- | ------------------------------------ |
| Prometheus           | Helm values          | Metrics — resizer/scaler data source |
| Alertmanager         | Helm values          | Alert delivery                       |
| kube-state-metrics   | pin-infra-to-pool.sh | KSM metrics for Prometheus           |
| Prometheus Operator  | pin-infra-to-pool.sh | Manages Prometheus config            |
| Autopilot controller | Helm chart           | Controller must survive drains       |

**Protected by PDB (GKE-managed, can't pin):**

| Component          | PDB                 | Protection                                           |
| ------------------ | ------------------- | ---------------------------------------------------- |
| kube-dns           | ops-ai-kube-dns     | minAvailable: 1 — at least 1 DNS pod survives drains |
| konnectivity-agent | ops-ai-konnectivity | minAvailable: 1 — at least 1 agent survives          |

**Verify:**

```bash theme={null}
POOL_NODE=$(kubectl get nodes -l ops-ai.io/role=controller -o name --context=xano-stage | head -1)
kubectl get pods -A -o wide --field-selector spec.nodeName=${POOL_NODE#node/} --context=xano-stage
kubectl get pdb -n kube-system --context=xano-stage
```

## CRD reference

For the full CRD schemas (RebalancePolicy, NodeScalingPolicy, PodResizePolicy) with all
fields and examples, see [CRD Reference](/reference/crds).