Getting Started

This guide walks DevOps engineers through the Autopilot by Xano Kubernetes optimization platform — from initial setup through production operations. It covers every major feature and how the pieces fit together.

What Autopilot does

Autopilot is a Kubernetes controller that continuously optimizes cluster cost and performance through four mechanisms:

Pod Resizing — Right-sizes container resource requests based on actual usage (in-place, no restarts on K8s 1.33+)
Node Scaling — Adds and removes nodes based on demand, with per-pod machine type selection (heterogeneous scaling)
Cluster Rebalancing — Redistributes pods across nodes to eliminate waste and enable node removal
Scheduled Scaling — Pre-warms HPA targets before predictable traffic bursts

It also provides observability for GCP managed services (Cloud SQL, Redis, Cloud NAT, etc.) and per-package cost attribution.

Architecture overview

                          +-------------------+
                          |  Admin Dashboard  |  (:9090)
                          |  Angular 20 SPA   |
                          +--------+----------+
                                   |
                          +--------v----------+
                          |   REST API        |
                          +--------+----------+
                                   |
              +--------------------+--------------------+
              |                    |                     |
    +---------v--------+  +-------v--------+  +---------v--------+
    | PodResize        |  | NodeScaling    |  | Rebalance        |
    | Reconciler       |  | Reconciler     |  | Reconciler       |
    | (5min cycle)     |  | (60s cycle)    |  | (cron schedule)  |
    +--------+---------+  +-------+--------+  +---------+--------+
             |                    |                      |
    +--------v--------+  +-------v--------+  +----------v-------+
    | Prometheus       |  | GCE Compute    |  | K8s Eviction     |
    | (usage metrics)  |  | (VM provision) |  | (pod movement)   |
    +------------------+  +----------------+  +------------------+

The controller runs as a single-replica Deployment with leader election. All state is externalized to CRDs, ConfigMaps, and Prometheus — restarts lose nothing.

Prerequisites

GKE cluster (K8s 1.33+ recommended for in-place pod resizing)
kubectl configured with cluster access
gcloud authenticated (gcloud auth application-default login)
Helm 3.x
Docker or Depot CLI (for building images)

Initial setup

1. Bootstrap a cluster

Every cluster needs a one-time setup that creates service accounts, installs Prometheus, and generates the environment config:

scripts/setup-cluster.sh --context=<cluster-name>

This automatically:

Creates a GCP service account with compute.admin + container.admin roles
Creates a node service account for heterogeneous VM provisioning
Configures Workload Identity bindings
Deploys kube-prometheus-stack to the monitoring namespace
Creates the ops-ai namespace
Generates deploy/envs/<cluster-name>.yaml with cluster-specific values

Autopilot’s runtime identifiers — the ops-ai namespace, the ops-ai.io CRD API group, the opsctl CLI, and ops-ai.io/* annotations — keep their original names. They are code and runtime identifiers, not brand names, and renaming them would break existing clusters.

2. Deploy Autopilot

# Build a release
scripts/release.sh patch

# Deploy to one or more clusters (always use comma-separated for parallel)
scripts/deploy-cluster.sh --context=xano-dev,xano-stage

3. Verify

# Open the dashboard
scripts/dashboard.sh --context=<cluster-name>

# Check cluster stats
scripts/cluster-stats.sh <cluster-name>

Navigate to the System Health page in the dashboard — all checks should be green.

The four optimization engines

Each engine has its own deep-dive page. Here is how they fit together:

Pod Resizer

Vertical optimization — right-sizes CPU/memory requests from Prometheus usage data.

Node Scaler

Horizontal infrastructure — NodePool or heterogeneous per-pod machine selection.

Cluster Rebalancer

Cost optimization — recomputes optimal node configs, provision-first or rolling drain.

Scheduled Scaler

Pre-warming — overrides HPA minReplicas during defined time windows.

Observability

Autopilot surfaces cluster and managed-service health in the same dashboard:

GCP managed services — read-only monitoring for Cloud SQL, Memorystore Redis, Cloud NAT, Storage, DNS, VPC Peering, Artifact Registry, and Secret Manager.
Network monitoring — cluster throughput, top talkers, and per-node errors/drops, powered by existing Prometheus metrics with no extra agents.
Package cost tracking — per-package cost attribution for tenant namespaces, with 6-hour snapshots for trend analysis.

See Observability & Cost for full details.

Admin dashboard

The controller embeds an Angular SPA served on port 9090. Access it via:

scripts/dashboard.sh --context=<cluster-name>

Dashboard pages

Section	Page	Description
Overview	Dashboard	Cluster summary with utilization, cost, scaling state, health score
Optimization	Vertical Optimization	Resize recommendations with savings estimates
	Recommendation Detail	Per-workload audit: usage charts, algorithm transparency, history
	Scheduled Scales	Time-based HPA scaling policies with activation history
Infrastructure	Policies	All CRD policies (Rebalance, NodeScaling, PodResize)
	Scaling Activity	Node scale-up/down event timeline
	Node List	All nodes with utilization, machine type, cost
	GCP Services	Managed service health cards
	Package Costs	Per-package cost attribution with history
	Namespaces	Namespace list with resource attribution
	Network	Cluster network throughput, top talkers, per-node errors/drops
Operations	Rebalancer	On-demand rebalance runs with cost comparison
	Maintenance	Maintenance window runs and consolidation history
	Problem Pods	Pods with OOM kills, crashes, resize issues
	Events	Kubernetes event timeline
	System Health	Health checks with latency and status

Configuration layers

Policy behavior is controlled at three layers. When changing defaults, all layers must be considered:

Code defaults — fallback when CRD fields are empty
Helm values (deploy/helm/ops-ai-controller/values.yaml) — installed as default CRDs on first deploy
Live CRDs on cluster — the actual runtime config (Helm only updates on install/upgrade)

Changing code defaults alone does not change behavior if the CRD already has an explicit value. You must also update values.yaml and patch the live CRD (or let a Helm upgrade apply it).

Per-cluster configuration

Each cluster gets its own env file in deploy/envs/<cluster-name>.yaml:

gke:
  project: my-gcp-project
  location: us-central1
  cluster: my-cluster

controller:
  mode: active              # active (make changes) or passive (observe only)

prometheus:
  url: http://prometheus-kube-prometheus-prometheus.monitoring:9090

gcpServices:
  enabled: true
  services:
    - type: CloudSQL
      instanceName: my-database
    - type: MemorystoreRedis
      instanceName: my-cache

Operational safety

What Autopilot never does

Evicts pods without checking PodDisruptionBudgets
Modifies Deployment templates (uses webhook injection instead)
Resizes its own namespace (ops-ai is always excluded)
Scales below minNodes in the NodeScalingPolicy
Deploys automatically — always requires explicit deploy-cluster.sh

Protection mechanisms

CAST AI conflict detector — forces all policies to passive mode if CAST AI agents are detected
Single-replica protection — temporarily scales to 2 replicas before evicting the sole pod
Limit safety net — the webhook enforces minimum memory limits on all pod creates
OOM death spiral detection — evicts pods stuck in OOM loops for webhook re-injection
Circuit breaker — Prometheus failures degrade gracefully to metrics-server

Controller replica count

The controller runs with 1 replica using the Recreate deployment strategy. Do not increase this — leader election and operational simplicity mean exactly 1 is correct, and a restart is a 30–60s gap during which nothing time-critical is missed (the webhook fails open).

Monitoring the controller

Slack alerts — infrastructure problems (pool capacity, Prometheus OOM/disk, pending pods, managed node count). Configure via slack.secretName in Helm values.
Sentry — captures all error+ level logs. Configure via sentry.dsn in the per-cluster env file.
Prometheus metrics — exposed at /metrics (port 8080); a ServiceMonitor ships with the Helm chart.
Grafana — a pre-built dashboard deploys automatically (scripts/grafana.sh).

Release workflow

# 1. Make code changes
# 2. Verify locally
go build ./...
go test ./...

# 3. Build release (auto-commits uncommitted changes, builds image, tags)
scripts/release.sh patch            # or minor, major
scripts/release.sh patch --skip-tests  # faster iteration

# 4. Deploy to all clusters (always parallel)
scripts/deploy-cluster.sh --context=xano-dev,xano-dev-d2,xano-stage

# 5. Push git tag
git push origin <version>

# 6. Verify
scripts/cluster-stats.sh

Next steps

opsctl CLI Guide

Detailed command reference for the CLI.

Operations Runbook

Day-to-day procedures, health checks, and disaster recovery.

Autopilot vs CAST AI

How Autopilot compares — transparency, cost model, and data residency.

API Reference

Every dashboard API endpoint.

Get Started

Platform

Operations

Migrating from CAST AI

What Autopilot does

Architecture overview

Prerequisites

Initial setup

1. Bootstrap a cluster

2. Deploy Autopilot

3. Verify

The four optimization engines

Pod Resizer

Node Scaler

Cluster Rebalancer

Scheduled Scaler

Observability

Admin dashboard

Dashboard pages

Configuration layers

Per-cluster configuration

Operational safety

What Autopilot never does

Protection mechanisms

Controller replica count

Monitoring the controller

Release workflow

Next steps

opsctl CLI Guide

Operations Runbook

Autopilot vs CAST AI

API Reference

​What Autopilot does

​Architecture overview

​Prerequisites

​Initial setup

​1. Bootstrap a cluster

​2. Deploy Autopilot

​3. Verify

​The four optimization engines

Pod Resizer

Node Scaler

Cluster Rebalancer

Scheduled Scaler

​Observability

​Admin dashboard

​Dashboard pages

​Configuration layers

​Per-cluster configuration

​Operational safety

​What Autopilot never does

​Protection mechanisms

​Controller replica count

​Monitoring the controller

​Release workflow

​Next steps

opsctl CLI Guide

Operations Runbook

Autopilot vs CAST AI

API Reference

What Autopilot does

Architecture overview

Prerequisites

Initial setup

1. Bootstrap a cluster

2. Deploy Autopilot

3. Verify

The four optimization engines

Observability

Admin dashboard

Dashboard pages

Configuration layers

Per-cluster configuration

Operational safety

What Autopilot never does

Protection mechanisms

Controller replica count

Monitoring the controller

Release workflow

Next steps