> ## Documentation Index > Fetch the complete documentation index at: https://autopilot.docs.xano.com/llms.txt > Use this file to discover all available pages before exploring further. # Getting Started > Bootstrap a cluster, deploy Autopilot, and tour every optimization engine. This guide walks DevOps engineers through the Autopilot by Xano Kubernetes optimization platform — from initial setup through production operations. It covers every major feature and how the pieces fit together. ## What Autopilot does Autopilot is a Kubernetes controller that continuously optimizes cluster cost and performance through four mechanisms: 1. **Pod Resizing** — Right-sizes container resource requests based on actual usage (in-place, no restarts on K8s 1.33+) 2. **Node Scaling** — Adds and removes nodes based on demand, with per-pod machine type selection (heterogeneous scaling) 3. **Cluster Rebalancing** — Redistributes pods across nodes to eliminate waste and enable node removal 4. **Scheduled Scaling** — Pre-warms HPA targets before predictable traffic bursts It also provides observability for GCP managed services (Cloud SQL, Redis, Cloud NAT, etc.) and per-package cost attribution. ## Architecture overview ``` +-------------------+ | Admin Dashboard | (:9090) | Angular 20 SPA | +--------+----------+ | +--------v----------+ | REST API | +--------+----------+ | +--------------------+--------------------+ | | | +---------v--------+ +-------v--------+ +---------v--------+ | PodResize | | NodeScaling | | Rebalance | | Reconciler | | Reconciler | | Reconciler | | (5min cycle) | | (60s cycle) | | (cron schedule) | +--------+---------+ +-------+--------+ +---------+--------+ | | | +--------v--------+ +-------v--------+ +----------v-------+ | Prometheus | | GCE Compute | | K8s Eviction | | (usage metrics) | | (VM provision) | | (pod movement) | +------------------+ +----------------+ +------------------+ ``` The controller runs as a single-replica Deployment with leader election. All state is externalized to CRDs, ConfigMaps, and Prometheus — restarts lose nothing. ## Prerequisites * GKE cluster (K8s 1.33+ recommended for in-place pod resizing) * `kubectl` configured with cluster access * `gcloud` authenticated (`gcloud auth application-default login`) * Helm 3.x * Docker or Depot CLI (for building images) ## Initial setup ### 1. Bootstrap a cluster Every cluster needs a one-time setup that creates service accounts, installs Prometheus, and generates the environment config: ```bash theme={null} scripts/setup-cluster.sh --context= ``` This automatically: * Creates a GCP service account with `compute.admin` + `container.admin` roles * Creates a node service account for heterogeneous VM provisioning * Configures Workload Identity bindings * Deploys kube-prometheus-stack to the `monitoring` namespace * Creates the `ops-ai` namespace * Generates `deploy/envs/.yaml` with cluster-specific values Autopilot's runtime identifiers — the `ops-ai` namespace, the `ops-ai.io` CRD API group, the `opsctl` CLI, and `ops-ai.io/*` annotations — keep their original names. They are code and runtime identifiers, not brand names, and renaming them would break existing clusters. ### 2. Deploy Autopilot ```bash theme={null} # Build a release scripts/release.sh patch # Deploy to one or more clusters (always use comma-separated for parallel) scripts/deploy-cluster.sh --context=xano-dev,xano-stage ``` ### 3. Verify ```bash theme={null} # Open the dashboard scripts/dashboard.sh --context= # Check cluster stats scripts/cluster-stats.sh ``` Navigate to the **System Health** page in the dashboard — all checks should be green. ## The four optimization engines Each engine has its own deep-dive page. Here is how they fit together: Vertical optimization — right-sizes CPU/memory requests from Prometheus usage data. Horizontal infrastructure — NodePool or heterogeneous per-pod machine selection. Cost optimization — recomputes optimal node configs, provision-first or rolling drain. Pre-warming — overrides HPA `minReplicas` during defined time windows. ## Observability Autopilot surfaces cluster and managed-service health in the same dashboard: * **GCP managed services** — read-only monitoring for Cloud SQL, Memorystore Redis, Cloud NAT, Storage, DNS, VPC Peering, Artifact Registry, and Secret Manager. * **Network monitoring** — cluster throughput, top talkers, and per-node errors/drops, powered by existing Prometheus metrics with no extra agents. * **Package cost tracking** — per-package cost attribution for tenant namespaces, with 6-hour snapshots for trend analysis. See [Observability & Cost](/platform/observability) for full details. ## Admin dashboard The controller embeds an Angular SPA served on port 9090. Access it via: ```bash theme={null} scripts/dashboard.sh --context= ``` ### Dashboard pages | Section | Page | Description | | ------------------ | --------------------- | ------------------------------------------------------------------- | | **Overview** | Dashboard | Cluster summary with utilization, cost, scaling state, health score | | **Optimization** | Vertical Optimization | Resize recommendations with savings estimates | | | Recommendation Detail | Per-workload audit: usage charts, algorithm transparency, history | | | Scheduled Scales | Time-based HPA scaling policies with activation history | | **Infrastructure** | Policies | All CRD policies (Rebalance, NodeScaling, PodResize) | | | Scaling Activity | Node scale-up/down event timeline | | | Node List | All nodes with utilization, machine type, cost | | | GCP Services | Managed service health cards | | | Package Costs | Per-package cost attribution with history | | | Namespaces | Namespace list with resource attribution | | | Network | Cluster network throughput, top talkers, per-node errors/drops | | **Operations** | Rebalancer | On-demand rebalance runs with cost comparison | | | Maintenance | Maintenance window runs and consolidation history | | | Problem Pods | Pods with OOM kills, crashes, resize issues | | | Events | Kubernetes event timeline | | | System Health | Health checks with latency and status | ## Configuration layers Policy behavior is controlled at three layers. When changing defaults, all layers must be considered: 1. **Code defaults** — fallback when CRD fields are empty 2. **Helm values** (`deploy/helm/ops-ai-controller/values.yaml`) — installed as default CRDs on first deploy 3. **Live CRDs on cluster** — the actual runtime config (Helm only updates on install/upgrade) Changing code defaults alone does **not** change behavior if the CRD already has an explicit value. You must also update `values.yaml` and patch the live CRD (or let a Helm upgrade apply it). ### Per-cluster configuration Each cluster gets its own env file in `deploy/envs/.yaml`: ```yaml theme={null} gke: project: my-gcp-project location: us-central1 cluster: my-cluster controller: mode: active # active (make changes) or passive (observe only) prometheus: url: http://prometheus-kube-prometheus-prometheus.monitoring:9090 gcpServices: enabled: true services: - type: CloudSQL instanceName: my-database - type: MemorystoreRedis instanceName: my-cache ``` ## Operational safety ### What Autopilot never does * Evicts pods without checking PodDisruptionBudgets * Modifies Deployment templates (uses webhook injection instead) * Resizes its own namespace (`ops-ai` is always excluded) * Scales below `minNodes` in the NodeScalingPolicy * Deploys automatically — always requires explicit `deploy-cluster.sh` ### Protection mechanisms * **CAST AI conflict detector** — forces all policies to passive mode if CAST AI agents are detected * **Single-replica protection** — temporarily scales to 2 replicas before evicting the sole pod * **Limit safety net** — the webhook enforces minimum memory limits on all pod creates * **OOM death spiral detection** — evicts pods stuck in OOM loops for webhook re-injection * **Circuit breaker** — Prometheus failures degrade gracefully to metrics-server ### Controller replica count The controller runs with **1 replica** using the `Recreate` deployment strategy. Do not increase this — leader election and operational simplicity mean exactly 1 is correct, and a restart is a 30–60s gap during which nothing time-critical is missed (the webhook fails open). ## Monitoring the controller * **Slack alerts** — infrastructure problems (pool capacity, Prometheus OOM/disk, pending pods, managed node count). Configure via `slack.secretName` in Helm values. * **Sentry** — captures all `error`+ level logs. Configure via `sentry.dsn` in the per-cluster env file. * **Prometheus metrics** — exposed at `/metrics` (port 8080); a ServiceMonitor ships with the Helm chart. * **Grafana** — a pre-built dashboard deploys automatically (`scripts/grafana.sh`). ## Release workflow ```bash theme={null} # 1. Make code changes # 2. Verify locally go build ./... go test ./... # 3. Build release (auto-commits uncommitted changes, builds image, tags) scripts/release.sh patch # or minor, major scripts/release.sh patch --skip-tests # faster iteration # 4. Deploy to all clusters (always parallel) scripts/deploy-cluster.sh --context=xano-dev,xano-dev-d2,xano-stage # 5. Push git tag git push origin # 6. Verify scripts/cluster-stats.sh ``` ## Next steps Detailed command reference for the CLI. Day-to-day procedures, health checks, and disaster recovery. How Autopilot compares — transparency, cost model, and data residency. Every dashboard API endpoint.