> ## Documentation Index
> Fetch the complete documentation index at: https://autopilot.docs.xano.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability & Cost

> GCP managed-service monitoring, cluster network observability, and per-package cost attribution.

Autopilot surfaces cluster and managed-service health, network traffic, and cost attribution
in the same dashboard — all from existing Prometheus metrics and the GCP Cloud Monitoring
API, with no extra agents.

## GCP managed services monitoring

Autopilot provides read-only monitoring for GCP services associated with each cluster. The
controller polls the GCP Cloud Monitoring API every 60s and surfaces health and metrics in
the dashboard.

| Service           | Metrics                                | Health thresholds                              |
| ----------------- | -------------------------------------- | ---------------------------------------------- |
| Cloud SQL         | CPU, disk, connections, state          | CPU >90%, disk >80/95%, connections >80/95%    |
| Memorystore Redis | Memory, evicted keys, hit ratio, state | Memory >80/95%, evictions >0, hit ratio `<50%` |
| Cloud NAT         | Port usage, dropped packets, NAT IPs   | Ports >80/95%, drops ≥1/s, IPs >40             |
| Cloud Storage     | Size, request rates, error rates       | Error rate >1%, high request rates             |
| Cloud DNS         | Query rates, latency, DNSSEC status    | Latency thresholds, DNSSEC validation failures |
| VPC Peering       | Peering status, route counts           | Inactive peerings, route quota proximity       |
| Artifact Registry | Image count, vulnerabilities, size     | Critical/high vulnerabilities, scan failures   |
| Secret Manager    | Secret count, rotation status, expiry  | Expired secrets, missing rotation              |

### Configuration

Per-cluster in `deploy/envs/<cluster>.yaml`:

```yaml theme={null}
gcpServices:
  enabled: true
  services:
    - type: CloudSQL
      instanceName: my-database
    - type: MemorystoreRedis
      instanceName: my-cache
    - type: CloudNAT
      routerName: my-router
```

The **GCP Services** page shows a card for each service with health indicators and key
metrics. Click through for a detail page with usage charts and events.

## Network monitoring

Cluster-wide network observability powered by Prometheus. The Network page provides
real-time visibility into traffic patterns, errors, and drops across the cluster without any
additional agents or configuration — it queries existing Prometheus container network
metrics.

**What it shows:**

* **Cluster throughput** — aggregate ingress/egress bandwidth (bytes/sec) with a time-series chart.
* **Top namespaces by traffic** — which namespaces generate the most network I/O.
* **Top pods by traffic** — individual pods ranked by bandwidth consumption.
* **Per-node network health** — errors/sec and drops/sec per node, flagging nodes with problems.
* **Error summary** — cluster-wide error and drop rates with node counts.

**Key design details:**

* Virtual interfaces (veth, calico, flannel, CNI bridges) are excluded — only physical NIC traffic is counted, to avoid double-counting.
* Data is cached with singleflight so dashboard polls don't hammer Prometheus.
* No configuration needed — it works automatically if Prometheus is available.

Navigate to **Infrastructure → Network**. The page shows summary cards, a throughput
time-series chart, and tables for top namespaces, top pods, and node health.

## Package cost tracking

Per-package cost attribution for tenant namespaces. Each pod's share of node cost is
computed using request-weighted blended CPU/memory:

```
pod_cost = node_hourly_cost × (0.5 × pod_cpu_req/node_allocatable_cpu
                              + 0.5 × pod_mem_req/node_allocatable_mem)
```

Pods are grouped by the `xano.com/package` label. Unattributed cost is tracked as spillover
in three categories: DaemonSet pods, system pods, and idle capacity. Cost snapshots are
recorded every 6 hours for trend analysis.

The **Package Costs** page (Infrastructure section) shows a cost breakdown table, spillover
analysis, and a historical cost chart.

## System health checks

The health endpoint (`GET /api/v1/health`) runs a battery of checks:

| Check                 | What it validates                                    |
| --------------------- | ---------------------------------------------------- |
| Kubernetes API        | API server reachable, response time                  |
| Prometheus            | Reachable, query latency, sample availability        |
| CRDs                  | All CRD types registered, at least one policy exists |
| Metrics Server        | K8s metrics-server responding (optional)             |
| Node Capacity         | Managed nodes within scaling policy limits           |
| Prometheus Storage    | PVC utilization below thresholds                     |
| Resizer Effectiveness | PodResizePolicy producing recommendations            |
| Scaler Freshness      | NodeScalingPolicy evaluating on schedule             |
| Metrics Quality       | Node count in Prometheus matches the K8s API         |
| OOM Death Spirals     | No pods in repeated OOM-restart loops                |
| CoreDNS               | CoreDNS pods healthy in kube-system                  |
| Webhook               | Mutation webhook registered and serving              |
| Nodes Not Ready       | All nodes in Ready condition                         |
| Ingress Controller    | Ingress controller pods healthy                      |
| GCP Services          | Managed service health                               |
| CAST AI Conflict      | No conflicting CAST AI agents detected               |