> ## Documentation Index
> Fetch the complete documentation index at: https://autopilot.docs.xano.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Verify It's Working

> A DevOps quick reference — what runs when, and how to confirm each optimization engine is healthy.

Everything you need to know about how Autopilot works, why it works that way, and how to
verify it's working.

## How often things run

| Component                 | Interval      | What it does                                                             |
| ------------------------- | ------------- | ------------------------------------------------------------------------ |
| Pod Resize evaluator      | Every 5 min   | Fetches usage from Prometheus, computes recommendations, applies resizes |
| Node Scaling evaluator    | Every 60 sec  | Checks for pending pods (scale-up) and underutilized nodes (scale-down)  |
| Rebalance checker         | Every 5 min   | Checks schedule, runs a rebalance if due                                 |
| Scheduled Scale evaluator | Every 1 min   | Checks if HPA windows need activation/deactivation                       |
| GCP service monitor       | Every 60 sec  | Polls the Cloud Monitoring API for SQL, Redis, NAT, etc.                 |
| Slack health alerts       | Every 2 min   | Checks pool capacity, pending pods, Prometheus health                    |
| Cost snapshots            | Every 6 hours | Records cluster + package cost to ConfigMaps                             |

## How to verify things are working

### Is the resizer producing recommendations?

```bash theme={null}
# Via dashboard
scripts/dashboard.sh --context=<cluster>
# Navigate to the Vertical Optimization page

# Via API
kubectl port-forward -n ops-ai deploy/ops-ai 9090:9090
curl localhost:9090/api/v1/recommendations | python3 -m json.tool | head -40

# Via CRD
kubectl get recommendations -A
```

Look for: `confidence` > 0.7, `sampleCount` > 100, `dataSource` = `prometheus` (not
`metrics-server`).

### Is the scaler evaluating?

```bash theme={null}
kubectl get nodescalingpolicy default -o yaml | grep -A5 status

# Key fields:
#   lastEvaluationTime — should be within the last 2 minutes
#   currentNodes / desiredNodes — should match (unless actively scaling)
#   lastScaleAction — "scale-up" or "scale-down" with timestamp
```

### Are cost snapshots being recorded?

```bash theme={null}
# Cluster cost history
kubectl get configmap ops-ai-cost-history -n ops-ai \
  -o jsonpath='{.data.history}' | python3 -m json.tool | tail -20

# Package cost history
kubectl get configmap ops-ai-package-cost-history -n ops-ai \
  -o jsonpath='{.data.history}' | python3 -m json.tool | tail -20
```

You should see snapshots every 6 hours. If the controller was recently deployed, there
should be at least one snapshot (recorded 30–45 seconds after startup).

### Is the webhook injecting resources?

```bash theme={null}
# Check if the webhook is registered
kubectl get mutatingwebhookconfigurations | grep ops-ai

# Compare a recent pod's resources vs its Deployment template
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[0].resources}'
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].resources}'
# If they differ, the webhook is injecting right-sized resources
```

### Health check — all in one

```bash theme={null}
# Dashboard health page
scripts/dashboard.sh --context=<cluster>
# Navigate to System Health

# API
curl localhost:9090/api/v1/health | python3 -m json.tool
# All checks should be "ok". "degraded" is acceptable for optional checks (Sentry, Metrics Server).
```

## Common "is it broken?" checklist

| Symptom                             | Check                                                   | Fix                                                                    |
| ----------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------- |
| No recommendations appearing        | `kubectl get recommendations -A` — empty?               | Check PodResizePolicy `includeNamespaces` matches your namespaces      |
| Recommendations say "awaiting data" | Check `algorithm.sampleCount` on the CRD                | Pod is too new (\< 5 min) or Prometheus has no data for it             |
| Scaler not scaling up               | Check `lastEvaluationTime` on NodeScalingPolicy         | If stale, check controller logs. If fresh, check `pendingPodThreshold` |
| Scaled up but pods still pending    | `kubectl describe pod <name>`                           | Likely a taint/affinity mismatch or PV zone issue                      |
| Dashboard shows no data             | `curl <prometheus-url>/-/healthy`                       | If down, check the monitoring namespace pods                           |
| Webhook not injecting               | `kubectl get mutatingwebhookconfigurations`             | If missing, check `webhook.enabled: true` in values                    |
| Cost shows \$0                      | `kubectl get nodes --show-labels \| grep instance-type` | Unknown machine type not in the catalog                                |

## Data retention

| Data                        | Where                                   | Retention                 |
| --------------------------- | --------------------------------------- | ------------------------- |
| Prometheus metrics          | monitoring namespace PVC                | 7 days                    |
| Recommendation history      | Annotation on Recommendation CRD        | 50 entries                |
| Schedule activation history | Annotation on ScheduledScalePolicy CRD  | 50 entries                |
| Cluster cost snapshots      | ConfigMap `ops-ai-cost-history`         | 360 snapshots (\~90 days) |
| Package cost snapshots      | ConfigMap `ops-ai-package-cost-history` | 180 snapshots (\~45 days) |
| Rebalance run history       | ConfigMaps `ops-ai-rebalance-*`         | 50 runs                   |
| K8s events                  | GKE default                             | \~1 hour                  |
| Controller logs             | Pod ephemeral storage                   | Until pod restart         |