Verify It's Working

Everything you need to know about how Autopilot works, why it works that way, and how to verify it’s working.

How often things run

Component	Interval	What it does
Pod Resize evaluator	Every 5 min	Fetches usage from Prometheus, computes recommendations, applies resizes
Node Scaling evaluator	Every 60 sec	Checks for pending pods (scale-up) and underutilized nodes (scale-down)
Rebalance checker	Every 5 min	Checks schedule, runs a rebalance if due
Scheduled Scale evaluator	Every 1 min	Checks if HPA windows need activation/deactivation
GCP service monitor	Every 60 sec	Polls the Cloud Monitoring API for SQL, Redis, NAT, etc.
Slack health alerts	Every 2 min	Checks pool capacity, pending pods, Prometheus health
Cost snapshots	Every 6 hours	Records cluster + package cost to ConfigMaps

How to verify things are working

Is the resizer producing recommendations?

# Via dashboard
scripts/dashboard.sh --context=<cluster>
# Navigate to the Vertical Optimization page

# Via API
kubectl port-forward -n ops-ai deploy/ops-ai 9090:9090
curl localhost:9090/api/v1/recommendations | python3 -m json.tool | head -40

# Via CRD
kubectl get recommendations -A

Look for: confidence > 0.7, sampleCount > 100, dataSource = prometheus (not metrics-server).

Is the scaler evaluating?

kubectl get nodescalingpolicy default -o yaml | grep -A5 status

# Key fields:
#   lastEvaluationTime — should be within the last 2 minutes
#   currentNodes / desiredNodes — should match (unless actively scaling)
#   lastScaleAction — "scale-up" or "scale-down" with timestamp

Are cost snapshots being recorded?

# Cluster cost history
kubectl get configmap ops-ai-cost-history -n ops-ai \
  -o jsonpath='{.data.history}' | python3 -m json.tool | tail -20

# Package cost history
kubectl get configmap ops-ai-package-cost-history -n ops-ai \
  -o jsonpath='{.data.history}' | python3 -m json.tool | tail -20

You should see snapshots every 6 hours. If the controller was recently deployed, there should be at least one snapshot (recorded 30–45 seconds after startup).

Is the webhook injecting resources?

# Check if the webhook is registered
kubectl get mutatingwebhookconfigurations | grep ops-ai

# Compare a recent pod's resources vs its Deployment template
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[0].resources}'
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].resources}'
# If they differ, the webhook is injecting right-sized resources

Health check — all in one

# Dashboard health page
scripts/dashboard.sh --context=<cluster>
# Navigate to System Health

# API
curl localhost:9090/api/v1/health | python3 -m json.tool
# All checks should be "ok". "degraded" is acceptable for optional checks (Sentry, Metrics Server).

Common “is it broken?” checklist

Symptom	Check	Fix
No recommendations appearing	`kubectl get recommendations -A` — empty?	Check PodResizePolicy `includeNamespaces` matches your namespaces
Recommendations say “awaiting data”	Check `algorithm.sampleCount` on the CRD	Pod is too new (< 5 min) or Prometheus has no data for it
Scaler not scaling up	Check `lastEvaluationTime` on NodeScalingPolicy	If stale, check controller logs. If fresh, check `pendingPodThreshold`
Scaled up but pods still pending	`kubectl describe pod <name>`	Likely a taint/affinity mismatch or PV zone issue
Dashboard shows no data	`curl <prometheus-url>/-/healthy`	If down, check the monitoring namespace pods
Webhook not injecting	`kubectl get mutatingwebhookconfigurations`	If missing, check `webhook.enabled: true` in values
Cost shows $0	`kubectl get nodes --show-labels \| grep instance-type`	Unknown machine type not in the catalog

Data retention

Data	Where	Retention
Prometheus metrics	monitoring namespace PVC	7 days
Recommendation history	Annotation on Recommendation CRD	50 entries
Schedule activation history	Annotation on ScheduledScalePolicy CRD	50 entries
Cluster cost snapshots	ConfigMap `ops-ai-cost-history`	360 snapshots (~90 days)
Package cost snapshots	ConfigMap `ops-ai-package-cost-history`	180 snapshots (~45 days)
Rebalance run history	ConfigMaps `ops-ai-rebalance-*`	50 runs
K8s events	GKE default	~1 hour
Controller logs	Pod ephemeral storage	Until pod restart

Get Started

Platform

Operations

Migrating from CAST AI

How often things run

How to verify things are working

Is the resizer producing recommendations?

Is the scaler evaluating?

Are cost snapshots being recorded?

Is the webhook injecting resources?

Health check — all in one

Common “is it broken?” checklist

Data retention

​How often things run

​How to verify things are working

​Is the resizer producing recommendations?

​Is the scaler evaluating?

​Are cost snapshots being recorded?

​Is the webhook injecting resources?

​Health check — all in one

​Common “is it broken?” checklist

​Data retention

How often things run

How to verify things are working

Is the resizer producing recommendations?

Is the scaler evaluating?

Are cost snapshots being recorded?

Is the webhook injecting resources?

Health check — all in one

Common “is it broken?” checklist

Data retention