Troubleshooting

Quick reference

Symptom	Likely cause	What to check
Recommendation oscillates	Prometheus step aliasing or restart spikes	`memHysteresis` on the Recommendation CRD
Recommendation too high	Startup spike not filtered	`algorithm.sampleCount` and `algorithm.dataSource`
Pods pending after scale-down	Scale-down removed labeled nodes too early	Node age (10-min grace period)
Webhook not injecting resources	Webhook pod not ready or `failurePolicy: Ignore`	Health page → Webhook check
`kubectl logs` fails “No agent available”	Konnectivity agent missing pool toleration	Re-run `scripts/create-opsai-pool.sh`
Cost shows $0.00	Unknown machine type not in catalog	Node `node.kubernetes.io/instance-type` label

Check if a NodeScalingPolicy exists:

kubectl get nodescalingpolicies -A   # shorthand: kubectl get nsp -A

Check the policy status:
kubectl describe nodescalingpolicy <name> -n <namespace>
Look at status.conditions for error messages and status.lastScaleAction.
Check controller logs:
kubectl -n ops-ai-system logs deploy/ops-ai-controller --tail=100
Look for:
- "failed to evaluate scaling" — metric collection or evaluation errors
- "cooldown active" — scaling is blocked by the cooldown period
- "dry-run mode" — scaling is blocked by the dry-run setting

Verify GKE credentials:

kubectl -n ops-ai-system describe pod -l app=ops-ai-controller | grep -A5 "Environment"

Ensure GKE_PROJECT_ID, GKE_LOCATION, and GKE_CLUSTER_NAME are set.

Verify Prometheus connectivity:

kubectl -n ops-ai-system exec deploy/ops-ai-controller -- \
  wget -qO- http://prometheus.monitoring:9090/-/healthy

Check if a RebalancePolicy exists:

kubectl get rebalancepolicies -A   # shorthand: kubectl get rbp -A

Check the imbalance score — it may be below the threshold:
opsctl --show-nodes
If all nodes show similar utilization percentages, the cluster is already balanced.
Check for PDB conflicts. If spec.respectPDB is true (default), pods protected by PodDisruptionBudgets that would be violated are not moved:
kubectl get pdb -A
Check status.lastPlan on the RebalancePolicy for the last analysis result.
Verify the schedule expression is correct — the controller uses standard cron syntax.

Check if the autoscaler pod is running:

kubectl -n ops-ai-system get pods -l app=ops-ai-autoscaler

Check autoscaler logs for pending pod detection:

kubectl -n ops-ai-system logs deploy/ops-ai-autoscaler --tail=100 | grep -i pending

Verify the pendingPodGracePeriod — pods must be pending for at least this duration (default 30s) before triggering a scale-up.
Check if maxNodes has been reached:
kubectl get nodes | wc -l

If CRDs are missing or outdated:

kubectl apply -f deploy/helm/ops-ai-controller/crds/
kubectl get crd rebalancepolicies.ops-ai.io -o jsonpath='{.spec.versions[*].name}'

If the controller fails to start with scheme registration errors, CRDs may need to be reinstalled before the controller pod starts.

If running multiple replicas with leader election enabled:

kubectl -n ops-ai-system get lease ops-ai-controller-leader -o yaml

If leader election is stuck, delete the lease to force re-election:

kubectl -n ops-ai-system delete lease ops-ai-controller-leader