> ## Documentation Index
> Fetch the complete documentation index at: https://autopilot.docs.xano.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Diagnose common problems with node scaling, rebalancing, the autoscaler, CRDs, and leader election.

## Quick reference

| Symptom                                   | Likely cause                                     | What to check                                      |
| ----------------------------------------- | ------------------------------------------------ | -------------------------------------------------- |
| Recommendation oscillates                 | Prometheus step aliasing or restart spikes       | `memHysteresis` on the Recommendation CRD          |
| Recommendation too high                   | Startup spike not filtered                       | `algorithm.sampleCount` and `algorithm.dataSource` |
| Pods pending after scale-down             | Scale-down removed labeled nodes too early       | Node age (10-min grace period)                     |
| Webhook not injecting resources           | Webhook pod not ready or `failurePolicy: Ignore` | Health page → Webhook check                        |
| `kubectl logs` fails "No agent available" | Konnectivity agent missing pool toleration       | Re-run `scripts/create-opsai-pool.sh`              |
| Cost shows \$0.00                         | Unknown machine type not in catalog              | Node `node.kubernetes.io/instance-type` label      |

## Controller not scaling nodes

1. Check if a NodeScalingPolicy exists:

   ```bash theme={null}
   kubectl get nodescalingpolicies -A   # shorthand: kubectl get nsp -A
   ```

2. Check the policy status:

   ```bash theme={null}
   kubectl describe nodescalingpolicy <name> -n <namespace>
   ```

   Look at `status.conditions` for error messages and `status.lastScaleAction`.

3. Check controller logs:

   ```bash theme={null}
   kubectl -n ops-ai-system logs deploy/ops-ai-controller --tail=100
   ```

   Look for:

   * `"failed to evaluate scaling"` — metric collection or evaluation errors
   * `"cooldown active"` — scaling is blocked by the cooldown period
   * `"dry-run mode"` — scaling is blocked by the dry-run setting

4. Verify GKE credentials:

   ```bash theme={null}
   kubectl -n ops-ai-system describe pod -l app=ops-ai-controller | grep -A5 "Environment"
   ```

   Ensure `GKE_PROJECT_ID`, `GKE_LOCATION`, and `GKE_CLUSTER_NAME` are set.

5. Verify Prometheus connectivity:

   ```bash theme={null}
   kubectl -n ops-ai-system exec deploy/ops-ai-controller -- \
     wget -qO- http://prometheus.monitoring:9090/-/healthy
   ```

## Pods not rebalancing

1. Check if a RebalancePolicy exists:

   ```bash theme={null}
   kubectl get rebalancepolicies -A   # shorthand: kubectl get rbp -A
   ```

2. Check the imbalance score — it may be below the threshold:

   ```bash theme={null}
   opsctl --show-nodes
   ```

   If all nodes show similar utilization percentages, the cluster is already balanced.

3. Check for PDB conflicts. If `spec.respectPDB` is `true` (default), pods protected by PodDisruptionBudgets that would be violated are not moved:

   ```bash theme={null}
   kubectl get pdb -A
   ```

4. Check `status.lastPlan` on the RebalancePolicy for the last analysis result.

5. Verify the schedule expression is correct — the controller uses standard cron syntax.

## Autoscaler not responding to pending pods

1. Check if the autoscaler pod is running:

   ```bash theme={null}
   kubectl -n ops-ai-system get pods -l app=ops-ai-autoscaler
   ```

2. Check autoscaler logs for pending pod detection:

   ```bash theme={null}
   kubectl -n ops-ai-system logs deploy/ops-ai-autoscaler --tail=100 | grep -i pending
   ```

3. Verify the `pendingPodGracePeriod` — pods must be pending for at least this duration (default 30s) before triggering a scale-up.

4. Check if `maxNodes` has been reached:

   ```bash theme={null}
   kubectl get nodes | wc -l
   ```

## CRD issues

If CRDs are missing or outdated:

```bash theme={null}
kubectl apply -f deploy/helm/ops-ai-controller/crds/
kubectl get crd rebalancepolicies.ops-ai.io -o jsonpath='{.spec.versions[*].name}'
```

If the controller fails to start with scheme registration errors, CRDs may need to be
reinstalled before the controller pod starts.

## Leader election issues

If running multiple replicas with leader election enabled:

```bash theme={null}
kubectl -n ops-ai-system get lease ops-ai-controller-leader -o yaml
```

If leader election is stuck, delete the lease to force re-election:

```bash theme={null}
kubectl -n ops-ai-system delete lease ops-ai-controller-leader
```