> ## Documentation Index
> Fetch the complete documentation index at: https://autopilot.docs.xano.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Node Scale-Down Behavior

> How and when Autopilot removes managed nodes — continuous scale-down, scheduled maintenance consolidation, and the guards that protect workloads.

This page explains how and when Autopilot removes managed nodes from a cluster.
Understanding it is important for capacity planning and debugging unexpected pod restarts.

## Overview

Autopilot has two distinct paths for removing nodes:

| Path                          | When it runs                          | What it targets                                  | Pod impact                                     |
| ----------------------------- | ------------------------------------- | ------------------------------------------------ | ---------------------------------------------- |
| **Scaler scale-down**         | Continuously (every 60s)              | Empty or stateless-only managed nodes            | Minimal — stateless pods reschedule in seconds |
| **Maintenance consolidation** | During configured maintenance windows | Any node, including those with stateful/PVC pods | Controlled — runs during low-traffic hours     |

## Scaler scale-down (continuous)

The NodeScalingPolicy reconciler evaluates managed node utilization every \~60 seconds. When
average managed node utilization drops below the `scaleDownThreshold` (default 30%), the
scaler selects the lowest-utilized managed nodes for removal.

**How many nodes per cycle:**

| Average managed utilization | Fraction removed       |
| --------------------------- | ---------------------- |
| \< 5%                       | 50% of managed nodes   |
| 5–15%                       | 33% of managed nodes   |
| 15–30%                      | 20% of managed nodes   |
| ≥ 30%                       | None (above threshold) |

**Consolidation mode:** Even when average utilization is above 30%, the scaler checks
whether removing one node would keep projected utilization below 80% of the scale-up
threshold. If so, it consolidates.

**What stops scale-down:**

* Average managed utilization reaches the `scaleDownThreshold` (30%)
* The cooldown period (`cooldownPeriod`, default 2m) hasn't elapsed since the last scale action
* Pending pods exist that aren't stale (stable count for \< 3 evaluation cycles)
* An active rebalance run is in progress
* The node count would drop below `minNodes`

### Node target selection guards

Before actually removing a node the evaluator selected, the reconciler applies several
guards. A node is **skipped** if:

1. **It has PVC-backed pods** — databases, Redis, or any pod with a PersistentVolumeClaim. These pods cause outages when evicted (volume detach/reattach takes 30–60 seconds, plus WAL replay and cache warming). PVC nodes are only consolidated during maintenance windows.
2. **It is the sole node with a taint profile and has workload pods** — removing it would make pods with that toleration permanently Pending until a replacement is provisioned (3–5 minute cold start).
3. **It was provisioned \< 10 minutes ago** — Prometheus metrics haven't stabilized yet. Without this, new nodes appear idle and get removed immediately.
4. **It is protected by an active rebalance run** — the rebalancer provisioned this node and it may still be receiving migrated pods.
5. **It is not a managed node** — GKE node pool nodes are never removed by the scaler.

### What happens during scale-down

1. Node is **cordoned** (marked unschedulable) — no new pods land on it.
2. Node is **drained** — existing pods are evicted via the K8s Eviction API (respects PodDisruptionBudgets).
3. **VolumeAttachments are detached** — remaining volume bindings are cleaned up so PVCs can reattach on other nodes immediately.
4. **Pod CIDR route is removed** — the alias IP range is released from the GCE NIC.
5. **GCE instance is deleted.**
6. **K8s node object is removed.**

Evicted pods reschedule on other nodes with available capacity. For stateless pods (no
PVCs), this takes 1–5 seconds and is invisible to users.

## Maintenance consolidation (scheduled)

The RebalancePolicy CRD supports maintenance windows for deep consolidation. During a
window, the consolidator identifies nodes that can be emptied (including those with fragile
and PVC-backed pods) and drains them to reduce node count.

```yaml theme={null}
spec:
  maintenanceWindows:
    - name: nightly-cleanup
      schedule:
        type: Weekly
        startTime: "03:15"
        endTime: "05:00"
        timezone: "America/Los_Angeles"
        daysOfWeek: [Monday, Tuesday, Wednesday, Thursday, Friday]
      consolidation:
        maxNodesPercent: 10
        maxParallel: 2
```

Maintenance consolidation has no PVC guard — it consolidates everything, targeting the most
expensive nodes first. This is the correct time to move databases and Redis pods because:

* Traffic is at its lowest.
* The operations team expects potential disruptions during this window.
* The consolidation respects PDBs and waits for new pods to be Ready.

## Interaction between scale-down and maintenance

1. **During the day:** the scaler removes empty/stateless-only nodes continuously. Nodes with PVC pods are left alone even if underutilized.
2. **At night:** the maintenance consolidator runs during the configured window and handles the PVC nodes the scaler couldn't touch. The evicted pods (including databases/Redis) reschedule on the remaining nodes.
3. **After maintenance:** the cluster is tightly packed. As new workloads arrive and the resizer shrinks pod requests, nodes become underutilized again — stateless-only nodes get removed continuously, PVC nodes wait for the next window.

## Debugging scale-down

Check the controller logs for scale decision audits:

```bash theme={null}
kubectl logs -n ops-ai deploy/ops-ai --tail=200 | grep "scale decision audit"
```

Example output:

```json theme={null}
{"msg":"scale decision audit","action":"scale_down","reason":"average utilization below threshold: 18.5%","pendingPods":0,"managedNodes":35,"targetNodes":["ops-ai-c2dhighcpu4-..."]}
```

If a node was skipped, look for the guard that blocked it:

```bash theme={null}
kubectl logs -n ops-ai deploy/ops-ai --tail=200 | grep "skipping scale-down"
```

Common skip reasons:

* `skipping scale-down of node with PVC pods` — has a database/Redis
* `skipping scale-down of sole taint profile node` — last node serving a workload type
* `skipping scale-down of recently-provisioned node` — \< 10 minutes old

## Configuration reference

| Field                      | Default | Description                                         |
| -------------------------- | ------- | --------------------------------------------------- |
| `spec.scaleDownThreshold`  | 0.4     | Average utilization below which scale-down triggers |
| `spec.cooldownPeriod`      | 2m      | Minimum time between scale actions                  |
| `spec.minNodes`            | 1       | Never scale below this count                        |
| `spec.pendingPodThreshold` | 2       | Pending pods above this trigger scale-up            |