Node Scale-Down Behavior

​

Overview

​

Scaler scale-down (continuous)

• Average managed utilization reaches the scaleDownThreshold (30%)
• The cooldown period (cooldownPeriod, default 2m) hasn’t elapsed since the last scale action
• Pending pods exist that aren’t stale (stable count for < 3 evaluation cycles)
• An active rebalance run is in progress
• The node count would drop below minNodes

​

Node target selection guards

1. It has PVC-backed pods — databases, Redis, or any pod with a PersistentVolumeClaim. These pods cause outages when evicted (volume detach/reattach takes 30–60 seconds, plus WAL replay and cache warming). PVC nodes are only consolidated during maintenance windows.
2. It is the sole node with a taint profile and has workload pods — removing it would make pods with that toleration permanently Pending until a replacement is provisioned (3–5 minute cold start).
3. It was provisioned < 10 minutes ago — Prometheus metrics haven’t stabilized yet. Without this, new nodes appear idle and get removed immediately.
4. It is protected by an active rebalance run — the rebalancer provisioned this node and it may still be receiving migrated pods.
5. It is not a managed node — GKE node pool nodes are never removed by the scaler.

​

What happens during scale-down

1. Node is cordoned (marked unschedulable) — no new pods land on it.
2. Node is drained — existing pods are evicted via the K8s Eviction API (respects PodDisruptionBudgets).
3. VolumeAttachments are detached — remaining volume bindings are cleaned up so PVCs can reattach on other nodes immediately.
4. Pod CIDR route is removed — the alias IP range is released from the GCE NIC.
5. GCE instance is deleted.
6. K8s node object is removed.

​

Maintenance consolidation (scheduled)

spec:
  maintenanceWindows:
    - name: nightly-cleanup
      schedule:
        type: Weekly
        startTime: "03:15"
        endTime: "05:00"
        timezone: "America/Los_Angeles"
        daysOfWeek: [Monday, Tuesday, Wednesday, Thursday, Friday]
      consolidation:
        maxNodesPercent: 10
        maxParallel: 2

• Traffic is at its lowest.
• The operations team expects potential disruptions during this window.
• The consolidation respects PDBs and waits for new pods to be Ready.

​

Interaction between scale-down and maintenance

1. During the day: the scaler removes empty/stateless-only nodes continuously. Nodes with PVC pods are left alone even if underutilized.
2. At night: the maintenance consolidator runs during the configured window and handles the PVC nodes the scaler couldn’t touch. The evicted pods (including databases/Redis) reschedule on the remaining nodes.
3. After maintenance: the cluster is tightly packed. As new workloads arrive and the resizer shrinks pod requests, nodes become underutilized again — stateless-only nodes get removed continuously, PVC nodes wait for the next window.

​

Debugging scale-down

kubectl logs -n ops-ai deploy/ops-ai --tail=200 | grep "scale decision audit"

{"msg":"scale decision audit","action":"scale_down","reason":"average utilization below threshold: 18.5%","pendingPods":0,"managedNodes":35,"targetNodes":["ops-ai-c2dhighcpu4-..."]}

kubectl logs -n ops-ai deploy/ops-ai --tail=200 | grep "skipping scale-down"

• skipping scale-down of node with PVC pods — has a database/Redis
• skipping scale-down of sole taint profile node — last node serving a workload type
• skipping scale-down of recently-provisioned node — < 10 minutes old

​

Configuration reference