PodResizePolicy (cluster-scoped)
How it works
- Every 5 minutes, the reconciler fetches 24h of usage data per workload.
- It computes percentile-based recommendations (P99 for memory, Max or P95 for CPU).
- It applies a configurable buffer (default 15%) for headroom.
- If the change exceeds the sensitivity threshold, it applies an in-place resize (K8s 1.33+) or records a recommendation.
The formula
- CPU uses the maximum observed usage over 24h, plus a 15% buffer.
- Memory takes the higher of: P99 of actual usage (+15% buffer) or P99 of 1-minute peaks (+7.5% buffer).
recommendationBuffer on the PodResizePolicy CRD.
Why these choices
Why Max for CPU (not P99)? CPU is compressible — if you’re wrong, the pod gets throttled, not killed. Using Max covers worst-case spikes. If the pod has an HPA, Autopilot switches to P95 instead (let the HPA handle the top 5% via horizontal scaling). Why P99 for memory (not Max)? Memory is not compressible — if you’re wrong, the pod gets OOM-killed. But absolute Max is unstable across Prometheus step-grid shifts (the same spike appears and disappears as the query window moves). P99 filters step-aliasing noise while still covering 99% of actual usage. Why a separate peak query for memory? The main memory query uses instantaneous values (good for percentiles). Short spikes (30–60 seconds) can be missed between Prometheus scrape points. Themax_over_time[1m] query captures these spikes. Autopilot takes P99 of the
peaks (not absolute max) for stability, with half the buffer (7.5%) since the peak query
already inflates values slightly.
Why 15% buffer and not target utilization? A targetUtilization divider double-inflated
recommendations when combined with the buffer. The old formula
percentile × (1 + buffer) / targetUtil — with a 15% buffer and 70% target — is 64%
overhead. The current formula percentile × (1 + buffer) is exactly 15% overhead. The
targetUtilization field still exists on the CRD but is display-only.
What gets filtered out
- Startup spikes — the first 5 minutes after pod creation are ignored (JIT warmup, cache loading, connection pools).
- Container restart spikes — samples within 5 minutes of a container restart are filtered (same initialization noise).
- Workload aggregation — data from old pod incarnations (same Deployment, prior restarts) is merged so you get the full 24h history even after a restart.
When recommendations change
A recommendation doesn’t change on every evaluation. Five stability mechanisms prevent churn:- Change sensitivity — the change must exceed a dynamic threshold (~25% for tiny pods like 64Mi, ~5% for large pods like 16Gi). Small fluctuations are ignored.
- Hysteresis — if the rounded recommendation bounces between two buckets (e.g. 128Mi ↔ 192Mi), the previous value is held when the pre-rounding value is near the boundary.
- Decrease bias — memory decreases require the value to move at least 50% of the way toward the new recommendation before being accepted. Memory is not compressible, so shrinks are conservative.
- P99 of peaks — filters the Prometheus step-aliasing that would cause Max to oscillate.
- Split queries — instantaneous values for percentiles,
max_over_timeonly for Max, preventing peak inflation of the P50–P99 range.
Rounding
- CPU: nearest 25m (below 500m) or 100m (500m+)
- Memory: nearest 32Mi (below 256Mi) or 64Mi (256Mi+)
Key behaviors
- HPA-aware — switches to P95 for CPU on HPA-managed workloads (lets the HPA handle spikes).
- OOM-responsive — bumps memory on OOM kills, never shrinks below OOM-bumped values.
- Change sensitivity — dynamic threshold prevents churn.
- Startup filtering — ignores the first 5 min of data after pod creation or container restart.