[monotek]

Problem description

The operator rotates Postgres pods itself (the StatefulSet uses updateStrategy: OnDelete). After deleting a pod, recreatePods only waits for the new pod to get its Patroni role label, and Patroni sets that label from inside the Spilo container. Pod readiness and sidecar containers are never checked.

So a broken sidecar image is rolled to every member of a cluster. We hit this with an exporter sidecar whose new image crashlooped:

1. Replica -1 was recreated, and its exporter started crashlooping.
2. 44s later the operator deleted replica -2, although -1's exporter had already restarted several times.
3. The operator switched over from -0 to -1, recreated -0, and emitted Rolling update done.

The result: every pod was NotReady. A single rotated member should have been enough to notice, but the operator kept going.

This PR adds an opt-in readiness check for rolling updates. It is off by default, so current behaviour does not change.

#1424 raised the concern that waiting for ready pods could keep users from fixing a broken config. That does not happen here: any new spec change, such as reverting the broken image, marks every pod for rotation again, so no pod counts as already rotated and the rollout proceeds. TestSyncStatefulSetRollingUpdateReadinessCheck covers that case.

Summary

Two new options in the kubernetes section:

• enable_rolling_update_readiness_check (default false)
• rolling_update_min_ready_time (default 30s): how long a recreated pod must stay Ready before the rollout continues. A container crash resets the Ready condition and with it this time, so a sidecar that crashes shortly after starting is caught too.

Within one rollout:

 recreatePods
   for each replica
-    recreatePod                    # waits for the role label only
+    recreatePodForRollingUpdate
+      recreatePod
+      waitForPodAvailable          # if enabled
+        fail at once if any container restarted
+        wait until Ready for rolling_update_min_ready_time
+        give up after resource_check_timeout -> rollout stops
   switchover
-  recreatePod(old master)
+  recreatePodForRollingUpdate(old master)

Across syncs: a stopped rollout leaves the remaining pods annotated with zalando-postgres-operator-rolling-update-required. Without a check, the next sync would rotate the next pod anyway. So:

 syncStatefulSet
   podsToRecreate = pods with rolling-update annotation
+  if enabled and a pod without the annotation is not available
+    postpone recreation ("already rotated pod(s) not available: ...")
   recreatePods(...)

MigrateMasterPod, used when draining a node, is unchanged.

Evidence

Unit tests at the levels the existing tests already use:

TestRecreatePodsReadinessCheck                  (2 replicas, the first is recreated broken)
  crashlooping container         -> 1 deletion, error "container has restarted"
  never Ready                    -> 1 deletion, error "did not become available within"
  Ready < min ready time         -> 1 deletion, error "did not become available within"
  crashlooping old master        -> error reported
  pod GET fails once             -> retried, both pods rotated
  check disabled, crashlooping   -> 2 deletions (today's behaviour)

TestNotAvailableRotatedPods
  rotated pods not Ready, or Ready < min ready time, hold the rollout
  annotated pods (incl. lazy upgrade) are ignored

TestSyncStatefulSetRollingUpdateReadinessCheck  (full sync, mocked Patroni API)
  rotated master not Ready           -> 0 pod deletions, rollout postponed
  rotated master Ready               -> pending replica is recreated
  new spec change (image revert)     -> rollout proceeds despite a NotReady pod

• Before: with the check disabled, the crashlooping case rotates both pods.
  After: with it enabled, the rollout stops after the first pod. I also disabled the sync check temporarily, and TestSyncStatefulSetRollingUpdateReadinessCheck failed as it should. go test ./... passes.

Merge Danger

Door: two-way

Opt-in and off by default. Turning it off restores the current behaviour.

Blast Radius: none by default; stalled rollouts when enabled

With the check enabled, a pod that is NotReady for an unrelated reason holds the rollout until it becomes ready, for example a lagging replica when enable_readiness_probe is on. Within one rollout, each pod also takes up to rolling_update_min_ready_time longer.

Linked issues

• Related to Sidecar without image specification causes all PostgreSQL pods to shutdown #2966: a broken sidecar spec takes down every pod, one per sync.
• Overlaps with fix(sync): prevent rolling update when unexpected pods are down #3112, which adds a similar always-on check before a rollout starts. This PR also stops a rollout that is already running, which fix(sync): prevent rolling update when unexpected pods are down #3112 doesn't cover: in our case all members were rotated within a single recreatePods call.

Checklist

Thanks for submitting a pull request to the Postgres Operator project.
Please, ensure your contribution matches the following items:

• Your go code is formatted. Your IDE should do it automatically for you.
• You have updated generated code when introducing new fields to the acid.zalan.do api package.
• New configuration options are reflected in CRD validation, helm charts and sample manifests.
• New functionality is covered by unit and/or e2e tests.
• You have checked existing open PRs for possible overlay and referenced them.

🤖 Generated with Claude Code