Skip to content

fix(xds): back off and jitter reconnects during overload - #37

Open
Kuromesi wants to merge 1 commit into
openkruise:masterfrom
Kuromesi:codex/reconnect-recovery-20261106
Open

Kuromesi wants to merge 1 commit into
openkruise:masterfrom
Kuromesi:codex/reconnect-recovery-20261106

Conversation

@Kuromesi

@Kuromesi Kuromesi commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Overload rejection, early stream closure, and some other ADS failures can cause clients to reconnect too aggressively or retain a large backoff after a successful configuration exchange. Large groups of clients also retry Sandbox restoration at the same fixed interval.

Apply exponential backoff consistently, use a nominal one-second minimum for ResourceExhausted returned during either stream establishment or streaming, and add ±20% jitter with an actual delay capped at 15 seconds. Reset accumulated backoff after an accepted response and retain fast retries for healthy stream completion. Jitter automatic Sandbox restoration retries from five seconds to four–six seconds; the per-attempt 30-second preparation timeout is unchanged.

Add TLS/gRPC regression tests covering overload, early EOF, and backoff reset after an accepted response.

Validation:

  • All 24 xDS-filtered tests and 76 Sandbox-filtered tests passed on my-ecs (the filtered sets overlap).
  • Release build completed.
  • Verified the patched image in 20 real injected ztunnels in ACK, covering already-bound Sandboxes and bindings delayed by 10 or 40 seconds. All ultimately prepared successfully; the 40-second cases required retry after the existing 30-second timeout. Sustained native traffic completed 44,422 requests without errors.
  • The 100,000-connection ACK load test used simulated clients rather than 100,000 real prepare RPCs; it does not establish the real prepare endpoint's throughput limit.

Test reports and generated load-test artifacts are excluded from this PR.

Signed-off-by: Kuromesi <blackfacepan@163.com>
@kruise-bot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from kuromesi by writing /assign @kuromesi in a comment. For more information see:The Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants