Repository navigation
Commit 2d5f9ed
authored
feat(gax): support transparent retries during mTLS certificate rotations (#13995)
## Description
This PR adds transparent retries for mTLS workload certificate rotation
to the gRPC and HTTP/JSON transports. When a request fails with
`UNAUTHENTICATED` and the workload certificate on disk has changed, the
transport is rebuilt with the new certificate and the request is retried
once, without interrupting in-flight RPCs or streams.
### 🚀 Core Features & Architectural Updates
• Rotation detection: `CertificateRotationTracker` compares the SHA-256
fingerprint of the workload certificate on disk with the certificate the
transport was built with. The check only runs after an auth failure
(never on the request path), and positive results are cached for at most
1 second so a burst of failures doesn't trigger a burst of file reads.
Empty or mid-write files are ignored.
• gRPC: `ChannelPool` replaces all of its channels when a rotation is
detected. Calls already in flight finish on their old channels, which
are shut down once idle. During a rotation refresh, a channel that can't
be recreated is dropped rather than kept, so no traffic keeps using the
old certificate; a statically sized pool is refilled in the background.
• HTTP/JSON: `RefreshingHttpJsonChannel` swaps the channel's
`HttpTransport` for one built with the new certificate. Calls already
created keep using the transport they started with, so in-flight
requests aren't interrupted.
• Retries: `AttemptCallable` (unary) and
`ServerStreamingAttemptCallable` (server streaming) refresh the
transport after an `UNAUTHENTICATED` failure. If the transport moved to
a new certificate during or after the attempt, the failure is flagged
for a single immediate retry on the refreshed channel; its configured
`isRetryable()` value is unchanged.
• `ApiResultRetryAlgorithm` gives that retry once, immediately, without
using a regular attempt.
• If the free retry also fails after another rotation, the call stops
instead of falling back to backoff retries.
• For server streams, the free retry becomes available again once the
stream has made progress, and the stream resumes through the existing
resumption strategy.
• Client and bidi streams refresh the transport on `UNAUTHENTICATED` but
don't retry; the next stream uses the new certificate.
• Plumbing: `TransportChannel` gains `shouldRefresh()`, `refresh()` and
`getGeneration()`, which default to no-ops.
`ApiCallContext.getTransportChannel()` gives retrying callables access
to the channel, and `GrpcCallContext` / `HttpJsonCallContext` carry it
through `merge()` and `withChannel()`.
### 🔒 System Hardening & Bug Fixes
• Outstanding RPC leak (`ChannelPool.ReleasingClientCall`): if a call
was cancelled before `start()`, `start()` threw without releasing the
channel entry, leaving the channel with a permanently outstanding RPC
count so it could never be cleaned up after a refresh. The entry is now
released on that path and on cancellation before start.
• HTTP/JSON refresh vs. shutdown: `shutdown()` / `shutdownNow()` are
serialized with `refresh()` so a transport swap can't race with
teardown.
### 1 parent acc7c9d commit 2d5f9ed
46 files changed
Lines changed: 6748 additions & 273 deletions
File tree
- google-auth-library-java/oauth2_http
- javatests/com/google/auth/mtls
- java/com/google/auth/mtls
- sdk-platform-java/gax-java
- gax-grpc
- src
- main/java/com/google/api/gax/grpc
- test/java/com/google/api/gax/grpc
- gax-httpjson/src
- main/java/com/google/api/gax/httpjson
- test/java/com/google/api/gax/httpjson
- gax/src
- main/java/com/google/api/gax
- retrying
- rpc
- mtls
- test/java/com/google/api/gax/rpc
- mtls
- testing
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
41 | 45 | | |
42 | 46 | | |
| 47 | + | |
43 | 48 | | |
44 | 49 | | |
45 | 50 | | |
| |||
57 | 62 | | |
58 | 63 | | |
59 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
60 | 227 | | |
61 | 228 | | |
62 | 229 | | |
| |||
65 | 232 | | |
66 | 233 | | |
67 | 234 | | |
68 | | - | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
69 | 238 | | |
70 | 239 | | |
71 | 240 | | |
72 | 241 | | |
73 | 242 | | |
74 | 243 | | |
75 | | - | |
| 244 | + | |
| 245 | + | |
76 | 246 | | |
77 | 247 | | |
78 | 248 | | |
| |||
92 | 262 | | |
93 | 263 | | |
94 | 264 | | |
95 | | - | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
96 | 268 | | |
97 | 269 | | |
98 | 270 | | |
| |||
106 | 278 | | |
107 | 279 | | |
108 | 280 | | |
109 | | - | |
| 281 | + | |
110 | 282 | | |
111 | 283 | | |
112 | 284 | | |
| |||
0 commit comments