Skip to content

Commit 2d5f9ed

Browse files
authored
feat(gax): support transparent retries during mTLS certificate rotations (#13995)
## Description This PR adds transparent retries for mTLS workload certificate rotation to the gRPC and HTTP/JSON transports. When a request fails with `UNAUTHENTICATED` and the workload certificate on disk has changed, the transport is rebuilt with the new certificate and the request is retried once, without interrupting in-flight RPCs or streams. ### 🚀 Core Features & Architectural Updates • Rotation detection: `CertificateRotationTracker` compares the SHA-256 fingerprint of the workload certificate on disk with the certificate the transport was built with. The check only runs after an auth failure (never on the request path), and positive results are cached for at most 1 second so a burst of failures doesn't trigger a burst of file reads. Empty or mid-write files are ignored. • gRPC: `ChannelPool` replaces all of its channels when a rotation is detected. Calls already in flight finish on their old channels, which are shut down once idle. During a rotation refresh, a channel that can't be recreated is dropped rather than kept, so no traffic keeps using the old certificate; a statically sized pool is refilled in the background. • HTTP/JSON: `RefreshingHttpJsonChannel` swaps the channel's `HttpTransport` for one built with the new certificate. Calls already created keep using the transport they started with, so in-flight requests aren't interrupted. • Retries: `AttemptCallable` (unary) and `ServerStreamingAttemptCallable` (server streaming) refresh the transport after an `UNAUTHENTICATED` failure. If the transport moved to a new certificate during or after the attempt, the failure is flagged for a single immediate retry on the refreshed channel; its configured `isRetryable()` value is unchanged. • `ApiResultRetryAlgorithm` gives that retry once, immediately, without using a regular attempt. • If the free retry also fails after another rotation, the call stops instead of falling back to backoff retries. • For server streams, the free retry becomes available again once the stream has made progress, and the stream resumes through the existing resumption strategy. • Client and bidi streams refresh the transport on `UNAUTHENTICATED` but don't retry; the next stream uses the new certificate. • Plumbing: `TransportChannel` gains `shouldRefresh()`, `refresh()` and `getGeneration()`, which default to no-ops. `ApiCallContext.getTransportChannel()` gives retrying callables access to the channel, and `GrpcCallContext` / `HttpJsonCallContext` carry it through `merge()` and `withChannel()`. ### 🔒 System Hardening & Bug Fixes • Outstanding RPC leak (`ChannelPool.ReleasingClientCall`): if a call was cancelled before `start()`, `start()` threw without releasing the channel entry, leaving the channel with a permanently outstanding RPC count so it could never be cleaned up after a refresh. The entry is now released on that path and on cancellation before start. • HTTP/JSON refresh vs. shutdown: `shutdown()` / `shutdownNow()` are serialized with `refresh()` so a transport swap can't race with teardown. ### ⚠️ Behavioral & Security Boundary Changes * mTLS is enabled automatically when a workload certificate config is present (`MtlsUtils`, auth library): * When `GOOGLE_API_CERTIFICATE_CONFIG`, or the default gcloud `certificate_config.json`, contains a `workload` certificate, client certificates are used even if `GOOGLE_API_USE_CLIENT_CERTIFICATE` is unset. * `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` still turns mTLS off. * mTLS misconfiguration now fails closed (`MtlsUtils`): * Previously, an invalid explicit config was swallowed and the client silently connected without mTLS. * Now client creation throws `IllegalStateException` in these cases: * the explicit `GOOGLE_API_CERTIFICATE_CONFIG` file is missing, unreadable or malformed; * the default config file exists but is unreadable or malformed; * a configured certificate or key file can't be read. * When mTLS is required but the transport can't load the client certificate, gRPC and HTTP/JSON channel creation fails with an `IOException` instead of falling back to a non-mTLS connection. ### 🧪 Testing #### Automated Testing • Added and updated comprehensive unit-tests reflecting the thread-safety fixes inside ChannelPoolTest.java and RefreshingHttpJsonChannelTest.java. • Corrected edge case test configurations to leverage realistic mocked X.509 certificates to properly exercise deep WorkloadCertificateUtils. getCertificateFingerprint() filesystem caching mechanisms. #### Manual Testing End-to-end tests with real generated GAPIC clients against local mTLS servers, run on this PR's head. Each scenario runs in its own JVM and checks the exact sequence of client certificates and outcomes the server saw. * **Setup** * Clients: `KeyManagementServiceClient` over gRPC and HTTP/JSON, and `GrpcBigQueryReadStub` for server-streaming `ReadRows` with offset-based resumption. * Servers: local gRPC and HTTPS servers that require a client certificate and return `UNAUTHENTICATED` / 401 for "revoked" certificates (by CN). * Rotation: the workload certificate under `GOOGLE_API_CERTIFICATE_CONFIG` is replaced by atomic rename, key first, then cert. * **gRPC unary:** * basic rotation: one rejected attempt, then a free retry on the new cert, and no extra attempts afterwards; * the server rejects every cert: one free retry, then stop; * a second rotation during the free retry: stops after 2 attempts, with no backoff retry; * UNAVAILABLE, then rotation; * corrupt cert on disk: no crash, the call fails, and the client recovers once a valid cert is written; * 20-call bursts across 2 rotations less than 1s apart: all succeed with exactly 1 refresh each; * a 3-channel pool: no stale channel after refresh; * 8 threads across 4 rotations, then `close()` with calls in flight: 70,624/70,624 calls succeeded, no hangs. * **gRPC server streaming:** * rotation before the stream starts; * rotation mid-stream: resumes at the right offset with no gaps or duplicates; * rotation retry re-armed after the stream makes progress, following an earlier UNAVAILABLE retry; * bounded: a second rotation without progress stops the stream, with no third attempt. * **HTTP/JSON:** basic rotation; reject-all; double rotation; a slow call started before rotation completes on the old cert while new calls use the new cert; corrupt cert, then restore; 8-thread stress with 4 rotations, then `close()` (49,561/49,561 calls succeeded, no hangs). The socket factory was verified for both Conscrypt and plain JDK TLS. * **No mTLS** (`GOOGLE_API_USE_CLIENT_CERTIFICATE=false`, or no certificate config): a single attempt with no client cert and no refresh, over both gRPC and HTTP/JSON, even when the cert files on disk change. * **Results:** * JDK 26 (JDK TLS): all 22 scenarios pass. * JDK 21 (Conscrypt): all gRPC, streaming and no-mTLS scenarios pass. * HTTP/JSON with mTLS and Conscrypt on TLS 1.3 fails with an existing `Unknown authType: GENERIC` handshake error, which #14556 fixes. With #14556 applied on top of this PR, all 22 scenarios pass on JDK 21 with Conscrypt active. * Harness source: https://paste.googleplex.com/6120902048219136 * Logs: https://paste.googleplex.com/5467233782988800 (this PR on JDK 26 and JDK 21), https://paste.googleplex.com/5563956459077632 (this PR + #14556 on JDK 21)
1 parent acc7c9d commit 2d5f9ed

46 files changed

Lines changed: 6748 additions & 273 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎google-auth-library-java/oauth2_http/java/com/google/auth/mtls/MtlsUtils.java‎

Lines changed: 176 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,17 @@
3434
import com.google.auth.oauth2.EnvironmentProvider;
3535
import com.google.auth.oauth2.PropertyProvider;
3636
import com.google.common.base.Strings;
37+
import com.google.common.io.BaseEncoding;
3738
import java.io.File;
3839
import java.io.FileInputStream;
3940
import java.io.IOException;
4041
import java.io.InputStream;
42+
import java.nio.file.Files;
43+
import java.nio.file.Paths;
44+
import java.security.MessageDigest;
4145
import java.util.Locale;
4246
import org.jspecify.annotations.NullMarked;
47+
import org.jspecify.annotations.Nullable;
4348

4449
/**
4550
* Utility class for mTLS related operations.
@@ -57,6 +62,168 @@ private MtlsUtils() {
5762
// Prevent instantiation for Utility class
5863
}
5964

65+
/**
66+
* Returns if mutual TLS client certificate should be used. Returns true if valid workload
67+
* certificates are configured or if GOOGLE_API_USE_CLIENT_CERTIFICATE is explicitly set to true
68+
* (e.g. for Enterprise Certificate Proxy or custom MtlsProviders), unless explicitly disabled via
69+
* GOOGLE_API_USE_CLIENT_CERTIFICATE=false.
70+
*/
71+
public static boolean useMtlsClientCertificate(
72+
EnvironmentProvider envProvider, PropertyProvider propProvider) {
73+
String useClientCertificate = envProvider.getEnv("GOOGLE_API_USE_CLIENT_CERTIFICATE");
74+
if ("false".equalsIgnoreCase(useClientCertificate)) {
75+
return false;
76+
}
77+
if (getWorkloadCertPath(envProvider, propProvider) != null) {
78+
return true;
79+
}
80+
return "true".equalsIgnoreCase(useClientCertificate);
81+
}
82+
83+
/**
84+
* Resolves and returns the path to the mutual TLS client certificate, or null if none should be
85+
* used.
86+
*
87+
* <p>Possible outcomes:
88+
*
89+
* <ol>
90+
* <li><b>Non-null {@link String} (Valid happy path):</b> A valid workload certificate
91+
* configuration was found and both the certificate and private key files exist and are
92+
* readable.
93+
* <li><b>{@link IllegalStateException} (Invalid state - fail closed):</b> An explicit {@code
94+
* GOOGLE_API_CERTIFICATE_CONFIG} path or an existing default well-known certificate
95+
* configuration file is missing, unreadable, malformed, or references missing/unreadable
96+
* certificate or private key files. This is treated as an unrecoverable misconfiguration.
97+
* <li><b>{@code null} (Safe fallback / fail open):</b> Client certificates are explicitly
98+
* disabled via {@code GOOGLE_API_USE_CLIENT_CERTIFICATE=false}, no explicit configuration
99+
* is set and the default well-known configuration file does not exist on disk, or the
100+
* configuration specifies an non-workload source (e.g., ECP/PKCS11 without a {@code
101+
* workload} section). Callers can proceed without workload certificate file polling.
102+
* </ol>
103+
*/
104+
public static @Nullable String getWorkloadCertPath(
105+
EnvironmentProvider envProvider, PropertyProvider propProvider) {
106+
String useClientCertificate = envProvider.getEnv("GOOGLE_API_USE_CLIENT_CERTIFICATE");
107+
if ("false".equalsIgnoreCase(useClientCertificate)) {
108+
return null;
109+
}
110+
111+
String explicitConfigPath = envProvider.getEnv(CERTIFICATE_CONFIGURATION_ENV_VARIABLE);
112+
113+
// 1. Explicit Configuration Path (Fail Closed)
114+
if (!Strings.isNullOrEmpty(explicitConfigPath)) {
115+
File configFile = new File(explicitConfigPath);
116+
if (!configFile.exists()) {
117+
throw new IllegalStateException(
118+
"Certificate configuration file specified via GOOGLE_API_CERTIFICATE_CONFIG at '"
119+
+ explicitConfigPath
120+
+ "' does not exist.");
121+
}
122+
if (!configFile.isFile() || !configFile.canRead()) {
123+
throw new IllegalStateException(
124+
"Failed to read certificate configuration file specified via"
125+
+ " GOOGLE_API_CERTIFICATE_CONFIG at '"
126+
+ explicitConfigPath
127+
+ "'.");
128+
}
129+
WorkloadCertificateConfiguration config;
130+
try {
131+
config = getWorkloadCertificateConfiguration(envProvider, propProvider, explicitConfigPath);
132+
} catch (CertificateSourceUnavailableException e) {
133+
// ECP / PKCS11 configuration without workload section; safe fallback
134+
return null;
135+
} catch (Exception e) {
136+
throw new IllegalStateException(
137+
"Certificate configuration file specified via GOOGLE_API_CERTIFICATE_CONFIG at '"
138+
+ explicitConfigPath
139+
+ "' is malformed: "
140+
+ e.getMessage(),
141+
e);
142+
}
143+
checkCertAndKeyFilesReadable(config, explicitConfigPath, false);
144+
return config.getCertPath();
145+
}
146+
147+
// 2. Implicit / Default gcloud Configuration Path
148+
File defaultConfigFile = null;
149+
try {
150+
defaultConfigFile = getWellKnownCertificateConfigFile(envProvider, propProvider);
151+
} catch (IOException e) {
152+
// APPDATA missing on Windows, etc. Safe fallback.
153+
}
154+
if (defaultConfigFile != null && defaultConfigFile.exists()) {
155+
if (!defaultConfigFile.isFile() || !defaultConfigFile.canRead()) {
156+
throw new IllegalStateException(
157+
"Default certificate configuration file at '"
158+
+ defaultConfigFile.getAbsolutePath()
159+
+ "' exists but could not be read.");
160+
}
161+
WorkloadCertificateConfiguration config = null;
162+
try {
163+
config = getWorkloadCertificateConfiguration(envProvider, propProvider, null);
164+
} catch (CertificateSourceUnavailableException e) {
165+
// ECP-only configuration without workload section; safe fallback
166+
} catch (Exception e) {
167+
throw new IllegalStateException(
168+
"Default certificate configuration file at '"
169+
+ defaultConfigFile.getAbsolutePath()
170+
+ "' is malformed: "
171+
+ e.getMessage(),
172+
e);
173+
}
174+
if (config != null) {
175+
checkCertAndKeyFilesReadable(config, defaultConfigFile.getAbsolutePath(), true);
176+
return config.getCertPath();
177+
}
178+
}
179+
180+
return null;
181+
}
182+
183+
private static void checkCertAndKeyFilesReadable(
184+
WorkloadCertificateConfiguration config, String configPath, boolean isDefaultConfig) {
185+
File certFile = new File(config.getCertPath());
186+
File keyFile = new File(config.getPrivateKeyPath());
187+
if (!certFile.isFile() || !certFile.canRead() || !keyFile.isFile() || !keyFile.canRead()) {
188+
String sourcePrefix =
189+
isDefaultConfig
190+
? "referenced by default configuration '"
191+
: "referenced by configuration '";
192+
throw new IllegalStateException(
193+
"Failed to read certificate/key file at '"
194+
+ config.getCertPath()
195+
+ "' or '"
196+
+ config.getPrivateKeyPath()
197+
+ "' "
198+
+ sourcePrefix
199+
+ configPath
200+
+ "'.");
201+
}
202+
}
203+
204+
/**
205+
* Computes the lower-case SHA-256 hex fingerprint of the certificate file at {@code certPath}.
206+
*
207+
* <p>Unlike {@link #getWorkloadCertPath}, which validates configuration at channel initialization
208+
* and fails closed on errors, this method is called dynamically at runtime during active RPCs to
209+
* detect certificate rotations on disk. External certificate rotators may temporarily delete,
210+
* truncate, or rewrite the certificate file mid-RPC. Returning {@code null} on read/digest
211+
* exceptions (which callers normalize to {@code ""}) allows runtime refresh checks to ignore
212+
* transient mid-write states and keep the active healthy channel without failing in-flight RPCs.
213+
*/
214+
public static @Nullable String getCertificateFingerprint(@Nullable String certPath) {
215+
if (certPath == null) {
216+
return null;
217+
}
218+
try {
219+
byte[] certBytes = Files.readAllBytes(Paths.get(certPath));
220+
byte[] digest = MessageDigest.getInstance("SHA-256").digest(certBytes);
221+
return BaseEncoding.base16().lowerCase().encode(digest);
222+
} catch (Exception e) {
223+
return null;
224+
}
225+
}
226+
60227
/**
61228
* Returns the path to the client certificate file specified by the loaded workload certificate
62229
* configuration.
@@ -65,14 +232,17 @@ private MtlsUtils() {
65232
* @throws IOException if the certificate configuration cannot be found or loaded.
66233
*/
67234
public static String getCertificatePath(
68-
EnvironmentProvider envProvider, PropertyProvider propProvider, String certConfigPathOverride)
235+
EnvironmentProvider envProvider,
236+
PropertyProvider propProvider,
237+
@Nullable String certConfigPathOverride)
69238
throws IOException {
70239
String certPath =
71240
getWorkloadCertificateConfiguration(envProvider, propProvider, certConfigPathOverride)
72241
.getCertPath();
73242
if (Strings.isNullOrEmpty(certPath)) {
74243
throw new CertificateSourceUnavailableException(
75-
"Certificate configuration loaded successfully, but does not contain a 'certificate_file' path.");
244+
"Certificate configuration loaded successfully, but does not contain a"
245+
+ " 'cert_configs.workload.cert_path' path.");
76246
}
77247
return certPath;
78248
}
@@ -92,7 +262,9 @@ public static String getCertificatePath(
92262
* @throws IOException if the configuration file cannot be found, read, or parsed
93263
*/
94264
static WorkloadCertificateConfiguration getWorkloadCertificateConfiguration(
95-
EnvironmentProvider envProvider, PropertyProvider propProvider, String certConfigPathOverride)
265+
EnvironmentProvider envProvider,
266+
PropertyProvider propProvider,
267+
@Nullable String certConfigPathOverride)
96268
throws IOException {
97269
File certConfig;
98270
if (certConfigPathOverride != null) {
@@ -106,7 +278,7 @@ static WorkloadCertificateConfiguration getWorkloadCertificateConfiguration(
106278
}
107279
}
108280

109-
if (!certConfig.isFile()) {
281+
if (!certConfig.isFile() || !certConfig.canRead()) {
110282
throw new CertificateSourceUnavailableException(
111283
"Certificate configuration file does not exist or is not a file: "
112284
+ certConfig.getAbsolutePath());

0 commit comments

Comments
 (0)