Skip to content

fix(transport): stop retrying certificate failures and name the fix (#461) - #463

Draft
thodson-usgs wants to merge 1 commit into
DOI-USGS:mainfrom
thodson-usgs:fix/ssl-error-message
Draft

thodson-usgs wants to merge 1 commit into
DOI-USGS:mainfrom
thodson-usgs:fix/ssl-error-message

Conversation

@thodson-usgs

@thodson-usgs thodson-usgs commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #461.

Problem

When a network proxy or firewall re-signs HTTPS traffic (common on agency and corporate networks), Python rejects the server certificate with CERTIFICATE_VERIFY_FAILED. Before this change:

  • The Water Data, NGWMN, and NWDC getters classed the failure as transient. They spent the retry budget, then raised ServiceInterrupted, which tells the caller to resume once the service recovers.
  • The legacy getters (nwis, wqp) raised a bare NetworkError that gave no remedy.
  • An unchunked OGC query reported Could not reach the service at unknown service.

Fix

The message gives two ways to trust the root certificate that the interception proxy uses:

  1. pip install truststore, then import truststore; truststore.inject_into_ssl() before the first request. This verifies against the operating system's trust store, where IT installs the organization root. truststore's docs say inject_into_ssl() is for applications and scripts, not libraries, so the library only suggests it to the user and never calls it.
  2. Set SSL_CERT_FILE to a PEM bundle that contains the organization's root certificate.

No dependency is added.

What changed

  • interruptions._deterministic_failure treats ssl.SSLCertVerificationError as deterministic, so it is neither retried nor offered as resumable. A new _certificate_rejected() predicate sits next to it, and transport.http.network_error() uses that predicate to append the remedy. One predicate decides both behaviors, so they can't drift apart.
  • FanOut._normalize_failure falls back to the failed request's URL when the plan has no canonical_url.
  • Added a "Certificate verification failures" section to userguide/errors.rst, and a NEWS entry.

Verification

  • Offline tests in tests/transport_test.py cover: not retried and not resumable, exactly one attempt, the remedy text, no remedy on other connection failures, and the URL fallback.
  • Ran it live by pointing SSL_CERT_FILE at a self-signed bundle. waterdata.get_daily and wqp.what_sites now raise NetworkError on the first attempt and name the real URL. In a scratch venv (not added to the project), I made certifi's bundle invalid and confirmed that truststore.inject_into_ssl() alone makes get_daily succeed.
  • All gates pass: ruff, mypy, lint-imports, xenon, complexipy, and the full suite (1764 passed, coverage 99.08%).

Known gap (follow-up)

NetworkError.retryable is a class-level True. A caller that branches on e.retryable, as the errors guide suggests, will still retry a certificate failure. Permanent DNS failures already behave this way. Fixing it would mean a per-instance retryable or a new subclass, which changes the public exception taxonomy, so I left it out of this PR.

…OI-USGS#461)

A server certificate that Python does not trust, usually because a network
proxy or firewall re-signs HTTPS with an organization root that certifi does
not contain, was classed as transient. The fan-out getters spent their retry
budget on it and raised ServiceInterrupted, telling the caller to resume once
the service recovered.

ssl.SSLCertVerificationError is now a deterministic failure, so it is neither
retried nor resumable, and network_error() appends a remedy: verify against
the operating system's certificates with truststore, or set SSL_CERT_FILE to
a bundle containing the organization's root certificate. No dependency is
added.

A NetworkError from an unchunked OGC query also names the request URL
instead of "unknown service".
@thodson-usgs
thodson-usgs force-pushed the fix/ssl-error-message branch from 51564b2 to 0bc2d30 Compare November 4, 2026 17:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add error message for SSL errors

1 participant