us-congress-pipeline

ETL that mirrors the workings of the US Congress as git repositories — the text of federal law with its history as commits, every bill as a branch, tags marking the law at a point in time, and each commit carrying who sponsored the measure and where it had got to.

Concretely: sponsor, cosponsor count and every roll-call vote are commit trailers, and sponsor, cosponsors, committees, actions and votes are in metadata.md beside the text, each filtered to the version it sits on. How each member voted is a file of its own under votes/, appearing on the commit for the text that was before the chamber when it voted.

Votes come from the chambers themselves — clerk.house.gov and senate.gov — not from the Congress.gov API, which publishes House votes only, for the 118th and 119th Congresses only, against a corpus that starts at the 108th and holds both. Experimental amendment execution is built, and carries out 18.5% of the instructions it reads — phase 7 in the roadmap; the rest are listed with the reason rather than guessed at. Said plainly rather than left to be discovered, which is the rule the generated repositories follow in their own GAPS.md.

This repository contains only the pipeline. It generates the data repositories; it does not contain them.

Why this exists

There is no actively maintained git repository of US federal law. A wave of them appeared in March–April 2026 and every one went silent within weeks; the older, better-known projects (divegeek/uscode, unitedstates/uscode) are archived. Meanwhile the upstream government feeds — govinfo bulk data, OLRC release points, the Congress.gov API — are healthy and publishing daily.

Maintained is the whole claim, so STATUS.md carries the date this last updated. That file is the point: a project like this does not die with an error. A disabled schedule or an expired token raises nothing and notifies nobody, so nothing has to fire for you to notice — the date simply stops moving. See Staying alive.

Repositories it produces

See REPOSITORIES.md for the full index, including live status for each repository. It is generated by uv run uscongress index from src/uscongress/registry.py, so it reports what actually exists rather than what is planned — repositories not yet created are shown as such.

Repository	Phase	Contents	State
us-congress-code	1	the codified US Code; one commit per OLRC release point, tagged	built
us-congress-bills-{congress}	2	one repo per Congress; one branch per measure	12 built, 108–119
us-congress-statutes	5	Statutes at Large, 136 volumes and 102,234 laws	built
us-congress-comps	11	Statute Compilations, one commit per daily snapshot	built
us-congress-record-{congress}	6	Congressional Record as text, 1994 to present	17 built, 103–119

Only repositories that exist are linked here; the rest are named but not linked, because a link to a repository that has not been created yet is a 404. REPOSITORIES.md is generated from live GitHub state and carries the full list, including a direct link to every shard.

Every repository is prefixed us-congress-. Sharding by Congress is a deliberate choice, not a capacity limit: measured against govinfo, the twelve Congresses from the 108th hold 171,881 measures in 6.60 GB of XML, which one repository could carry. GitHub publishes no branch-count ceiling, and the whole corpus packs to roughly 1–2 GB.

It is sharded because a finished Congress never changes again, so frozen shards never rebuild and their clones stay valid; because a defect in recent data should not force a rebuild of 2003; because reading the 118th should not mean downloading 6.6 GB; and because twelve repositories build in parallel where one serializes. Branch counts run 10,637 to 19,315 per Congress.

All of these repositories are public. The federal text they carry is public domain under 17 U.S.C. § 105; each one states its own terms in a LICENSE file, and the pipeline's own terms are separate — see License.

What this is — and what it is not

Diffing across a bill branch's own commits — hr-1234~2..hr-1234, or against the commit that introduced it — shows how the bill changed, not how the US Code would change. Bills are written as amendatory instructions ("strike subsection (b) and insert…"), not as diffs, and executing them automatically is an unsolved problem.

derived/amendments.md on each branch does the part that is solvable and stops there. Across the whole corpus it read 1,155,101 amendatory instructions and carried out 214,139 of them (18.5%) — the ones where the bill states both the text removed and the text inserted, so the result follows from the bill alone and can be checked against it. The other 940,962 are listed with the reason, most often that the bill names the law by structure — strike subsection (k) — and the words being changed are in the US Code rather than in the bill. Nothing guesses them.

The rate is mostly a fact about the year, not the bill. An instruction can only be placed if GPO tagged the citation it names: 64% of the 108th Congress's documents carry a machine-readable US Code citation and 55% of the 113th's, against 5% of the 111th's and 5% of the 112th's. So the share carried out runs from 1.3% in the 112th to 23.9% in the 113th with no change in the reading of them at all.

Everything under derived/ is marked derived and unofficial, and unapplied instructions are stated rather than guessed at.

Usage

Requires uv, which installs the Python pinned in .python-version if you do not already have it.

Starting from a fresh clone, get the generated repositories rather than rebuilding them. data/ is gitignored and runs to about 93 GB on a machine that has been building, but the repositories already exist on GitHub and cloning them takes minutes where a rebuild takes days — the Congressional Record alone is a 1.37 million request crawl:

uv run uscongress bootstrap        # clone the 32 data repos into data/repos/

The cached upstream XML under data/raw/ is deliberately not restored and does not need to be: it is an optimization, not state, and every job refetches what it needs. See CLAUDE.md for the full orientation.

uv sync
cp .env.example .env                  # then add your govinfo key

uv run uscongress comps               # snapshot Statute Compilations (do this first)
uv run uscongress comps --fresh       # ignore today's manifest, refetch everything
uv run uscongress releasepoints       # list the current year's OLRC release points, oldest first
uv run uscongress seed-code           # build every listed release point not yet tagged
uv run uscongress seed-code --limit 5 # build only the oldest 5
uv run uscongress seed-bills --congress 113   # build us-congress-bills-113
uv run uscongress seed-bills --congress 113 --limit 25   # first 25 measures only
uv run uscongress seed-bills --congress 113 --rebuild    # rewrite every branch from its root
uv run uscongress republish --congress 113 --dry-run     # what a rebuild changed vs GitHub
uv run uscongress republish --congress 113               # force-push only what moved
uv run uscongress seed-statutes       # build us-congress-statutes, volumes 1–138
uv run uscongress seed-record --congress 115  # build us-congress-record-115
uv run uscongress bootstrap           # clone the generated repos instead of rebuilding
uv run uscongress seed-record --congress 115 --limit 5   # first 5 issue days per edition
uv run uscongress index               # regenerate REPOSITORIES.md

uv run uscongress update              # the daily job: rebuild whatever changed
uv run uscongress update --dry-run    # list what changed upstream, build nothing
uv run uscongress update --check      # exit non-zero if the daily loop has stopped
uv run uscongress update --state-path /tmp/x.json --status-path /tmp/x.md
                                      # a local run that leaves the tracked heartbeat alone

uv run uscongress update-record       # the second daily job: the current Congress's Record
uv run uscongress update-record --publish    # …and push the issue days it added

uv run uscongress attention           # what needs a person, computed from live state
uv run uscongress attention --check   # …and change nothing; exits non-zero if due

uv run pytest                         # tests

Every job is resumable and idempotent: comps skips packages already recorded, seed-code skips release points whose tag exists, seed-bills skips measures whose branch exists, seed-statutes skips volumes whose tag exists, seed-record skips issue days already committed, and all of them cache their downloads. Re-running a completed job fetches nothing — re-running a finished Congress takes seconds and rebuilds no branches.

OLRC's listing is not the history. When us-congress-code was built, priorreleasepoints.htm listed 386 release points but only 383 were distinct — pl-113-165, pl-115-95not91 and pl-115-117not91not96not97 each appeared twice at adjacent positions. The tag check that makes seed-code resumable absorbed the duplicates, so the build came to 383 commits and 383 tags, not 386. That page was replaced in October 2026 by priorreleasepoints.html, which names only the current year's release points: seed-code can still extend the repository but could no longer rebuild it from nothing, so its tags, not OLRC's page, are the complete list.

Inside a bills repository

Each measure is a branch named from its citation — hr-588, s-1339, sconres-13 — holding one commit per text version, oldest first. Two files per commit:

File	Contents
bill.md	the bill text as of that version
metadata.md	sponsor, cosponsors, committees and actions as of that version

metadata.md is filtered to the version's date on purpose. BILLSTATUS is a single present-day snapshot of the whole measure, so writing it unfiltered would have a bill's introduced text already reporting that it became law — the same trap as Table III's present-day classification in us-congress-code.

Committees follow the same rule and name their chamber, because both chambers run an Armed Services, a Judiciary and an Appropriations Committee. H.R. 7283 of the 119th was referred to House Oversight on 2026-01-30 and to Senate Homeland Security on 2026-07-23, so only the first appears on its introduced version — a bill that had not yet passed the House should not already be sitting in a Senate committee.

A main branch carries GAPS.md, listing measures that have no branch because govinfo records them in BILLSTATUS but links no text. This is not evenly spread, and the older Congresses are mostly gaps:

Congress	branches	no text
108	1,912	8,755
109	10,766	2,302
110	13,724	314
111	13,520	155
112–119	120,260	173 total

Ordering comes from BILLSTATUS textVersions, not from the bill documents: action-date is absent from engrossed, enrolled and received versions, and a reported version repeats the introduction date, so the files cannot sequence themselves. Where BILLSTATUS gives no date — usually the enrolled bill — the preceding version's date is carried forward and the commit message says so.

Bill text is legacy bill.dtd XML, not USLM. GPO publishes a parallel USLM 2.0 tree, but it covers 1.8% of versions (2,443 of 134,013), so billtext.py renders the legacy format.

Inside the statutes repository

One commit per volume, holding every session law that volume prints as one Markdown file each. Laws live at volume-NNN/{public,private,resolutions,organic}/, named the way they are cited: public-law-108-1.md, private-law-82-1.md, and chapter-1-1-i.md for the numbered chapters that preceded public-law numbering.

Tags are named for the citation — stat-001 to stat-138, as in 117 Stat. — rather than matching the directories. volume-117 would be both a tag and a path, and git refuses an argument that is both: git log volume-117 fails with ambiguous argument. The tag moved rather than the directory, because the directory listing is what a reader browses first.

Per-public-law commits would have made git log the chronology of enactment since 1789, and were rejected: a session law is printed as passed and never amended, so no commit would ever touch what an earlier one wrote, and ~102,000 commits of pure addition is a sorted list rather than a history. The one diff that is real here is GPO's re-transcription — volume 1 was re-digitised on 2025-11-03 — and that is expressible only when the volume is the commit unit.

Commit dates before 1970 are all 1970-01-01. git stores no timestamp earlier than the Unix epoch: git commit refuses 1799-03-03 outright, and writing a negative one through fast-import succeeds only for git log to render it blank. 82 of the 138 volumes close before then, so each commit's subject line carries the years instead, and every law carries its own approved: date.

Statutes at Large XML is USLM 2.0 (GPO), not USLM 1.0 (OLRC), and statutetext.py is a sibling of render.py rather than an extension of it. The namespace differs, but the reason is that the printed page puts marginalia inside the sentence: 5,293 <sidenote> elements in volume 1 alone, one of them opening between "United" and "States" in the 1789 oath act. Flattening with itertext(), which is right for the US Code, splices a marginal note into the middle of the clause and produces fluent English that is not the law.

GAPS.md records what is deliberately absent — chiefly the 13,387 treaties, proclamations and executive agreements the volumes also print, which are not acts of Congress. Volumes 7 and 8 contain nothing else and so have no commit at all.

Inside a Congressional Record repository

Three branches. daily is CREC, the edition printed overnight; bound is CRECB, the permanent edition GPO republishes years later with corrections folded in and pages renumbered into one continuous run; main carries the artifacts and GAPS.md. They are two publications of the same proceedings rather than versions of one document, so keeping them apart makes git diff bound daily -- 2018/07-23/ answer a real question — what changed between what was said and what was printed permanently.

One commit per issue day, and the history accumulates: each commit adds YYYY/MM-DD/{senate,house,extensions,daily-digest}/NNN-slug.md and leaves every earlier day in place, so git log is the legislative calendar and a commit's diff is exactly what was published that day. The Record is a serial publication — an issue succeeds its predecessor rather than revising it — so replacing the tree each commit, which is what fast-import does by default, would leave the branch holding only the most recent issue.

Each document carries its granule, page, 172 Cong. Rec. S4415-style citation, the members recorded as speaking with their bioguide identifiers, and the measures it refers to. All of that comes from one MODS request per package rather than one per granule: a 288-granule day is 3.4 MB of MODS against 288 separate calls, which across the corpus is the difference between a 34-hour crawl and a 67-hour one. Measures are named as plain citations, not links — us-congress-bills exists only for the 108th to the 119th, and the Record reaches back further.

The machine-readable Record begins in 1994, not 1873. CRECB holds 2,420 volume parts back to 1873 and only the 337 named CRECB-{year}-pt{n} (1999–2018) have an HTML rendition; the other 2,083 are scanned page images whose /htm answers HTTP 400. CREC starts on 1994-01-01. So a shard for a Congress that ended before 1994 is empty, and its GAPS.md says so rather than leaving the absence to be read as a build that failed. Shards from the 116th on have no bound branch either — the newest bound volume is 2018.

Placement is the trap. A day can be several packages: 11 March 2025 is published as three overlapping ones whose granules share 267 identifiers, and 4 January 2023 is two genuinely distinct issues. They are merged per day and deduplicated on the granule identifier, which is the only stable identity. And a package is placed by its own declared Congress, never by its date — CREC-2025-01-03-v170 is dated the day the 119th convened and declares the 118th, which adjourned sine die that morning. Both shards hold a 2025/01-03/, which is what actually happened.

Changing Python or a dependency

A changed byte in a rendered file is a changed SHA on a published commit, so render the cache under both toolchains and compare before switching. Python 3.15 was adopted this way: 520,091 files, not one different.

uv run uscongress render-digest /tmp/before              # in the current environment
# in a worktree with the new .python-version or lockfile:
uv --directory <worktree> run uscongress render-digest /tmp/after --raw "$PWD/data/raw"
uv run uscongress render-compare /tmp/before /tmp/after  # exits 1 if anything differs

It reads data/raw/ only, and records a cache miss rather than fetching it, so both runs read exactly the same inputs.

Nothing upgrades by itself. Every dependency, the build backend and each GitHub Action is pinned to an exact version, CI installs with uv sync --locked, and tests/test_toolchain.py fails on a range. Dependabot opens a pull request a week per ecosystem, plus security fixes as they appear; no workflow tests them, so run uv run pytest — and the comparison above, for a change that could alter rendering — before merging one.

Staying alive

uv run uscongress update is the daily job, and it runs in .github/workflows/update.yml rather than on a machine somewhere, because the run history of a public workflow is itself a liveness record that anyone can read.

Only what changed is rebuilt. govinfo's incremental collections endpoint reported 1,183 BILLSTATUS packages modified in the six days to 2026-08-07 — about 170 a day, against roughly 18,000 to re-poll a Congress blindly. Each changed measure has its branch rewritten from the root, not appended to: BILLSTATUS is one present-day snapshot and metadata.md is filtered to each version's date, so a correction upstream can change what an older commit should say, and appending cannot express that.

Rewriting costs nothing when nothing moved. The render is a pure function of the source document and the commit timestamps come from the version dates, so an unchanged measure rebuilds to byte-identical commits with the same SHAs, git records no change, and there is nothing to push. That is what makes the hour of deliberate overlap on every run free, and it is why the watermark can advance only on success: a crash re-fetches rather than skips, and re-fetching is cheap while missing a bill is not.

The failure mode is silence, so the signal is inverted. Nothing has to fire:

Where	What it says
STATUS.md	written every run, successful or not. A date that stops moving is legible to a stranger who knows nothing about this project
update --check	exits non-zero once the last success is more than two days old, like check-links and describe --check
the workflow	raises a GitHub notification on failure by itself
state/update.json	the watermark, committed rather than gitignored — a scheduled runner is a fresh machine every time, so git log state/update.json is the history of the loop
state/ci-*.txt	written in shell by a step that always runs, so a failure early enough to skip the Python job still commits. Without it a broken dependency produces red runs and no repository activity, and GitHub disables a schedule after 60 days of that — the mechanism taking itself down
uscongress attention	what a schedule cannot do, computed rather than remembered. Renders a Needs a person section on STATUS.md and keeps one GitHub issue in step with it

Committing the heartbeat also keeps the schedule alive: GitHub disables a scheduled workflow after 60 days without repository activity, and the job's own output is what prevents that.

The Congressional Record has a loop of its own, in .github/workflows/record.yml, two hours behind the first. It is separate because the two fail differently and one of them is the liveness signal: a Record fetch error must not be able to take the bills heartbeat down with it. Only the bills loop renders STATUS.md, and it renders both heartbeats — from state/update.json and state/record.json. That is the point. A Record loop that has stopped shows up as a date that no longer moves on a page something else is still writing every morning, because neither loop can be trusted to report its own death.

That loop is append-only: it asks which issue days a shard already holds and builds the rest, never --rebuild. The Record is one cumulative branch, so rewriting a day in the middle rewrites every commit after it — and govinfo restamps already-published days in bulk without changing them, nine of them on 2026-08-12 with 1,469 documents before and after. Reacting to those stamps would force-push the entire history to produce identical trees.

The credential is scoped to the repositories that existed when it was made. DATA_REPO_TOKEN is a fine-grained token carrying Contents: read/write on the us-congress-* repositories and nothing else, so a compromised workflow can push data and do nothing more. It does not expire. The one thing it cannot do is reach a repository created after it: when the 120th Congress convenes and update builds us-congress-bills-120, the push will fail until that repository is added to the token's list. That fails loudly — the run fails, GitHub notifies, the heartbeat records it, and the watermark holds so nothing is skipped — which is the intended shape of the error rather than a surprise to debug.

New US Code release points are reported, not built. A release point is a full snapshot of ~60,000 files built against the one before it — the guard that stops a truncated archive recording 336 repeals and reversing them two commits later compares the two trees — so it is built where that history already is, with seed-code. The daily job puts the backlog on STATUS.md instead of hiding it.

Why comps runs first

govinfo replaces Statute Compilations in place and keeps no version archive. Once a compilation is superseded, the previous text is gone. Non-codified law — the Social Security Act, for example, which bills amend by act section rather than by US Code citation — exists nowhere else in versioned form. Every day without a snapshot is history that cannot be recovered, which makes this the only genuinely time-sensitive job in the project.

Snapshots are content-addressed: blobs live under data/comps/objects/ keyed by SHA-256, with one manifest per run under data/comps/snapshots/. A second run against an unchanged collection costs one manifest, not another 270 MB.

Layout

src/uscongress/
├── config.py        filesystem layout, credentials, rate limits
├── govinfo.py       rate-limited, retrying async client
├── gitbuild.py      GitRepo and the fast-import writer
├── render.py        USLM 1.0 → Markdown (the US Code)
├── billtext.py      legacy bill-DTD XML → Markdown (bills)
├── statutetext.py   USLM 2.0 → Markdown (the Statutes at Large)
├── xmlrepair.py     unbalanced tags and bare ampersands in government XML
├── registry.py      the repositories, and the roadmap beside them
└── jobs/
    ├── comps.py     Statute Compilations snapshot
    ├── uscode.py    → us-congress-code
    ├── bills.py     → us-congress-bills-{congress}
    ├── statutes.py  → us-congress-statutes
    ├── record.py    → us-congress-record-{congress}
    ├── table3.py    per-law attribution trailers
    ├── update.py    the daily loop, and both loops' heartbeats
    ├── recordloop.py the Congressional Record's own daily loop, append-only
    ├── attention.py what needs a person, computed rather than remembered
    ├── publish.py   fetching and pushing refs, with GitHub's real failure modes
    ├── index.py     REPOSITORIES.md
    ├── artifacts.py README and LICENSE into each generated repo
    ├── describe.py  GitHub description and topics
    ├── links.py     check-links
    └── renderdigest.py what two toolchains render, compared
state/update.json    the watermark — committed, so the loop survives a fresh runner
state/record.json    the Record loop's watermark, rendered onto STATUS.md by the bills loop
data/                gitignored — corpora and generated repos (~50 GB)

data/ is gitignored in full. It holds the fetched corpora and the generated repositories, which carry their own git histories; nesting them inside this repository's history would be painful to undo. state/ is the exception that is not gitignored: it is a few hundred bytes, and a watermark a scheduled runner cannot carry between runs is not a watermark.

Data sources

Federal and public domain under 17 U.S.C. § 105, with one exception noted below.

• US Code — OLRC release points, USLM 1.0 XML (uscode.house.gov/download/)

• Statutes at Large / PLAW / COMPS — govinfo bulk data, USLM 2.0 XML

• Bills — govinfo BILLS (text) and BILLSTATUS (metadata, back to the 108th Congress)

• Attribution — OLRC Table III, mapping public-law sections to US Code sections. Note it records present-day classification: PL 113-40 (2013) is listed under Title 54, which did not exist until December 2014. Trailers answer "where does this law live now", not "what did this commit change" — for that, read the diff.

• Roll-call votes — the chambers themselves: clerk.house.gov/evs/ and senate.gov/legislative/LIS/roll_call_votes/. BILLSTATUS names every vote taken on a measure and links it to the chamber that took it, so the index is already in the bill metadata and only the positions are fetched. Neither host is keyed.

• Members — unitedstates/congress-legislators (CC0), for the LIS↔bioguide crosswalk. The one source here not published by the federal government, and the only one that is vendored rather than fetched: it is edited continuously upstream, and reading it live would re-render every affected vote file the day someone corrects a spelling. A pinned extract lives in src/uscongress/members.py — 328 rows, small enough to read in review. A mapping whose surname and state do not agree with the vote document is refused rather than guessed.

Not the Congress.gov API. It was listed here as a source for years and is read nowhere: its roll-call endpoint covers the House alone for the 118th and 119th Congresses, and this corpus starts at the 108th and holds both chambers.

Two mutually incompatible USLM schemas are in production: OLRC emits v1.0.15, GPO emits v2.0. The pipeline needs both, and GPO is not internally consistent either — the 137 Statutes at Large volumes declare four different minor versions (2.0.10, 2.0.12, 2.0.13, 2.0.17).

Bulk listing endpoints return HTTP 406 without an Accept: application/json header — they work in a browser and fail from a script. The client handles this.

The same trap runs the other way for files, and there it is silent rather than fatal. STATUTE/107, 108 and 109 answer HTTP 200 with 67,225 bytes of "Govinfo Bulkdata Service Error" HTML under a default Accept: */*, while the directory listing for the same path advertises 13.7 MB of application/xml. With Accept: application/xml all three return in full. No other volume does it, so a sampled check would never see it — which is why every fetch validates its payload before caching, rather than trusting the status code.

License

This pipeline is proprietary — © 2026 Jade Naaman, all rights reserved; see LICENSE. No license is granted. Terms may be opened up later; nothing here is offered under an open-source license today.

The generated repositories are licensed separately, because they hold two different kinds of material. The federal text is a work of the United States Government, so 17 U.S.C. § 105 puts it in the public domain and no rights are asserted over it — claiming otherwise would be a claim with nothing behind it. The layer this project authored around that text — the Markdown conversion, the structure of the history, the commit messages and the explanatory prose — is © 2026 Jade Naaman, all rights reserved. Each generated repository carries its own LICENSE saying exactly that.

Keeping the generated repositories self-describing

uv run uscongress artifacts         # write README.md and LICENSE into every generated repo
uv run uscongress describe          # set each repo's GitHub description and topics
uv run uscongress describe --check  # report any whose metadata has drifted
uv run uscongress check-links       # verify every link in every generated document resolves

Publishing a new repository — after seed-code or seed-bills has built it:

1. gh repo create junxit/<name> --private
2. uv run uscongress artifacts — README and license
3. uv run uscongress describe — GitHub description, homepage and topics
4. push its branches, then set its default branch to main
5. uv run uscongress check-links and uscongress describe --check to confirm

Steps 2, 3 and 5 derive everything from src/uscongress/registry.py, so a repository added there is covered without anything being written by hand. describe --check and check-links both exit non-zero when something has drifted, so neither depends on anyone remembering to look.

Three things GitHub gets wrong on its own, all handled in jobs/publish.py so the daily job does not have to rediscover them:

• A repository's default branch becomes the first branch pushed, which for a bills repository is hconres-1 — landing every visitor on a random bill instead of the README. It is set to main explicitly.
• A single push of ~10,000 refs is rejected outright with Internal Server Error, atomically, after transferring everything: zero refs land. Pushes go out in batches of 2,000.
• Individual refs fail transiently with the same error — about one in 2,000 during the 119th — while git reports the push as successful. Push output is therefore not evidence: what landed is read back with ls-remote, compared against local, and the difference is pushed again.

A generated repository is published without its pipeline, so it has to explain itself: what a branch means, how to read a diff, what the data does not say, and where it came from. artifacts writes that into each one, along with the cross-reference table linking back here and across to the sibling repositories, so the set stays navigable as phases land.

It is idempotent — the counts it embeds deliberately exclude the commit it makes, so re-running it after a new phase changes only what actually changed.