ccc's map is static: it links a test to the code the test calls. A test that reaches code another way is invisible to it, for example over HTTP to another service, through a framework, or through a table of handlers. A change in one service then leaves you running every suite that might touch it.
Runtime instrumentation fills that gap. Record one run of your suites with ccc's probes in, and ccc knows which tests actually executed which functions, across every service and language the tests passed through. For any change, it picks the tests that ran the changed code.
- What the examples prove
- How it works
- Instrumenting a project
- Recording in CI
- What is built and what isn't yet
examples/ holds two copies of the same blob-store SaaS (disclosure: all examples are generated by Agents): five Go services (gateway, auth, metadata, storage and billing) behind one public gateway, and 50 tests.
| Suite | Tests | How it reaches the services |
|---|---|---|
| Go unit | 20 | In process |
| Go integration | 6 | Over HTTP, through test servers |
| Python integration: customer flows | 12 | Over HTTP |
| Python contract: response shapes | 8 | Over HTTP |
| Python load: throughput and latency | 4 | Over HTTP |
- Project-A is the plain project. All ccc has is its static map.
- Project-BBB is the same source, byte for byte, plus ccc's probes and one recorded run of all 50 tests.
The same three changes went into each project. The numbers are how many of the 50 tests each project has to run.
| Scenario | Change | Changed functions | Without instrumentation (Project-A) | With: safe (Project-BBB) | With: minimal (Project-BBB) |
|---|---|---|---|---|---|
| 1 | metadata counts list and delete calls for a metrics endpoint | 2 | 50 | 11 | 2 |
| 2 | storage hashes chunks through a streaming hasher | 1 | 50 | 22 | 1 |
| 3 | a hardening release across all five services | 6 | 50 | 42 | 5 |
- safe: every test the recorded run saw execute a changed function.
- minimal: the cheapest tests that still execute every changed function at least once.
1. A simple change that still needs more than one test. Index.Delete and Index.List sit on different request paths.
- Without instrumentation, ccc's call graph links no test to either function. The blast radius covers all eight services, so every test runs.
- The recording shows 11 tests executing one of the two: 2 unit, 6 integration, 1 contract and 2 load.
minimalneeds two unit tests, one per path:
go test -run '^(TestDeleteObject|TestListObjectsSortedWithPrefix)$' ./services/metadata2. A one-function change deep in the stack. Checksum runs on every upload and download.
- The call graph finds the 4 storage unit tests that call it directly. It can't see the 18 tests that also run it over HTTP: 10 integration, 4 contract and 4 load.
- The recording sees all 22.
minimalneeds one test:
go test -run '^(TestPutGetChunk)$' ./services/storage3. A complex change across every service. It changes six functions in five services.
- The call graph links 2 tests, while 42 tests actually ran at least one changed function.
minimalneeds five, one per service:
go test -run '^(TestVerifyKnownKey|TestRecordAccumulatesUsage|TestJoinChunksReversesSplit|TestCreateBucketRejectsBadNames|TestPutGetChunk)$' \
./services/auth ./services/billing ./services/gateway ./services/metadata ./services/storage- Without runtime data, the suites that cross services can't be narrowed. Every scenario puts all eight services in ccc's blast radius, so every scenario runs the whole suite.
- The call graph's own short list isn't safe to run alone. It finds 0, 4 and 2 tests for the three scenarios, while 11, 22 and 42 tests actually ran the changed code.
- One recorded run is enough to narrow it. The safe set runs 78%, 56% and 16% fewer tests than the whole suite, and the minimal set 96%, 98% and 90% fewer.
- The services' source doesn't change. The probes go into a build-time copy, and with recording off the tests behave exactly as they do in Project-A.
minimal promises that the changed code runs, not that every regression is caught. It prefers cheap tests: cost is the recorded duration weighted by kind, from unit 1 up to load 6. When a change matters enough, run safe. In these scenarios minimal never needs a load test, while safe keeps every load test that ran the change.
The probes also count how many times each test ran each function. ccc prints the count beside every test it selects, as in Checksum ×2, and the VS Code hover shows it too. The counts don't change which tests are selected. They show how hard each test works the changed code:
| Scenario | minimal |
The busiest test in safe |
|---|---|---|
| 1 | TestDeleteObject: Delete ×1TestListObjectsSortedWithPrefix: List ×2 |
test_list_under_load: List ×101 |
| 2 | TestPutGetChunk: Checksum ×2 |
test_upload_throughput: Checksum ×600 |
| 3 | Five unit tests, each running its function 1 to 6 times | test_download_latency_p95: Verify ×202, PriceCents ×201, joinChunks ×200, Get ×200, validBucketName ×1 |
A minimal test runs the change a handful of times. A regression that only appears under repetition, such as a slowdown, a leak or a race, is far more likely to show in a test that runs the code hundreds of times.
To see it yourself, run bash examples/run-scenarios.sh. It needs only ccc, git and Python 3.
- Each test is a fixture. A fixture has the name the test runner selects it by, a kind (
unit,contract,interop,integration,smokeorload), its file and its language. - Its identity travels with its requests. Every request a fixture makes carries a W3C
baggageheader,ccc.fixture=…,ccc.file=…,ccc.lang=…,ccc.kind=…, and each service passes it on to the services it calls. - Services record what ran for it. Each function counts, on entry, one run for the fixture behind the current request.
- The records are
ccc-observed/1files. Each line covers one fixture in one process: the fixture, how long it took, and the functions it ran, by file, name and line, with how many times each ran. CLI.md has the format. - ccc reads them from
observedin.ccc/map.json, from--observed, or from.ccc/observed/. It ties each recorded function to its own map by file, name and owner, so a recording still matches after the code has moved. - ccc selects. In
ccc insightsand thetest_triggersMCP tool,test_triggers.observedlists the safe and minimal sets, each with its run commands.ccc changesand the VS Code hints count a function a fixture ran as tested, with evidenceobserved. Each of these shows how many times the fixture ran the function.
ccc probe writes an instrumented copy of the project. The source in the repository never changes.
ccc probe go,python . --out /tmp/probed
cd /tmp/probed
export CCC_OBSERVED_DIR=$PWD/observed
go test ./... && pytestCLI.md has every option. In the Go copy, for example:
-package storage
+package storage; import cccprobe "ccc.invalid/probe"
-func (s *Store) Put(id string, data []byte, checksum string) error {
+func (s *Store) Put(id string, data []byte, checksum string) error { cccprobe.Hit("services/storage/storage.go", "Store.Put", 39);
-func TestPutGetChunk(t *testing.T) {
+func TestPutGetChunk(t *testing.T) { cccprobe.Test(t, "services/storage/storage_test.go", "unit");
- srv := &http.Server{Handler: h, ReadHeaderTimeout: 5 * time.Second}
+ srv := &http.Server{Handler: cccprobe.Handler(h), ReadHeaderTimeout: 5 * time.Second}The probes are inserted as text, on the line they belong to, so every comment stays where it was and every line keeps its number. Each function is named the way ccc's own map names it, so the records match the map exactly.
What goes into the copy, by language:
| Language | Counts its runs | Names itself a fixture | Takes the fixture in | Sends the fixture out | Hands it to other threads |
|---|---|---|---|---|---|
| Go | every function and method except init |
TestXxx(t *testing.T), with its subtests |
net/http servers (http.Server, ListenAndServe, Serve, httptest), gRPC servers |
http.DefaultTransport, http.Client transports, gRPC dials |
goroutines, through the goroutine that started them (Go 1.21+) |
| Rust | every function except const fn |
#[test], #[tokio::test], #[rstest], #[test_case] and the like |
axum 0.7+ and tonic 0.12+ through a tower layer, actix-web through wrap_fn |
reqwest .send(), tonic::Request::new |
tokio::spawn, spawn_blocking, std::thread::spawn |
| C++ | every function except constexpr |
gtest TEST, TEST_F, TEST_P, Catch2 and doctest TEST_CASE, SCENARIO, Boost.Test |
cpp-httplib routes, grpc++ servers | cpp-httplib clients, grpc::CreateChannel |
std::thread, std::jthread, std::async |
| TypeScript | every function, method and arrow bound to a name | it, test and specify callbacks, under jest, vitest, mocha and node --test |
Node http and https servers (express, fastify, koa and the rest), grpc-js servers |
http and https requests (axios, got, node-fetch), fetch, grpc-js clients |
every async continuation, through AsyncLocalStorage |
| Python | every function and method | pytest tests, unittest methods | WSGI (Flask, Django, wsgiref), ASGI (Starlette, FastAPI, Django), http.server, grpcio |
http.client (urllib, requests, urllib3), httpx, aiohttp, grpcio |
threads and ThreadPoolExecutor |
How each runtime reaches the build:
- Go: a module,
ccc.invalid/probe, in.ccc-probe/go. Each probedgo.modrequires it and replaces it with that directory, and so dogo.workandvendor/where there are any. A module that requires gRPC also getsinternal/cccprobegrpc. - Rust: a crate,
ccc_probe, in.ccc-probe/rust, added to each probed package as a path dependency, with itstowerfeature on where axum or tonic is used. It needs Rust 1.70. A--lockedbuild needs its lock file updated first. - C++: a header in
.ccc-probe/cpp, included by relative path with a#lineafter it. No build file changes. It needs C++17. - TypeScript: a package in the copy's
node_modules/ccc-probe, imported by every probed file. The copy'snode_moduleslinks each installed package back to the original. - Python: a module in
.ccc-probe/pythonthat every probed file finds by itself. It registers with pytest as the test files load, so nothing needs installing.
A test's kind comes from its file name, then its directories, innermost first: contract, interop, integration (or e2e, it, acceptance, functional), smoke, and load (or perf, stress, bench). A Rust crate's tests/ directory is integration. Everything else is unit. --kind 'qa/**=smoke' overrides it.
Every probe stays off unless CCC_OBSERVED_DIR names a directory, so the copy runs unchanged otherwise. CCC_SERVICE names the process in its records, CCC_RUN names the run (local by default), and CCC_COMMIT overrides the commit the copy was made from.
A test's record is written when the test ends. A service writes what its fixtures ran every two seconds, and again when it exits.
Limits:
- Only functions the map names are counted. Function literals and anonymous lambdas aren't.
- Work that no test, request or parent claims goes to the process's one running test, and to nobody when two or more run at once. Run tests serially, for example with
--test-threads=1, to attribute everything. - A Rust async test on a multi-threaded runtime is followed only when the tests run serially.
- A grpc++ server binds the fixture on the thread that receives each call, so synchronous services are followed and callback services aren't. Its interceptor replaces any set on the builder before it.
- In a project that also has
.cfiles,.hfiles stay unprobed, because a C source might include them. - A Python function that opens with a block statement gets its count on a line of its own, so the lines after it shift in the copy. The records still carry each function's original line.
- JavaScript files aren't probed, only TypeScript.
On the default branch, nightly or on every merge:
ccc probe <languages> . --out $WORKmakes the instrumented copy.- Build and run every suite in
$WORKwithCCC_OBSERVED_DIRset. Start each service the suites call from the copy too, withCCC_SERVICEnaming it. - Stop the services cleanly, or give them two seconds to write what they ran.
cat $CCC_OBSERVED_DIR/*.jsonl > observed.jsonl, and publish the file as a build artifact.
Then point ccc at it:
"observed": [{ "from": "https://ci.example.com/artifacts/observed.jsonl", "auth": "env:CI_TOKEN" }]The example's record.sh predates ccc probe and uses the hand-written probes in examples/project-bbb/ccc. ccc probe go,python examples/project-a produces the same 50 records from the plain project, function for function and call for call.
ccc fetches a URL at most once every ten minutes. Records are matched by function name, so they stay useful as the code moves. A function written since the recording has no record yet. It's listed under unobserved, and the static call-graph list covers it until the next recording.
Available Now
- The
ccc-observed/1format and the reader. - Observed evidence in
ccc changes,test_triggersand the VS Code hints. - The safe and minimal selections, with run commands.
- How many times each test ran each changed function.
ccc probe, for Go, Rust, C++, TypeScript and Python, carrying the fixture over HTTP and gRPC.
Should be available in 2.0.0
- JavaScript files.
- Carrying the fixture through message queues.
- Line-level precision for
safe. - Drift reports that say when a test stops reaching what it used to.