Check regression coverage for added tests

.github/workflows/regression-check.yml

@@ -0,0 +1,49 @@
+name: Regression coverage
+
+permissions:
+  contents: read
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, labeled, unlabeled]
+
+jobs:
+  regression-check:
+    name: Added test regression check
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    env:
+      BUNDLE_GEMFILE: gemfiles/rails_8.1.gemfile
+    steps:
+    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
+      with:
+        fetch-depth: 0
+        persist-credentials: false
+    - name: Setup Ruby
+      uses: ruby/setup-ruby@7f797815d8adceb1c412e7abefdd1ab4abc47a5f # v1.335.0
+      with:
+        ruby-version: 4.0
+        bundler-cache: true
+    - name: Compare added and modified tests without the implementation
+      env:
+        BASE_SHA: ${{ github.event.pull_request.base.sha }}
+        REQUIRE_REGRESSION: ${{ contains(github.event.pull_request.labels.*.name, 'regression-test') }}
+      shell: bash
+      run: |
+        args=(--base "$BASE_SHA" --output tmp/regression-check)
+        if [ "$REQUIRE_REGRESSION" = true ]; then
+          args+=(--strict)
+        fi
+        ruby script/regression_check.rb "${args[@]}"
+    - name: Summarize regression results
+      if: always()
+      run: |
+        if [ -f tmp/regression-check/summary.md ]; then
+          cat tmp/regression-check/summary.md >> "$GITHUB_STEP_SUMMARY"
+        fi
+    - name: Upload regression results
+      uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5
+      if: always()
+      with:
+        name: regression-check
+        path: tmp/regression-check

docs/CHANGELOG.md

@@ -10,6 +10,10 @@ nav_order: 6
 
 ## main
 
+* Add an informational CI check that runs added and modified tests without a pull request's implementation changes to identify regression coverage.
+
+    *Joel Hawksley*
+
 * Speed up rendering with direct compiled-template calls and avoid intermediate method allocations when invoking lambda-backed slots.
 
     *Joel Hawksley*

docs/CONTRIBUTING.md

@@ -102,6 +102,39 @@ bundle exec ruby performance/measure.rb tmp/performance/head.json
 bundle exec ruby performance/compare.rb tmp/performance/base.json tmp/performance/head.json
 ```
 
+### Regression coverage
+
+The **Added test regression check** CI job runs added and modified Minitest tests on Ruby 4.0 and Rails 8.1. It repeats the same tests with packaged `app/` and `lib/` implementation files restored to the pull request's merge base. Both runs retain the pull request's dependencies, tests, and fixtures, including `app/views/test_mailer/`.
+
+Each test has its own result:
+
+| Result | Meaning |
+| --- | --- |
+| `demonstrated` | Passes on the pull request. An assertion fails without the implementation. |
+| `not_demonstrated` | Also passes without the implementation. Review whether it exercises the change. |
+| `head_failed` | Fails or errors on the pull request. |
+| `inconclusive` | Errors, skips, times out, makes no assertions, or doesn't run. Inspect the logs. |
+
+Results are informational by default. The `regression-test` label requires every selected test to demonstrate a regression, with no warnings or unsupported cases. An assertion failure demonstrates a behavioral difference, not necessarily a bug. New APIs can produce legitimate pre-fix exceptions. Those remain inconclusive rather than counting arbitrary errors as evidence.
+
+The job selects individual `def test_*` methods and literal Rails `test "..."` blocks using Prism. RSpec and dynamic definitions are reported as unsupported. Changes only to helpers, shared setup, or fixtures don't automatically select existing tests. Dependency changes are reported but not reverted. The Actions summary and `regression-check` artifact contain results, exceptions, and process logs.
+
+To run locally against committed changes:
+
+```command
+ruby script/regression_check.rb --base origin/main
+```
+
+Use `--strict` to require demonstrated regressions and `--install` to install missing dependencies in the temporary checkout. The working tree is never modified.
+
+To audit the latest 100 pull requests, including open and closed pull requests:
+
+```command
+ruby script/regression_check_history.rb --limit 100
+```
+
+The audit uses `gh`, freezes the PR list, fetches their revisions, and writes a per-test report and aggregate summary under `tmp/regression-history`. Repeating the command resumes comparisons made with the same checker from that directory. Historical dependency or runtime incompatibilities aren't counted as coverage results.
+
 ## Documentation
 
 ### Previewing changes locally