[dgenio]

Description

Fixes #21607.

Pre-existing broken tests in dependent packages were causing the Unit Testing / test (3.10|3.11|3.12) matrix to fail on every PR that touched llama-index-core — because llama-dev test computes the dependent packages of changed packages and tests them too. Any PR that changed core inherited ~23 broken integration test failures it couldn't fix, making it impossible to get a green CI signal.

This PR implements Option 2 from the issue: distinguish between failures in directly-changed packages (hard failures) and failures in dependent-only packages (warnings by default, hard failures with --strict). Only direct failures cause exit(1) in the default mode.

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No (not a new package)

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No (llama-dev package — internal tooling, no version bump needed per repo policy)

Type of Change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• I added new unit tests to cover this change

Test commands and results

cd llama-dev
uv run -- pytest tests/test/test_test.py -v
# 24 passed in ~30s

uv run -- ruff check llama_dev/test/__init__.py tests/test/test_test.py
uv run -- ruff format --check llama_dev/test/__init__.py tests/test/test_test.py
# All checks pass; files already formatted.

uvx codespell --toml pyproject.toml llama-dev/llama_dev/test/__init__.py llama-dev/tests/test/test_test.py
# clean

Changes

llama-dev/llama_dev/test/__init__.py

• Split the failed list into failed_direct (packages whose path is in changed_packages) and failed_dependent (everything else that failed). exit(1) only fires when failed_direct is non-empty.
• Dependent failures print a ⚠️ warning section listing the affected packages but do not block CI by default.
• New --strict CLI flag restores the old "any failure → exit 1" contract. With --strict, dependent-only failures also cause exit(1) and a Failing due to --strict message prints. Default remains the new lenient behavior.
• The warning section also emits a GitHub Actions ::warning:: workflow command so dependent-package regressions surface in the PR Checks tab, not only in the collapsed CI test log.

llama-dev/tests/test/test_test.py

• Fixed 5 pre-existing broken tests (test_coverage_failures, test_install_failures, test_skip_failures_no_tests, test_skip_failures_unsupported_python, test_success) — they used Path("/fake/repo/package1") for changed_packages while the mock returned data_path / "test_integration", causing Path.relative_to() to raise a ValueError in the progress display loop.
• Added 4 new tests:
  • test_dependent_failure_is_warning — when only a dependent package fails, exit_code == 0, the warning message appears, and a ::warning:: annotation is emitted.
  • test_direct_failure_exits_nonzero_with_dependent_failures — when both a direct and a dependent package fail, exit_code == 1 and both sections appear in the output.
  • test_dependent_success_no_warning — when dependents are present and all packages pass, no warning section or ::warning:: annotation is emitted.
  • test_strict_flag_exits_nonzero_on_dependent_failures — with --strict, a dependent-only failure exits 1 and the strict-mode message is printed.

Risks

• Visibility: Pre-existing failures in dependent packages are still printed and visible in CI — both as a ⚠️ warning section in stdout and as a GitHub Actions ::warning:: annotation in the PR Checks tab. Regressions introduced by a PR in a dependent package will still appear in the warning section — they won't be missed, just won't block the PR by default.
• Contract change: The default exit-code behavior changes from "any failure → exit 1" to "direct failure → exit 1". Callers that depended on the old contract (e.g., release-validation pipelines) can pass --strict to restore it.
• Backward compatibility: No change to existing CLI flags or public API. --fail-fast, explicit package names, --cov, --cov-fail-under, --base-ref, --workers all work unchanged. The new --strict flag is opt-in.
• Scope: No changes to workflow YAML, no skip-list file to maintain.

Suggested Checklist

• I have performed a self-review of my own code
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran uv run -- ruff check and uvx codespell to satisfy lint (pre-commit lint job in CI is green)

[logan-markewich]

Ngl I would rather not do this, and just fix fhe actual tests 😅

[dgenio]

Totally fair pushback — quick clarification first: the broken tests still run and their failures still print, just demoted from exit(1) to a ⚠️ N dependent packages had pre-existing test failures section, so regressions stay visible but don't block green CI on unrelated PRs. Agree though that fixing the actual tests is the real solution; #21607 lists both paths and they're orthogonal.

Would you rather I (a) keep this PR as a stopgap plus open a tracking issue for per-package fixes, or (b) close this and pursue per-package fixes directly? Either works — happy to follow your call.

[github-actions]

This PR is stale because it has been open 50 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was closed because it has been stalled for 10 days with no activity.