Skip to content

feat: support sparse boolean masks in COO and GCXS - #962

Merged
hameerabbasi merged 8 commits into
pydata:mainfrom
Doribelove:codex/sparse-boolean-indexing
Nov 13, 2026
Merged

hameerabbasi merged 8 commits into
pydata:mainfrom
Doribelove:codex/sparse-boolean-indexing

Conversation

@Doribelove

@Doribelove Doribelove commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • 🪄 Feature
  • 🐞 Bug Fix
  • 🔧 Optimization
  • 📚 Documentation
  • 🧪 Test
  • 🛠️ Other

Related issues

Closes #304. Addresses the retrieval part of #860; assignment remains unsupported.

Checklist

  • Code follows style guide
  • Tests added
  • Documented the changes

Please explain your changes below.

x[x > 0] and x[x == 0] currently raise IndexError for COO and GCXS arrays. This adds a single sparse boolean mask over matching leading dimensions, including full-shape, scalar and zero-length masks. Python/NumPy boolean scalar keys insert the same leading axis as a zero-dimensional mask. The output retains the source fill value.

The COO path searches stored mask coordinates. With a True fill, it subtracts the number of preceding exclusions to compute output coordinates without enumerating implicit True entries. GCXS delegates through COO. An integer coordinate buffer avoids float promotion when source coordinates are uint64.

Both indexing paths use SparseArray for the mask check, as suggested in review. COO, GCXS, and DOK masks are covered by the regressions. The branch includes main through 9fbb898, including broadcast integer-array indexing from #1026 and GCXS validation from #1015. The merge preserves those paths alongside sparse-mask dispatch, and the operations guide describes both. The sparse-mask selection function is unchanged. The single-coordinate uint64 masking fixture is explicitly sorted and duplicate-free; the independent constructor-validation precision regression found during this update is fixed separately in #1029.

The documentation explicitly limits this to standalone masks (x[mask] or x[(mask,)]); mixed indices, sparse integer indices, and assignment are outside this change. Source arrays are COO or GCXS. Other backends and the general Array API boolean-indexing capability flag are unchanged.

Array API review follow-up

Per review, removed the test-only plugin and its README and restored pixi.toml. The final change removes the whole test_getitem_masking entry from the Numba xfails file. With the existing CI setting --max-examples=2, this property passes; leaving the whole-property xfail produces XPASS(strict).

There is a remaining limitation in the pinned external test: it also generates NumPy scalar sources. With --max-examples=200 it reaches np.False_[<COO: shape=(), dtype=bool, ...>], which raises IndexError inside NumPy. The pinned suite reduces the example limit by ten for this unvectorized property, with a minimum of one. The test has a single pytest ID, so the existing xfails file cannot target only that scalar case. The 243 local regressions exercise sparse source arrays directly. No change to Sparse's scalar-return semantics was made to influence test generation. I would welcome guidance on how you prefer to track this upstream mixed-operand case without a collection plugin.

Validation on Python 3.12 / NumPy 2.5 / Numba 0.67:

  • 243 parametrized regressions pass, including DOK masks, large shapes, True-fill masks without densification, empty masks, and boolean scalar keys.
  • Numba backend tests and module doctests after merging main (sparse/numba_backend): 7,374 passed, 1 skipped, 113 xfailed, 41 xpassed, 7 warnings. Another 3,504 fixed-seed public-API comparisons against NumPy pass. The masking property passes with CI's --max-examples=2 setting and four workers. The previously reproduced --max-examples=200 limitation for NumPy scalar sources remains under review.
  • Pinned Array API suite with CI's --max-examples=2 setting, four workers and an explicit test root: 1,180 passed, 84 skipped, 119 xfailed; one strict XPASS for complex acosh remains. A focused run on unmodified main at 9fbb898, with the baseline import path verified, reproduces the same strict XPASS for complex acosh; masking remains xfailed there. This is not an all-green local Array API result.
  • Repository pre-commit hooks and git diff --check pass. The earlier revision passed a wheel build and 3,504 fixed-seed comparisons against NumPy. Finch/MLIR and other platforms were not tested locally.

Prepared with assistance from OpenAI Codex; the reported validation was executed locally.

Signed-off-by: 李永祺 <doribelove@gmail.com>
@github-actions github-actions Bot added the enhancement Indicates new feature requests label Sep 22, 2026

@hameerabbasi hameerabbasi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello! Thanks for the contribution. It's broadly okay, you'd just need to run the Array API tests with the Numba backend; and remove the passing tests from the xfails file.

Comment thread sparse/numba_backend/_coo/indexing.py Resolved
Signed-off-by: 李永祺 <doribelove@gmail.com>
Comment thread xp-tests/Numba-array-api-skips.txt Resolved
…indexing

Signed-off-by: 李永祺 <doribelove@gmail.com>
Signed-off-by: 李永祺 <doribelove@gmail.com>
Comment thread xp-tests/numba_masking.py Resolved
Comment thread pixi.toml Resolved
Signed-off-by: 李永祺 <doribelove@gmail.com>
@Doribelove

Copy link
Copy Markdown
Contributor Author

@hameerabbasi, the current head (996182f) addresses the review points: the mask check uses SparseArray, the test-only plugin and pixi.toml edits are gone, and test_getitem_masking is no longer marked as a whole-test xfail. The Numba Array API check and the Linux, macOS, and Windows test matrices pass on this head. The remaining NumPy-scalar case generated by the pinned property at a larger Hypothesis budget is documented in the PR description; I would appreciate your guidance on its treatment.

Two CI checks are still red. The benchmark job errors in Finch's Reflector setup, also seen on #964, before the new masking path is exercised. codecov/patch passes at 100% of the diff, while codecov/project reports 90.65% against a 100% project target; #964 has the same project-check failure. Could you take another look at the implementation and advise whether any further PR-side change is needed? Thank you.

@Doribelove Doribelove mentioned this pull request Sep 24, 2026
6 of 10 tasks
@hameerabbasi

Copy link
Copy Markdown
Collaborator

The remaining NumPy-scalar case generated by the pinned property at a larger Hypothesis budget is documented in the PR description; I would appreciate your guidance on its treatment.

Could you perhaps post the failure + stacktrace here?

@Doribelove

Copy link
Copy Markdown
Contributor Author

Reproduced on the current PR head, 996182fd, using the pinned Array API tests revision 55fcc60179efa2680ddd6cd926ddf17b83530e2b. Environment: Python 3.12.14, NumPy 2.5.3, Numba 0.67.0, Hypothesis 6.168.0 and pytest 9.1.1.

The failing draw is x = np.False_ and a zero-dimensional COO boolean mask with fill_value=False. Here is a standalone reproduction and its complete traceback:

import numpy as np
import sparse

x = np.bool_(False)
key = sparse.COO.from_numpy(np.array(False))
x[key]
Traceback (most recent call last):
  File "repro.py", line 7, in <module>
    x[key]
    ~^^^^^
IndexError: invalid index to scalar variable.

I also reran the original property with the repository's current xfail/skip files. From the Array API checkout, with this PR installed and the files at ../xp-tests:

SPARSE_BACKEND=Numba ARRAY_API_TESTS_MODULE=sparse \
  python -m pytest array_api_tests/test_array_object.py::test_getitem_masking \
  --max-examples=200 --derandomize --disable-deadline --disable-warnings \
  -o strict_xfail=True \
  --xfails-file ../xp-tests/Numba-array-api-xfails.txt \
  --skips-file ../xp-tests/Numba-array-api-skips.txt --tb=long -q

This reports 1 failed. For precision, --max-examples=200 is the CLI setting: the pinned suite reduces the limit to 20 for this unvectorized property. Similarly, CI's --max-examples=2 setting becomes a limit of 1 for this property. I have clarified those settings in the PR description.

Complete pytest failure output
F                                                                        [100%]
=================================== FAILURES ===================================
_____________________________ test_getitem_masking _____________________________

    @pytest.mark.unvectorized
>   @pytest.mark.data_dependent_shapes
                   ^^^

array_api_tests/test_array_object.py:167: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

shape = (), data = data(...)

    @pytest.mark.unvectorized
    @pytest.mark.data_dependent_shapes
    @given(hh.shapes(), st.data())
    def test_getitem_masking(shape, data):
        x = data.draw(hh.arrays(hh.all_dtypes, shape=shape), label="x")
        mask_shapes = st.one_of(
            st.sampled_from([x.shape, ()]),
            st.lists(st.booleans(), min_size=x.ndim, max_size=x.ndim).map(
                lambda l: tuple(s if b else 0 for s, b in zip(x.shape, l))
            ),
            hh.shapes(),
        )
        key = data.draw(hh.arrays(dtype=xp.bool, shape=mask_shapes), label="key")
    
        if key.ndim > x.ndim or not all(
            ks in (xs, 0) for xs, ks in zip(x.shape, key.shape)
        ):
            with pytest.raises(IndexError):
                x[key]
            return
    
        repro_snippet = ph.format_snippet(f"out = {x!r}[{key!r}]")
        try:
>           out = x[key]
                  ^^^^^^
E           IndexError: invalid index to scalar variable.
E           
E           ========== FAILING CODE SNIPPET:
E           out = np.False_[<COO: shape=(), dtype=bool, nnz=0, fill_value=False>]
E           ====================
E           
E           Failing test case: test_getitem_masking(
E               shape=(),
E               data=data(...),
E           )
E           Draw 1 (x): np.False_
E           Draw 2 (key): _f(<...>, False)
E           Explanation:
E               These lines were always and only run by failing test cases:
E                   <sparse-checkout>/sparse/numba_backend/_sparse_array.py:286

array_api_tests/test_array_object.py:189: IndexError
=========================== short test summary info ============================
FAILED array_api_tests/test_array_object.py::test_getitem_masking - IndexErro...
1 failed, 6 warnings in 3.11s

For comparison, using x = sparse.COO.from_numpy(np.array(False)) with the same key returns a COO with shape (0,). The failing generated operand above is a NumPy scalar, so the standalone traceback contains no Sparse indexing implementation frames.

Reproduction and response prepared with Codex assistance; the commands above were executed locally.

Integrate upstream through d632550 and retain the COO import required
for boolean scalar keys while resolving the GCXS indexing conflict.
The existing sparse boolean-mask selection algorithm is unchanged.

Signed-off-by: 李永祺 <doribelove@gmail.com>
Merge main through 9fbb898 and retain the broadcast integer-indexing
implementation alongside sparse boolean-mask dispatch. Resolve the
operations guide conflict by describing both supported indexing paths.

Declare the one-coordinate uint64 masking fixture sorted and duplicate-
free, so it tests masking independently of main's constructor precision
regression. That constructor regression is being fixed separately.

Validation: 7,374 Numba backend tests/doctests passed; 3,504 fixed-seed
NumPy comparisons passed. The pinned Array API suite reports 1,180 passed,
84 skipped, 119 xfailed and the same acosh strict XPASS reproduced on main.
Applicable pre-commit hooks and git diff --check pass.

Signed-off-by: 李永祺 <doribelove@gmail.com>
@Doribelove

Copy link
Copy Markdown
Contributor Author

Synced this branch with main through 9fbb898 in 587412a, resolving the operations-guide conflict while retaining #1026's broadcast integer-indexing support. The masking function is unchanged.

The large-uint64 masking regression exposed a separate constructor-validation bug from #1015; its fix and constructor tests are in #1029. Here the one-coordinate fixture now explicitly declares its known sorted/duplicate-free input, so it continues testing masking's integer precision independently.

Validation: 7,374 Numba backend tests/doctests and 3,504 seeded NumPy comparisons pass. The pinned Array API run reports 1,180 passed, 84 skipped, 119 xfailed and the same complex-acosh strict XPASS reproduced on unmodified main. Applicable pre-commit hooks pass. The previously reported NumPy-scalar masking limitation at a larger Hypothesis budget remains documented in the PR.

@hameerabbasi
hameerabbasi enabled auto-merge (squash) November 13, 2026 10:29
@hameerabbasi
hameerabbasi merged commit fd40544 into pydata:main Nov 13, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Indicates new feature requests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support boolean indexing with a COO sparse matrix

2 participants