Repository navigation
feat: support sparse boolean masks in COO and GCXS - #962
Conversation
Signed-off-by: 李永祺 <doribelove@gmail.com>
hameerabbasi
left a comment
There was a problem hiding this comment.
Hello! Thanks for the contribution. It's broadly okay, you'd just need to run the Array API tests with the Numba backend; and remove the passing tests from the xfails file.
Signed-off-by: 李永祺 <doribelove@gmail.com>
…indexing Signed-off-by: 李永祺 <doribelove@gmail.com>
Signed-off-by: 李永祺 <doribelove@gmail.com>
Signed-off-by: 李永祺 <doribelove@gmail.com>
|
@hameerabbasi, the current head ( Two CI checks are still red. The benchmark job errors in Finch's |
Could you perhaps post the failure + stacktrace here? |
|
Reproduced on the current PR head, The failing draw is import numpy as np
import sparse
x = np.bool_(False)
key = sparse.COO.from_numpy(np.array(False))
x[key]I also reran the original property with the repository's current xfail/skip files. From the Array API checkout, with this PR installed and the files at SPARSE_BACKEND=Numba ARRAY_API_TESTS_MODULE=sparse \
python -m pytest array_api_tests/test_array_object.py::test_getitem_masking \
--max-examples=200 --derandomize --disable-deadline --disable-warnings \
-o strict_xfail=True \
--xfails-file ../xp-tests/Numba-array-api-xfails.txt \
--skips-file ../xp-tests/Numba-array-api-skips.txt --tb=long -qThis reports 1 failed. For precision, Complete pytest failure outputFor comparison, using Reproduction and response prepared with Codex assistance; the commands above were executed locally. |
Integrate upstream through d632550 and retain the COO import required for boolean scalar keys while resolving the GCXS indexing conflict. The existing sparse boolean-mask selection algorithm is unchanged. Signed-off-by: 李永祺 <doribelove@gmail.com>
Merge main through 9fbb898 and retain the broadcast integer-indexing implementation alongside sparse boolean-mask dispatch. Resolve the operations guide conflict by describing both supported indexing paths. Declare the one-coordinate uint64 masking fixture sorted and duplicate- free, so it tests masking independently of main's constructor precision regression. That constructor regression is being fixed separately. Validation: 7,374 Numba backend tests/doctests passed; 3,504 fixed-seed NumPy comparisons passed. The pinned Array API suite reports 1,180 passed, 84 skipped, 119 xfailed and the same acosh strict XPASS reproduced on main. Applicable pre-commit hooks and git diff --check pass. Signed-off-by: 李永祺 <doribelove@gmail.com>
|
Synced this branch with main through The large-uint64 masking regression exposed a separate constructor-validation bug from #1015; its fix and constructor tests are in #1029. Here the one-coordinate fixture now explicitly declares its known sorted/duplicate-free input, so it continues testing masking's integer precision independently. Validation: 7,374 Numba backend tests/doctests and 3,504 seeded NumPy comparisons pass. The pinned Array API run reports 1,180 passed, 84 skipped, 119 xfailed and the same complex-acosh strict XPASS reproduced on unmodified main. Applicable pre-commit hooks pass. The previously reported NumPy-scalar masking limitation at a larger Hypothesis budget remains documented in the PR. |
What type of PR is this? (check all applicable)
Related issues
Closes #304. Addresses the retrieval part of #860; assignment remains unsupported.
Checklist
Please explain your changes below.
x[x > 0]andx[x == 0]currently raiseIndexErrorfor COO and GCXS arrays. This adds a single sparse boolean mask over matching leading dimensions, including full-shape, scalar and zero-length masks. Python/NumPy boolean scalar keys insert the same leading axis as a zero-dimensional mask. The output retains the source fill value.The COO path searches stored mask coordinates. With a True fill, it subtracts the number of preceding exclusions to compute output coordinates without enumerating implicit True entries. GCXS delegates through COO. An integer coordinate buffer avoids float promotion when source coordinates are uint64.
Both indexing paths use
SparseArrayfor the mask check, as suggested in review. COO, GCXS, and DOK masks are covered by the regressions. The branch includes main through9fbb898, including broadcast integer-array indexing from #1026 and GCXS validation from #1015. The merge preserves those paths alongside sparse-mask dispatch, and the operations guide describes both. The sparse-mask selection function is unchanged. The single-coordinate uint64 masking fixture is explicitly sorted and duplicate-free; the independent constructor-validation precision regression found during this update is fixed separately in #1029.The documentation explicitly limits this to standalone masks (
x[mask]orx[(mask,)]); mixed indices, sparse integer indices, and assignment are outside this change. Source arrays are COO or GCXS. Other backends and the general Array API boolean-indexing capability flag are unchanged.Array API review follow-up
Per review, removed the test-only plugin and its README and restored
pixi.toml. The final change removes the wholetest_getitem_maskingentry from the Numba xfails file. With the existing CI setting--max-examples=2, this property passes; leaving the whole-property xfail producesXPASS(strict).There is a remaining limitation in the pinned external test: it also generates NumPy scalar sources. With
--max-examples=200it reachesnp.False_[<COO: shape=(), dtype=bool, ...>], which raisesIndexErrorinside NumPy. The pinned suite reduces the example limit by ten for this unvectorized property, with a minimum of one. The test has a single pytest ID, so the existing xfails file cannot target only that scalar case. The 243 local regressions exercise sparse source arrays directly. No change to Sparse's scalar-return semantics was made to influence test generation. I would welcome guidance on how you prefer to track this upstream mixed-operand case without a collection plugin.Validation on Python 3.12 / NumPy 2.5 / Numba 0.67:
sparse/numba_backend): 7,374 passed, 1 skipped, 113 xfailed, 41 xpassed, 7 warnings. Another 3,504 fixed-seed public-API comparisons against NumPy pass. The masking property passes with CI's--max-examples=2setting and four workers. The previously reproduced--max-examples=200limitation for NumPy scalar sources remains under review.--max-examples=2setting, four workers and an explicit test root: 1,180 passed, 84 skipped, 119 xfailed; one strict XPASS for complexacoshremains. A focused run on unmodified main at9fbb898, with the baseline import path verified, reproduces the same strict XPASS for complexacosh; masking remains xfailed there. This is not an all-green local Array API result.git diff --checkpass. The earlier revision passed a wheel build and 3,504 fixed-seed comparisons against NumPy. Finch/MLIR and other platforms were not tested locally.Prepared with assistance from OpenAI Codex; the reported validation was executed locally.