Repository navigation
perf: speed up sparse NaN reductions and n-ary coordinate evaluation - #1031
Merged
hameerabbasi merged 5 commits intoNov 16, 2026
Merged
hameerabbasi merged 5 commits into
hameerabbasi merged 5 commits into
Conversation
hameerabbasi
requested changes
Nov 13, 2026
hameerabbasi
left a comment
Collaborator
There was a problem hiding this comment.
Broadly looks okay, caught one code smell.
Contributor
Author
|
Updated in 002c954: the general matcher and coordinate result now use an explicit if/else and share the COO construction. This keeps the union optimization when alignment is unavailable. All 727 elementwise tests pass, along with Ruff and whitespace checks. |
hameerabbasi
previously approved these changes
Nov 15, 2026
hameerabbasi
enabled auto-merge (squash)
November 15, 2026 07:20
Cover logical-index overflow, constant sparse operands, multi-block support unions and cancellation, and array-valued NaN reduction identities.
auto-merge was automatically disabled
November 16, 2026 04:29
Head branch was pushed to by a user without write access
hameerabbasi
approved these changes
Nov 16, 2026
hameerabbasi
enabled auto-merge (squash)
November 16, 2026 07:06
hameerabbasi
disabled auto-merge
November 16, 2026 07:36
hameerabbasi
enabled auto-merge (squash)
November 16, 2026 07:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (check all applicable)
Related issues
Refs #908. This follows @peanutfun's profiling and the invitation to contribute in that discussion.
Checklist
Please explain your changes below.
COO operations with matching stored coordinates can evaluate their data directly. For three or more same-shape sparse operands with different supports, this evaluates the stored-coordinate union in bounded blocks instead of enumerating every stored/fill combination. Operations on unequal binary supports and sparse shape expansion retain the general matching path.
NaN replacement works directly on stored data and the fill value. Dense operands are sampled at stored coordinates, avoiding a full logical broadcast view. Regression cases cover very large logical shapes and scalar/singleton operands passed to dtype-aware callables. Custom callables receive owned stored-data buffers when multiple nonempty aligned sparse operands are matched, preserving input arrays even when the callable reuses a buffer or inputs are read-only. Ufuncs keep the direct data path. Output format, dtype selection and pruning use the existing machinery.
On the original 100 × 100 × 100, density-0.1 workload with seed 42, all seven baseline/candidate outputs and dtypes matched. These are warmed median wall times from seven alternating pairs, with three operations per sample, on one shared Linux machine.
The unrelated-coordinate control varied by a few percent between the two local runs. The xarray NaN-sum case used 26,881,437 versus 12,906,992 peak bytes measured by tracemalloc.
Validation used Python 3.12.14, NumPy 2.3.5, Numba 0.68.1 and xarray 2026.9.0:
test_elemwise.py: 727 passed. Eight new cases cover input preservation when a callable reuses the first or last operand, with two or three aligned sparse inputs and writable or read-only data.Edge-case validation
Additional regressions cover the logical-index overflow guard, constant-filled empty sparse operands in all argument positions and supported formats, support unions across the 65,536-element block boundary, cancellation/pruning, and array-valued NaN reduction identities.
Validation of the proposed merge with main
fd405444bbff191d01ef14c569f7483ccec00af8used Python 3.12.14, NumPy 2.5.4, Numba 0.68.1, and SciPy 1.18.1: