Skip to content

perf: speed up sparse NaN reductions and n-ary coordinate evaluation - #1031

Merged
hameerabbasi merged 5 commits into
pydata:mainfrom
woahwhattheheck:perf/coo-coordinate-evaluation-908
Nov 16, 2026
Merged

hameerabbasi merged 5 commits into
pydata:mainfrom
woahwhattheheck:perf/coo-coordinate-evaluation-908

Conversation

@woahwhattheheck

@woahwhattheheck woahwhattheheck commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • 🪄 Feature
  • 🐞 Bug Fix
  • 🔧 Optimization
  • 📚 Documentation
  • 🧪 Test
  • 🛠️ Other

Related issues

Refs #908. This follows @peanutfun's profiling and the invitation to contribute in that discussion.

Checklist

  • Code follows style guide
  • Tests added
  • Documented the changes

Please explain your changes below.

COO operations with matching stored coordinates can evaluate their data directly. For three or more same-shape sparse operands with different supports, this evaluates the stored-coordinate union in bounded blocks instead of enumerating every stored/fill combination. Operations on unequal binary supports and sparse shape expansion retain the general matching path.

NaN replacement works directly on stored data and the fill value. Dense operands are sampled at stored coordinates, avoiding a full logical broadcast view. Regression cases cover very large logical shapes and scalar/singleton operands passed to dtype-aware callables. Custom callables receive owned stored-data buffers when multiple nonempty aligned sparse operands are matched, preserving input arrays even when the callable reuses a buffer or inputs are read-only. Ufuncs keep the direct data path. Output format, dtype selection and pruning use the existing machinery.

On the original 100 × 100 × 100, density-0.1 workload with seed 42, all seven baseline/candidate outputs and dtypes matched. These are warmed median wall times from seven alternating pairs, with three operations per sample, on one shared Linux machine.

Operation Upstream ms Candidate ms
where, zero fill 19.308 6.606
nansum, zero fill 31.979 17.932
xarray sum, zero fill 45.218 20.274
where, NaN fill 49.745 9.519
nansum, NaN fill 68.537 18.294
xarray sum, NaN fill 84.946 27.456
addition with unrelated coordinates 61.465 58.719

The unrelated-coordinate control varied by a few percent between the two local runs. The xarray NaN-sum case used 26,881,437 versus 12,906,992 peak bytes measured by tracemalloc.

Validation used Python 3.12.14, NumPy 2.3.5, Numba 0.68.1 and xarray 2026.9.0:

  • Full test_elemwise.py: 727 passed. Eight new cases cover input preservation when a callable reuses the first or last operand, with two or three aligned sparse inputs and writable or read-only data.
  • Twelve original/candidate public-API comparisons passed for writable dense callable operands, values, dtypes and pruning.
  • All ten added benchmark cases executed successfully.
  • Ruff check, Ruff formatting and Git whitespace checks passed on all four changed files.

Edge-case validation

Additional regressions cover the logical-index overflow guard, constant-filled empty sparse operands in all argument positions and supported formats, support unions across the 65,536-element block boundary, cancellation/pruning, and array-valued NaN reduction identities.

Validation of the proposed merge with main fd405444bbff191d01ef14c569f7483ccec00af8 used Python 3.12.14, NumPy 2.5.4, Numba 0.68.1, and SciPy 1.18.1:

  • The two affected test modules: 4,178 passed, 80 xfailed, 33 xpassed, with the two existing warnings and no unexpected failures.
  • 100% of the PR's 113 added executable production lines covered, using the repository's unchanged coverage rules.
  • Ruff check, Ruff formatting, and Git whitespace checks passed.

@hameerabbasi hameerabbasi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broadly looks okay, caught one code smell.

Comment thread sparse/numba_backend/_umath.py Resolved
@woahwhattheheck

Copy link
Copy Markdown
Contributor Author

Updated in 002c954: the general matcher and coordinate result now use an explicit if/else and share the COO construction. This keeps the union optimization when alignment is unavailable. All 727 elementwise tests pass, along with Ruff and whitespace checks.

hameerabbasi
hameerabbasi previously approved these changes Nov 15, 2026
@hameerabbasi
hameerabbasi enabled auto-merge (squash) November 15, 2026 07:20
Cover logical-index overflow, constant sparse operands, multi-block support
unions and cancellation, and array-valued NaN reduction identities.
auto-merge was automatically disabled November 16, 2026 04:29

Head branch was pushed to by a user without write access

@hameerabbasi
hameerabbasi enabled auto-merge (squash) November 16, 2026 07:06
@hameerabbasi
hameerabbasi enabled auto-merge (squash) November 16, 2026 07:36
@hameerabbasi
hameerabbasi merged commit f9b079a into pydata:main Nov 16, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants