Repository navigation
ensemble_tides: dedupe locations before idw(), not after - #155
Open
yurithefury wants to merge 1 commit into
Open
yurithefury wants to merge 1 commit into
yurithefury wants to merge 1 commit into
Conversation
Ran into out-of-memory (OOM) errors during ensemble modelling. x/y passed to idw() come straight from tide_df's full index, so each location gets interpolated once per timestep and model instead of once. Dedupe before calling idw(), not after. Memory now scales with unique locations, not locations x times x models.
yurithefury
force-pushed
the
fix-ensemble-tides-duplicate-idw
branch
from
August 14, 2026 00:28
cb1394a to
8df4d21
Compare
yurithefury
marked this pull request as ready for review
August 14, 2026 00:34
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Ran into out-of-memory (OOM) errors running ensemble tide modelling.
ensemble_tides()interpolates model rankings usingtide_df's full (time, x, y, tide_model) index, soidw()gets called with each location repeated once per timestep and model, even though rankings only depend on location.Hit this generating a DEM over a 10km AOI, 5 years of exposure() at the default 30min cadence: 64 locations turned into ~50M query rows (64 x 87,648 timesteps x 9 models), idw() tried to allocate 33.8 GiB and died.
Fix
Dedupe x/y before calling
idw()instead of after (there's already adrop_duplicates()call, it just runs on the result). The join back onto tide_df a few lines down already broadcasts each location's ranking to every matching row, so this is just a perf fix. Checked original vs patched with duplicated query points, same output.Test
Added
test_ensemble_tides_deduplicates_ranking_queries, which wrapsidw()to check how many points it's called with, and confirms it only gets called once per unique location. Fails on main, passes with this fix.