Skip to content

Fix NumPy 2 overflow and quadratic point selection on large stacks - #162

Merged
Andreas-Piter merged 3 commits into
luhipi:mainfrom
A220N1:fix/large-stacks
Sep 14, 2026
Merged

Andreas-Piter merged 3 commits into
luhipi:mainfrom
A220N1:fix/large-stacks

Conversation

@A220N1

@A220N1 A220N1 commented Sep 10, 2026 •

Copy link
Copy Markdown

Two independent problems met while processing Sentinel-1 stacks of a few million
second-order points.

prog_bar.update(..., every=np.int16(...)) raises OverflowError: Python integer 32768 out of bounds for int16 under NumPy 2, at the very end of step 4 and during
triangulation = the whole run is lost.

The three point selection loops (objects.removePoints,
processing.runDensificationTimeAndSpace twice) are quadratic in the number of
points and never complete at that size. np.isin gives the same mask, duplicates
included.

Checked on a 55-date Sentinel-1 stack: steps 0 to 4 run twice, with and without the
vectorization, give 49 140 points, the same identifiers and a maximum difference of
0 over the 62 numeric columns of the export.

No test is added: both changes are internal and the existing suite covers them
through test_processing.

AZZONI MAXIME added 3 commits September 10, 2026 17:18
NumPy 2 refuses to convert a Python integer larger than 32767 to int16,
so densification and triangulation crash at the very end of a large run.
The three "for p in point_id: if p not in other_id" loops are quadratic
and do not end on a stack of a few million points.
gstools allocates one (conditioning points x evaluated points) float64 array
per call, hundreds of gigabytes on a large stack, and the worker is killed.
@A220N1

A220N1 commented Sep 11, 2026

Copy link
Copy Markdown
Author

A third problem, same family. gstools evaluates a kriging model in a single
block by default (chunk_size=None), so filtering.py allocates one
(conditioning points x evaluated points) float64 array, with a few thousand
first-order points interpolated onto a few million second-order ones, that is
hundreds of gigabytes and the worker is killed.

Evaluating in blocks of at most 128 MB is bit-identical and costs nothing.

@Andreas-Piter
Andreas-Piter self-requested a review September 14, 2026 09:14
@Andreas-Piter Andreas-Piter added bug Something isn't working enhancement New feature or request labels Sep 14, 2026
@Andreas-Piter

Copy link
Copy Markdown
Member

Dear @A220N1,
Thanks for pointing to these issues in the implementation and for providing a fix. I have tested it and it can be merged.
Thank you so much for contributing to our community software! If you have any further comments, suggestions or feedback, please send me an email or add an issue here on github.
Best, Andreas

@Andreas-Piter
Andreas-Piter merged commit 192b3d1 into luhipi:main Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants