Repository navigation
DEMBenchmark: reuse a translated DEM newer than its inputs - #215
Merged
Merged
Conversation
Re-scoring from cached pc_align runs rewrote every ~500 MB translated copy, about 6 of the 20 minutes the lidar notebook spent re-scoring the 39 full-scene DEMs. Reuse the copy when it is newer than its DEM and pc_align output, and write new copies under a temporary name then rename, so an interrupted run leaves no partial file to reuse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ac1db60)
The cherry-picked entry quoted one study's numbers (39 full-scene DEMs, ~500 MB copies, minutes in one notebook). A package changelog says what the package does differently; the study and its timings live elsewhere.
Closed
9 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DEMBenchmark.run(pc_align=True)already reuses thepc_alignproducts under its folder, so a re-run makes nopc_aligncall. It still rewrote each DEM's translated copy (dem_benchmark/<label>/<stem>_pc_align_translated.tif) on every run, a full-size raster write per DEM. Re-scoring the 39 full-scene DEMs of #207 from cached alignments spent about 6 of its 20 minutes on those writes.Change
pc_align/<prefix>-*file (_is_newer_than). Otherwise it is rewritten.*_pc_align_translated.partial.tifand is then moved into place withos.replace, so an interrupted run never leaves a partial copy that the next run would take as current.directoryargument and ofrun(pc_align=...)say so.[Unreleased] / Changedline in the CHANGELOG.Tests
TestRunnow also checks three things:pc_alignoutput is rewritten;.partialfile is left behind.pytest tests/test_dem_benchmark.py: 34 passed.pre-commit run --all-files: passed.Commits
-x) ofac1db60from Full-scene scene-combination benchmarks: Atlanta and UCSD, lidar check, Atlanta jitter (#207) #210, where the change was first made.#210 itself will be closed: its notebooks and write-up are moving out of this repo.