Repository navigation
Release netCDF memory-map pages after each frame - #5463
erikfransson wants to merge 6 commits into
Conversation
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read. The map is read only, so dropping is safe: pages are faulted back in if the data are read again. MADV_DONTNEED does not exist on Windows, where the call is skipped and the reader is unchanged.
Documentation build overview
162 files changed ·
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #5463 +/- ##
========================================
Coverage 93.92% 93.92%
========================================
Files 184 184
Lines 22661 22665 +4
Branches 3223 3224 +1
========================================
+ Hits 21284 21288 +4
Misses 917 917
Partials 460 460 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| SimpleNamespace(madvise=advice.append), | ||
| ) | ||
| universe.trajectory[1] | ||
| assert advice == [mmap.MADV_DONTNEED] |
There was a problem hiding this comment.
The test is a bit convoluted with monkeypatch--could we avoid that?
Also, am I understanding correctly that on this branch something like https://github.com/pythonprofilers/memory_profiler should show a constant level of physical memory consumption, while it should increase linearly when reading in/iterating over a trajecotry of the appropriate type on the develop branch?
We could probably capture that with https://asv.readthedocs.io/en/stable/writing_benchmarks.html#peak-memory as well, assuming you're saying that physical memory is being used excessively on develop in these scenarios.
There was a problem hiding this comment.
Yea I agree, I removed the test.
There was a problem hiding this comment.
But yes I ran memory profilers locally on 4GB sized trajectory.
Reading frame by frame with develop leads to memory usage climbing steadily and reaching peak memory of 4GB since nothing unmaps the scipys memory map.
With the fix in place the memory didnt climb at all and behaved as expected when reading frame by frame.
Review feedback: a versionchanged directive is not needed for a bug fix with no user-facing behaviour change, and the test only asserted that madvise is called with a particular flag, which is an implementation detail. The values read from the trajectory are unchanged and already covered by the existing NCDF reader tests.
|
@tylerjereddy @orbeckst Anything else you would like me to do? |
|
@tylerjereddy would you be comfortable giving an "approve" or "request changes" review? |
orbeckst
left a comment
orbeckst
left a comment
There was a problem hiding this comment.
Overall this looks sensible to me. However, we don't know if the problem also exists on Windows (or macOS). I'd like to be able to state clearly which OS is affected and on which the fix solves the problem.
| that provides `MADV_DONTNEED`, i.e. not Windows (Discussion #4792, | ||
| PR #5463). |
There was a problem hiding this comment.
| that provides `MADV_DONTNEED`, i.e. not Windows (Discussion #4792, | |
| PR #5463). | |
| that provides `MADV_DONTNEED`, i.e. not Windows (#5541, PR #5463) |
I raised a proper issue from the discussion #5541
I'm a bit slow getting to this, but I'll make a calendar reminder to look more closely. I may still be slow (few days) though. |
fixes #5541
Description
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read, as documented in discussion here. (ping @orbeckst )
MADV_DONTNEED does not exist on Windows, so the call is guarded by hasattr(mmap, "MADV_DONTNEED") and the reader is unchanged there.
Changes made in this Pull Request:
NCDFReader._read_framenow drops the pages of scipy's memory map once theframe has been read into the
Timestep.Added a test that the pages are dropped when a frame is read.LLM / AI generated code disclosure
LLMs or other AI-powered tools (beyond simple IDE use cases) were used in this contribution: yes test written by AI
PR Checklist
package/CHANGELOGfile updated?package/AUTHORS?Developers Certificate of Origin
I certify that I can submit this code contribution as described in the Developer Certificate of Origin, under the MDAnalysis LICENSE.