Skip to content

memory allocation size accuracy fix - #1377

Merged
biochem-fan merged 1 commit into
3dem:masterfrom
nebiolabs:fix/gpu-allocator-stale-free-state
Oct 15, 2026
Merged

biochem-fan merged 1 commit into
3dem:masterfrom
nebiolabs:fix/gpu-allocator-stale-free-state

Conversation

@bwlang

@bwlang bwlang commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

without this fix i was seeing GPU runs crashing with errors like these:

  • "illegal memory access" errors inside RELION's GPU allocator (custom_allocator.cuh)
  • segfaults in CudaCustomAllocator::_free
  • an out-of-memory report asking for 564 TB
  • "No orientation was found as better than any other"
  • NVIDIA Xid 31 and Xid 13 ("Out Of Range Address") in the kernel log, on two different GPU

with this fix it's working.

@biochem-fan

biochem-fan commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Thank you very much. I merged an earlier PR #1375 that had the same fix.

Your PR contains another fix about memory alignment.
Could you please provide the detail? Does it cause crash, or inefficiency?

@bwlang
bwlang force-pushed the fix/gpu-allocator-stale-free-state branch from a0580bb to 18bea71 Compare October 15, 2026 11:07
@bwlang

bwlang commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

I've rebased...

This PR is now a "correctness" fix... i didn't test it separately from the #1375 parallel fix though so I'm not sure of often it bites in practice.

Allocations above 16MB will round incorrectly I think, leading to a too small allocation and potential writes past the end of the allocated region if the entire block is used.

@bwlang bwlang changed the title Fix stale free state handling in custom allocator for CUDA and HIP memory allocation size accuracy fix Oct 15, 2026
@biochem-fan

Copy link
Copy Markdown
Member

OK, I understood the logic. float32 has 9 exponent bits and 23 fraction bits and 16 MB = 2^(23 + 1). So beyond 16 MB some lower bits are rounded. I agree with your point.

I merge this. Thank you very much.

@biochem-fan
biochem-fan merged commit afdd816 into 3dem:master Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants