Refactor/lazy parsing remove sentinel in parser#26
Merged
Conversation
DragerBinParser.parse() now returns the raw memmap directly without copying to RAM or replacing sentinel values (-1000.0, 0xFF7FC99E). Sanitization is the responsibility of the preprocessing layer, which ensures files are never loaded into RAM during batch parsing. - Remove steps 6-8 (copy + replace_sentinels for pixels and Medibus) - Remove _FLOAT_SENTINELS/_BIT_SENTINELS/_replace_no_data_sentinels imports - Remove __init__ sentinel lists (no longer needed in parser) - Update test: sentinel values are now preserved in raw parser output
parse_memmap() now returns raw memmap pixels without calling replace_no_data_sentinels — consistent with the parser layer change. Sentinel values are preserved; replacement is preprocessing responsibility.
parse_memmap() now returns a lazy view with no pixel array allocated in RAM. Update interpretation to reflect that sentinel replacement is deferred and memmap peak RAM is near zero (only virtual mapping overhead).
Re-run after removing sentinel replacement from parse_memmap(): - memmap time: 0.000 s at all sizes (pure mapping setup, no pixel reads) - memmap peak RAM: 0.0 MB (no pixel array allocated — lazy view only) - struct loop unchanged as expected Fix grammar in interpretation cell. Remove redundant inline comment in test_bin_parser.py.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Owner
Author
|
failed gitworkflow |
Owner
Author
|
failed gitworkflow |
Owner
Author
|
fixed. GH action |
Owner
Author
|
close issue #24 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
avoid completely to upload data in ram during parsing phase. Solve issue #24