Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR addresses high memory usage issues by optimizing how file contents are retrieved from containers and how submission data is transmitted to the backend. The changes focus on reducing memory footprint during file operations and data transmission.
- Modified the
getmethod inrunner/sandbox.pyto retrieve multiple files in a single operation and use streaming approach - Updated
dispatcher/dispatcher.pyto write submission data to a temporary file instead of keeping it in memory during HTTP requests - Removed debug logging that was truncating stdout/stderr for display purposes
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| runner/sandbox.py | Refactored file retrieval to batch multiple files and use streaming approach to reduce memory usage |
| dispatcher/dispatcher.py | Changed submission data handling to use temporary files instead of in-memory JSON for HTTP requests |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
runner/sandbox.py
Outdated
| tar.extractall(extract_path) | ||
| return [ | ||
| open( | ||
| f'{extract_path}/{dir}/{filename}', |
There was a problem hiding this comment.
Path construction is incorrect. The dir parameter already contains the full path (e.g., '/result/'), so concatenating extract_path/{dir}/{filename} will create an invalid path like /tmp/tmpdir//result//filename. Should be f'{extract_path}/{filename}' instead.
dumping large json payloads was consuming up to 1GB in some cases. write the payload to file before send it to lower memory consumption. in the future we might need to change the API about reporting submission result.
to prevent breaking changes in python 3.14
2 participants
No description provided.