Problem
When using the HuggingFace source in this repository—which may spawn many concurrent simd-r-drive handles—the code does not currently account for the system hitting OS ulimits (like the maximum number of open file descriptors). If the ulimit is reached, it is unclear and currently undefined how these handles should be managed, often leading to failures or unhandled exceptions.
Desired Behavior
- Proactively detect when OS ulimits (such as file handles) are approached or reached while handling concurrent simd-r-drive operations from a HuggingFace source.
- Clearly define and implement a strategy for how simd-r-drive handles should be managed or evicted when limits are reached (e.g., queue, delay, close, or evict handles appropriately).
- Ensure the mechanism is robust, prevents panics/crashes, and logs actions for observability.
- Retain high performance and predictable resource usage wherever possible.
Example Use Case
- Using the HuggingFace source to parallelize multiple downloads, leading to a large number of simd-r-drive handles being open at once. If the process exceeds
ulimit -n, the system should handle this situation gracefully based on a well-defined eviction or waiting policy, rather than failing unpredictably.
Label: enhancement
Problem
When using the HuggingFace source in this repository—which may spawn many concurrent simd-r-drive handles—the code does not currently account for the system hitting OS ulimits (like the maximum number of open file descriptors). If the ulimit is reached, it is unclear and currently undefined how these handles should be managed, often leading to failures or unhandled exceptions.
Desired Behavior
Example Use Case
ulimit -n, the system should handle this situation gracefully based on a well-defined eviction or waiting policy, rather than failing unpredictably.Label: enhancement