[QDP] Numpy IO support

[ryankert01]

Purpose of PR

ran some benchmark + test(maybe) on colab.
https://colab.research.google.com/drive/1NuAyAFsko3uD8MMBTDQpob4zSxg_GeiC?usp=sharing

Related Issues or PRs

Closes #722

Changes Made

• Bug fix
• New feature
• Refactoring
• Documentation
• Test
• CI/CD pipeline
• Other

Breaking Changes

• Yes
• No

Checklist

• Added or updated unit tests for all changes
• Added or updated documentation for all changes
• Successfully built and ran all unit tests or manual tests locally
• PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
• Code follows ASF guidelines

[rich7420]

@ryankert01 thanks for the patch

[rich7420]

Should we add a file size check like other format to prevent OOM?

[rich7420]

I think we could add this path.exists() in other format.

[rich7420]

I think this one it's great!

[rich7420]

overall LGTM

[400Ping]

Should we add a file size check like other format to prevent OOM?

I agree with this one.

[400Ping]

I have thought of 2 more memory-friendly alternatives to the current read_npy -> Array2 -> flatten Vec flow:

• streaming/iterator reading: we parse the .npy header (dtype/shape/order) and then iterate the flat data from the file in small chunks (so the file doesn’t need to fit in RAM).
• memory-mapping (mmap): we map the .npy file into memory (OS loads pages on demand), parse the header to locate the data region, and avoid an extra “read + flatten copy” peak while also enabling easy slicing/random access.

We can do this in a follow up if needed.

[400Ping]

Overall LGTM

[guan404ming]

Should we add a file size check like other format to prevent OOM?

This one is a nice suggestion!