MAHOUT-725: [QDP] PyTorch Tensor Detection and CPU Path

[CheyuWu]

Purpose of PR

This issue asks the ability to detect and handle PyTorch tensors, focusing on cpu tensors.

1. add PyTorch tensor detection utilities
2. add tensor validation
3. add cpu tensor path in encode_tensor()
4. add error handling
5. add basic tests

Related Issues or PRs

Closes #725

Changes Made

• Bug fix
• New feature
• Refactoring
• Documentation
• Test
• CI/CD pipeline
• Other

Breaking Changes

• Yes
• No

Checklist

• Added or updated unit tests for all changes
• Added or updated documentation for all changes
• Successfully built and ran all unit tests or manual tests locally
• PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
• Code follows ASF guidelines

[CheyuWu]

Manual Testing

(mahout) user@DESKTOP-UDTSV2K:~/code/mahout$ cd ./qdp/qdp-python/
(mahout) user@DESKTOP-UDTSV2K:~/code/mahout/qdp/qdp-python$ cp ../target/debug/libmahout_qdp.so mahout_qdp.so && PYTHONPATH=. pytest .
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.13, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/user/code/mahout/qdp/qdp-python
configfile: pyproject.toml
collected 21 items                                                                                                                                                      

tests/test_bindings.py ...s.....                                                                                                                                  [ 42%]
tests/test_high_fidelity.py ............                                                                                                                          [100%]

===================================================================== 20 passed, 1 skipped in 3.30s =====================================================================

[400Ping]

please fix the ci error

[CheyuWu]

Hi @400Ping, I have fixed the problem PTAL

[400Ping]

I think overall is good but we are going to merge #753 into dev-qdp, I think it is best to wait for it to merge and fix the conflicts for this pr.

[400Ping]

Its merged, please taken account how other inputs are written and refactor this pr.

[CheyuWu]

Its merged, please taken account how other inputs are written and refactor this pr.

OK, I will address this tomorrow

[rich7420]

plz fix precommit errors

[CheyuWu]

Hi @rich7420 @400Ping , I’ve addressed the issue.
BTW, the numpy package was missing from pyproject.toml, so I added it.

Manual Testing

(qdp-python) user@DESKTOP-UDTSV2K:~/code/mahout/qdp/qdp-python$ PYTHONPATH=. python -m pytest .
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.13, pytest-9.0.1, pluggy-1.6.0
rootdir: /home/user/code/mahout/qdp/qdp-python
configfile: pyproject.toml
collected 21 items                                                                                                                                                      

tests/test_bindings.py ...s.....                                                                                                                                  [ 42%]
tests/test_high_fidelity.py ............                                                                                                                          [100%]

===================================================================== 20 passed, 1 skipped in 2.63s =====================================================================

[400Ping]

@CheyuWu see this #752 (comment)

[ryankert01]

@CheyuWu feel free to help review this related PR #777 ~ It's kind of complex to add a input data format.

[400Ping]

cc @guan404ming @rich7420 @ryankert01 to review
I think we can merge this first and I will open a pr for handling PyTorch input format.

[rich7420]

@CheyuWu plz fix this pre-commit error

[CheyuWu]

Let me check what is going on

[rich7420]

It seems here do twice copy.

PyTorch tensor → Python list
Python list → Rust Vec<f64>

We should improve this in follow-up PR use like numpy() or PinnedHostBuffer way to decrease memory copy time.
Please comments in this part.

[CheyuWu]

OK no problem
I have added the comment PTAL

[rich7420]

I think overall LGTM
It will be merged when all green.

[CheyuWu]

This is modified by ruff format

[rich7420]

I think this is a unrelated format error. It should be reverted.
cc @ryankert01

[CheyuWu]

I have reverted the code PTAL

[rich7420]

@400Ping do you want to take another look?