-
|
Hi all, Currently I'm trying to quantize Qwen 7B model so it can fit into Jetson Orin Nano. The previous repo works on target side but my desktop's Nvidia seems too new (RTX 5060 Ti 16GB). So, it won't work. However, this repo can works on my desktop but I'm not sure if it's going to work on Orin according to the guide. Perhaps any other idea? Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Thanks for the discussion! We are planning to be compatible with Jetpack 6.2 series in our next releases. If your Jetson Orin Nano is such platform we could support it. The same code should work on your desktop GPU and Jetson if the precision is int4 or fp16. (Orin does not support fp8 or fp4) |
Beta Was this translation helpful? Give feedback.
Thanks for the discussion! We are planning to be compatible with Jetpack 6.2 series in our next releases. If your Jetson Orin Nano is such platform we could support it. The same code should work on your desktop GPU and Jetson if the precision is int4 or fp16. (Orin does not support fp8 or fp4)