cuSparseLt python bindings & performance questions

[dxqb]

Hi all,

1. it would be great to have some python bindings for cuSparseLt, because it'll be a while until pytorch supports this for all dtypes, especially low-precision such as int8
2. Using the C++ API, I'm only getting about 30% more speed for sparse int8 x int8 compared to dense torch._int_mm. Is that expected? I would have expected more given the hardware claims of twice the speed.

I'm pretty sure int8xint8 isn't bandwidth-limited on any modern GPU, is it?

[j4yan]

Hi @dxqb

1. Thanks for the request. We don’t currently plan to provide official Python wrappers. We will re-evaluate based on demand and resourcing.
2. What's your problem dimensions and GPU? 30% could be reasonable for small problems. For large problems it could be up to 70%. 2x is the theoretical speedup and in practice it's almost impossible to achieve it.

[dxqb]

Thanks!
I don't have it in front of me, but I usually test with a sequence len of a few thousand and a feature dim of 3072. So probably something like
(2048, 3072) x (3072, 3072)

[j4yan]

Please share the GPU you are using.

I'm pretty sure int8xint8 isn't bandwidth-limited on any modern GPU, is it?
Whether it's bandwidth bound or not mainly depends on the problem sizes rather than data time.