CUDA error: an illegal memory access was encountered

[zacharykzhao]

I encountered a cuda access error.

Traceback (most recent call last):
File "/Quantization/MIXQ-main/benchflops.py", line 313, in
main(args)
File "/Quantization/MIXQ-main/benchflops.py", line 268, in main
stats, model_version = run_round(
^^^^^^^^^^
File "/Quantization/MIXQ-main/benchflops.py", line 222, in run_round
raise RuntimeError(ex)

File "/Quantization/MIXQ-main/benchflops.py", line 121, in generate
out = model(inputs, use_cache=True)

File "/Quantization/MIXQ-main/mixquant/modules/linear.py", line 199, in forward
if cache.x_scale[0:M].max() > self.sigma / (( 2 ** (self.bit - 1) - 1 ) ) :

RuntimeError: CUDA error: an illegal memory access was encountered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

I have located that the cache.x_sacle is not accessible at this point. May I have potential source of the errors?

Will the cuda toolkit version cause the error?

My envs:

CUDA Version: 12.4

torch 2.4.0+cu124
transformers 4.45.1

[coyslab]

I encountered a cuda access error.

Traceback (most recent call last): File "/Quantization/MIXQ-main/benchflops.py", line 313, in main(args) File "/Quantization/MIXQ-main/benchflops.py", line 268, in main stats, model_version = run_round( ^^^^^^^^^^ File "/Quantization/MIXQ-main/benchflops.py", line 222, in run_round raise RuntimeError(ex)

File "/Quantization/MIXQ-main/benchflops.py", line 121, in generate out = model(inputs, use_cache=True)

File "/Quantization/MIXQ-main/mixquant/modules/linear.py", line 199, in forward if cache.x_scale[0:M].max() > self.sigma / (( 2 ** (self.bit - 1) - 1 ) ) :

RuntimeError: CUDA error: an illegal memory access was encountered Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

I have located that the cache.x_sacle is not accessible at this point. May I have potential source of the errors?

Will the cuda toolkit version cause the error?

My envs:

CUDA Version: 12.4

torch 2.4.0+cu124 transformers 4.45.1

I have encountered the same problem with you at the line "if cache.x_scale[0:M].max() > self.sigma / (( 2 ** (self.bit - 1) - 1 ) ) :". How did you solve it?

[Qcompiler]

hello, may I ask that the bug is found in the 4 bit mixed-precision or 8 bit mixed-precesion case?
may I ask which platform (H100 or A100) is used ? thanks a lot

[coyslab]

hello, may I ask that the bug is found in the 4 bit mixed-precision or 8 bit mixed-precesion case? may I ask which platform (H100 or A100) is used ? thanks a lot

I have only tested the 8 bit mixed-precesion method in H20 platform. Then encountered this error

[wangsen31]

I have the same problem on H20. It seems all the tensors including the input tensor x cannot be accessed at this point
(Pdb) p x
*** RuntimeError: CUDA error: an illegal memory access was encountered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(Pdb) p x.shape
torch.Size([1, 13, 4096])