nvcomp: Unnecessary sync events and host memory transfers

[dxqb]

Please consider the sample code below.

In torch profiler, you can see that there are

• sync events, even though everything is executed on the same stream
• MemCpy (Device -> Pageable) and MemCpy (Device -> Pinned) even though nothing ever leaves GPU
• as a result, gaps in the GPU stream. GPU becomes idle and waiting

This kills performance if the decompression is interleaved with any other GPU ops.

This is what it should look like, when I remove the nvcomp decode operation:

• GPU operations are scheduled
• command buffer is full
• no gaps

import cupy
import torch
from nvidia import nvcomp
from modules.util.profiling_util import TorchProfiler
from torch import Tensor
from torch.utils.dlpack import to_dlpack
import tqdm

device='cuda'
act = torch.randint(-127, 127, (3072, 3072), device=device, dtype=torch.int8)
weight = torch.randint(-127, 127, (3072, 3072), device=device, dtype=torch.int8)

stream = torch.cuda.current_stream().cuda_stream
codec = nvcomp.Codec(algorithm="Zstd", uncomp_chunk_size=2048, cuda_stream=stream)


def encode(x: Tensor):
    array = nvcomp.as_array(x)
    compressed = codec.encode(array).cuda()
    tensor = torch.utils.dlpack.from_dlpack(compressed.to_dlpack()).clone()
    return tensor, x.shape, x.dtype



def decode(compressed_tensor: Tensor, shape: list[int], dtype: torch.dtype) -> Tensor:
    compressed_array = nvcomp.as_array(compressed_tensor, cuda_stream=stream)
    return codec.decode(compressed_array)

compressed = encode(weight)

with TorchProfiler("decompress.json", enabled=True):
    for _ in tqdm.tqdm(range(10000)):
        decode(*compressed)
        #note: this is just another GPU operation to demonstrate the sync. it doesn't even use the result of decode:
        torch._int_mm(act, weight.T)

this might be related to NVIDIA/nvcomp#105 but I'm not entirely sure because I use a different API

[naveenaero]

Hi @dxqb,
Thank you for reaching out regarding this concern.
nvCOMP's python API is built upon the nvCOMP HLIF API. During decompression, we do several checks on the encoded stream header to make sure it can be decompressed correctly. For example, we check

• Whether the codec being used to decompress is the same as the one which was used to encode
• Whether the compressed stream was generated from an old version of nvCOMP
• To calculate the output buffer size which is stored in the compressed stream header.

To be able to do this, we have to copy over the stream header to the pinned memory and do a stream synchronize.
Hope I answered your questions.

[dxqb]

Thanks!

Hi @dxqb, Thank you for reaching out regarding this concern. nvCOMP's python API is built upon the nvCOMP HLIF API.

Are you saying that this issue can be avoided by using the C++ API instead (and using self-made python bindings)?

During decompression, we do several checks on the encoded stream header to make sure it can be decompressed correctly. For example, we check

To be able to do this, we have to copy over the stream header to the pinned memory and do a stream synchronize. Hope I answered your questions.

I see. Don't you think it should be possible to do these checks on GPU-only and CPU-only without any transfers?

* Whether the codec being used to decompress is the same as the one which was used to encode

* Whether the compressed stream was generated from an old version of nvCOMP

* To calculate the output buffer size which is stored in the compressed stream header.

all of these could be separated into

• the data stream (that lives on GPU)
• metadata (that could live on CPU)

Effectively syncing CPU with GPU after each decompression is a dealbreaker for any application that seeks to decompress on GPU for parallelization and then use the decompressed data on GPU. And why else would you even want to decompress on GPU?

I have tried an implementation that schedules the decompression in advance on a separate CUDA stream, so that when the main CUDA stream wants to use the decompressed data, the performance bottleneck has already been passed.
That works, but it's quite involved and shouldn't be necessary.