Running Example Error Occur

[VLadImirluren]

root@app-033c564329764e5895175944eb838871-66d8ddb486-ptb4v:/mnt/afs/xx/codes/GSH/Extend3D# python example2.py
/usr/local/lib/python3.10/dist-packages/torch/utils/_pytree.py:185: FutureWarning: optree is installed but the version is too old to support PyTorch Dynamo in C++ pytree. C++ pytree support is disabled. Please consider upgrading optree using python3 -m pip install --upgrade 'optree>=0.13.0'.
warnings.warn(
[SPARSE] Backend: spconv, Attention: flash_attn
[SPARSE][CONV] spconv algo: native
[ATTENTION] Using backend: flash_attn
/usr/local/lib/python3.10/dist-packages/torch/utils/_pytree.py:185: FutureWarning: optree is installed but the version is too old to support PyTorch Dynamo in C++ pytree. C++ pytree support is disabled. Please consider upgrading optree using python3 -m pip install --upgrade 'optree>=0.13.0'.
warnings.warn(
/mnt/afs/xx/codes/GSH/Extend3D/dinov2/layers/swiglu_ffn.py:51: UserWarning: xFormers is not available (SwiGLU)
warnings.warn("xFormers is not available (SwiGLU)")
/mnt/afs/xx/codes/GSH/Extend3D/dinov2/layers/attention.py:33: UserWarning: xFormers is not available (Attention)
warnings.warn("xFormers is not available (Attention)")
/mnt/afs/xx/codes/GSH/Extend3D/dinov2/layers/block.py:40: UserWarning: xFormers is not available (Block)
warnings.warn("xFormers is not available (Block)")
Sparse Structure Sampling: 100%|████████████████████████████████████████████| 25/25 [00:59<00:00, 2.38s/it]
Sparse Structure Sampling: 100%|████████████████████████████████████████████| 25/25 [00:59<00:00, 2.36s/it]
Sparse Structure Sampling: 100%|████████████████████████████████████████████| 25/25 [00:58<00:00, 2.35s/it]
100%|█████████████████████████████████████████████████████████████████████████| 3/3 [02:58<00:00, 59.40s/it]
Structured Latent Sampling: 100%|███████████████████████████████████████████| 25/25 [01:43<00:00, 4.12s/it]
[Exception|indice_conv|subm]feat=torch.Size([882752, 96]),w=torch.Size([96, 3, 3, 3, 96]),pair=torch.Size([2, 27, 882752]),pairnum=tensor([741779, 814132, 741789, 770896, 846144, 770440, 742256, 814188, 741420,
771244, 847472, 771668, 803120, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0],
device='cuda:0', dtype=torch.int32),act=882752,algo=ConvAlgo.Native
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
Traceback (most recent call last):
File "/mnt/afs/xx/codes/GSH/Extend3D/example2.py", line 10, in
output = pipeline.run(image)
File "/mnt/afs/xx/codes/GSH/Extend3D/extend3d.py", line 756, in run
decoded = self.decode_slat(slat, width, length, formats=formats)
File "/mnt/afs/xx/codes/GSH/Extend3D/extend3d.py", line 517, in decode_slat
patch_hr = mesh_decoder.forward_features(patch_latent)
File "/mnt/afs/xx/codes/GSH/Extend3D/trellis/models/structured_latent_vae/decoder_mesh.py", line 167, in forward_features
h = block(h)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/afs/xx/codes/GSH/Extend3D/trellis/models/structured_latent_vae/decoder_mesh.py", line 67, in forward
h = self.out_layers(h)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 250, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/afs/xx/codes/GSH/Extend3D/trellis/modules/sparse/conv/conv_spconv.py", line 26, in forward
new_data = self.conv(x.data)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/spconv/pytorch/conv.py", line 755, in forward
return self._conv_forward(self.training,
File "/usr/local/lib/python3.10/dist-packages/spconv/pytorch/conv.py", line 327, in _conv_forward
out_features = Fsp.indice_subm_conv(
File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py", line 514, in decorate_fwd
return fwd(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/spconv/pytorch/functional.py", line 327, in forward
raise e
File "/usr/local/lib/python3.10/dist-packages/spconv/pytorch/functional.py", line 308, in forward
return ops.indice_conv(features,
File "/usr/local/lib/python3.10/dist-packages/spconv/pytorch/ops.py", line 860, in indice_conv
ConvGemmOps.indice_conv(alloc, ext_mm, GEMM_CPP, ALL_WEIGHT_IS_KRSC,
RuntimeError: TensorStorage /tmp/pip-build-env-36bib9o0/overlay/lib/python3.10/site-packages/cumm/include/tensorview/tensor.h:172
cuda failed with error 2 out of memory. use CUDA_LAUNCH_BLOCKING=1 to get correct traceback.

Device: H100

example2.py code:

from extend3d import Extend3D
from PIL import Image
import imageio

from trellis.utils import render_utils, postprocessing_utils

pipeline = Extend3D.from_pretrained('/mnt/afs/xx/ckpt/trellis').cuda()
image = Image.open("assets/examples/2.png")

output = pipeline.run(image)

video = render_utils.render_video(output['gaussian'][0], r=1.6, resolution=1024)['color']

imageio.mimsave('sample_gs.mp4', video, fps=30)

glb = postprocessing_utils.to_glb(
output['gaussian'][0],
output['mesh'][0],
simplify=0.9,
texture_size=1024
)
glb.export(os.path.join(args.output_dir, 'sample.glb'))

[Seungwoo-Yoon]

Hi @VLadImirluren, thank you for your interest in our work!

It seems that the mesh decoding of TRELLIS produces OOM error. Although the minimum requirement of memory is 24GB in our cases, it may depend on devices, input, or random seed. In this case, I recommend to output Gaussian only, as our method primarily focuses on the Gaussian results. As Gaussian decoding is also used is SLat generation, I think it will work. Especially for mesh_decoder.forward_features, as it is not part of our contribution, you may follow some guide of TRELLIS.

Thank you!

[yjx2851]

I encounter the same problem as my device is H20.