about hardware requirement and checkpoint release

[NeutrinoLiu]

Hi great project! thx for releasing the code!

I have two questions, first is there any minimum requirement for each step (data preprocess, generative model training, and model inference). Sad truth is that not every lab has a bundle of 8x80G GPU.
Another question is, instead of training our own generative model (vae, diffusion, GSM), will it be possible for you to release the well trained checkpoint directly?

Thx for your great effort again!

[yifanlu0227]

Hi!
The minimum requirements would be

1. data preprocessing: at least 24 GB VRAM GPU. You can shrink the chunk size in segformer & metric3d inference to save memory
2. generative model training: 80 GB VRAM GPU. This is a must.
3. model inference: 20 GB VRAM GPU

Unfortunately, due to the company constraint, we can not release the checkpoints. Sorry for the inconvenience.

[Debrove]

1. data preprocessing: at least 24 GB VRAM GPU. You can shrink the chunk size in segformer & metric3d inference to save memory

Hi! Thanks for your great work!

I have tried to shrink the chunk size in metric3d inference, i.e chunking_num=1. , but it still cause OOM. BTW, the GPU is 24GB 4090.

Looking forward to your reply! Thank you!

[yifanlu0227]

You can append cpu tensor of pred_depth instead of cuda tensor

SCube/datagen/scube_data_utils.py

Line 228 in ed0a62f

pred_depth_chunks.append(pred_depth) # shape (T, 1, H, W)

by

 pred_depth_chunks.append(pred_depth.cpu()) # shape (T, 1, H, W) 

This also saves some memory

[Debrove]

@yifanlu0227 Thanks for your quick reply and your suggestion!!

It still cause OOM. The OOM occurs during the model inference.

It sovled by using ViT-small insted of ViT-Large. And I set chunking_num = 200 (batch size = 1).

[ZhAnGToNG1]

May I ask if 4 GPUs with 49GB can be used to train generative model? @yifanlu0227

[yifanlu0227]

we don't have a 49GB gpu for testing. but the peak memory in our experiment exceeds that.