HBEVOcc: Height-aware Bird's-Eye-View Representation for 3D Occupancy Prediction from Multi-Camera Images

🤗 Model Weights

We utilize 8 RTX2080Ti GPUs to train our model.

Setting	Epochs	Training MeM (Batch Size=1)	RayIoU	mAVE	Weights (Batch Size=2) Logs
r50_256x704_1f_openocc	24	7.3GB	39.4	0.52	Google-drive
r50_256x704_8f_openocc	24	7.5GB	40.8	0.41	Google-drive
r50_256x704_8f_openocc	48	7.5GB	41.4	0.39	Google-drive

The code for the Occ3D dataset is coming soon..

🛠️Environment

Install Pytorch 1.13 + CUDA 11.6

conda create --name hbevocc python=3.8
conda activate hbevocc
pip install torch==1.13.0+cu116 torchvision==0.14.0+cu116 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu116
pip install mmcv-full==1.7.0 -f https://download.openmmlab.com/mmcv/dist/cu117/torch1.13/index.html
pip install mmdet==2.28.2
pip install mmsegmentation==0.30.0
pip install mmengine
pip install -v -e .
pip install numpy==1.23.4
pip install yapf==0.40.1
pip install setuptools==59.5.0
pip install ninja
pip install einops
pip install open3d==0.16.0 -i https://pypi.tuna.tsinghua.edu.cn/simple

🏗 Prepare Dataset

1. Download nuScenes from nuScenes
2. Download Occ3D-nus from Occ3D-nus
3. Download OpenOcc from OpenOcc-OpenDataLab or OpenOcc-Google Drive
4. Download the generated info file from Google Drive and unzip it to the data/nuscenes folder. Rename them to 'hbevocc-nuscenes_infos_train.pkl' and 'hbevocc-nuscenes_infos_val.pkl'. These *pkl files can be generated by running the tools/create_data_bevdet.py
5. Download the pretrained weights from BEVDet
6. Organize your folder structure as below:

├── project
├── ckpts/
│   ├── bevdet-r50-4d-stereo-cbgs.pth
│   ├── bevdet-r50-4dlongterm-stereo-cbgs.pth
├── data/
│   ├── nuscenes/
│   │   ├── samples/ 
│   │   ├── v1.0-trainval/
│   │   ├── gts/ (Occ3D-nus)
│   │   ├── openocc_v2/
│   │   ├── hbevocc-nuscenes_infos_train.pkl
│   │   ├── hbevocc-nuscenes_infos_val.pkl

💡 Training and Evaluation

Train HBEVOcc with 8 GPUs:

bash tools/dist_train.sh config/hbevocc/hbevocc-r50-256x704-1f-openocc.py 8
bash tools/dist_train.sh config/hbevocc/hbevocc-r50-256x704-8f-openocc.py 8

Evaluate HBEVOcc with 1 GPU or 8 GPUs:

bash tools/dist_test.sh config/hbevocc/hbevocc-r50-256x704-1f-openocc.py path/to/ckpts 1
bash tools/dist_test.sh config/hbevocc/hbevocc-r50-256x704-8f-openocc.py path/to/ckpts 8

🧩 Results

🙏 Acknowledgement

Many thanks to these excellent projects:

• STCOcc
• FlashOCC
• BEVDet
• FB-Occ
• BEVFormer
• DAT
• mmdetection3d