CLIP-Head

Official Implementation of the paper "CLIP-Head: Text-Guided Generation of Textured Neural Parametric 3D Head Models" accepted at SIGGRAPH Asia 2023 Technical Communications.

Project Page | PDF

If you like our project, please give us a star ⭐ on GitHub for latest update.

TODO

• Initial Code Release
• Release Checkpoints
• Add Dreambooth support
• Release code for rendering pipeline
• Code Optimization
• Gradio demo
• Add SDXL Refiner
• Texture Synthesis with SDXL and LoRA
• Parameterization code for FLAME head model

Instructions

The code was tested on an RTX 4090 (24 GB VRAM)

• In case you face issues while building pytorch3d, try using spack to build with a different gcc version

Step 1

# Clone the repo
git clone https://github.com/raipranav384/CLIP-Head.git
cd CLIP-Head


# Create a new Anaconda environment
conda create -n clip_head python=3.9

conda activate clip_head
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

# Install NPHM dependencies
cd NPHM
mkdir checkpoints
pip install -e .
cd ..

# Install other requirements
pip install -r requirements.txt

# Install PyTorch3D
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

Step 2

• Download the NPHM's pretrained checkpoints from here and place it in ./NPHM/checkpoints
• Download checkpoints for $ControlNet_{uv}$ from here. Unzip in ./checkpoints

Step 3

python run.py --prompt "face of a man smiling with joker makeup"

Citation (will be added soon)

Acknowledgement

• Thanks to NPHM authors for providing the dataset and code.
• Thanks to the amazing work by lllyasviel on ControlNet.

Related Works

• ClipFace : Text-guided Editing of Textured 3D Morphable Models
• High-fidelity 3D Face Generation from Natural Language Descriptions