CTRG

Implemention of paper 《Structure Observation Driven Image-Text Contrastive Learning for Computed Tomography Report Generation》

Data Preparation

1. Download the dataset from CT-RATE. And then preprocess data use scripts in ./data_preprocess (update the datepath in corresponding script first).
2. Download the CT-CLIP model from https://github.com/ibrahimethemhamamci/CT-CLIP.
3. Download the pretrained text encoder: CXR-BERT-general.
4. Download the LLM text decoder: LLaMA-2-7B.
5. Download the pre-extracted text embedding from Here.

Environment Setup

1. Install dependencies:

   pip install -r requirements_final.txt
   

2. Install CT-CLIP package following its official instructions.
3. install ctvit:

   cd ctvit
   pip install -e .
   

Training

1. Pretrain: Update the model and data paths in /CTRG/config.py, run

   ./run_scripts/pretrain_3D.sh
   

2. To reduce memory consumption during finetuning, extract visual features after pretraining: Set the pretrained model path in pretrained_visual_feature_extract.py, run

   ./run_scripts/visual_feature_extract.sh
   

3. Finetuning: Update /CTRG/config.py with the correct model and data paths (including the pretraining checkpoint from step 1).

   ./run_scripts/finetune_rg.sh
   

Inference

Run the inference script, then a .txt file containing the generated reports will be saved in the checkpoint folder. You can then compute your desired evaluation metrics (e.g., BLEU, ROUGE, etc.) on this output:

./run_scripts/finetune_rg_test.sh