Error loading weights for overall survival

[lucbuijs]

Dear @DivyanshuTak, @bini-a,

First of all, thank you for making this very interesting project publicly available! I am currently trying to run overall survival inference for glioma using the provided pretrained checkpoints, but I encountered an issue when loading the task-specific checkpoint.

I downloaded the following checkpoints:

• os_model.pt
• BrainIAC.ckpt

and placed them in:

src/checkpoints/

My inference configuration is:

DATASETS = {
    "survival_task": {
        "checkpoint_path": "./src/checkpoints/os_model.pt",
        "test_csv_path": "./data/csvs/survival_test.csv",
        "root_dir": "./data/images",
        "output_csv_path": "./inference/model_output/survival_predictions.csv",
        "task_type": "classification",
        "image_type": "quad",
        "num_classes": 1
    },
}


SIMCLR_CKPT_PATH = "./src/checkpoints/BrainIAC.ckpt"

But when trying to run inference I get this error:

Loading model from ./checkpoints/os_model.pt
Backbone weights loaded!!
Error : Error(s) in loading state_dict for SingleScanModelQuad:
        Missing key(s) in state_dict: "backbone.backbone.patch_embedding.position_embeddings", "backbone.backbone.patch_embedding.patch_embeddings.weight", "backbone.backbone.patch_embedding.patch_embeddings.bias", "backbone.backbone.blocks.0.mlp.linear1.weight", "backbone.backbone.blocks.0.mlp.linear1.bias", "backbone.backbone.blocks.0.mlp.linear2.weight", "backbone.backbone.blocks.0.mlp.linear2.bias", "backbone.backbone.blocks.0.norm1.weight", "backbone.backbone.blocks.0.norm1.bias", "backbone.backbone.blocks.0.attn.out_proj.weight", "backbone.backbone.blocks.0.attn.out_proj.bias", "backbone.backbone.blocks.0.attn.qkv.weight", "backbone.backbone.blocks.0.norm2.weight", "backbone.backbone.blocks.0.norm2.bias", "backbone.backbone.blocks.1.mlp.linear1.weight", "backbone.backbone.blocks.1.mlp.linear1.bias", "backbone.backbone.blocks.1.mlp.linear2.weight", "backbone.backbone.blocks.1.mlp.linear2.bias", "backbone.backbone.blocks.1.norm1.weight", "backbone.backbone.blocks.1.norm1.bias", "backbone.backbone.blocks.1.attn.out_proj.weight", "backbone.backbone.blocks.1.attn.out_proj.bias", "backbone.backbone.blocks.1.attn.qkv.weight", "backbone.backbone.blocks.1.norm2.weight", "backbone.backbone.blocks.1.norm2.bias", "backbone.backbone.blocks.2.mlp.linear1.weight", "backbone.backbone.blocks.2.mlp.linear1.bias", "backbone.backbone.blocks.2.mlp.linear2.weight", "backbone.backbone.blocks.2.mlp.linear2.bias", "backbone.backbone.blocks.2.norm1.weight", "backbone.backbone.blocks.2.norm1.bias", "backbone.backbone.blocks.2.attn.out_proj.weight", "backbone.backbone.blocks.2.attn.out_proj.bias", "backbone.backbone.blocks.2.attn.qkv.weight", "backbone.backbone.blocks.2.norm2.weight", "backbone.backbone.blocks.2.norm2.bias", "backbone.backbone.blocks.3.mlp.linear1.weight", "backbone.backbone.blocks.3.mlp.linear1.bias", "backbone.backbone.blocks.3.mlp.linear2.weight", "backbone.backbone.blocks.3.mlp.linear2.bias", "backbone.backbone.blocks.3.norm1.weight", "backbone.backbone.blocks.3.norm1.bias", "backbone.backbone.blocks.3.attn.out_proj.weight", "backbone.backbone.blocks.3.attn.out_proj.bias", "backbone.backbone.blocks.3.attn.qkv.weight", "backbone.backbone.blocks.3.norm2.weight", "backbone.backbone.blocks.3.norm2.bias", "backbone.backbone.blocks.4.mlp.linear1.weight", "backbone.backbone.blocks.4.mlp.linear1.bias", "backbone.backbone.blocks.4.mlp.linear2.weight", "backbone.backbone.blocks.4.mlp.linear2.bias", "backbone.backbone.blocks.4.norm1.weight", "backbone.backbone.blocks.4.norm1.bias", "backbone.backbone.blocks.4.attn.out_proj.weight", "backbone.backbone.blocks.4.attn.out_proj.bias", "backbone.backbone.blocks.4.attn.qkv.weight", "backbone.backbone.blocks.4.norm2.weight", "backbone.backbone.blocks.4.norm2.bias", "backbone.backbone.blocks.5.mlp.linear1.weight", "backbone.backbone.blocks.5.mlp.linear1.bias", "backbone.backbone.blocks.5.mlp.linear2.weight", "backbone.backbone.blocks.5.mlp.linear2.bias", "backbone.backbone.blocks.5.norm1.weight", "backbone.backbone.blocks.5.norm1.bias", "backbone.backbone.blocks.5.attn.out_proj.weight", "backbone.backbone.blocks.5.attn.out_proj.bias", "backbone.backbone.blocks.5.attn.qkv.weight", "backbone.backbone.blocks.5.norm2.weight", "backbone.backbone.blocks.5.norm2.bias", "backbone.backbone.blocks.6.mlp.linear1.weight", "backbone.backbone.blocks.6.mlp.linear1.bias", "backbone.backbone.blocks.6.mlp.linear2.weight", "backbone.backbone.blocks.6.mlp.linear2.bias", "backbone.backbone.blocks.6.norm1.weight", "backbone.backbone.blocks.6.norm1.bias", "backbone.backbone.blocks.6.attn.out_proj.weight", "backbone.backbone.blocks.6.attn.out_proj.bias", "backbone.backbone.blocks.6.attn.qkv.weight", "backbone.backbone.blocks.6.norm2.weight", "backbone.backbone.blocks.6.norm2.bias", "backbone.backbone.blocks.7.mlp.linear1.weight", "backbone.backbone.blocks.7.mlp.linear1.bias", "backbone.backbone.blocks.7.mlp.linear2.weight", "backbone.backbone.blocks.7.mlp.linear2.bias", "backbone.backbone.blocks.7.norm1.weight", "backbone.backbone.blocks.7.norm1.bias", "backbone.backbone.blocks.7.attn.out_proj.weight", "backbone.backbone.blocks.7.attn.out_proj.bias", "backbone.backbone.blocks.7.attn.qkv.weight", "backbone.backbone.blocks.7.norm2.weight", "backbone.backbone.blocks.7.norm2.bias", "backbone.backbone.blocks.8.mlp.linear1.weight", "backbone.backbone.blocks.8.mlp.linear1.bias", "backbone.backbone.blocks.8.mlp.linear2.weight", "backbone.backbone.blocks.8.mlp.linear2.bias", "backbone.backbone.blocks.8.norm1.weight", "backbone.backbone.blocks.8.norm1.bias", "backbone.backbone.blocks.8.attn.out_proj.weight", "backbone.backbone.blocks.8.attn.out_proj.bias", "backbone.backbone.blocks.8.attn.qkv.weight", "backbone.backbone.blocks.8.norm2.weight", "backbone.backbone.blocks.8.norm2.bias", "backbone.backbone.blocks.9.mlp.linear1.weight", "backbone.backbone.blocks.9.mlp.linear1.bias", "backbone.backbone.blocks.9.mlp.linear2.weight", "backbone.backbone.blocks.9.mlp.linear2.bias", "backbone.backbone.blocks.9.norm1.weight", "backbone.backbone.blocks.9.norm1.bias", "backbone.backbone.blocks.9.attn.out_proj.weight", "backbone.backbone.blocks.9.attn.out_proj.bias", "backbone.backbone.blocks.9.attn.qkv.weight", "backbone.backbone.blocks.9.norm2.weight", "backbone.backbone.blocks.9.norm2.bias", "backbone.backbone.blocks.10.mlp.linear1.weight", "backbone.backbone.blocks.10.mlp.linear1.bias", "backbone.backbone.blocks.10.mlp.linear2.weight", "backbone.backbone.blocks.10.mlp.linear2.bias", "backbone.backbone.blocks.10.norm1.weight", "backbone.backbone.blocks.10.norm1.bias", "backbone.backbone.blocks.10.attn.out_proj.weight", "backbone.backbone.blocks.10.attn.out_proj.bias", "backbone.backbone.blocks.10.attn.qkv.weight", "backbone.backbone.blocks.10.norm2.weight", "backbone.backbone.blocks.10.norm2.bias", "backbone.backbone.blocks.11.mlp.linear1.weight", "backbone.backbone.blocks.11.mlp.linear1.bias", "backbone.backbone.blocks.11.mlp.linear2.weight", "backbone.backbone.blocks.11.mlp.linear2.bias", "backbone.backbone.blocks.11.norm1.weight", "backbone.backbone.blocks.11.norm1.bias", "backbone.backbone.blocks.11.attn.out_proj.weight", "backbone.backbone.blocks.11.attn.out_proj.bias", "backbone.backbone.blocks.11.attn.qkv.weight", "backbone.backbone.blocks.11.norm2.weight", "backbone.backbone.blocks.11.norm2.bias", "backbone.backbone.norm.weight", "backbone.backbone.norm.bias", "classifier.fc.weight", "classifier.fc.bias".  
        Unexpected key(s) in state_dict: "model_state_dict", "loss", "epoch", "f1", "auc_roc".

So I was wondering whether the released checkpoints (os_model.pt, BrainIAC.ckpt) are compatible with the current implementation of SingleScanModelQuad? And if so, could you provide the correct checkpoint format or instructions to convert the provided weights?

Any guidance would be greatly appreciated!

Thanks again for sharing this work.