Train on High Resolution RGB Images

[ibrahim-azista]

I am very excited about this project, I love and appreciate the time and effort you guys have taken to advance the EO community. Also, loved the introductory webinar, learned a lot.

I understand if you think this is a stupid question, but, I want help with a task. I am currently thinking about working on a semantic segmentation task, I was envisioning using the Clay model as the feature extractor (to create embeddings) and then use U-net as a decoder. I wanted help with implementing this, It would be very helpful if you could refer me to a notebook that has roughly the type of thing I am trying to do.

I read the documentation on finetuning the model.

Thank you so much in advance.

[shub-garg]

https://github.com/qubvel/segmentation_models

Try this repository. Doesn't have the Clay model, but might be useful.

[ibrahim-azista]

I do have a pipeline built using segmentation models pytorch. I think I may not have been clear sorry, suppose I used the smp.Unet() as my backbone (decoder), is it possible for me to integrate the clay model to the pipeline as the encoder?

[srmsoumya]

@ibrahim-azista It is possible to add an SMP decoder to CLAY's encoder for finetuning, but the results may not be optimal. This is because we are extracting only the final layers of the model, which are downscaled 16 times, while the task of the model is to predict the segmentation mask at the native image resolution.

For segmentation using MAE-style architectures, it is better to extract features from intermediate layers as well and then attach a neck/head (decoder) for segmentation. We have a basic implementation using a simple decoder from Segformer, and it works with a small subset of training data in just a few epochs. I recommend starting with this and then trying the following approaches in order:

Replace the Segformer decoder with PSPNet or UPerNet to see if there are any improvements.
Add adapters (to inject information back into the network), extract features, and then use a decoder for segmentation.
Unfreeze the encoder to finetune it as well.
We will also experiment with these methods and share the notebooks/code in the future. Please feel free to open PRs if you implement any of these ideas.

[lauracchen]

Hi @ibrahim-azista, @srmsoumya just merged this segmentation decoder: https://github.com/Clay-foundation/model/tree/main/finetune/segment. Feel free to check that out!

If you want help with just generating embeddings to plug into your U-net, this tutorial shows you how to go through the full model workflow. You'd likely pause at step 5 outlined here, and then use those embeddings in your own decoder. Hope that helps!