Running hovernet on wsi mrxs file #295
Description
Hi, I have been trying to run the wsi mode of run_infer.py on mrxs files and have been having a series of memory/time-related issues with my most recent error (and exact parameters used) copied below. I was hoping you can share some suggestions on how to process large .mrxs files both to overcome these errors but also speed up the process. Thanks.
Few additional notes:
- Just for context I am able to run the same script successfully and much faster on whole slide .svs files.
- Do you recommend converting mrxs files to svs (and is this possible?).
- I have noticed that before it crashes, each "Process Chunk" takes longer and longer to process, going from ~1 minute to even >1 hour.
- I have also gotten errors related to too many files being open.
--gpu='0' \
--nr_types=6 \
--type_info_path=type_info.json \
--batch_size=8 \
--model_mode=fast \
--model_path=hovernet_fast_pannuke_type_tf2pytorch.tar \
--nr_inference_workers=4 \
--nr_post_proc_workers=4
Process Chunk 586/756: 100%|##################| 450/450 [02:35<00:00, 2.89it/s]
Process Chunk 587/756: 100%|####################| 23/23 [01:00<00:00, 2.62s/it]
Process Chunk 588/756: 100%|################| 450/450 [1:11:54<00:00, 9.59s/it]
Process Chunk 589/756: 0%| | 0/450 [00:00<?, ?it/s]Traceback (most recent call last):
RuntimeError: DataLoader worker (pid 55817) is killed by signal: Killed.