Develop a better laser point detection algorithm

[mzur]

We collected the following usage statistics from biigle.de:

Laser point detection was attempted on 159368 images. Of these detections, 103059 (65 %) were successful. Of the successful cases, 23423 (23 %) were automatic detections. This means that only 15 % of the attempted laser point detections were successful automatic detections.

If we assume that users prefer automatic detections and only annotate manually if the detection fails, we could conclude that the laser point detection algorithm (as implemented here) does not work very well. There could be a need for a better and carefully tested algorithm that can replace DELPHI.

[mzur]

It's probably possible to train a deep learning laser point detection model based on the data we have gathered in BIIGLE. This could be investigated in a student project.

[mzur]

We want to start working on this now and replace Delphi with a DL-based approach. A few notes:

• The detection for a whole volume is initiated in the ProcessVolumeDelphiJob. it first processes all images with manual laser point annotations. This should work the same later. Then it executes the "gather" step of Delphi with a subset of annotated images. Finally, it submits a ProcessDelphiJob for all images that have no manually annotated laser points.

• The jobs use the DelphiGather and DelphiApply support classes to execute Python scripts.

• The Python scripts are delphi_gather.py, delphi_gather_finish.py and delphi_apply.py.

• The ProcessImageDelphiJob does the Delphi detection on a single image.

All these places have to be updated for the new DL-based approach. Instead of running one job for each image on the CPU, a single job for all (or maybe a chunk of 1000) images on the GPU may be more efficient. This is because loading a DL model on the GPU takes time and once the model is loaded, it is much faster to process a batch of files already lying around on the local disk.

[mads-reher]

We have created a new colour-threshold based LP detection algorithm, that also tries to predict the possible LP positions using lines fitted over detected points. The algorithm works by:

1. Calibrating its parameters over a small number of hand annotated LPs, similar to Delphi.
2. Detecting LPs over all images (or a subset) to create the lines.
3. Detecting LPs over all images, with everything but the lines masked out.

As this algorithm needs to iterate over all images and gather results multiple times, it would also profit from being run in a single large job. I would propose as input:

• A list of all image paths of the volume.
• A list of lists of hand annotated LPs.
• A list of the image paths for the hand annotated LPs.

I'm not quite sure which output format would be best for integration into the current workflow.

[mzur]

Typically, a job like this gets a JSON file that includes a map of image IDs to their local file paths and additional information. In your case the additional information can be the list of laser points (with their associated image IDs). This should give you all the input you need. The output can be a JSON file that maps each image ID to a list of detected laser points or, if there was an error, additional information about the error.

That said, let's talk about your point 2., creating the lines. If this can be done on a subset of images, it could be implemented as a preprocessing step similar to Delphi. Did you do an experiment like calculating the lines only on 100 randomly chosen images of a total of 1000, or something like this? Do you think this could work?

The thing is, if we implement a job that wants to run on all images of a volume, at some point there will be a volume that is too large. If we can split the work between a preprocessing step (creating parameters and lines) that works on a subset of images and then the actual processing step (detecting laser points on each image), where the second step can be parallelized for each image, this would scale much better.

[mads-reher]

Splitting the steps like this should work. Line creation depends on how difficult the LPs are to detect and the movement and objects in the image, but should not take more than 100-200 images from my experience.
I will start to work on the python files for this workflow.

[mzur]

A test with one of your datasets and random sampling of 100 images to create the lines would also be great, so we can compare the detection results.

[dlangenk]

Some more notes:

delphi_gather.py <image_path> <laserpoints> <delphi_gather_output_path>

gets an image_path with manually annotated laserpoints and saves a mask_image (everything 0 except for 24 pixels around lps), lp_prototypes (the rgb values of the marked lps), lp_neg_prototypes (rgb values of points around the marked lps), all_laserpoints (the cocordinates of the lps) saved as a compressed npz

delphi_gather_finish.py <delphi_gather_output_path>

post_processes the positive lps and eliminates lps that have a cityblock distance of less than 49 and removes the negative lps from the output.

delphi_apply.py <delphi_gather_output_path> <imgFile> <laserdistance>

actually finds the lps and computes the footprint

detect.py <imgFile> <laserdistance>

computes the footprint for manually annotated files