Using Segment Anything (SAM) and Autodistill to Automatically Label Multispectral Data While Rapidly Decreasing Model Creation Time

Abstract

While serving in the military, Maj. Gallagher saw that real-time object identification and image labeling both need a significant amount of time. Segment Anything (SAM) is drastically accelerating the speed and accuracy of automatically labeling large datasets by means of easy-to-use prompts from the COCO dataset. However, SAM is unable to effectively label images outside of the visible light spectrum. This research will create a method to label LWIR and fused RGB-LWIR imagery by utilizing segmentation masks from RGB imagery as the keystone layer to transpose segmentation masks from the RGB layer to the LWIR and RGB-LWIR layers. 

 We decided to tackle this issue by fusing thermal imagery (LWIR) with RGB imagery to create a multispectral object detection model that is more resilient than traditional stand-alone RGB object detection. By fusing LWIR with RGB, our object detection model will work during any time of the day and in all types of visibility environments. By fusing LWIR and RGB we are also increasing edge redundancy, which will therefore increase the number of unique edges for a specific object, thus increasing object detection model performance.