Combining Meta’s SAM with traditional machine learning methods to improve crop type classification

Abstract

Crop mapping via remotely-sensed images is growing increasingly important in the agricultural sector; oftentimes machine learning is employed to expedite the process of creating these maps. However, the results of traditional machine learning approaches, such as random forest and SVM, can be noisy: these methods are applied to satellite imagery pixel-by-pixel and thus can result in misclassified pixels within a cropland unit. This can be improved upon in postprocessing – by using computer vision models to automatically segment the original satellite image, we can identify the boundaries of each cropland unit and use this data to clean up the image. Specifically, our experiment goes through each segmented area, finding the mode of the cropland values and reclassifying any erroneous pixels in that area to this value. We used Meta’s Segment Anything Model (SAM) to perform image segmentation: this system has zero-shot generalization, meaning that it can accurately segment images that it has never encountered in training. Using Google Earth Engine as the primary data analysis platform, we applied this model to two key agricultural regions in the United States as study areas (one in California’s Central Valley and the other in the U.S. Corn Belt) to assess the utility of this approach. As compared with the U.S. Department of Agriculture’s Cropland Data Layer, the accuracy of the crop maps improved by 17% in the California study area and 8% in the Illinois area. We conclude that this method can be used to improve crop map accuracy and could be applied across the entire United States, lending itself to more accurate crop yield estimation and thus more informed agricultural decisions.