A Game-Changing Model for Object Segmentation from Meta AI

Published on April 12, 2023

As part of their Segment Anything Model (SAM) initiative, Meta AI has introduced a new task, dataset, and model that aims to democratize the process of image segmentation. This project includes the Segment Anything Model (SAM) and the Segment Anything 1-Billion Mask dataset (SA-1B), which is the most comprehensive segmentation dataset available.

Researchers at Meta AI have developed the largest segmentation dataset to date, containing more than 1 billion masks on 11 million licensed and privacy-respecting images.

In order to enable zero-shot transfer to new image distributions and tasks, the model has been designed and trained with the purpose of being promptable. As a result of evaluating the model’s capabilities across a wide variety of tasks, it has been determined that its zero-shot performance is often comparable to, or even superior to, the performance of previous fully supervised models.

There have been two main categories of methods available for solving segmentation problems up to this point. First, interactive segmentation permits the segmentation of any object category but relies on human guidance to refine the mask iteratively. As a second approach, automatic segmentation is available, which enables the segmentation of predefined categories of objects (such as chairs or cats), but training the segmentation model requires significant amounts of manually annotated objects (sometimes thousands or tens of thousands of examples of segmented cats), in addition to substantial computing resources and technical expertise. However, neither of these methods was able to provide a universal, fully automated approach to segmentation.

SAM is a synthesis of these two approaches. This model is capable of handling both interactive and automatic segmentation tasks effectively. By engineering the right prompt for the model, such as clicks, boxes, or text, the model is suitable for a wide range of segmentation tasks. Moreover, SAM is trained on a diverse and high-quality dataset of over 1 billion masks, which was collected as part of the project. It is therefore able to generalize well to new types of objects and images beyond those on which it was trained. The ability to generalize significantly reduces the need for practitioners to collect their own segmentation data and to fine-tune a model to meet their specific needs.

Using their research and dataset, Meta hopes to facilitate further advances in segmentation and image and video understanding. Its promptable segmentation model can be incorporated into a larger system to perform segmentation tasks. The composition approach allows one model to be used in a variety of extensible ways, potentially achieving tasks that were unknown at the time of the model’s conception.

SAM is expected to be useful for a wide range of applications including AR/VR, content creation, scientific research, and more general artificial intelligence applications with prompt engineering techniques.