Meta released two new computer vision models on November 19: SAM 3, which detects, segments and tracks objects in images and video based on text or visual prompts, and SAM 3D, which reconstructs objects, scenes and human bodies in three dimensions from a single photograph. Both are part of the company’s Segment Anything project, and Meta says they are already being used in its own products.
The models matter for anyone who edits video, builds visual effects, labels data for machine learning or works with 3D content, because they replace laborious manual steps with a short instruction.
Segmenting by concept
Earlier Segment Anything models required a user to click on an object or draw a box around it. SAM 3 accepts a short noun phrase instead. Type “yellow school bus” and the model finds and outlines every school bus in the image or across a video. Meta says the model also handles more specific descriptions, and in combination with a multimodal language model it can resolve complex requests such as “people sitting down but not wearing a red baseball cap.” The model can be prompted with example images as well as text.
According to Meta, SAM 3 achieves a zero-shot mask average precision of 47.0 on the LVIS benchmark, which tests recognition of a long tail of object categories, a 22 percent improvement over the previous best result. Meta reports that the model processes an image with more than 100 detected objects in about 30 milliseconds on an Nvidia H200 GPU, and that it maintains near real-time performance when tracking roughly five objects in video.
From one photo to 3D
SAM 3D comes in two parts. SAM 3D Objects reconstructs the shape, texture and layout of objects and full scenes from a single image, and SAM 3D Body estimates the pose and shape of a human body from one photo. Meta says the models are designed to cope with the clutter, occlusion and unusual viewpoints of real-world images rather than clean studio shots.
The first consumer application is on Facebook Marketplace, where a new View in Room feature uses SAM 3D to let shoppers see how a piece of furniture would look in their own space before buying it. Meta also points to uses in robotics, sports science, e-commerce and video production.
Products, partners and open release
Inside Meta’s own apps, SAM 3 now powers effects in Edits, the company’s video editing app, and in Vibes within the Meta AI app, allowing users to apply changes to specific people or objects in a scene. A web tool called Segment Anything Playground lets anyone try the models on their own images and video without writing code.
For developers, Meta is releasing the SAM 3 model weights, evaluation benchmarks and research paper, along with the checkpoints and inference code for SAM 3D. The company has also partnered with Roboflow, a computer vision platform, so that developers can annotate data, fine-tune SAM 3 and deploy it for specialized tasks in fields such as medical imaging, drone vision, traffic monitoring and manufacturing inspection.
Meta also released a new benchmark, SA-Co, built to measure how well models recognize a very large open vocabulary of concepts, and says it hopes the open release will let researchers extend the models further.
