Object Detection and Tracking

With ML Kit's on-device object detection and tracking API, you can localize and track in real time the most prominent objects in an image or live camera feed. You can also optionally classify detected objects into one of several general categories.

Object detection and tracking with coarse classification is useful for building live visual search experiences. Because object detection and tracking happens quickly and completely on the device, it works well as the front end of a longer visual search pipeline. After you detect and filter objects, you can pass them to a cloud backend, such as Cloud Vision Product Search, or to a custom model, such as one you trained using AutoML Vision Edge.

iOSAndroid

Key capabilities

Fast object detection and trackingDetect objects and get their location in the image. Track objects across images.
Optimized on-device modelThe object detection and tracking model is optimized for mobile devices and intended for use in real-time applications, even on lower-end devices.
Prominent object detectionAutomatically determine the most prominent object in an image.
Coarse classificationClassify objects into broad categories, which you can use to filter out objects you're not interested in. The following categories are supported: home goods, fashion goods, food, plants, places, and unknown.

Example results

Tracking the most prominent object across images

Tracking ID0
Bounds(95, 45), (496, 45), (496, 240), (95, 240)
CategoryPLACE
Classification confidence0.9296875
Tracking ID0
Bounds(84, 46), (478, 46), (478, 247), (84, 247)
CategoryPLACE
Classification confidence0.8710938
Tracking ID0
Bounds(53, 45), (519, 45), (519, 240), (53, 240)
CategoryPLACE
Classification confidence0.8828125

Photo: Christian Ferrer [CC BY-SA 4.0]

Multiple objects in a static image

Object 0
Bounds(1, 97), (332, 97), (332, 332), (1, 332)
CategoryFASHION_GOOD
Classification confidence0.95703125
Object 1
Bounds(186, 80), (337, 80), (337, 226), (186, 226)
CategoryFASHION_GOOD
Classification confidence0.84375
Object 2
Bounds(296, 80), (472, 80), (472, 388), (296, 388)
CategoryFASHION_GOOD
Classification confidence0.94921875
Object 3
Bounds(439, 83), (615, 83), (615, 306), (439, 306)
CategoryFASHION_GOOD
Classification confidence0.9375