Custom detections using Mapbox Vision SDK for Android
Familiarity with Android Studio, Java or Kotlin.
The Mapbox Vision SDK for Android is a library for interpreting road scenes in real time directly on Android devices using the device’s built-in camera. The Vision SDK detects many types of objects including cars, people, road signs, and more. In this tutorial, you will identify any cars in view and highlight them using a green circle.
When you use the end product on a road scene that includes cars, the result will look like the image below.
Getting started
Before starting this tutorial, go through the Install and configure steps in the Vision SDK for Android documentation. This will walk you through how to install the Vision SDK and configure your application. Skip the Add VisionView
to the activity layout (optional) step since you will be drawing object detections in this guide.
Prepare an ImageView
Start by preparing a regular Android ImageView
.
Using android:scaleType="centerCrop"
is important because the frame's aspect ratio coming from Vision SDK likely does not match the aspect ratio of actual device.
<ImageView
android:id="@+id/detections_view"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:scaleType="centerCrop"/>
Prepare the Vision SDK to identify objects
The VisionManager
class is the main object for registering events from the Vision SDK and starting and stopping their delivery. Inside the Activity.onStart()
callback:
- Check that all permissions are granted and that you have not already started the Vision SDK.
- Create a
VisionManager
instance. - Configure the detection frequency using
VisionManager.setModelPerformance()
. - Set up
VisionEventsListener
. - Start delivering events from the
VisionManager
.
You do not need your device to be calibrated to receive detections.
@Overridepublic void onStart() {super.onStart();startVisionManager();} private void startVisionManager() {if (allPermissionsGranted() && !visionManagerWasInit) {VisionManager.create();VisionManager.setModelPerformance(new ModelPerformance.On(ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE));VisionManager.setVisionEventsListener(visionEventsListener);VisionManager.start();visionManagerWasInit = true;}}
You will also want to stop the Vision SDK on Activity.onStop()
.
@Overridepublic void onStop() {super.onStop();stopVisionManager();} private void stopVisionManager() {if (visionManagerWasInit) {VisionManager.stop();VisionManager.destroy();visionManagerWasInit = false;}}
Compile and run your code. Make sure after running your application, Vision SDK frames with detections arrive shortly in VisionEventsListener.onFrameDetectionsUpdated
callback.
For more information on running and testing the Vision SDK in a development environment, see the Testing and development guide.
Prepare detections for drawing
You'll use Android's Canvas
class to draw the green circles around cars that are detected by the Vision SDK. To prepare detections for drawing you will:
- Create a
Paint
object and define the color, style, and stroke width - Create a
Bitmap
from a Vision SDK frame (Image
) - Create a
Canvas
using the resultingBitmap
- Create a function to draw a single detection using
drawCircle
- Draw all car detections on
Bitmap
Create a Paint
object
Create a new Paint
object to specify the appearance of the green circles that will be displayed when a car is detected.
private void preparePaint() {paint = new Paint(Paint.ANTI_ALIAS_FLAG);paint.setColor(Color.GREEN);paint.setStrokeWidth(5f);paint.setStyle(Paint.Style.STROKE);}
Call preparePaint
on Activity.onCreate()
.
public void onCreate(Bundle savedInstanceState) {super.onCreate(savedInstanceState);detectionsView = findViewById(R.id.detections_view);preparePaint();}
Create a Bitmap
from a Vision SDK frame
Detections from the Vision SDK are represented as an actual frame (stored in Image
) and an array of Detection
s. You will need to convert the Image
to Android Bitmap
to use with Canvas
.
private Bitmap convertImageToBitmap(final Image originalImage) {final Bitmap bitmap = Bitmap.createBitmap(originalImage.getSize().getImageWidth(),originalImage.getSize().getImageHeight(),Bitmap.Config.ARGB_8888);// prepare direct ByteBuffer that will hold camera frame dataByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());// associate underlying native ByteBuffer with our bufferoriginalImage.copyPixels(buffer);buffer.rewind();// copy ByteBuffer to bitmapbitmap.copyPixelsFromBuffer(buffer);return bitmap;}
Create a Canvas
Create a Canvas
using the Bitmap
you created in the previous step.
final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());// now we will draw current detections on canvas with frame bitmapfinal Canvas canvas = new Canvas(frameBitmap);
Create a function to draw a single detection
Write a function called drawSingleDetection
that draws a circle for a single detection. Detections contain RectF
describing how to position the detection using vertices located in a [0, 1] range. Before drawing on Canvas
you need to transform the detection values to absolute values. Then, use drawCircle
to draw a circumscribed circle of the detection rectangle using the absolute coordinates.
private void drawSingleDetection(final Canvas canvas, final Detection detection) {// first thing we get coordinates of bounding boxfinal RectF relativeBbox = detection.getBoundingBox();// we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size// we do not care about screen resolution at all - we will use cropCenter modefinal RectF absoluteBbox = new RectF(relativeBbox.left * canvas.getWidth(),relativeBbox.top * canvas.getHeight(),relativeBbox.right * canvas.getWidth(),relativeBbox.bottom * canvas.getHeight());// we want to draw circle bounds, we need radius and center for thatfinal float radius = (float) Math.sqrt(Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2));canvas.drawCircle(absoluteBbox.centerX(),absoluteBbox.centerY(),radius,paint);}
Draw all car detections on Bitmap
Because more than one object can be detected in a frame, use for
to iterate through all detections. Then, filter the detections to only those with the class Car
and with a confidence level greater than 0.6
and use drawSingleDetection
to draw a green circle around each.
for (final Detection detection : frameDetections.getDetections()) {// we will draw only detected cars// and filter detections which we are not confident withif (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {drawSingleDetection(canvas, detection);}}
Set final Bitmap
with detections to ImageView
Bitmap
with all detections is ready. But you've prepared it on background thread in the VisionEventsListener.onFrameDetectionsUpdated
callback. That means you should switch to main thread and start drawing.
runOnUiThread(() -> {// finally we update our image view on main threaddetectionsView.setImageBitmap(frameBitmap);});
Finished product
package com.mapbox.vision.examples; import android.graphics.Bitmap;import android.graphics.Canvas;import android.graphics.Color;import android.graphics.Paint;import android.graphics.RectF;import android.os.Bundle;import android.widget.ImageView; import com.mapbox.vision.VisionManager;import com.mapbox.vision.mobile.core.interfaces.VisionEventsListener;import com.mapbox.vision.mobile.core.models.AuthorizationStatus;import com.mapbox.vision.mobile.core.models.Camera;import com.mapbox.vision.mobile.core.models.Country;import com.mapbox.vision.mobile.core.models.FrameSegmentation;import com.mapbox.vision.mobile.core.models.classification.FrameSignClassifications;import com.mapbox.vision.mobile.core.models.detection.Detection;import com.mapbox.vision.mobile.core.models.detection.DetectionClass;import com.mapbox.vision.mobile.core.models.detection.FrameDetections;import com.mapbox.vision.mobile.core.models.frame.Image;import com.mapbox.vision.mobile.core.models.position.VehicleState;import com.mapbox.vision.mobile.core.models.road.RoadDescription;import com.mapbox.vision.mobile.core.models.world.WorldDescription;import com.mapbox.vision.performance.ModelPerformance;import com.mapbox.vision.performance.ModelPerformanceMode;import com.mapbox.vision.performance.ModelPerformanceRate; import org.jetbrains.annotations.NotNull; import java.nio.ByteBuffer; public class CustomDetectionActivity extends BaseActivity { private Boolean visionManagerWasInit = false;private ImageView detectionsView;private Paint paint; // VisionEventsListener handles events from Vision SDK on background thread.private VisionEventsListener visionEventsListener = new VisionEventsListener() { @Overridepublic void onAuthorizationStatusUpdated(@NotNull AuthorizationStatus authorizationStatus) { } @Overridepublic void onCameraUpdated(@NotNull Camera camera) { } @Overridepublic void onCountryUpdated(@NotNull Country country) { } @Overridepublic void onFrameDetectionsUpdated(@NotNull FrameDetections frameDetections) {final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());// now we will draw current detections on canvas with frame bitmapfinal Canvas canvas = new Canvas(frameBitmap);for (final Detection detection : frameDetections.getDetections()) {// we will draw only detected cars// and filter detections which we are not confident withif (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {drawSingleDetection(canvas, detection);}}runOnUiThread(() -> {// finally we update our image view on main threaddetectionsView.setImageBitmap(frameBitmap);});} @Overridepublic void onFrameSegmentationUpdated(@NotNull FrameSegmentation frameSegmentation) { } @Overridepublic void onFrameSignClassificationsUpdated(@NotNull FrameSignClassifications frameSignClassifications) { } @Overridepublic void onRoadDescriptionUpdated(@NotNull RoadDescription roadDescription) { } @Overridepublic void onUpdateCompleted() { } @Overridepublic void onVehicleStateUpdated(@NotNull VehicleState vehicleState) { } @Overridepublic void onWorldDescriptionUpdated(@NotNull WorldDescription worldDescription) { }}; @Overridepublic void onCreate(Bundle savedInstanceState) {super.onCreate(savedInstanceState);detectionsView = findViewById(R.id.detections_view);preparePaint();} @Overridepublic void onPermissionsGranted() {startVisionManager();} @Overridepublic void initViews() {setContentView(R.layout.activity_custom_detection);} @Overridepublic void onStart() {super.onStart();startVisionManager();} @Overridepublic void onStop() {super.onStop();stopVisionManager();} private void startVisionManager() {if (allPermissionsGranted() && !visionManagerWasInit) {VisionManager.create();VisionManager.setModelPerformance(new ModelPerformance.On(ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE));VisionManager.setVisionEventsListener(visionEventsListener);VisionManager.start();visionManagerWasInit = true;}} private void stopVisionManager() {if (visionManagerWasInit) {VisionManager.stop();VisionManager.destroy();visionManagerWasInit = false;}} private Bitmap convertImageToBitmap(final Image originalImage) {final Bitmap bitmap = Bitmap.createBitmap(originalImage.getSize().getImageWidth(),originalImage.getSize().getImageHeight(),Bitmap.Config.ARGB_8888);// prepare direct ByteBuffer that will hold camera frame dataByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());// associate underlying native ByteBuffer with our bufferoriginalImage.copyPixels(buffer);buffer.rewind();// copy ByteBuffer to bitmapbitmap.copyPixelsFromBuffer(buffer);return bitmap;} private void drawSingleDetection(final Canvas canvas, final Detection detection) {// first thing we get coordinates of bounding boxfinal RectF relativeBbox = detection.getBoundingBox();// we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size// we do not care about screen resolution at all - we will use cropCenter modefinal RectF absoluteBbox = new RectF(relativeBbox.left * canvas.getWidth(),relativeBbox.top * canvas.getHeight(),relativeBbox.right * canvas.getWidth(),relativeBbox.bottom * canvas.getHeight());// we want to draw circle bounds, we need radius and center for thatfinal float radius = (float) Math.sqrt(Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2));canvas.drawCircle(absoluteBbox.centerX(),absoluteBbox.centerY(),radius,paint);} private void preparePaint() {paint = new Paint(Paint.ANTI_ALIAS_FLAG);paint.setColor(Color.GREEN);paint.setStrokeWidth(5f);paint.setStyle(Paint.Style.STROKE);}}