Help
All docschevron-rightHelpchevron-rightarrow-leftTutorialschevron-rightCustom detections using Mapbox Vision SDK for Android

Custom detections using Mapbox Vision SDK for Android

Intermediate
Java, Kotlin
Prerequisite

Familiarity with Android Studio, Java or Kotlin.

The Mapbox Vision SDK for Android is a library for interpreting road scenes in real time directly on Android devices using the device’s built-in camera. The Vision SDK detects many types of objects including cars, people, road signs, and more. In this tutorial, you will identify any cars in view and highlight them using a green circle.

When you use the end product on a road scene that includes cars, the result will look like the image below.

Getting started

Before starting this tutorial, go through the Install and configure steps in the Vision SDK for Android documentation. This will walk you through how to install the Vision SDK and configure your application. Skip the Add VisionView to the activity layout (optional) step since you will be drawing object detections in this guide.

Prepare an ImageView

Start by preparing a regular Android ImageView.

book
Note

Using android:scaleType="centerCrop" is important because the frame's aspect ratio coming from Vision SDK likely does not match the aspect ratio of actual device.

<ImageView
        android:id="@+id/detections_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:scaleType="centerCrop"/>

Prepare the Vision SDK to identify objects

The VisionManager class is the main object for registering events from the Vision SDK and starting and stopping their delivery. Inside the Activity.onStart() callback:

  1. Check that all permissions are granted and that you have not already started the Vision SDK.
  2. Create a VisionManager instance.
  3. Configure the detection frequency using VisionManager.setModelPerformance().
  4. Set up VisionEventsListener.
  5. Start delivering events from the VisionManager.
book
Note

You do not need your device to be calibrated to receive detections.

@Override
public void onStart() {
super.onStart();
startVisionManager();
}
 
private void startVisionManager() {
if (allPermissionsGranted() && !visionManagerWasInit) {
VisionManager.create();
VisionManager.setModelPerformance(
new ModelPerformance.On(
ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE
)
);
VisionManager.setVisionEventsListener(visionEventsListener);
VisionManager.start();
visionManagerWasInit = true;
}
}

You will also want to stop the Vision SDK on Activity.onStop().

@Override
public void onStop() {
super.onStop();
stopVisionManager();
}
 
private void stopVisionManager() {
if (visionManagerWasInit) {
VisionManager.stop();
VisionManager.destroy();
visionManagerWasInit = false;
}
}

Compile and run your code. Make sure after running your application, Vision SDK frames with detections arrive shortly in VisionEventsListener.onFrameDetectionsUpdated callback.

book
Note

For more information on running and testing the Vision SDK in a development environment, see the Testing and development guide.

Prepare detections for drawing

You'll use Android's Canvas class to draw the green circles around cars that are detected by the Vision SDK. To prepare detections for drawing you will:

  • Create a Paint object and define the color, style, and stroke width
  • Create a Bitmap from a Vision SDK frame (Image)
  • Create a Canvas using the resulting Bitmap
  • Create a function to draw a single detection using drawCircle
  • Draw all car detections on Bitmap

Create a Paint object

Create a new Paint object to specify the appearance of the green circles that will be displayed when a car is detected.

private void preparePaint() {
paint = new Paint(Paint.ANTI_ALIAS_FLAG);
paint.setColor(Color.GREEN);
paint.setStrokeWidth(5f);
paint.setStyle(Paint.Style.STROKE);
}

Call preparePaint on Activity.onCreate().

public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
detectionsView = findViewById(R.id.detections_view);
preparePaint();
}

Create a Bitmap from a Vision SDK frame

Detections from the Vision SDK are represented as an actual frame (stored in Image) and an array of Detections. You will need to convert the Image to Android Bitmap to use with Canvas.

private Bitmap convertImageToBitmap(final Image originalImage) {
final Bitmap bitmap = Bitmap.createBitmap(
originalImage.getSize().getImageWidth(),
originalImage.getSize().getImageHeight(),
Bitmap.Config.ARGB_8888
);
// prepare direct ByteBuffer that will hold camera frame data
ByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());
// associate underlying native ByteBuffer with our buffer
originalImage.copyPixels(buffer);
buffer.rewind();
// copy ByteBuffer to bitmap
bitmap.copyPixelsFromBuffer(buffer);
return bitmap;
}

Create a Canvas

Create a Canvas using the Bitmap you created in the previous step.

final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());
// now we will draw current detections on canvas with frame bitmap
final Canvas canvas = new Canvas(frameBitmap);

Create a function to draw a single detection

Write a function called drawSingleDetection that draws a circle for a single detection. Detections contain RectF describing how to position the detection using vertices located in a [0, 1] range. Before drawing on Canvas you need to transform the detection values to absolute values. Then, use drawCircle to draw a circumscribed circle of the detection rectangle using the absolute coordinates.

private void drawSingleDetection(final Canvas canvas, final Detection detection) {
// first thing we get coordinates of bounding box
final RectF relativeBbox = detection.getBoundingBox();
// we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size
// we do not care about screen resolution at all - we will use cropCenter mode
final RectF absoluteBbox = new RectF(
relativeBbox.left * canvas.getWidth(),
relativeBbox.top * canvas.getHeight(),
relativeBbox.right * canvas.getWidth(),
relativeBbox.bottom * canvas.getHeight()
);
// we want to draw circle bounds, we need radius and center for that
final float radius = (float) Math.sqrt(
Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +
Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2)
);
canvas.drawCircle(
absoluteBbox.centerX(),
absoluteBbox.centerY(),
radius,
paint
);
}

Draw all car detections on Bitmap

Because more than one object can be detected in a frame, use for to iterate through all detections. Then, filter the detections to only those with the class Car and with a confidence level greater than 0.6 and use drawSingleDetection to draw a green circle around each.

for (final Detection detection : frameDetections.getDetections()) {
// we will draw only detected cars
// and filter detections which we are not confident with
if (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {
drawSingleDetection(canvas, detection);
}
}

Set final Bitmap with detections to ImageView

Bitmap with all detections is ready. But you've prepared it on background thread in the VisionEventsListener.onFrameDetectionsUpdated callback. That means you should switch to main thread and start drawing.

runOnUiThread(() -> {
// finally we update our image view on main thread
detectionsView.setImageBitmap(frameBitmap);
});

Finished product

CustomDetectionActivity.java
github View on GitHub
package com.mapbox.vision.examples;
 
import android.graphics.Bitmap;
import android.graphics.Canvas;
import android.graphics.Color;
import android.graphics.Paint;
import android.graphics.RectF;
import android.os.Bundle;
import android.widget.ImageView;
 
import com.mapbox.vision.VisionManager;
import com.mapbox.vision.mobile.core.interfaces.VisionEventsListener;
import com.mapbox.vision.mobile.core.models.AuthorizationStatus;
import com.mapbox.vision.mobile.core.models.Camera;
import com.mapbox.vision.mobile.core.models.Country;
import com.mapbox.vision.mobile.core.models.FrameSegmentation;
import com.mapbox.vision.mobile.core.models.classification.FrameSignClassifications;
import com.mapbox.vision.mobile.core.models.detection.Detection;
import com.mapbox.vision.mobile.core.models.detection.DetectionClass;
import com.mapbox.vision.mobile.core.models.detection.FrameDetections;
import com.mapbox.vision.mobile.core.models.frame.Image;
import com.mapbox.vision.mobile.core.models.position.VehicleState;
import com.mapbox.vision.mobile.core.models.road.RoadDescription;
import com.mapbox.vision.mobile.core.models.world.WorldDescription;
import com.mapbox.vision.performance.ModelPerformance;
import com.mapbox.vision.performance.ModelPerformanceMode;
import com.mapbox.vision.performance.ModelPerformanceRate;
 
import org.jetbrains.annotations.NotNull;
 
import java.nio.ByteBuffer;
 
public class CustomDetectionActivity extends BaseActivity {
 
private Boolean visionManagerWasInit = false;
private ImageView detectionsView;
private Paint paint;
 
// VisionEventsListener handles events from Vision SDK on background thread.
private VisionEventsListener visionEventsListener = new VisionEventsListener() {
 
@Override
public void onAuthorizationStatusUpdated(@NotNull AuthorizationStatus authorizationStatus) {
 
}
 
@Override
public void onCameraUpdated(@NotNull Camera camera) {
 
}
 
@Override
public void onCountryUpdated(@NotNull Country country) {
 
}
 
@Override
public void onFrameDetectionsUpdated(@NotNull FrameDetections frameDetections) {
final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());
// now we will draw current detections on canvas with frame bitmap
final Canvas canvas = new Canvas(frameBitmap);
for (final Detection detection : frameDetections.getDetections()) {
// we will draw only detected cars
// and filter detections which we are not confident with
if (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {
drawSingleDetection(canvas, detection);
}
}
runOnUiThread(() -> {
// finally we update our image view on main thread
detectionsView.setImageBitmap(frameBitmap);
});
}
 
@Override
public void onFrameSegmentationUpdated(@NotNull FrameSegmentation frameSegmentation) {
 
}
 
@Override
public void onFrameSignClassificationsUpdated(@NotNull FrameSignClassifications frameSignClassifications) {
 
}
 
@Override
public void onRoadDescriptionUpdated(@NotNull RoadDescription roadDescription) {
 
}
 
@Override
public void onUpdateCompleted() {
 
}
 
@Override
public void onVehicleStateUpdated(@NotNull VehicleState vehicleState) {
 
}
 
@Override
public void onWorldDescriptionUpdated(@NotNull WorldDescription worldDescription) {
 
}
};
 
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
detectionsView = findViewById(R.id.detections_view);
preparePaint();
}
 
@Override
public void onPermissionsGranted() {
startVisionManager();
}
 
@Override
public void initViews() {
setContentView(R.layout.activity_custom_detection);
}
 
@Override
public void onStart() {
super.onStart();
startVisionManager();
}
 
@Override
public void onStop() {
super.onStop();
stopVisionManager();
}
 
private void startVisionManager() {
if (allPermissionsGranted() && !visionManagerWasInit) {
VisionManager.create();
VisionManager.setModelPerformance(
new ModelPerformance.On(
ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE
)
);
VisionManager.setVisionEventsListener(visionEventsListener);
VisionManager.start();
visionManagerWasInit = true;
}
}
 
private void stopVisionManager() {
if (visionManagerWasInit) {
VisionManager.stop();
VisionManager.destroy();
visionManagerWasInit = false;
}
}
 
private Bitmap convertImageToBitmap(final Image originalImage) {
final Bitmap bitmap = Bitmap.createBitmap(
originalImage.getSize().getImageWidth(),
originalImage.getSize().getImageHeight(),
Bitmap.Config.ARGB_8888
);
// prepare direct ByteBuffer that will hold camera frame data
ByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());
// associate underlying native ByteBuffer with our buffer
originalImage.copyPixels(buffer);
buffer.rewind();
// copy ByteBuffer to bitmap
bitmap.copyPixelsFromBuffer(buffer);
return bitmap;
}
 
private void drawSingleDetection(final Canvas canvas, final Detection detection) {
// first thing we get coordinates of bounding box
final RectF relativeBbox = detection.getBoundingBox();
// we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size
// we do not care about screen resolution at all - we will use cropCenter mode
final RectF absoluteBbox = new RectF(
relativeBbox.left * canvas.getWidth(),
relativeBbox.top * canvas.getHeight(),
relativeBbox.right * canvas.getWidth(),
relativeBbox.bottom * canvas.getHeight()
);
// we want to draw circle bounds, we need radius and center for that
final float radius = (float) Math.sqrt(
Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +
Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2)
);
canvas.drawCircle(
absoluteBbox.centerX(),
absoluteBbox.centerY(),
radius,
paint
);
}
 
private void preparePaint() {
paint = new Paint(Paint.ANTI_ALIAS_FLAG);
paint.setColor(Color.GREEN);
paint.setStrokeWidth(5f);
paint.setStyle(Paint.Style.STROKE);
}
}
Related tutorials
Advanced
codeKotlin, Java
Use the Mapbox Vision SDK for Android with an external USB camera
Implement a custom VideoSource that feeds video stream from the USB camera to the Mapbox Vision SDK for Android.
Advanced
codeJava, Kotlin
Draw points of interest using the Mapbox Vision SDK for Android
Use coordinate transformations to draw a point of interest (POI) on the screen from its geolocation using the Vision SDK for Android.
Advanced
codeSwift
Video: Build a navigation app for iOS
Integrate navigation into any iOS application.
Advanced
codeSwift
Build a navigation app for iOS
Integrate navigation into any iOS application.
Join the Mapbox Developers Discord Communityarrow-right