Custom detections using Mapbox Vision SDK for Android

Join the Mapbox Developers Discord Communityarrow-right

Prerequisite

Familiarity with Android Studio, Java or Kotlin.

The Mapbox Vision SDK for Android is a library for interpreting road scenes in real time directly on Android devices using the device’s built-in camera. The Vision SDK detects many types of objects including cars, people, road signs, and more. In this tutorial, you will identify any cars in view and highlight them using a green circle.

When you use the end product on a road scene that includes cars, the result will look like the image below.

Getting started

Before starting this tutorial, go through the Install and configure steps in the Vision SDK for Android documentation. This will walk you through how to install the Vision SDK and configure your application. Skip the Add VisionView to the activity layout (optional) step since you will be drawing object detections in this guide.

Prepare an ImageView

Start by preparing a regular Android ImageView.

Note

Using android:scaleType="centerCrop" is important because the frame's aspect ratio coming from Vision SDK likely does not match the aspect ratio of actual device.

<ImageView
        android:id="@+id/detections_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:scaleType="centerCrop"/>

Prepare the Vision SDK to identify objects

The VisionManager class is the main object for registering events from the Vision SDK and starting and stopping their delivery. Inside the Activity.onStart() callback:

1. Check that all permissions are granted and that you have not already started the Vision SDK.
2. Create a VisionManager instance.
3. Configure the detection frequency using VisionManager.setModelPerformance().
4. Set up VisionEventsListener.
5. Start delivering events from the VisionManager.

Note

You do not need your device to be calibrated to receive detections.

Java

Kotlin

@Override
public void onStart() {
    super.onStart();
    startVisionManager();
}

private void startVisionManager() {
if (allPermissionsGranted() && !visionManagerWasInit) {
VisionManager.create();
VisionManager.setModelPerformance(
new ModelPerformance.On(
ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE
)
);
VisionManager.setVisionEventsListener(visionEventsListener);
VisionManager.start();
visionManagerWasInit = true;
}
}

You will also want to stop the Vision SDK on Activity.onStop().

Java

Kotlin

@Override
public void onStop() {
    super.onStop();
    stopVisionManager();
}

private void stopVisionManager() {
if (visionManagerWasInit) {
VisionManager.stop();
VisionManager.destroy();
visionManagerWasInit = false;
}
}

Compile and run your code. Make sure after running your application, Vision SDK frames with detections arrive shortly in VisionEventsListener.onFrameDetectionsUpdated callback.

Note

For more information on running and testing the Vision SDK in a development environment, see the Testing and development guide.

Prepare detections for drawing

You'll use Android's Canvas class to draw the green circles around cars that are detected by the Vision SDK. To prepare detections for drawing you will:

• Create a Paint object and define the color, style, and stroke width
• Create a Bitmap from a Vision SDK frame (Image)
• Create a Canvas using the resulting Bitmap
• Create a function to draw a single detection using drawCircle
• Draw all car detections on Bitmap

Create a Paint object

Create a new Paint object to specify the appearance of the green circles that will be displayed when a car is detected.

Java

Kotlin

private void preparePaint() {
    paint = new Paint(Paint.ANTI_ALIAS_FLAG);
    paint.setColor(Color.GREEN);
    paint.setStrokeWidth(5f);
    paint.setStyle(Paint.Style.STROKE);
}

Call preparePaint on Activity.onCreate().

Java

Kotlin

public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    detectionsView = findViewById(R.id.detections_view);
    preparePaint();
}

Create a Bitmap from a Vision SDK frame

Detections from the Vision SDK are represented as an actual frame (stored in Image) and an array of Detections. You will need to convert the Image to Android Bitmap to use with Canvas.

Java

Kotlin

private Bitmap convertImageToBitmap(final Image originalImage) {
    final Bitmap bitmap = Bitmap.createBitmap(
            originalImage.getSize().getImageWidth(),
            originalImage.getSize().getImageHeight(),
            Bitmap.Config.ARGB_8888
    );
    // prepare direct ByteBuffer that will hold camera frame data
    ByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());
    // associate underlying native ByteBuffer with our buffer
    originalImage.copyPixels(buffer);
    buffer.rewind();
    // copy ByteBuffer to bitmap
    bitmap.copyPixelsFromBuffer(buffer);
    return bitmap;
}

Create a Canvas

Create a Canvas using the Bitmap you created in the previous step.

Java

Kotlin

final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());
// now we will draw current detections on canvas with frame bitmap
final Canvas canvas = new Canvas(frameBitmap);

Create a function to draw a single detection

Write a function called drawSingleDetection that draws a circle for a single detection. Detections contain RectF describing how to position the detection using vertices located in a [0, 1] range. Before drawing on Canvas you need to transform the detection values to absolute values. Then, use drawCircle to draw a circumscribed circle of the detection rectangle using the absolute coordinates.

Java

Kotlin

private void drawSingleDetection(final Canvas canvas, final Detection detection) {
    // first thing we get coordinates of bounding box
    final RectF relativeBbox = detection.getBoundingBox();
    // we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size
    // we do not care about screen resolution at all - we will use cropCenter mode
    final RectF absoluteBbox = new RectF(
            relativeBbox.left * canvas.getWidth(),
            relativeBbox.top * canvas.getHeight(),
            relativeBbox.right * canvas.getWidth(),
            relativeBbox.bottom * canvas.getHeight()
    );
    // we want to draw circle bounds, we need radius and center for that
    final float radius = (float) Math.sqrt(
            Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +
                    Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2)
    );
    canvas.drawCircle(
            absoluteBbox.centerX(),
            absoluteBbox.centerY(),
            radius,
            paint
    );
}

Draw all car detections on Bitmap

Because more than one object can be detected in a frame, use for to iterate through all detections. Then, filter the detections to only those with the class Car and with a confidence level greater than 0.6 and use drawSingleDetection to draw a green circle around each.

Java

Kotlin

for (final Detection detection : frameDetections.getDetections()) {
    // we will draw only detected cars
    // and filter detections which we are not confident with
    if (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {
        drawSingleDetection(canvas, detection);
    }
}

Set final Bitmap with detections to ImageView

Bitmap with all detections is ready. But you've prepared it on background thread in the VisionEventsListener.onFrameDetectionsUpdated callback. That means you should switch to main thread and start drawing.

Java

Kotlin

runOnUiThread(() -> {
    // finally we update our image view on main thread
    detectionsView.setImageBitmap(frameBitmap);
});

Finished product

CustomDetectionActivity.java

github View on GitHub

Java

Kotlin

package com.mapbox.vision.examples;

import android.graphics.Bitmap;
import android.graphics.Canvas;
import android.graphics.Color;
import android.graphics.Paint;
import android.graphics.RectF;
import android.os.Bundle;
import android.widget.ImageView;

import com.mapbox.vision.VisionManager;
import com.mapbox.vision.mobile.core.interfaces.VisionEventsListener;
import com.mapbox.vision.mobile.core.models.AuthorizationStatus;
import com.mapbox.vision.mobile.core.models.Camera;
import com.mapbox.vision.mobile.core.models.Country;
import com.mapbox.vision.mobile.core.models.FrameSegmentation;
import com.mapbox.vision.mobile.core.models.classification.FrameSignClassifications;
import com.mapbox.vision.mobile.core.models.detection.Detection;
import com.mapbox.vision.mobile.core.models.detection.DetectionClass;
import com.mapbox.vision.mobile.core.models.detection.FrameDetections;
import com.mapbox.vision.mobile.core.models.frame.Image;
import com.mapbox.vision.mobile.core.models.position.VehicleState;
import com.mapbox.vision.mobile.core.models.road.RoadDescription;
import com.mapbox.vision.mobile.core.models.world.WorldDescription;
import com.mapbox.vision.performance.ModelPerformance;
import com.mapbox.vision.performance.ModelPerformanceMode;
import com.mapbox.vision.performance.ModelPerformanceRate;

import org.jetbrains.annotations.NotNull;

import java.nio.ByteBuffer;

public class CustomDetectionActivity extends BaseActivity {

    private Boolean visionManagerWasInit = false;
    private ImageView detectionsView;
    private Paint paint;

    // VisionEventsListener handles events from Vision SDK on background thread.
    private VisionEventsListener visionEventsListener = new VisionEventsListener() {

        @Override
        public void onAuthorizationStatusUpdated(@NotNull AuthorizationStatus authorizationStatus) {

        }

        @Override
        public void onCameraUpdated(@NotNull Camera camera) {

        }

        @Override
        public void onCountryUpdated(@NotNull Country country) {

        }

        @Override
        public void onFrameDetectionsUpdated(@NotNull FrameDetections frameDetections) {
            final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());
            // now we will draw current detections on canvas with frame bitmap
            final Canvas canvas = new Canvas(frameBitmap);
            for (final Detection detection : frameDetections.getDetections()) {
                // we will draw only detected cars
                // and filter detections which we are not confident with
                if (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {
                    drawSingleDetection(canvas, detection);
                }
            }
            runOnUiThread(() -> {
                // finally we update our image view on main thread
                detectionsView.setImageBitmap(frameBitmap);
            });
        }

        @Override
        public void onFrameSegmentationUpdated(@NotNull FrameSegmentation frameSegmentation) {

        }

        @Override
        public void onFrameSignClassificationsUpdated(@NotNull FrameSignClassifications frameSignClassifications) {

        }

        @Override
        public void onRoadDescriptionUpdated(@NotNull RoadDescription roadDescription) {

        }

        @Override
        public void onUpdateCompleted() {

        }

        @Override
        public void onVehicleStateUpdated(@NotNull VehicleState vehicleState) {

        }

        @Override
        public void onWorldDescriptionUpdated(@NotNull WorldDescription worldDescription) {

        }
    };

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        detectionsView = findViewById(R.id.detections_view);
        preparePaint();
    }

    @Override
    public void onPermissionsGranted() {
        startVisionManager();
    }

    @Override
    public void initViews() {
        setContentView(R.layout.activity_custom_detection);
    }

    @Override
    public void onStart() {
        super.onStart();
        startVisionManager();
    }

    @Override
    public void onStop() {
        super.onStop();
        stopVisionManager();
    }

    private void startVisionManager() {
        if (allPermissionsGranted() && !visionManagerWasInit) {
            VisionManager.create();
            VisionManager.setModelPerformance(
                    new ModelPerformance.On(
                            ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE
                    )
            );
            VisionManager.setVisionEventsListener(visionEventsListener);
            VisionManager.start();
            visionManagerWasInit = true;
        }
    }

    private void stopVisionManager() {
        if (visionManagerWasInit) {
            VisionManager.stop();
            VisionManager.destroy();
            visionManagerWasInit = false;
        }
    }

    private Bitmap convertImageToBitmap(final Image originalImage) {
        final Bitmap bitmap = Bitmap.createBitmap(
                originalImage.getSize().getImageWidth(),
                originalImage.getSize().getImageHeight(),
                Bitmap.Config.ARGB_8888
        );
        // prepare direct ByteBuffer that will hold camera frame data
        ByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());
        // associate underlying native ByteBuffer with our buffer
        originalImage.copyPixels(buffer);
        buffer.rewind();
        // copy ByteBuffer to bitmap
        bitmap.copyPixelsFromBuffer(buffer);
        return bitmap;
    }

    private void drawSingleDetection(final Canvas canvas, final Detection detection) {
        // first thing we get coordinates of bounding box
        final RectF relativeBbox = detection.getBoundingBox();
        // we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size
        // we do not care about screen resolution at all - we will use cropCenter mode
        final RectF absoluteBbox = new RectF(
                relativeBbox.left * canvas.getWidth(),
                relativeBbox.top * canvas.getHeight(),
                relativeBbox.right * canvas.getWidth(),
                relativeBbox.bottom * canvas.getHeight()
        );
        // we want to draw circle bounds, we need radius and center for that
        final float radius = (float) Math.sqrt(
                Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +
                        Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2)
        );
        canvas.drawCircle(
                absoluteBbox.centerX(),
                absoluteBbox.centerY(),
                radius,
                paint
        );
    }

    private void preparePaint() {
        paint = new Paint(Paint.ANTI_ALIAS_FLAG);
        paint.setColor(Color.GREEN);
        paint.setStrokeWidth(5f);
        paint.setStyle(Paint.Style.STROKE);
    }
}

Join the Mapbox Developers Discord Communityarrow-right

Was this page helpful?