メインコンテンツまでスキップ

Custom detections using Mapbox Vision SDK for Android

追加の開発者リソース
前提条件
Familiarity with Android Studio, Java or Kotlin.

The Mapbox Vision SDK for Android is a library for interpreting road scenes in real time directly on Android devices using the device’s built-in camera. The Vision SDK detects many types of objects including cars, people, road signs, and more. In this tutorial, you will identify any cars in view and highlight them using a green circle.

When you use the end product on a road scene that includes cars, the result will look like the image below.

Getting started​

Before starting this tutorial, go through the Install and configure steps in the Vision SDK for Android documentation. This will walk you through how to install the Vision SDK and configure your application. Skip the Add VisionView to the activity layout (optional) step since you will be drawing object detections in this guide.

Prepare an ImageView​

Start by preparing a regular Android ImageView.

Note

Using android:scaleType="centerCrop" is important because the frame's aspect ratio coming from Vision SDK likely does not match the aspect ratio of actual device.

<ImageView
        android:id="@+id/detections_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:scaleType="centerCrop"/>

Prepare the Vision SDK to identify objects​

The VisionManager class is the main object for registering events from the Vision SDK and starting and stopping their delivery. Inside the Activity.onStart() callback:

  1. Check that all permissions are granted and that you have not already started the Vision SDK.
  2. Create a VisionManager instance.
  3. Configure the detection frequency using VisionManager.setModelPerformance().
  4. Set up VisionEventsListener.
  5. Start delivering events from the VisionManager.
Note
You do not need your device to be calibrated to receive detections.
@Override
public void onStart() {
  super.onStart();
  startVisionManager();
}

private void startVisionManager() {
if (allPermissionsGranted() && !visionManagerWasInit) {
VisionManager.create();
VisionManager.setModelPerformance(
new ModelPerformance.On(
ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE
)
);
VisionManager.setVisionEventsListener(visionEventsListener);
VisionManager.start();
visionManagerWasInit = true;
}
}

You will also want to stop the Vision SDK on Activity.onStop().

@Override
public void onStop() {
  super.onStop();
  stopVisionManager();
}

private void stopVisionManager() {
if (visionManagerWasInit) {
VisionManager.stop();
VisionManager.destroy();
visionManagerWasInit = false;
}
}

Compile and run your code. Make sure after running your application, Vision SDK frames with detections arrive shortly in VisionEventsListener.onFrameDetectionsUpdated callback.

Note

For more information on running and testing the Vision SDK in a development environment, see the Testing and development guide.

Prepare detections for drawing​

You'll use Android's Canvas class to draw the green circles around cars that are detected by the Vision SDK. To prepare detections for drawing you will:

  • Create a Paint object and define the color, style, and stroke width
  • Create a Bitmap from a Vision SDK frame (Image)
  • Create a Canvas using the resulting Bitmap
  • Create a function to draw a single detection using drawCircle
  • Draw all car detections on Bitmap

Create a Paint object​

Create a new Paint object to specify the appearance of the green circles that will be displayed when a car is detected.

private void preparePaint() {
  paint = new Paint(Paint.ANTI_ALIAS_FLAG);
  paint.setColor(Color.GREEN);
  paint.setStrokeWidth(5f);
  paint.setStyle(Paint.Style.STROKE);
}

Call preparePaint on Activity.onCreate().

public void onCreate(Bundle savedInstanceState) {
  super.onCreate(savedInstanceState);
  detectionsView = findViewById(R.id.detections_view);
  preparePaint();
}

Create a Bitmap from a Vision SDK frame​

Detections from the Vision SDK are represented as an actual frame (stored in Image) and an array of Detections. You will need to convert the Image to Android Bitmap to use with Canvas.

private Bitmap convertImageToBitmap(final Image originalImage) {
  final Bitmap bitmap = Bitmap.createBitmap(
          originalImage.getSize().getImageWidth(),
          originalImage.getSize().getImageHeight(),
          Bitmap.Config.ARGB_8888
  );
  // prepare direct ByteBuffer that will hold camera frame data
  ByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());
  // associate underlying native ByteBuffer with our buffer
  originalImage.copyPixels(buffer);
  buffer.rewind();
  // copy ByteBuffer to bitmap
  bitmap.copyPixelsFromBuffer(buffer);
  return bitmap;
}

Create a Canvas​

Create a Canvas using the Bitmap you created in the previous step.

final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());
// now we will draw current detections on canvas with frame bitmap
final Canvas canvas = new Canvas(frameBitmap);

Create a function to draw a single detection​

Write a function called drawSingleDetection that draws a circle for a single detection. Detections contain RectF describing how to position the detection using vertices located in a [0, 1] range. Before drawing on Canvas you need to transform the detection values to absolute values. Then, use drawCircle to draw a circumscribed circle of the detection rectangle using the absolute coordinates.

private void drawSingleDetection(final Canvas canvas, final Detection detection) {
  // first thing we get coordinates of bounding box
  final RectF relativeBbox = detection.getBoundingBox();
  // we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size
  // we do not care about screen resolution at all - we will use cropCenter mode
  final RectF absoluteBbox = new RectF(
          relativeBbox.left * canvas.getWidth(),
          relativeBbox.top * canvas.getHeight(),
          relativeBbox.right * canvas.getWidth(),
          relativeBbox.bottom * canvas.getHeight()
  );
  // we want to draw circle bounds, we need radius and center for that
  final float radius = (float) Math.sqrt(
          Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +
                  Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2)
  );
  canvas.drawCircle(
          absoluteBbox.centerX(),
          absoluteBbox.centerY(),
          radius,
          paint
  );
}

Draw all car detections on Bitmap​

Because more than one object can be detected in a frame, use for to iterate through all detections. Then, filter the detections to only those with the class Car and with a confidence level greater than 0.6 and use drawSingleDetection to draw a green circle around each.

for (final Detection detection : frameDetections.getDetections()) {
  // we will draw only detected cars
  // and filter detections which we are not confident with
  if (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {
      drawSingleDetection(canvas, detection);
  }
}

Set final Bitmap with detections to ImageView​

Bitmap with all detections is ready. But you've prepared it on background thread in the VisionEventsListener.onFrameDetectionsUpdated callback. That means you should switch to main thread and start drawing.

runOnUiThread(() -> {
  // finally we update our image view on main thread
  detectionsView.setImageBitmap(frameBitmap);
});

Finished product​

CustomDetectionActivity.java
github View on GitHub
package com.mapbox.vision.examples;

import android.graphics.Bitmap;
import android.graphics.Canvas;
import android.graphics.Color;
import android.graphics.Paint;
import android.graphics.RectF;
import android.os.Bundle;
import android.widget.ImageView;

import com.mapbox.vision.VisionManager;
import com.mapbox.vision.mobile.core.interfaces.VisionEventsListener;
import com.mapbox.vision.mobile.core.models.AuthorizationStatus;
import com.mapbox.vision.mobile.core.models.Camera;
import com.mapbox.vision.mobile.core.models.Country;
import com.mapbox.vision.mobile.core.models.FrameSegmentation;
import com.mapbox.vision.mobile.core.models.classification.FrameSignClassifications;
import com.mapbox.vision.mobile.core.models.detection.Detection;
import com.mapbox.vision.mobile.core.models.detection.DetectionClass;
import com.mapbox.vision.mobile.core.models.detection.FrameDetections;
import com.mapbox.vision.mobile.core.models.frame.Image;
import com.mapbox.vision.mobile.core.models.position.VehicleState;
import com.mapbox.vision.mobile.core.models.road.RoadDescription;
import com.mapbox.vision.mobile.core.models.world.WorldDescription;
import com.mapbox.vision.performance.ModelPerformance;
import com.mapbox.vision.performance.ModelPerformanceMode;
import com.mapbox.vision.performance.ModelPerformanceRate;

import org.jetbrains.annotations.NotNull;

import java.nio.ByteBuffer;

public class CustomDetectionActivity extends BaseActivity {

    private Boolean visionManagerWasInit = false;
    private ImageView detectionsView;
    private Paint paint;

    // VisionEventsListener handles events from Vision SDK on background thread.
    private VisionEventsListener visionEventsListener = new VisionEventsListener() {

        @Override
        public void onAuthorizationStatusUpdated(@NotNull AuthorizationStatus authorizationStatus) {

        }

        @Override
        public void onCameraUpdated(@NotNull Camera camera) {

        }

        @Override
        public void onCountryUpdated(@NotNull Country country) {

        }

        @Override
        public void onFrameDetectionsUpdated(@NotNull FrameDetections frameDetections) {
            final Bitmap frameBitmap = convertImageToBitmap(frameDetections.getFrame().getImage());
            // now we will draw current detections on canvas with frame bitmap
            final Canvas canvas = new Canvas(frameBitmap);
            for (final Detection detection : frameDetections.getDetections()) {
                // we will draw only detected cars
                // and filter detections which we are not confident with
                if (detection.getDetectionClass() == DetectionClass.Car && detection.getConfidence() > 0.6) {
                    drawSingleDetection(canvas, detection);
                }
            }
            runOnUiThread(() -> {
                // finally we update our image view on main thread
                detectionsView.setImageBitmap(frameBitmap);
            });
        }

        @Override
        public void onFrameSegmentationUpdated(@NotNull FrameSegmentation frameSegmentation) {

        }

        @Override
        public void onFrameSignClassificationsUpdated(@NotNull FrameSignClassifications frameSignClassifications) {

        }

        @Override
        public void onRoadDescriptionUpdated(@NotNull RoadDescription roadDescription) {

        }

        @Override
        public void onUpdateCompleted() {

        }

        @Override
        public void onVehicleStateUpdated(@NotNull VehicleState vehicleState) {

        }

        @Override
        public void onWorldDescriptionUpdated(@NotNull WorldDescription worldDescription) {

        }
    };

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        detectionsView = findViewById(R.id.detections_view);
        preparePaint();
    }

    @Override
    public void onPermissionsGranted() {
        startVisionManager();
    }

    @Override
    public void initViews() {
        setContentView(R.layout.activity_custom_detection);
    }

    @Override
    public void onStart() {
        super.onStart();
        startVisionManager();
    }

    @Override
    public void onStop() {
        super.onStop();
        stopVisionManager();
    }

    private void startVisionManager() {
        if (allPermissionsGranted() && !visionManagerWasInit) {
            VisionManager.create();
            VisionManager.setModelPerformance(
                    new ModelPerformance.On(
                            ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE
                    )
            );
            VisionManager.setVisionEventsListener(visionEventsListener);
            VisionManager.start();
            visionManagerWasInit = true;
        }
    }

    private void stopVisionManager() {
        if (visionManagerWasInit) {
            VisionManager.stop();
            VisionManager.destroy();
            visionManagerWasInit = false;
        }
    }

    private Bitmap convertImageToBitmap(final Image originalImage) {
        final Bitmap bitmap = Bitmap.createBitmap(
                originalImage.getSize().getImageWidth(),
                originalImage.getSize().getImageHeight(),
                Bitmap.Config.ARGB_8888
        );
        // prepare direct ByteBuffer that will hold camera frame data
        ByteBuffer buffer = ByteBuffer.allocateDirect(originalImage.sizeInBytes());
        // associate underlying native ByteBuffer with our buffer
        originalImage.copyPixels(buffer);
        buffer.rewind();
        // copy ByteBuffer to bitmap
        bitmap.copyPixelsFromBuffer(buffer);
        return bitmap;
    }

    private void drawSingleDetection(final Canvas canvas, final Detection detection) {
        // first thing we get coordinates of bounding box
        final RectF relativeBbox = detection.getBoundingBox();
        // we need to transform them from relative (range [0, 1]) to absolute in terms of canvas(frame) size
        // we do not care about screen resolution at all - we will use cropCenter mode
        final RectF absoluteBbox = new RectF(
                relativeBbox.left * canvas.getWidth(),
                relativeBbox.top * canvas.getHeight(),
                relativeBbox.right * canvas.getWidth(),
                relativeBbox.bottom * canvas.getHeight()
        );
        // we want to draw circle bounds, we need radius and center for that
        final float radius = (float) Math.sqrt(
                Math.pow(absoluteBbox.centerX() - absoluteBbox.left, 2) +
                        Math.pow(absoluteBbox.centerY() - absoluteBbox.top, 2)
        );
        canvas.drawCircle(
                absoluteBbox.centerX(),
                absoluteBbox.centerY(),
                radius,
                paint
        );
    }

    private void preparePaint() {
        paint = new Paint(Paint.ANTI_ALIAS_FLAG);
        paint.setColor(Color.GREEN);
        paint.setStrokeWidth(5f);
        paint.setStyle(Paint.Style.STROKE);
    }
}
追加の開発者リソース
このpageは役に立ちましたか？