Vision SDK for Android
Beta

Current version: v0.3.0

  • AI and AR features for drivers that run on today’s mobile and embedded applications
  • Build augmented reality navigation with turn-by-turn directions and custom objects
  • Create custom alerts for speeding, lane departures, tailgating, and more
  • Neural networks run on-device: real time performance without taxing your data plan

The Mapbox Vision SDK for Android is a library for interpreting road scenes in real-time directly on Android devices using the device’s built-in camera. Features include:

  • Classification and display of regulatory and warning signs
  • Object detection for vehicles, pedestrians, road signs, and traffic lights
  • Semantic segmentation of the roadway into 14 different classes
  • Augmented reality navigation with global coverage
  • Support for external cameras: WiFi or wired connection

SDK structure

There are three Vision-related SDKs you can interact with directly. All three depend on the Vision Core module.

Available SDKs

  • Vision SDK (vision) is the primary SDK, needed for any application of Mapbox Vision. Its components enable camera configuration, display of classification, detection, and segmentation layers, lane feature extraction, and other interfaces. Vision accesses real-time inference running in VisionCore.
  • Vision AR SDK (vision-ar) is an add-on module for Vision used to create customizable augmented reality experiences. It allows configuration of the user’s route visualization: lane material (shaders, textures), lane geometry, occlusion, custom objects, and more. Read more in the AR navigation guide.
  • Vision Safety SDK (vision-safety) is an add-on module for Vision used to create customizable alerts for speeding, nearby vehicles, cyclists, pedestrians, lane departures, and more. Read more in the Safety alerts guide.

Core logic

Vision Core is the core logic of the system, including all machine learning models. Importing any of the Vision-related SDKs listed above into your project automatically brings VisionCore along.

Requirements

VisionSDK requires Android 6 (API 23) and higher, with QC Snapdragon 650, 710, 8xx (but less than 855) with Open CL support. Some of devices that will work with the Vision SDK:

  • Samsung Galaxy S8, S8+, S9, S9+, Note 8
  • QC Snapdragon 650, 710, 8xx (but less than 855) with Open CL support
  • Xiaomi Mi 6, 8
  • HTC U11, U11+, U12, U12+
  • OnePlus 5, 6

You can also check more details at Vision SDK FAQ.

Besides software and hardware requirements, use of the Vision SDK requires that the device is pointed with a view of the road. Use a dashboard or windshield mount to keep your phone oriented correctly while you drive, such as Car Phone Mount Windshield and Car Mount Holder.

Getting started

To set up the Vision SDK you will need to download the SDK, install the dependencies relevant to your project, and complete a few configuration steps.

Download and install the SDK

Download from vision.mapbox.com/install

You must download the relevant dependencies from vision.mapbox.com/install before continuing. You can download the dependency directly or import it into your project using Maven. This will require that you are logged into your Mapbox account.

SDK configuration

After downloading or importing the SDK into your project, configure the following in your Android Studio project.

Set your Mapbox access token

Mapbox APIs require a Mapbox account and access token:

  1. Get an access token from the Mapbox account page.
  2. Open the strings.xml file, create a new string with name="mapbox_access_token", and paste the access token.
<string name="mapbox_access_token" translatable="false">YOUR_ACCESS_TOKEN_HERE</string>
  1. Create a new class, initialize VisionManager, and pass it your Mapbox access token:
public class VisionApp extends Application {
@Override
public void onCreate() {
super.onCreate();
VisionManager.init(this, getResources().getString(R.string.mapbox_access_token));
}
}

Configure permissions

The Mapbox Vision SDK will require the following permissions:

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />

These should all be granted before calling the SDK.

Add VisionView to the activity layout (optional)

VisionView will render the image that was produced by Vision SDK. You can add it with the following snippet:

<com.mapbox.vision.view.VisionView
  android:id="@+id/vision_view"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  app:visualization_mode="detection" />

app:visualization_mode specifies what will be rendered on VisionView: clear video source, segmentation, or detections.

Lifecycle methods

You will need to call lifecycle methods of VisionManager. To listen for events, that Vision SDK produces, you pass an instance of VisionEventsListener to the VisionManager.start() method.

@Override
protected void onResume() {
super.onResume();
...
VisionManager.create();
VisionManager.start(myVisionEventsListenerInstance);
}
@Override
protected void onPause() {
super.onPause();
...
VisionManager.stop();
VisionManager.destroy();
}

Device setup

Besides installing the framework, you will need to set up the device in the vehicle. Some things to consider when choosing and setting up a mount:

  • Generally, shorter length mounts will vibrate less. Mounting to your windshield or to the dashboard itself are both options.
  • Place the phone near or behind your rearview mirror. Note that your local jurisdiction may have limits on where mounts may be placed.
  • Make sure the phone’s camera view is unobstructed (you will be able to tell with any of the video screens open).

Testing and development

Read more about setting up your development environment for testing the capabilities of the Vision SDK in the Testing and development guide.

Conditions

  • Pricing: For details on pricing, read the Vision FAQ.
  • Attribution: While the Vision SDK is using the camera you must display the Mapbox watermark on screen. Read more about attribution requirements in our terms of service.