Incremental Updates
Incremental updates are in public beta. The feature is subject to changes.
Overview
MTS incremental updates enable updating and re-processing a specific part of an existing tileset instead of updating and re-processing the entire tileset. For example, if your tileset contains 1,000 points and only 1 point changed recently, you can use MTS incremental updates to update and re-process only that 1 updated point. Without incremental updates, MTS will re-process and re-tile all 1,000 points in the tileset, even the points that did not change.
For large datasets, traditional full tileset updates may be time-consuming, leading to stale and out-of-date data. You have to export and upload a full dataset (which may be multiple gigabytes), which may take hours to prepare. With incremental updates, you only need to export, upload, and re-tile the data that recently changed, which may only be a few megabytes. This process is quicker and enables you to update your tileset more often.
Incremental updates is only available for vector tilesets created with Vector MTS.
Common use cases
Common use cases for incremental updates include:
- Updating global trail data with yesterday’s updates
- Hourly updates to a global parking dataset
- Incremental updates to a few roads in a large road network
- Daily or hourly updates to real estate portfolios and other building data, including indoor maps
- Updating the location of one buoy in a large dataset of global ocean buoys
- Consolidating multiple tilesets (e.g., US regions) updated separately into a single incrementally updated tileset (e.g., the entire US)
When to use incremental updates
Your tileset is a good candidate for incremental updates if it meets the following criteria:
- Stable, unique feature IDs (required)
- Less than 15% of the source data changes per update
- One or more of the following conditions:
- A maximum zoom level of 11 or higher
- Source data of 1 GB or more (regardless of zoom levels)
Except for unique stable feature IDs, the guidelines above are recommendations and not hard requirements. Even if your tileset does not meet all the above conditions, incremental updates may still be helpful. For example, an incremental update job where only 16% of the data changes or where the source data is 750 MB may still find incremental updates to be useful.
When to not use incremental updates
- If your source data does not have stable unique feature IDs, such as weather data, you cannot use incremental updates.
- MTS incremental updates are not recommended for low zoom level tilesets with a maximum zoom level of 10 or less (for example z0-z10). For low zoom level tilesets, updating and re-tiling the entire tileset may be quicker and more efficient than using incremental updates.
- If more than 15% of your data has changed then full tileset updates may be more efficient. You are still welcome to use incremental updates if sending changesets is easier.
- Incremental updates require managing and keeping track of changesets. If you do not want the overhead of managing changesets then you should not use incremental updates.
- Tilesets with frequent recipe changes may not be ideal candidates for incremental updates. This is because you cannot update a recipe when publishing a changeset for an incremental update. You can only update recipes in regular full updates. You can still update the recipe of a tileset with incremental updates enabled using initial publish or reset jobs, which are full update jobs.
Feature IDs
Feature IDs are required for MTS incremental updates to work. Additionally, feature IDs should be unique and stable - they should not change over time or between updates. As a result, to use MTS incremental updates, you will need to assign unique, stable feature IDs to your tileset source data and guarantee they stay consistent over time. You cannot use auto-generated feature IDs created by MTS. Without stable unique feature IDs, MTS may incrementally update or delete incorrect or unexpected GeoJSON features.
If you try to use incremental updates without feature IDs, MTS will throw an error message that you are missing feature IDs. MTS will also throw an error if one or more of your features is missing a feature ID.
Using MTS incremental updates with duplicate feature IDs will result in unexpected and unpredictable behavior. To guarantee feature uniqueness, MTS will randomly drop features with duplicate feature IDs in initial publish and reset jobs. Similarly, changesets with duplicate feature IDs will update tilesets in an unpredictable way.
Additionally, incremental updates requires each layer to have unique feature IDs. If there are multiple features with the same ID in a layer, only one of the features will stay. There are no guarantees about which feature remains when resolving duplicate IDs. To avoid unexpected results, make sure there are no features with duplicate IDs in a tileset source or in a changeset.
Adding feature IDs
By default, MTS expects a top-level id key in the GeoJSON. If your stable feature ID is not in the top-level ID field of your source GeoJSON, you can use the ID expression in your MTS recipe. More helpful tips on managing feature IDs are in the MTS recipe specification and MTS FAQ.
Here is an example of a GeoJSON feature with a top-level ID:
{
"id": 1001,
"type": "Feature",
"properties": {},
"geometry": {
"type": "Point",
"coordinates": [ -122, 37 ]
}
}
Here is an example of a recipe that uses the ID expression:
{
"layers": {
"{layer_name}": {
"minzoom": 1,
"maxzoom": 3,
"source": "mapbox://tileset-source/{mapbox_account}/{tileset_source}",
"features": {
"id": [ "get", "custom_id" ]
}
}
}
}
Here is an example of a GeoJSON feature that uses an attribute feature ID:
{
"type": "Feature",
"properties": {
"custom_id": 1001
},
"geometry": {
"type": "Point",
"coordinates": "coordinates": [ -122, 37 ]
}
}
Enabling incremental updates
New tilesets
To enable incremental updates on a new tileset, set incremental to true in the recipe.
{
"version": 1,
"incremental": true,
"layers": {
"{layer_name}": {
"minzoom": 10,
"maxzoom": 14,
"source": "mapbox://tileset-source/{username}/{tileset_source}"
}
}
}
There are four steps to create and publish a new tileset with incremental updates enabled:
- Create a new tileset source, with stable unique feature IDs.
- Create a new tileset recipe with
incrementalset totrue. - Create a new empty tileset, using the recipe from step 2.
- Publish the tileset.
The initial publish in step 4 is a regular full update. Any future publish jobs to the tileset can use changesets for incremental updates. You can also submit regular full updates or “reset jobs” to existing tilesets with incremental updates enabled.
Existing tilesets
To enable MTS incremental updates on an existing tileset (with stable unique feature IDs), first update the recipe of the tileset and set incremental to true.
{
"version": 1,
"incremental": true,
"layers": {
"{layer_name}": {
"minzoom": 10,
"maxzoom": 14,
"source": "mapbox://tileset-source/{mapbox_account}/{tileset_source}"
}
}
}
Second, publish the tileset again. You should receive an API response like the example below that you submitted a reset publish.
{"message": "Processing {tileset_id}", "jobId": "{job_id}", "incremental": "reset"}
The publish will trigger a full re-tile of the tileset (using the tileset source in the recipe) and is called a “reset publish.” Any future reset publish jobs will remove any changesets that have been published or applied beforehand.
Disabling incremental updates
To disable or remove MTS incremental updates on an existing tileset, first update the recipe to remove "incremental": true. Second, re-publish the tileset.
Updating recipes
You can only update the recipe of a tileset using incremental updates in initial publish or reset jobs. Updating a recipe when publishing a changeset may result in unexpected and unpredictable behavior. As a result, we do not recommend updating the recipe when you publish a changeset Tilesets that require frequent recipe changes may not be good candidates for incremental updates.
Changesets
A changeset is one or more line-delimited GeoJSON files uploaded to a Mapbox account that updates a layer in an existing tileset. Changesets allow you to add, update, and delete individual GeoJSON features without re-processing your entire tileset. Changesets are the same format as MTS tileset sources, but with the additional option to delete specific GeoJSON features.
A changeset may consist of up to 10 individual files. Each changeset has a unique changeset ID. The maximum combined total size of all files for a changeset is 50 GB.
Line-delimited GeoJSON
Like MTS tileset sources, changesets are also line delimited GeoJSON files. The code sample below shows a changeset for three points representing earthquakes. The first two GeoJSON features are new earthquakes, while the last GeoJSON feature is an earthquake point that will be deleted with "delete": true. The id field "id": "16994521" is the stable feature ID that MTS uses for incremental updates to properly work.
{ "type": "Feature", "properties": { "id": 16994521, "mag": 2.3, "time": 1507425650893, "felt": null, "tsunami": 0 }, "geometry": { "type": "Point", "coordinates": [ -151.5129, 63.1016, 0.0 ] } }
{ "type": "Feature", "properties": { "id": 16994519, "mag": 1.7, "time": 1507425289659, "felt": null, "tsunami": 0 }, "geometry": { "type": "Point", "coordinates": [ -150.4048, 63.1224, 105.5 ] } }
{ "delete": true, "properties": { "id": 16994517 } }
Adding features
To add a feature in an incremental update, add a new feature with a unique feature ID to your changeset.
{ "type": "Feature", "properties": { "id": 16994519, "mag": 1.7, "time": 1507425289659, "felt": null, "tsunami": 0 }, "geometry": { "type": "Point", "coordinates": [ -150.4048, 63.1224, 105.5 ] } }
Updating features
To update existing features in an incremental update, add the updated feature information to your changeset. You must use the same unique feature IDs in your updated information.
{ "type": "Feature", "properties": { "id": 16994521, "mag": 2.3, "time": 1507425650893, "felt": null, "tsunami": 0 }, "geometry": { "type": "Point", "coordinates": [ -151.5129, 63.1016, 0.0 ] } }
Do not delete and update the same feature or features in the same changeset. Because changesets are de-duplicated by layer and feature ID, the new feature or features may be deleted instead of updated.
Deleting features
To delete features in an incremental updates, use the delete key in your changeset. You only need to provide the unique feature ID to delete a feature. You do not need to provide the complete set of feature attribute or feature property information. The syntax of the delete key depends on whether you are using top-level feature IDs or property level feature IDs.
Top-level feature IDs
Here is an example of how you delete a feature that used a top-level feature ID.
{ "id": 12345, "delete": true }
Property level feature IDs
Here is an example of how you delete a feature that used a property level feature ID called custom_id.
{ "delete": true, "properties": { "custom_id": 2 } }
Validating changesets
You can validate a changeset before uploading using the validate-source command of the Tilesets CLI. This command will validate your changeset GeoJSON locally for any errors. For example, for your changeset file called changeset.geojson.ld, you can run the following command to validate your changeset GeoJSON:
$tilesets validate-source changeset.geojson.ld
The changeset API endpoints do not carry out any validation. If you upload an invalid changeset and then publish an incremental updates job, your jobs may fail with errors like “No GeoJSON features processed. Make sure your GeoJSON is line-delimited.”
Uploading changesets
To upload or create a changeset, use the create a changeset endpoint in the MTS API or the upload-changeset command in the Tilesets CLI. Creating a changeset does not publish or apply a changeset to a tileset, it only uploads a new changeset to your Mapbox account.
Publishing changesets
To publish a changeset, use the publish changesets endpoint in the MTS API or the publish-changesets command in the Tilesets CLI. Both methods require a changeset configuration JSON file that maps the layers in a tileset to a changeset. Here is an example changeset configuration file:
{
"layers": {
"{layer_name}": {
"changeset": "mapbox://tileset-changeset/{username}/{changeset_id}"
},
"{layer_name}": {
"changeset": "mapbox://tileset-changeset/{username}/{changeset_id}"
}
}
}
Deleting changesets
To delete a changeset, use the delete a changeset endpoint in the MTS API or the delete-changeset command in the Tilesets CLI. Deleting a changeset removes the changeset from your Mapbox account. It does not delete or remove the changeset from your tileset.
To remove previous changesets from your tileset, you can start a reset publish which removes all changesets from a tileset and reverts to the tileset source in the recipe. To start a reset publish, publish your tileset without using a changeset. Your tileset recipe can still have "incremental": true for reset publishes to work.
Replacing changesets
To replace (update or overwrite) an existing changeset, use the replace a changeset endpoint in the MTS API or the upload-changeset command in the Tilesets CLI. You can use this endpoint to update or overwrite the changeset data, but you cannot update the changeset ID.
Listing changesets
To list existing changesets in your Mapbox account, use the list changesets endpoint in the MTS API. The Tilesets CLI does not support listing changesets.
Pricing
Incremental updates uses the same pricing as regular full MTS jobs. Incremental update jobs are billed for the processing resources each job consumes, including processed file size (in megabytes) and processed compute units (CUs). Hosted tilesets with incremental updates enabled are also billed in tileset hosting days.
You can read more about pricing for incremental updates in the tilesets pricing guide.