Upload large source files to Mapbox Tiling Service
If you upload large amounts of source data to Mapbox Tiling Service or generate a pricing estimate with a large amount of source data, the process may take a long time to finish or even time out. There are a few strategies you can use to make the upload or estimation processes shorter.
Tileset sources
When you create or replace a tileset source, the upload time for the source data depends partially on the file size. There are two strategies you can use to make the upload process shorter: split the file into smaller files, and skip validation.
Split the file
If the upload process takes a long time to finish or times out, you can split your source file into up to 10 smaller files and upload them individually.
Remember that the maximum combined size of all files that make up a single tileset source is 50 GB. If you need to increase your limit, contact us.
- To create a new tileset source from a large source data file, split the source file into smaller files so that no file is larger than 5 GB. Issue a POST request with a single source data file to the tileset source, then append the other source data to the tileset source. For example, if you have a 45 GB file, write a script to issue 9 POST requests, each with a 5 GB file.
- To replace the data in an existing tileset source, split the source file into smaller files so that no file is larger than 5 GB. Issue a request to the PUT endpoint with a single source data file. Then use the POST endpoint to append any additional files.
Skip validation
When you use the Tilesets CLI upload-source command to either create a new tileset source or replace the data in an existing tileset source, by default it validates the data by processing the entire file in memory before beginning the upload. For large files, this validation means the upload process may take a long time. You can use the --no-validation option to bypass this behavior, which will significantly decrease your upload time.
Area estimation
If you are using the Tilesets CLI estimate-area command or the MTS Data Sync estimate command to estimate the tiled area of source data, the estimation process can take a long time if the file is large. There are two strategies you can use to make the process shorter:
- Split your file into smaller files (1 GB or less). These smaller files should be geographically bound; if you don’t split the original file geographically, you run the risk of having an inaccurate estimate with overlapping areas.
- Skip validation. The Tilesets CLI
estimate-areacommand estimates the tiled area of source data in square kilometers, which can help you generate a pricing estimate. By default,estimate-areavalidates data so it processes the entire file in memory before beginning the upload. For large files, this validation means it may take a long time to generate a square kilometer estimate. You can use the--no-validationoption to bypass this behavior, which will significantly decrease your upload time. (The MTS Data Syncestimatecommand, while similar other regards, does not allow you to skip validation.)