ECCV 2020 Tutorial: International, City-Scale Computer Vision Benchmarking with Mapillary MegaCities

Mapillary Research is organizing a tutorial entitled International, City-Scale Computer Vision Benchmarking with Mapillary MegaCities at the ECCV 2020 conference. It will be held as a half-day tutorial on the morning of Friday, August 28th, 2020 in Room M1 of the Scottish Event Campus (SEC) in Glasgow, UK.

Tutorial Motivation

Advancing computer vision (CV) technologies is at the core of every ECCV, CVPR and ICCV conference. Many CV tasks have seen extraordinary performance gains in the past decade, often driven by the ability to rigorously benchmark methods on datasets like ImageNet, Pascal VOC, or the KITTI benchmark suite, to name just a few. Autonomous driving, i.e. a concrete example depending on a number of CV algorithms, can only be safely deployed in real-life conditions after benchmarking existing solutions in rich testing environments and at scale, and with test results made available to the public. Conversely, benchmarking of autonomously driving systems is often tackled by using proprietary datasets not available to the public research community, on datasets with limited geographical coverage or by testing against simulated data.

To overcome this limitation, we introduce a novel, publicly available large-scale dataset called Mapillary MegaCities. This dataset is designed with the goal of creating a completely novel benchmarking paradigm for training and testing various computer vision algorithms. It comprises multiple cities from different continents, each of them built with annotations registered across different representations and modalities where available and possible. The difficulty of capturing data at scale, from many different areas at different conditions and at multiple times of season/day is sidestepped by Mapillary's technology that allows to collaboratively collect imagery (and other modality) data.

Cities can be accessed as highly accurate, reconstructed 3D point clouds, geolocated via landmarks captured at land-surveying grade accuracy. Correspondences of objects are provided across images, 3D models, aerial imagery, street-level LiDAR, aerial LiDAR and according to actual elevation information, where available. The dataset dynamically grows with new images uploaded through contributors and all information will be fully accessible through Mapillary’s platform and APIs. MegaCities will empower benchmarking of object recognition (bounding box based, 2D and 3D), object tracking, depth estimation, 3D reconstruction, domain adaptation, object re-identification, optical flow, object property and occlusion estimation, change detection, etc.

Given the complexity of the new MegaCities dataset and the positive impact it might have on the community, the goal of this tutorial is to provide an early introduction to the dataset, related benchmarking tools and pre-trained models in order to facilitate experimentation by interested researchers and engineers.

Provided Course Material

We will make the following material publicly available before the tutorial date: Online registration for data access, code for online access and evaluation of trained models, pre-trained models serving as baselines for all benchmarking tasks, technical report with dataset description. All material will be available from this website.

Tentative Tutorial Agenda

The tutorial will focus on three main topics: i) introducing MegaCities and its novel design philosophy as a living-and-breathing dataset, ii) describing the data gathering and annotation process, and iii) overviewing the dataset's usage, both as a training and benchmarking tool. The tentative tutorial agenda is:

Introduction (10 min)
The MegaCities Dataset (30 min)
Dataset Creation Process (60 min)
Coffee Break (15 min)
Benchmarks (30 min)
Dataset Usage (60 min)
Pre-trained models (10 min)
Conclusions (10 min)
Questions and Discussions (15 min)