Joint COCO and Mapillary Recognition Workshop at ECCV 2018

Mapillary Research is co-organizing the Joint COCO and Mapillary Recognition workshop at this year’s ECCV conference on September 9 in Munich, Germany. Previous COCO workshops have significantly contributed to pushing the state-of-the-art in object recognition and this year we are hosting challenges for Object Detection with Instance Segmentation and a new task on Panoptic Image Segmentation using images from the Mapillary Vistas dataset 1. We are looking forward to receiving high-quality submissions for advancing the field of visual object recognition.


The goals and participation rules for both tasks are described next and the Mapillary challenges are based on the Mapillary Vistas Research Dataset (MVD). Please register here to receive your download instructions via email to access training (18,000) and validation (2,000) images together with labels and for a more detailed description of the dataset. You will receive a link to the test images (5,000) in the same email. After downloading, please verify the integrity of the downloaded zip files by comparing the md5 checksums, which should match cf59d2f0db54d8b4ae952c00e275f015 for training and validation data and 4d1b70d1ba2fb9da600bd31f3410c125 for test data, respectively. In its research edition, MVD comprises 66 object categories (28 stuff classes, 37 thing classes, 1 void class), and corresponding label IDs can be found by running

> python

in a console from the extracted training data root folder. We kindly ask participating teams to check back on this site for further updates and upload instructions for results.

Mapillary Vistas Object Detection Task

The goal of this task is to provide instance-specific segmentation results for a subset of 37 object classes on MVD. In such a way, results allow to count individual instances of classes like e.g. the number of cars or pedestrians in an image. Details on the submission format will be provided soon, but will follow the specification of the corresponding COCO task to ease participation in both dataset challenges.

The main performance metric used is Average Precision (AP) computed on the basis of instance-level segmentations per object category and averaged over a range of overlaps 0.5:0.05:0.95 with inclusive start and end, following 2.

Mapillary Vistas Panoptic Segmentation Task

This task is an image labeling problem where each pixel in an image under test has to be assigned to a discrete label from a set of pre-defined object categories or to an ignore label void. In addition to labeling stuff classes, each category with instance-specific annotations should be labeled as in the object detection task above such that object instances are separately segmented and enumerable. This task follows the definition recently introduced in 3, and will be evaluated with the Panoptic metric introduced therein.

Performance Evaluation

Code for evaluation will be announced soon and identical for corresponding tasks on COCO and Mapillary Vistas datasets, respectively. All test results have to be stored in a single .zip file per task and can be uploaded to the benchmark server once available.

Important Dates

Training, validation and test data Available here
Challenge submission deadline August 10, 2018 (11:59 PST)
Challenge winners notified August 26, 2018
Winners present at ECCV 2018 Workshop September 9, 2018

Confirmed Speaker

Andreas Geiger is a full professor at the University of Tübingen and a group leader at the Max Planck Institute for Intelligent Systems. Prior to this, he was a visiting professor at ETH Zürich and a research scientist in the Perceiving Systems department of Dr. Michael Black at the MPI-IS. He received his PhD degree in 2013 from the Karlsruhe Institute of Technology. His research interests are at the intersection of 3D reconstruction, motion estimation and visual scene understanding. His work has been recognized with several prizes, including the 3DV best paper award, the CVPR best paper runner up award, the Heinz Maier Leibnitz Prize and the German Pattern Recognition Award. He serves as an area chair and associate editor for several computer vision conferences and journals (CVPR, ICCV, ECCV, PAMI, IJCV).


  1. G. Neuhold, T. Ollmann, S. Rota Bulò, and P. Kontschieder. The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes. In International Conference on Computer Vision (ICCV), 2017, pdf.

  2. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV), 2014, pdf.

  3. A. Kirillov, K. He, R. B. Girshick, C. Rother, and P. Dollár. Panoptic Segmentation. arXiv Tech Report, 2018, pdf.