Overview

In this page, you can find information on how to download the data and make a submission to the leaderboard.

Download data

You can download the data in json format from the Tasks page for each task. Each data example comes with the following fields:

• task: Task identifier. One of dialogue, summarization, intent, safety, stance, mt_zh_en, mt_en_de, mt_en_ru, mt_en_fr.
• example_id: Unique ID of the example. This is the unique ID that identifies a (context, target) pair across all examples for a task.
• {context_field}: This field contains the context data and varies across tasks. For example, in Dialogue task, this is the dialogue given as a list of turns.
• {target_field}: This field contains the target data and varies across tasks. For example, in Dialogue task, this is the response (i.e. final turn) to the given dialogue.

Make a leaderboard submission

In order to make a submission to our leaderboard, please follow the steps below:

1. Download the data.
2. Prepare a predictions file by adding prediction field (with values 0 or 1) to each example in the downloaded data. While each example has multiple fields as described above, the only fields we require for each submission in addition to prediction field are task and example_id fields which are already included in our data. Here is the prediction value mapping for each task:
   • Dialogue: 1=[plausible response], 0=[implausible response]
   • Dialogue Summarization: 1=[correct summary], 0=[incorrect summary]
   • Intent Detection: 1=[true intent], 0=[false intent]
   • Safety Detection: 1=[safe action], 0=[unsafe action]
   • Stance Classification: 1=[supporting argument], 0=[counter argument]
   • Machine Translation: 1=[correct translation], 0=[incorrect translation]
   Please, use the python script we have prepared to check if your prediction files are valid for submission. You can find the script here.
3. Go to Submit page which will redirect you to a Google Form where you can specify details of your submission (e.g. contributors, model details etc.) and upload the prediction file(s). You can submit only for one model and up to 10 prediction files at a time.
4. We will quickly process your submission and publish the results on the leaderboard.

Note: Each user is limited to 5 submissions per week and a total of 10 submissions per month.