SMART task is a dataset for the answer type prediction task. Question Answering is a popular task in the field of Natural Language Processing and Information Retrieval, in which, the goal is to answer a natural language question (going beyond the document retrieval). Question or answer type classification plays a key role in question answering. The questions can be generally classified based on Wh-terms (Who, What, When, Where, Which, Whom, Whose, Why). Similarly, the answer type classification is determining the type of the expected answer based on the query. Such answer type classifications in literature are performed as a short-text classification task using a set of coarse-grained types, for instance, either 6 or 50 types with TREC QA task. A granular answer type classification is possible with popular Semantic Web ontologies such as DBpedia (~760 classes) and Wikidata (~50K classes).

In this challenge, given a question in natural language, the task is to predict type of the answer using a set of candidates from a target ontology.

[1] A Methodology for Hierarchical Classification of Semantic Answer Types of Questions
Ammar Ammar, Shervin Mehryar, and Remzi Celebi

[2] Question Embeddings for Semantic Answer Type Prediction
Eleanor Bill and Ernesto Jiménez-Ruiz

[3] Hierarchical Contextualized RepresentationModels for Answer Type Prediction
Natthawut Kertkeidkachorn, Rungsiman Nararatwong, Phuc Nguyen, Ikuya Yamada, Hideaki Takeda, and Ryutaro Ichise

[4] Two-stage Semantic Answer Type Prediction for QA using BERT and Class-Specificity Rewarding
Christos Nikas, Pavlos Fafalios, and Yannis Tzitzikas

[5] Augmentation-based Answer Type Classification of the SMART dataset
Aleksandr Perevalov and Andreas Both

[6] Semantic Answer Type Prediction using BERT: IAI at the ISWC SMART Task 2020
Vinay Setty and Krisztian Balog

[7] COALA - A Rule-Based Approach to Answer Type Prediction
Nadine Steinmetz and Kai-Uwe Sattler

[8] Fine and Ultra-Fine Entity Type Embeddings for Question Answering
Sai Vallurupalli, Jennifer Sleeman, and Tim Finin

The following table illustrates some example questions and expected answer types from DBpedia ontology and Wikidata ontology.

We provide two datasets for this task, one using the DBpedia ontology and the other using the Wikidata ontology. Both follow the structure shown below.

Each question will have a (a) question id, (b) question text in natural language, (c) an answer category ("resource"/"literal"/"boolean"), and (d) answer type.

If the category is "resource", answer types are ontology classes from either the DBpedia ontology or the Wikidata ontology. If category is "literal", answer types are either "number", "date", or "string". "boolean" answer type. If the category is "boolean", answer type is always "boolean".

  [
    {
      "id": "dbpedia_1",
      "question": "Who are the gymnasts coached by Amanda Reddin?",
      "category": "resource",
      "type": ["dbo:Gymnast", "dbo:Athlete", "dbo:Person", "dbo:Agent"]
    },
    {
    "id": "dbpedia_2",
    "question": "How many superpowers does wonder woman have?",
    "category": "literal",
    "type": ["number"]
    }
    {
      "id": "dbpedia_3",
      "question": "When did Margaret Mead marry Gregory Bateson?",
      "category": "literal",
      "type": ["date"]
    },
    {
      "id": "dbpedia_4",
      "question": "Is Azerbaijan a member of European Go Federation?",
      "category": "boolean",
      "type": ["boolean"]
    }
  ]
        

Dataset statistics:

The DBpedia dataset contains 21,964 (train - 17,571, test - 4,393) questions and the Wikidata dataset contains 22,822 (train - 18,251, test - 4,571) questions.

DBpedia training set consists of 9,584 resource questions, 2,799 boolean questions, and 5,188 literal (number - 1,634, date - 1,486, string - 2,068) questions.

Wikidata training set consists of 11,683 resource questions, 2,139 boolean questions, and 4,429 literal questions.

For each natural language question in the test set, the participating systems are expected to provide two predictions: answer category and answer type. Answer category can be either "resource", "literal" or "boolean".

If answer category is "resource", the answer type should be an ontology class (DBpedia or Wikidata, depending on the dataset). The systems could predict a ranked list of classes from the corresponding ontology. If answer category is "literal", the answer type can be either "number", "date" or "string".

Category predication will be considered as a multi-class classification problem and accuracy score will be used as the metric. For type predication, we will use the metric lenient NDCG@k with a Linear decay from the paper from Balog and Neumayer.

We will provide more details on evaluation and the evaluations scripts soon.

Participants are requested to submit the system output for the test data. The format is as same as the training data. In addition, the participants are requested to submit a system description that will be included in a joint ISWC challenge proceedings volume in CEUR. System descriptions must be in English either in PDF or HTML, formatted in the style of LNCS, and no longer than 12 pages. Submissions can be sent via email or through the slack workspace. The accepted systems will get the opportunity to show their results during the ISWC 2020 poster and demo session.