Knowledge Base Question Answering (KBQA) is a popular task in the field of Natural Language Processing and Information Retrieval, in which the goal is to answer a natural language question using the facts in a Knowledge Base. KBQA can involve several subtasks such as entity linking, relation linking, and answer type prediction. In the SMART 2022 Semantic Web Challenge, we focus on three subtasks in KBQA.

The questions can be generally classified based on Wh-terms (Who, What, When, Where, Which, Whom, Whose, Why). A granular answer type classification is possible with popular Semantic Web ontologies such as DBpedia (~760 classes) and Wikidata (~50K classes). Relation linking for question is a hard task, some relations are semantically far and sometimes tokens deciding the relations are spread across the question, some relations are implicit in text, and there are lexical gaps in relation surface forms and KG property labels. Similarly, entity linking is an essential subtask for formulating queries for Knowledge Base Question Answering.

Thus, in the third iteration of SMART challenge in the ISWC 2022, we have three independent tasks:

Task 1 (AT) - Answer Type Prediction: Given a question in natural language, the task is to predict type of the answer using a set of candidates from a target ontology.

Task 2 (RL) - Relation Linking: Given a question in natural language, the task is to predict relation to used for identifying the correct answer.

Task 3 (EL) - Entity Linking: Given a question in natural language, the task is to identify the entities nd link them to the KG, so that they can be used for retrieving the correct answer.

We provide two datasets for this answer prediction task, one using the DBpedia ontology and the other using the Wikidata ontology. Both follow the structure shown below.

Each question will have a (a) question id, (b) question text in natural language, (c) an answer category ("resource"/"literal"/"boolean"), and (d) answer type.

If the category is "resource", answer types are ontology classes from either the DBpedia ontology or the Wikidata ontology. If category is "literal", answer types are either "number", "date", or "string". "boolean" answer type. If the category is "boolean", answer type is always "boolean".

  [
    {
      "id": "dbpedia_1",
      "question": "Who are the gymnasts coached by Amanda Reddin?",
      "category": "resource",
      "type": ["dbo:Gymnast", "dbo:Athlete", "dbo:Person", "dbo:Agent"]
    },
    {
    "id": "dbpedia_2",
    "question": "How many superpowers does wonder woman have?",
    "category": "literal",
    "type": ["number"]
    }
    {
      "id": "dbpedia_3",
      "question": "When did Margaret Mead marry Gregory Bateson?",
      "category": "literal",
      "type": ["date"]
    },
    {
      "id": "dbpedia_4",
      "question": "Is Azerbaijan a member of European Go Federation?",
      "category": "boolean",
      "type": ["boolean"]
    }
  ]
        

Task 2: Relation Prediction (SMART2022-RL)

Similar to AT, we provide two datasets for the relation prediction task, one using the DBpedia ontology and the other using the Wikidata ontology. Both follow the structure shown below.

Each question will have a (a) question id, (b) question text in natural language, (c) a list of relation blocks (n blocks for multihop questions involving n relations). Each relation block can have multiple interchangeable relations.

We also provide a restricted vocabulary of all the relations that are used in the dataset (both training and test sets).

[
  {
    "id": "smart-2022-rl-dbpedia-0"
    "question": "Where was richard sprigg steuart born?",
    "relations": [
      ["dbo:birthPlace"]
    ]
  }, {
    "id": "smart-2022-rl-dbpedia-1",
    "question": "What are the awards won by the producer of Puss in Boots (film)?",
    "relations": [
      ["dbo:producer"],
      ["dbo:award"]
    ]
  },{
    "id": "smart-2022-rl-dbpedia-2"
    "question": "Name a college in california",
    "relations": [
      ["dbo:locatedInArea", "dbo:city", "dbo:location", "dbo:region", "dbo:state"]
    ]
  }
]
            

Task 3: Entity Linking (SMART2022-EL)

Similar to previous two tasks, we provide two datasets for the entity linking task, one using the DBpedia ontology and the other using the Wikidata knowledge graphs. Both follow the structure shown below.

Each question will have a (a) question id, (b) question text in natural language, (c) a list of entities

[
  {
    "id": "smart-2022-rl-dbpedia-0"
    "question": "Where was richard sprigg steuart born?",
    "entities": ["dbr:Richard_Sprigg_Steuart"]
  }, {
    "id": "smart-2022-rl-dbpedia-1",
    "question": "What are the awards won by the producer of Puss in Boots (film)?",
    "entities": [dbr:Puss_in_Boots_(2011_film)]
  },{
    "id": "smart-2022-rl-dbpedia-2"
    "question": "Name a college in california",
    "entities": [dbr:California]
  }
]
            

Task 1: Answer Type Prediction

For each natural language question in the test set, the participating systems are expected to provide two predictions: answer category and answer type. Answer category can be either "resource", "literal" or "boolean". The format is as the same as the training data.

If answer category is "resource", the answer type should be an ontology class (DBpedia or Wikidata, depending on the dataset). The systems could predict a ranked list of classes from the corresponding ontology. If answer category is "literal", the answer type can be either "number", "date" or "string".

Category predication will be considered as a multi-class classification problem and accuracy score will be used as the metric. For type predication, we will use the metric lenient NDCG@k with a Linear decay from the paper from Balog and Neumayer.

The evaluation scripts can be found here.

Task 2: Relation Prediction

For each question, the participating systems are supposed to provide a list of relations. The format of the expected system output is as follows:

[
  {
    "id": "smart-2022-rl-dbpedia-test-0",
    "relations": ["dbo:location"]
  },
  {
    "id": "smart-2022-rl-wikidata-test-1",
    "relations": ["dbo:award", "dbo:director"]
  }
]
            

[
  {
    "id": "smart-2022-rl-wikidata-test-0",
    "relations": ["P4599"]
  },
  {
    "id": "smart-2022-rl-wikidata-test-1",
    "relations": ["P166", "P805"]
  }
]
            

The list of system predicted relations will be compared with the gold list of relations and precision, recall, and F1 metrics will be calculated for all questions. Micro F1 will be used to rank the systems.

The evaluation scripts can be found here.

Participants are requested to submit the system output for the test data. The format is as same as the training data. In addition, the participants are requested to submit a system description that will be included in a joint ISWC challenge proceedings volume in CEUR. System descriptions must be in English either in PDF or HTML, formatted in the style of LNCS, and no longer than 12 pages. Submissions can be sent via EasyChair. There is a slack workspace for the challenge related discussions, please send an email for an invite. The accepted systems will get the opportunity to show their results during the ISWC 2022 poster and demo session.