Multimodal Spatial Role Labeling

News

• 9/12/2017. Lab date, Tuesday, from 9:00a-11:00am.
• 5/15/2017. Gold data and the evaluation results delivered.
• 5/5/2017. Test data is released.
• 5/1/2017. Make sure you use the latest release of the training data and the evaluation script, we cleaned some annotation mistakes and typos in the data (Sprl2017_train_v03.xml), download-able in the data section on this page.
• 11/19/2016. The training data is available.
• 11/10/2016. The Registration is open, click here to register.
• 09/07/2016. The lab proposal accepted.

Tweets by mSpRL2017

Lab Schedule

Tuesday, 9/12/2017, 9:00-11:00 am.

09:00-09:20	CLEF 2017: Multimodal Spatial Role Labeling (mSpRL) Task Overview [PDF]	Parisa Kordjamshidi, Taher Rahgooy, Marie-Francine Moens, James Pustejovsky, Umar Manzoor, Kirk Roberts
09:20-09:50	LIP6@CLEF2017: Multi-Modal Spatial Role Labeling using Word Embeddings [PDF]	Eloi Zablocki, Patrick Bordes, Laure Soulier, Benjamin Piwowarski, Patrick Gallinari
09:50-10:20	Declarative Learning based Programming for Spatial Language Understanding	Parisa Kordjamshidi
10:20-11:00	Group discussion

Lab setting

The workshop has two types of submissions, research papers and shared task participant papers. The main topics for the general research papers include the following topics and the related areas :

• Extraction of spatial information form text.
• Extraction of spatial information form images.
• Combining various modalities for information extraction.
• Combining modalities to expand the feature space for machine learning models.
• Combining modalities to help resolving inconsistencies and ambiguity in the extractions.
• Spatial representations appropriate for information extraction from image and text.

Pilot Task Description

Multi-modal spatial role labeling is an extension of the spatial role labeling task to exploit multi-modal data including image and text for extraction of spatial information. The task of spatial role labeling (SpRL)[1] formalizes the representation of spatial concepts and relations in the natural language text to be mapped to qualitative spatial representation models by means of machine learning techniques. Let us illustrate the spatial role labeling task with the following example:

About 20 kids in traditional clothing and hats waiting on stairs. A house and a green wall with gate in the background. A sign saying that plants can't be picked up on the right.

1. The first level of this task is to extract spatial roles including,
   • Spatial indicator : Spatial indicators are triggers that indicate the existence of spatial information in a sentence.
   • Trajector : a Trajector is an entity whose location is described
   • Landmark: a Landmark is a reference object for describing the location of a trajector.
   
   

   About 20 [kids]_TRAJECTOR in traditional clothing and hats waiting [on]_{SPATIAL_INDICATOR} [stairs]_LANDMARK.

   For example, in the above sentences the location of kids that is the {trajector} has been described with respect to the {stairs} that is the {landmark} using the preposition {on} that is the {spatial indicator}. These are examples of some spatial roles that we aim to extract from the sentence.
   Figure[1], shows the picture related to the language descriptions. Extraction of the objects from the image might help the extraction of trajector and landmark from sentences.
2. The second level of this task is to extract spatial relationships. In the same example we have the triplet

   spatial_relation(kids, on, stairs)

   that form a kind of spatial relation/link between the three above mentioned roles. Recognizing the spatial relations is very challenging because there could be several spatial roles in the sentence and the model should be able to recognize the right connections. For example (waiting, on, stairs) is a wrong relation here because "kids" is the trajector in this sentence not "waiting". For this level also the information from the image can be very helpful.
3. The third level of the task is to recognize the type of spatial relations. The type of relations is expressed in terms of topological models, according to the well-known RCC (regional connection calculus) qualitative representation, in addition to 6 directional relations. RCC5 includes (Externally connected, Disconnected, Overlapping, Proper part, and Equality). Directional includes relative directions such as left, right, above, below, behind, front. In the above example we can state the type of relation between the roles in the triplet as

   above(kids, stairs)

The data has been annotated with the above mentioned roles and relations and the annotated data can be used to train machine leaning models to do this kind of extractions automatically. Since doing this task has been appeared to be very challenging due to the ambiguity and polysemy in the text[2], we have decided to exploit the images that come along with CLEF textual data to improve the recognition of the spatial objects and their relations. Specifically, we believe the images will certainly help to improve the recognition of the type of relations given that the geometrical features of the boundaries of the objects in the images are closer to the formal qualitative representations of the relationships compared to the counterpart linguistic descriptions.

Data

The dataset we prepared for this task has been based on CLEF, IAPRTC-12 Image Benchmark. The original data includes touristic pictures along with textual description of the pictures. We have annotated the texts of a subset of images according to spatial role labeling annotation scheme[1].

• Download mSpRL-2017 Testing data without annotation (published 05/05/2017)
• Download mSpRL-2017 Training and Gold data (published 11/19/2016 and 5/15/2017 respectively)

Evaluation

We split the human annotated data into a training set and a testing set. The extractions of the models are compared to human annotations. The evaluation metrics are precision, recall and F1 measures. These metrics are used to evaluate the performance on recognizing each type of role and each type of relation separately. We divide the prediction task to into following subtasks.

• Task1: prediction of TR, LM, SP
• Task2: prediction of spatial triplets, relation(sp, tr, lm)
• Task3: prediction of the type of the relations (General types: Region, Direction, Distance; Specific-values: Above, Below,...)

The submitted systems should be able to perform at least Task1 and Task2. The evaluation is phrase-based and it can be done with exact matching between phrases or counting the overlapping phrases as correct perditions. Please see the usage instructions in the ReadMe file included in the package.

• Download the evaluation scripts published (5/15/2017)
  (bug fixed in counting rare classes for relation types)
• Download the evaluation scripts published (5/7/2017)
  (has options for excluding the rare classes in the evaluation and reporting the overall evaluation)
• Download the evaluation scripts published (4/20/2017)
• Download the evaluation scripts (outdated) (published 4/18/2017)

Baselines

The baseline models have been implemented in Saul which has been equipped by various NLP tools for extraction of linguistic features. This is a very flexible platform to facilitate programming and designing various models for the interested participants and it is publicly available. The SpRL models and various number of features have been implemented based on the previous related works.

Important dates

• Labs registration opens: 4 November 2016
• Registration closes: 21 April 2017
• Test Data release: 5 May 2017.
• Return the results by participants to organizers: 10 May 2017.
• Return the evaluation outcome from organizers to participants: 15 May 2017.
• Submission of Participant Papers [CEUR-WS]: 26 May 2017.
• Review process of participant papers: 26 May-16 June 2017
• Submission of Lab Overviews [LNCS]: 9 June 2017
• Notification of Acceptance Lab Overviews [LNCS]: 16 June 2017
• Camera Ready Copy of Lab Overviews [LNCS] due: 23 June 2017
• Notification of Acceptance Participant Papers [CEUR-WS]: 16 June 2017
• Camera Ready Copy of Participant Papers and Extended Lab Overviews [CEUR-WS] due: 3 July 2017
• CEUR-WS Working Notes Preview for checking: 21 July 2017
• Feedback from lab organizers/participants to correct any error [NO CHANGES/UPDATES ALLOWED]: 28 July 2017

Organizers

• Parisa Kordjamshidi, Tulane University, pkordjam@tulane.edu

• Taher Rahgooy, Tulane University, taher.rahgooy@gmail.com

• Marie-Francine Moens, KULeuven, sien.moens@cs.kuleuven.be

• James Pustejovsky, Brandeis University, jamesp@cs.brandeis.edu

• Umar Manzoor, King Abdulaziz University, umarmanzoor@gmail.com

• Kirk Roberts, UT Health Science Center at Houston, kirk.roberts@uth.tmc.edu

Send Mail

References

[1]. Kordjamshidi, P., Moens, M., van Otterlo, M., (2010). Spatial Role Labeling: Task Definition and Annotation Scheme, LREC'10.

[2]. James Pustejovsky, Jessica Moszkowicz, and Marc Verhagen. A linguistically grounded annotation language for spatial information. TAL, 53(2), 2012. 6

[3]. Kordjamshidi, P., van Otterlo, M., Moens, M. (2011). Spatial role labeling: Towards extraction of spatial relations from natural language. ACM Transactions on Speech and Language Processing, 8(3), 4-36.

[4]. Kordjamshidi, P., Moens, M. (2015). Global machine learning for spatial ontology population. Journal of Web Semantics, 30, 3-21. Download

[5]. James Pustejovsky and Zachary Yochum. Image Annotation with ISO-Space: Distinguishing Content from Structure. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), 2014.

[6]. Kordjamshidi, P., van Otterlo, M., Moens, M. (2015). Spatial role labeling annotation scheme. In: Pustejovsky J., Ide N. (Eds.), Handbook of Linguistic Annotation Springer Verlag. Download

[7]. James Pustejovsky and Zachary Yocum. Capturing motion in ISO-SpaceBank. In Workshop on Interoperable Semantic Annotation, page 25, 2013.