About the task
Negation is a complex linguistic phenomenon of growing interest in computational linguistics. Detection and treatment of negation is relevant in a wide range of applications, such as information extraction, machine translation or sentiment analysis, where it is crucial to know when a part of the text should have a different meaning due to the presence of negation. In recent years, several challenges and shared tasks have focused on processing negation: NeSp-NLP 2010 (Morante and Sporleder, 2010), CoNLL-2010 share task (Farkas et al., 2010) and SEM 2012 shared task (Morante and Blanco, 2012). However, most of the research on negation has been done for English. Therefore, this task aims to advance the study of this phenomenon in Spanish, the second most widely spoken language in the world and the third most widely used on the Internet. The main objective is to bring together the scientific community that is working on this issue to discuss how this phenomenon is being addressed, what are the main problems encountered, as well as sharing resources and tools aimed at negation in Spanish.
This task was also organized last year as a workshop, NEGES: Workshop on Negation in Spanish (Jiménez-Zafra et al., 2018a; Jiménez-Zafra et al., 2018b), held as part of the XXXIV edition of the International Conference of the Spanish Society for Natural Language Processing (SEPLN 2018). This year, it is presented in IberLEF (Iberian Languages Evaluation Forum), that will be collocated with SEPLN 2019 Conference, with the aim of joining forces with other researchers to create a reference forum in Spanish with tasks of relevance to processing some of the languages spoken in the Iberian Peninsula.
Sub-task A: Negation cues detection
The aim of this task is to advance in the identification of negation cues automatically. To do this, the participants must develop a system able to identify all the negation cues present in a document. For example, in sentence (1) the systems will have to identify three negation cues: i) No, ii) en mi vida and iii) sin.
(1) No recomiendo el libro, en mi vida he leído un libro peor, sin sentido.
Data
The SFU Review SP-NEG corpus (Jiménez-Zafra et al., 2018c) will be used to train and test the systems. The quality of the dataset was measured in terms of Kappa coefficient being of 0.97 for negation cues. A detailed discussion of the main sources of disagreements can be found in (Jiménez-Zafra et al., 2016).
You must have an account in order to download the dataset. If you don't have an account, go to register form. If you have an account,
login here.Evaluation measures
The measures used to evaluate the systems will be Precision, Recall and F-score. For this, the evaluation script used in the *SEM2012 Shared Task: "Resolving the Scope and Focus of Negation" (Morante and Blanco, 2012) will be provided.
Submission
Submitted files must have the format specified in the "OUTPUT" section of the README file provided with the dataset and they must be named as follows, team_domain_subtaskA.txt.
To submit the files, participants must
login here.Papers format
Format details will be communicated shortly, according to the specifications of IberLEF organizers.
Sub-task B: Role of negation in sentiment analysis
It is a specific task whose objective is to evaluate the role of negation in sentiment analysis. In this case, participants should develop a system that uses the negation information contained in the SFU ReviewSP-NEG corpus (Jiménez-Zafra et al., 2018c) to improve the task of polarity classification. Systems will have to classify each review as positive or negative using a negation processing heuristic.
Data
The SFU ReviewSP-NEG corpus (Jiménez-Zafra et al., 2018c) will be used to train and test the systems. The quality of the dataset was measured in terms of Kappa coefficient being of 0.97 for negation cues, 0.94 for scopes, 0.95 for events, 0.95 for the type of negation structure and 0.99 for the type of change in the polarity. A detailed discussion of the main sources of disagreements can be found in (Jiménez-Zafra et al., 2016).
You must have an account in order to download the dataset. If you don't have an account, go to register form. If you have an account,
login here.Evaluation measures
The measures used to evaluate the systems will be Precision, Recall and F-score. They will be measured per class and averaged using macro-average method.
Submission
Submitted files must have the format specified in the "OUTPUT FORMAT" section of the README file provided with the dataset and they must be named as follows, team_subtaskB.txt.
To submit the files, participants must
login here.Papers format
Format details will be communicated shortly, according to the specifications of IberLEF organizers.
Registration
NOTE: Submitting this form only implies registration in NEGES task. To attend it, the registration must be made through the space provided for that purpose on the page of the XXXV International Conference of the Spanish Society for Natural Language Processing (http://hitz.eus/sepln2019).
Schedule
-
February 18, 2019
Registration open
-
February 25, 2019
Training and development sets released
-
April 12, 2019
Test set released
-
May 24, 2019 (23:59 UTC+2)
Submission end
-
June 1, 2019
Results posted
-
June 7, 2019 (23:59 UTC+2)
Deadline for paper submission
-
June 14, 2019:
Notification of acceptance
-
June 21, 2019 (23:59 UTC+2)
Deadline for camera ready paper submission
-
24 September, 2019
IberLEF, SEPLN 2019, Bilbao (Spain)
-
9.00 AM
Registration
-
10.00 AM
Welcome and presentation of the workshop
-
11.00 AM
Presentation of works: Task 2
Task 2: Negation cues detection -
12.00 AM
Presentation of works: Task 3
Task 3: Sentiment analysis -
12.30 AM
Coffee Break
-
1.00 PM
Panel discussion Task 1
Task 1: Annotation guidelines -
2.00 PM
Conclusions and closure
References
- Farkas, R., Vincze, V., Móra, G., Csirik, J., & Szarvas, G. (2010, July). The CoNLL- 2010 shared task: learning to detect hedges and their scope in natural language text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning---Shared Task (pp. 1-12). Association for Computational Linguistics.
- Jiménez-Zafra, S. M., Martín-Valdivia, M. T., Ureña-Lopez, L. A., Marti, T., & Taulé, M. (2016). Problematic cases in the annotation of negation in Spanish. In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics (ExProM) (pp. 42-48).
- Jiménez-Zafra, S. M., Díaz, N. P. C., Morante, R., & Martın-Valdivia, M. T. (2018a). Tarea 1 del Taller NEGES 2018: Guías de Anotación. In Proceedings of NEGES 2018: Workshop on Negation in Spanish (Vol. 2174, pp. 15-21).
- Jiménez-Zafra, S. M., Cruz-Díaz, N. P., Morante, R., & Martín-Valdivia, M. T. (2018b). Tarea 2 del Taller NEGES 2018: Detección de Claves de Negación. In Proceedings of NEGES 2018: Workshop on Negation in Spanish (Vol. 2174, pp. 35- 41).
- Jiménez-Zafra, S. M., Taulé, M., Martín-Valdivia, M. T., Ureña-López, L. A., & Martí, M. A. (2018c). SFU Review SP-NEG: a Spanish corpus annotated with negation for sentiment analysis. A typology of negation patterns. Language Resources and Evaluation, 52(2), 533-569. First published online May 22, 2017. https://doi.org/10.1007/s10579-017-9391-x
- Morante, R., & Blanco, E. (2012, June). * SEM 2012 shared task: Resolving the scope and focus of negation. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (pp. 265-274). Association for Computational Linguistics.
- Morante, R., & Sporleder, C. (2010). Proceedings of the Workshop on Negation and Speculation in Natural Language Processing. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing.
Organizing committee
Salud María Jiménez Zafra
Universidad de Jaén
Maite Martín Valdivia
Universidad de Jaén
Noa Cruz Díaz
Bankia, Madrid
Roser Morante
VU Amsterdam
Program committee
- Altuna, Begoña (Unversidad del País Vasco)
- Bel, Nuria (Universitat Pompeu Fabra, España)
- Blanco, Eduardo (University of North Texas, USA)
- Carrillo de Albornoz, Jorge (UNED, España)
- Casillas, Arantza (Universidad del País Vasco, España)
- Cruz Díaz, Noa (Bankia, Madrid. España)
- Fernández-Gavilanes, Milagros (Centro Universitario de la Defensa - Universidad de Vigo, España)
- García Méndez, Silvia (Universidad de Vigo, España)
- Jiménez Zafra, Salud María (Universidad de Jaén, España)
- Maña, Manuel (Universidad de Huelva, España)
- Marimón, Montserrat (Barcelona Supercomputing Center, España)
- Martí, M. Antonia (Universitat de Barcelona, España)
- Martín Valdivia, Maite (Universidad de Jaén, España)
- Martínez, Paloma (Universidad Carlos III, Madrid, España)
- Martínez Barco, Patricio (Universidad de Alicante, España)
- Martínez-Cámara, Eugenio (Universidad de Granada, España)
- Morante, Roser (VU Amsterdam, Países Bajos)
- Segura, Isabel (Universidad Carlos III, Madrid, España)
- Taboada, Maite (Simon Fraser University, Canadá)
- Taulé, Mariona (Universitat de Barcelona, España)
- Vilares, David (Universidad de la Coruña, España)
- Vivaldi, Jorge (Universitat Pompeu Fabra, España)