Skip to main content
Springer Nature Link
Log in

Overview of EXIST 2024 — Learning with Disagreement for Sexism Identification and Characterization in Tweets and Memes

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14959))

  • 546 Accesses

  • 4 Citations

Abstract

In recent years, the rapid increase in the dissemination of offensive and discriminatory material aimed at women through social media platforms has emerged as a significant concern. This trend has had adverse effects on women’s well-being and their ability to freely express themselves. The EXIST campaign has been promoting research in online sexism detection and categorization in English and Spanish since 2021. The fourth edition of EXIST, hosted at the CLEF 2024 conference, consists of three groups of tasks, which are a continuation of EXIST 2023: sexism identification, source intention identification, and sexism categorization. However, while EXIST 2023 focused on processing tweets, the novelty of this edition is that the three tasks are also applied to memes, resulting in a total of six tasks. The “learning with disagreement” paradigm is adopted to address disagreements in the labelling process and promote the development of equitable systems that are able to learn from different perspectives on the sexism phenomena. The 2024 edition of EXIST has exceeded the success of previous editions, with the participation of 57 teams submitting 412 runs. This lab overview describes the tasks, dataset, evaluation methodology, participant approaches and results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
eBook
USD 59.99
Price excludes VAT (USA)
Softcover Book
USD 74.99
Price excludes VAT (USA)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Notes

  1. 1.

    http://nlp.uned.es/exist2024/. Accessed 28 May 2024.

  2. 2.

    No personally identifiable information about the crowd workers was collected. Crowd workers were informed that the tweets could contain offensive information and were allowed to withdraw voluntarily at any time. Full consent was obtained.

  3. 3.

    In the case of zero variance, we must consider that the probability for values equals or below the mean is 1 (zero IC) and the probability for values above the mean must be smoothed. But this is not the case of the EXIST datasets.

References

  1. Amigó, E., Delgado, A.: Evaluating extreme hierarchical multi-label classification. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, vol. Volume 1: Long Papers, pp. 5809–5819. ACL, Dublin, Ireland (2022)

    Google Scholar 

  2. Aru, G., Emmolo, N., Piras, A., Marzeddu, S., Raffi, J., Passaro, L.C.: RoBEXedda: enhancing sexism detection in tweets for the EXIST 2024 challenge. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  3. Azadi, A., Ansari, B., Zamani, S.: Bilingual sexism classification: fine-tuned XLM-RoBERTa and GPT-3.5 few-shot learning. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  4. Barua, D.D., et al.: Penta ML at EXIST 2024: tagging sexism in online multimodal content with attention-enhanced modal context. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  5. Billig, M.: Humour and hatred: the racist jokes of the Ku Klux Klan. Discourse Soc. 12(3), 267–289 (2014)

    Article  Google Scholar 

  6. Carrillo-Casado, Á., Román-Pásaro, J., Mata-Vázquez, J., Pachón-Álvarez, V.: I2C-UHU at EXIST 2024: transformer-based detection of sexism and source intention in memes using a learning with disagreement approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  7. Chakravarthi, B.R., et al.: Overview of shared task on multitask meme classification - unraveling misogynistic and trolls in online memes. In: Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion, pp. 139–144 (2024)

    Google Scholar 

  8. Chulvi, B., Fontanella, L., Labadie-Tamayo, R., Rosso, P.: Social or individual disagreement? Perspectivism in the annotation of sexist jokes. In: Proceedings of the NLPerspectives 2023: 2nd Workshop on Perspectivist Approaches to Disagreement in NLP, co-locotaed with ECAI-2023 (2023)

    Google Scholar 

  9. Fan, S., Frick, R.A., Steinebach, M.: FraunhoferSIT@EXIST2024: leveraging stacking ensemble learning for sexism detection. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  10. Fang, Y.Z., Lee, L.H., Huang, J.D.: NYCU-NLP at EXIST 2024 – leveraging transformers with diverse annotations for sexism identification in social networks. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  11. Fersini, E., et al.: SemEval-2022 Task 5: multimedia automatic misogyny identification. In: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 533–549 (2022)

    Google Scholar 

  12. Gasparini, F., Rizzi, G., Saibene, A., Fersini, E.: Benchmark dataset of memes with text transcriptions for automatic detection of multi-modal misogynistic content. Data Brief 44, 108526 (2022)

    Article  Google Scholar 

  13. Guerrero-García, M., Cerrejón-Naranjo, M., Mata-Vázquez, J., Pachón-Álvarez, V.: I2C-UHU at EXIST2024: learning from divergence and perspectivism for sexism identification and source intent classification. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  14. Hodson, G., Rush, J., MacInnis, C.C.: A joke is just a joke (except when it isn���t): cavalier humor beliefs facilitate the expression of group dominance motives. J. Pers. Soc. Psychol. 99(4), 660–682 (2010)

    Article  Google Scholar 

  15. Jimenez-Martinez, M.P., Raygoza-Romero, J.M., Sánchez-Torres, C.E., Lopez-Nava, I.H., Montes-y Gómez, M.: Enhancing sexism detection in tweets with annotator-integrated ensemble methods and multimodal embeddings for memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  16. Keinan, R.: Sexism identification in social networks using TF-IDF embeddings, preproccessing, feature selection, word/Char N-grams and various machine learning models in Spanish and English. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  17. Khan, S., Pergola, G., Jhumka, A.: Multilingual sexism identification via fusion of large language models. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  18. Kirk, H.R., Yin, W., Vidgen, B., Röttger, P.: SemEval-2023 task 10: explainable detection of online sexism. In: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval) (2023)

    Google Scholar 

  19. Labadie-Tamayo, R., Chulvi, B., Rosso, P.: Everybody hurts, sometimes. Overview of HUrtful HUmour at IberLEF 2023: detection of humour spreading prejudice in Twitter. In: Procesamiento del Lenguaje Natural (SEPLN), pp. 383–395, No. 71 (2023)

    Google Scholar 

  20. Ma, J., Li, R.: RoJiNG-CL at EXIST 2024: sexism identification in memes by integrating prompting and fine-tuning. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  21. Maqbool, F., Fersini, E.: A contrastive learning based approach to detect sexism in memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  22. Maqbool, N.: Sexism identification in social networks: advances in automated detection – a report on the exist task at CLEF. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  23. Martinez, E., Cuadrado, J., Martinez-Santos, J.C., Puertas, E.: VerbaNex AI at CLEF EXIST 2024: detection of online sexism using transformer models and profiling techniques. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  24. Mendiburo-Seguel, A., Ford, T.E.: The effect of disparagement humor on the acceptability of prejudice. Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues, pp. No Pagination Specified–No Pagination Specified (2019)

    Google Scholar 

  25. Menárguez Box, A., Torres Bertomeu, D.: DiTana-PV at sEXism identification in social networks (EXIST) tasks 4 and 6: the effect of translation in sexism identification. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  26. Naebzadeh, A., Nobakhtian, M., Eetemadi, S.: NICA at EXIST CLEF tasks 2024. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  27. Obrador Reina, M., García Cucó, A.: LightGMB for sexism identification in memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  28. Pan, R., García Díaz, J.A., Bernal Beltrán, T., Valencia-Garcia, R.: UMUTeam at EXIST 2024: multi-modal identification and categorization of sexism by feature integration. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  29. Pasha, U.W.: Multilingual sexism detection in memes, a CLIP-enhanced machine learning approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  30. Petrescu, A., Truică, C.O., Apostol, E.S.: Language-based mixture of transformers for EXIST2024. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  31. Plaza, L., et al.: Overview of EXIST 2023 – learning with disagreement for sexism identification and characterization (Extended Overview). In: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  32. Plaza, L., et al.: Overview of EXIST 2024 – learning with disagreement for sexism identification and characterization in social networks and memes (Extended Overview). In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  33. Plaza, L., et al.: Overview of EXIST 2023 – learning with disagreement for sexism identification and characterization (Extended Overview). In: Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, M. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), vol. 497, pp. 813–854. CEUR Working Notes (2023)

    Google Scholar 

  34. Quan, L.M., Thin, D.V.: Sexism identification in social networks with generation-based approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  35. Rizzi, G., Gimeno-Gómez, D., Fersini, E., Martínez-Hinarejos, C.D.: PINK at EXIST2024: a cross-lingual and multi-modal transformer approach for sexism detection in memes. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  36. Rodríguez-Sánchez, F., et al.: Overview of EXIST 2021: sexism identification in social networks. Procesamiento del Lenguaje Natural 67, 195–207 (2021)

    Google Scholar 

  37. Rodríguez-Sánchez, F., et al.: Overview of EXIST 2022: sexism identification in social networks. Procesamiento del Lenguaje Natural 69, 229–240 (2022)

    Google Scholar 

  38. Ruiz, V., Carrillo-de-Albornoz, J., Plaza, L.: Concatenated transformer models based on levels of agreements for sexism detection. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  39. Shah, A., Gokhale, A.: Team Aditya at EXIST 2024 – detecting sexism in multilingual tweets using contrastive learning approach. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  40. Shanbhag, A., Jadhav, S., Date, A., Joshi, S., Sonawane, S.: The wisdom of weighing: stacking ensembles for a more balanced sexism detector. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  41. Shifat, F.T., et al.: Penta-NLP at EXIST 2024 Task 1–3: sexism identification, source intention, sexism categorization in tweets. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  42. Shimi, G., Mahibha, J., Thenmozhi, D.: Automatic classification of gender stereotypes in social media post. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  43. Siino, M., Tinnirello, I.: Prompt engineering for identifying sexism using GPT mistral 7B. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  44. Smith, T., Nie, R., Trippas, J., Spina, D.: RMIT-IR at EXIST lab at CLEF 2024. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  45. Murari Sreekumar, S.K., Thenmozhi, D., Gopalakrishnan, S., Swaminathan, K.: Sexism identification in tweets using traditional machine learning approaches. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  46. Tavarez-Rodríguez, J., Sánchez-Vega, F., Rosales-Pérez, A., López-Monroy, A.P.: Better together: LLM and neural classification transformers to detect sexism. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  47. Uma, A., et al.: SemEval-2021 task 12: learning with disagreements. In: Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pp. 338–347. Association for Computational Linguistics, Online, August 2021

    Google Scholar 

  48. Usmani, M., Siddiqui, R., Rizwan, S., Khan, F., Alvi, F., Samad, A.: Sexism identification in tweets using BERT and XLM – Roberta. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  49. Vetagiri, A., Mogha, P., Pakray, P.: Cracking down on digital misogyny with MULTILATE a MULTImodaL hATE detection system. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

  50. Villarreal-Haro, K., Sánchez-Vega, F., Rosales-Pérez, A., López-Monroy, A.P.: Stacked reflective reasoning in large neural language models. In: Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum (2024)

    Google Scholar 

Download references

Acknowledgments

This work has been financed by the European Union (NextGenerationEU funds) through the “Plan de Recuperación, Transformación y Resiliencia”, by the Ministry of Economic Affairs and Digital Transformation and by the UNED University. It has also been financed by the Spanish Ministry of Science and Innovation (project FairTransNLP (PID2021-124361OB-C31 and PID2021-124361OB-C32)) funded by MCIN/AEI/10.13039/501100011033 and by ERDF, EU A way of making Europe, and by the Australian Research Council (DE200100064 and CE200100005).

Author information

Authors and Affiliations

  1. Universidad Nacional de Educación a Distancia (UNED), 28040, Madrid, Spain

    Laura Plaza, Jorge Carrillo-de-Albornoz, Víctor Ruiz, Enrique Amigó, Julio Gonzalo & Roser Morante

  2. Universitat Politècnica de València (UPV), 46022, Valencia, Spain

    Alba Maeso, Berta Chulvi & Paolo Rosso

  3. ValgrAI - Valencian Graduate School and Research Network of Artificial Intelligence, 46022, Valencia, Spain

    Paolo Rosso

  4. RMIT University, 3000, Melbourne, Australia

    Damiano Spina

Authors
  1. Laura Plaza
    View author publications

    Search author on:PubMed Google Scholar

  2. Jorge Carrillo-de-Albornoz
    View author publications

    Search author on:PubMed Google Scholar

  3. Víctor Ruiz
    View author publications

    Search author on:PubMed Google Scholar

  4. Alba Maeso
    View author publications

    Search author on:PubMed Google Scholar

  5. Berta Chulvi
    View author publications

    Search author on:PubMed Google Scholar

  6. Paolo Rosso
    View author publications

    Search author on:PubMed Google Scholar

  7. Enrique Amigó
    View author publications

    Search author on:PubMed Google Scholar

  8. Julio Gonzalo
    View author publications

    Search author on:PubMed Google Scholar

  9. Roser Morante
    View author publications

    Search author on:PubMed Google Scholar

  10. Damiano Spina
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Laura Plaza .

Editor information

Editors and Affiliations

  1. Université Grenoble Alpes, CNRS, Grenoble, France

    Lorraine Goeuriot

  2. Université Grenoble Alpes, CNRS, Grenoble, France

    Philippe Mulhem

  3. Université Grenoble Alpes, CNRS, Grenoble, France

    Georges Quénot

  4. Université Grenoble Alpes, CNRS, Grenoble, France

    Didier Schwab

  5. University of Padova, Padua, Italy

    Giorgio Maria Di Nunzio

  6. Sorbonne University, Paris, France

    Laure Soulier

  7. University of Stavanger, Stavanger, Norway

    Petra Galuščáková

  8. University of Essex, Colchester, UK

    Alba García Seco de Herrera

  9. University of Padova, Padua, Italy

    Guglielmo Faggioli

  10. University of Padova, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Plaza, L. et al. (2024). Overview of EXIST 2024 — Learning with Disagreement for Sexism Identification and Characterization in Tweets and Memes. In: Goeuriot, L., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2024. Lecture Notes in Computer Science, vol 14959. Springer, Cham. https://doi.org/10.1007/978-3-031-71908-0_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-71908-0_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-71907-3

  • Online ISBN: 978-3-031-71908-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Keywords

Publish with us

Policies and ethics