Skip to main content
Springer Nature Link
Log in

Overview of BioASQ 2024: The Twelfth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14959))

Abstract

This is an overview of the twelfth edition of the BioASQ challenge in the context of the Conference and Labs of the Evaluation Forum (CLEF) 2024. BioASQ is a series of international challenges promoting advances in large-scale biomedical semantic indexing and question answering. This year, BioASQ consisted of new editions of the two established tasks b and Synergy, and two new tasks: a) MultiCardioNER on the adaptation of clinical entity detection to the cardiology domain in a multilingual setting, and b) BIONNE on nested NER in Russian and English. In this edition of BioASQ, 37 competing teams participated with more than 700 distinct submissions in total for the four different shared tasks of the challenge. Similarly to previous editions, most of the participating systems achieved competitive performance, suggesting the continuous advancement of the state-of-the-art in the field.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
eBook
USD 59.99
Price excludes VAT (USA)
Softcover Book
USD 74.99
Price excludes VAT (USA)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Notes

  1. 1.

    https://zenodo.org/doi/10.5281/zenodo.6458078.

  2. 2.

    https://zenodo.org/doi/10.5281/zenodo.11065432.

  3. 3.

    https://zenodo.org/doi/10.5281/zenodo.10948354.

  4. 4.

    https://github.com/nerel-ds/NEREL-BIO/tree/master/bio-nne/.

  5. 5.

    https://codalab.lisn.upsaclay.fr/competitions/16464.

  6. 6.

    http://participants-area.bioasq.org/Tasks/b/eval_meas_2022/.

  7. 7.

    http://participants-area.bioasq.org/results/12b/phaseA/.

  8. 8.

    http://participants-area.bioasq.org/results/12b/phaseAplus/.

  9. 9.

    http://participants-area.bioasq.org/results/12b/phaseB/.

  10. 10.

    http://participants-area.bioasq.org/results/synergy_v2024/.

References

  1. Aksenova, A., Datseris, A., Vassileva, S., Boytcheva, S.: Transformer-based disease and drug named entity recognition in multilingual clinical texts: MultiCardioNER challenge. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  2. Almeida, T., Jonker, R., Reis, J., Almeida, J., Matos, S.: From retrieval to answer generation: insights from BioASQ 12 task B. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  3. Anaya, C., Fernandes, M., Couto, F.: LLM fine-tuning with biomedical open-source data. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  4. Ateia, S., Kruschwitz, U.: Can open-source LLMs compete with commercial models? Exploring the few-shot performance of current GPT models in biomedical tasks. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  5. Baldwin, B., Carpenter, B.: Lingpipe. Available from World Wide Web (2003). http://alias-i.com/lingpipe

  6. Balikas, G., et al.: Evaluation framework specifications. Project deliverable D4.1, UPMC (2013)

    Google Scholar 

  7. Beltagy, I., Lo, K., Cohan, A.: SciBERT: pretrained language model for scientific text. In: EMNLP (2019)

    Google Scholar 

  8. Buonocore, T.M., Crema, C., Redolfi, A., Bellazzi, R., Parimbelli, E.: Localizing in-domain adaptation of transformer-based biomedical language models. J. Biomed. Inform. 144, 104431 (2023)

    Article  Google Scholar 

  9. Chih, B.C., Han, J.C., Tzong-Han Tsai, R.: NCU-IISR: enhancing biomedical question answering with GPT-4 and retrieval augmented generation in BioASQ 12b phase B. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  10. Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. CoRR abs/1911.02116 (2019). http://arxiv.org/abs/1911.02116

  11. Danu, M.D., Marica, V.G., Suciu, C., Itu, L.M., Farri, O.: Multilingual clinical NER for diseases and medications recognition in cardiology texts using BERT embeddings. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  12. Davydova, V., Loukachevitch, N., Tutubalina, E.: Overview of BioNNE task on biomedical nested named entity recognition at BioASQ 2024. In: CLEF Working Notes (2024)

    Google Scholar 

  13. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805

  14. Galat, D., Moshkin, S.: Refining zero-short approaches for biomedical question answering. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  15. Gao, Y., Zong, L., Li, Y.: Enhancing biomedical question answering with parameter-efficient fine-tuning and hierarchical retrieval augmented generation. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  16. Gasco, L., et al.: Overview of BioASQ 2021-MESINESP track. Evaluation of advance hierarchical classification techniques for scientific literature, patents and clinical trials (2021)

    Google Scholar 

  17. Gonçalves, R., Lamúrias, A.: Team NOVA LINCS @ BIOASQ12 MultiCardioNER track: entity recognition with additional entity types. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  18. Gu, Y., et al.: Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. (HEALTH) 3(1), 1–23 (2021)

    Google Scholar 

  19. He, P., Gao, J., Chen, W.: DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing (2021)

    Google Scholar 

  20. He, P., Liu, X., Gao, J., Chen, W.: DeBERTa: decoding-enhanced BERT with disentangled attention. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=XPZIaotutsD

  21. Huang, B.W.: Generative large language models augmented hybrid retrieval system for biomedical question answering. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  22. Jiang, A.Q., et al.: Mixtral of experts (2024)

    Google Scholar 

  23. Jonker, R., Almeida, T., Matos, S.: BIT.UA at MultiCardioNER: adapting a multi-head CRF for cardiology. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  24. Krithara, A., Nentidis, A., Bougiatiotis, K., Paliouras, G.: BioASQ-QA: a manually curated corpus for biomedical question answering. Sci. Data 10(1), 170 (2023)

    Article  Google Scholar 

  25. Krithara, A., Nentidis, A., Paliouras, G., Krallinger, M., Miranda, A.: BioASQ at CLEF2021: large-scale biomedical semantic indexing and question answering. In: Hiemstra, D., Moens, M.-F., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds.) ECIR 2021. LNCS, vol. 12657, pp. 624–630. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72240-1_73

    Chapter  Google Scholar 

  26. Lee, C., Simpson, T.I., Posma, J.M., Lain, A.D.: Comparative analyses of multilingual drug entity recognition systems for clinical case reports in cardiology. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  27. Li, J., et al.: BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database J. Biol. Databases Curation 2016 (2016). https://doi.org/10.1093/database/baw068

  28. Lima-López, S., Farré-Maduell, E., Brivá-Escalada, V., Gascó, L., Krallinger, M.: MEDDOPLACE Shared Task overview: recognition, normalization and classification of locations and patient movement in clinical texts. Procesamiento del Lenguaje Natural 71, 301–311 (2023)

    Google Scholar 

  29. Lima-López, S., Farré-Maduell, E., Gasco-Sánchez, L., Rodríguez-Miret, J., Krallinger, M.: Overview of SympTEMIST at BioCreative VIII: corpus, guidelines and evaluation of systems for the detection and normalization of symptoms, signs and findings from text. In: Proceedings of the BioCreative VIII Challenge and Workshop: Curation and Evaluation in the era of Generative Models (2023)

    Google Scholar 

  30. Lima-López, S., et al.: Overview of MedProcNER task on medical procedure detection and entity linking at BioASQ 2023. In: Working Notes of CLEF 2023 (2023)

    Google Scholar 

  31. Lima-López, S., et al.: Overview of MultiCardioNER task at BioASQ 2024 on medical speciality and language adaptation of clinical NER systems for Spanish, English and Italian. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  32. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  33. Loukachevitch, N., et al.: NEREL: a Russian information extraction dataset with rich annotation for nested entities, relations, and wikidata entity links. Lang. Resour. Eval. 58, 547–583 (2023)

    Article  Google Scholar 

  34. Loukachevitch, N., et al.: NEREL-BIO: a dataset of biomedical abstracts annotated with nested named entities. Bioinformatics 39(4), btad161 (2023). https://doi.org/10.1093/bioinformatics/btad161

  35. Miranda-Escalada, A., et al.: Overview of DisTEMIST at BioASQ: automatic detection and normalization of diseases from clinical texts: results, methods, evaluation and multilingual resources (2022)

    Google Scholar 

  36. Nentidis, A., et al.: Overview of BioASQ 2023: the eleventh BioASQ challenge on large-scale biomedical semantic indexing and question answering. In: Arampatzis, A., et al. (eds.) CLEF 2023. LNCS, vol. 14163, pp. 227–250. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-42448-9_19

    Chapter  Google Scholar 

  37. Nentidis, A., Katsimpras, G., Krithara, A., Paliouras, G.: Overview of BioASQ tasks 12b and Synergy12 in CLEF2024. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  38. Nentidis, A., et al.: Overview of BioASQ 2021: the ninth BioASQ challenge on large-scale biomedical semantic indexing and question answering. In: Candan, K.S., et al. (eds.) CLEF 2021. LNCS, vol. 12880, pp. 239–263. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85251-1_18

    Chapter  Google Scholar 

  39. Nentidis, A., et al.: Overview of BioASQ 2022: the tenth BioASQ challenge on large-scale biomedical semantic indexing and question answering. In: Barrón-Cedeño, A., et al. (eds.) CLEF 2022. LNCS, vol. 13390, pp. 337–361. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-13643-6_22

    Chapter  Google Scholar 

  40. Nentidis, A., et al.: Overview of BioASQ 2020: the eighth BioASQ challenge on large-scale biomedical semantic indexing and question answering. In: Arampatzis, A., et al. (eds.) CLEF 2020. LNCS, vol. 12260, pp. 194–214. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58219-7_16

    Chapter  Google Scholar 

  41. Nentidis, A., Krithara, A., Paliouras, G., Farre-Maduell, E., Lima-Lopez, S., Krallinger, M.: BioASQ at CLEF2023: the eleventh edition of the large-scale biomedical semantic indexing and question answering challenge. In: Kamps, J., et al. (eds.) ECIR 2023, Part III. LNCS, vol. 13982, pp. 577–584. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28241-6_66

    Chapter  Google Scholar 

  42. Nentidis, A., Krithara, A., Paliouras, G., Gasco, L., Krallinger, M.: BioASQ at CLEF2022: the tenth edition of the large-scale biomedical semantic indexing and question answering challenge. In: Hagen, M., et al. (eds.) ECIR 2022. LNCS, vol. 13186, pp. 429–435. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99739-7_53

    Chapter  Google Scholar 

  43. Nentidis, A., et al.: BioASQ at CLEF2024: the twelfth edition of the large-scale biomedical semantic indexing and question answering challenge. In: Goharian, N., et al. (eds.) ECIR 2024. LNCS, vol. 14612, pp. 490–497. Springer, Cham (2024). https://doi.org/10.1007/978-3-031-56069-9_67

    Chapter  Google Scholar 

  44. Panou, D., Dimopoulos, A., Reczko, M.: Farming open LLMs for biomedical question answering. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  45. Rehana, H., et al.: Nested named entity recognition using multilayer BERT-based model. In: CLEF Working Notes (2024)

    Google Scholar 

  46. Reimer, J.H., Bondarenko, A., Hagen, M., Viehweger, A.: MiBi at BioASQ 2024: retrieval-augmented generation for answering biomedical questions. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  47. Romano, A., Riccio, G., Postiglione, M., Moscato, V.: Identifying cardiological disorders in spanish via data augmentation and fine-tuned language models. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  48. Rozhkov, I., Loukachevitch, N.: Prompts in few-shot named entity recognition. Pattern Recogn. Image Anal. 33(2), 122–131 (2023)

    Article  Google Scholar 

  49. Styll, P., Campillos-Llanos, L., Kusa, W., Hanbury, A.: Cross-linguistic disease and drug detection in cardiology clinical texts: methods and outcomes. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  50. Tsatsaronis, G., et al.: An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinform. 16, 138 (2015)

    Article  Google Scholar 

  51. Yang, Z., Zhou, Y., Eric, N.: Learning to answer biomedical questions: OAQA at bioASQ 4B. In: ACL 2016, p. 23 (2016)

    Google Scholar 

  52. Yasunaga, M., Leskovec, J., Liang, P.: LinkBERT: pretraining language models with document links. In: Association for Computational Linguistics (ACL) (2022)

    Google Scholar 

  53. Zhang, S., Cheng, H., Gao, J., Poon, H.: Optimizing bi-encoder for named entity recognition via contrastive learning. In: The Eleventh International Conference on Learning Representations (2022)

    Google Scholar 

  54. Zhou, W.: Biomedical nested NER with large language model and UMLS heuristics. In: CLEF Working Notes (2024)

    Google Scholar 

  55. Zhou, W., Ngo, T.H.: Using pretrained large language model with prompt engineering to answer biomedical questions. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

  56. Şerbetçi, O., Wang, X.D., Leser, U.: HU-WBI at BioASQ12B phase A: exploring rank fusion of dense retrievers for biomedical question answering. In: Faggioli, G., Ferro, N., Galuščáková, P., García Seco de Herrera, A. (eds.) CLEF Working Notes (2024)

    Google Scholar 

Download references

Acknowledgments

Google was a proud sponsor of the BioASQ Challenge in 2023. Ovid is also sponsoring this edition of BioASQ. The twelfth edition of BioASQ is also sponsored by Elsevier. Atypon Systems Inc. is also sponsoring this edition of BioASQ. The MEDLINE/PubMed data resources considered in this work were accessed courtesy of the U.S. National Library of Medicine. BioASQ is grateful to the CMU team for providing the exact answer baselines for task 12b. The MultiCardioNER track was funded by Spanish and European projects such as DataTools4Heart (Grant Agreement No. 101057849), AI4HF (Grant Agreement No. 101080430), BARITONE (Proyectos de Transición Ecológica y Transición Digital 2021. Expediente \(\textrm{N}^{\underline{\textrm{o}}}\) TED2021-129974B-C21) and AI4ProfHealth (PID2020-119266RA-I00). The work on the BioNNE task was supported by the Russian Science Foundation [grant number 23-11-00358].

Author information

Authors and Affiliations

  1. National Center for Scientific Research “Demokritos”, Athens, Greece

    Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara & Georgios Paliouras

  2. Aristotle University of Thessaloniki, Thessaloniki, Greece

    Anastasios Nentidis

  3. Barcelona Supercomputing Center, Barcelona, Spain

    Salvador Lima-López, Eulàlia Farré-Maduell & Martin Krallinger

  4. Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  5. Sber AI, Moscow, Russia

    Vera Davydova & Elena Tutubalina

  6. Artificial Intelligence Research Institute, Moscow, Russia

    Elena Tutubalina

  7. Kazan Federal University, Kazan, Russia

    Elena Tutubalina

Authors
  1. Anastasios Nentidis
    View author publications

    Search author on:PubMed Google Scholar

  2. Georgios Katsimpras
    View author publications

    Search author on:PubMed Google Scholar

  3. Anastasia Krithara
    View author publications

    Search author on:PubMed Google Scholar

  4. Salvador Lima-López
    View author publications

    Search author on:PubMed Google Scholar

  5. Eulàlia Farré-Maduell
    View author publications

    Search author on:PubMed Google Scholar

  6. Martin Krallinger
    View author publications

    Search author on:PubMed Google Scholar

  7. Natalia Loukachevitch
    View author publications

    Search author on:PubMed Google Scholar

  8. Vera Davydova
    View author publications

    Search author on:PubMed Google Scholar

  9. Elena Tutubalina
    View author publications

    Search author on:PubMed Google Scholar

  10. Georgios Paliouras
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Anastasios Nentidis .

Editor information

Editors and Affiliations

  1. Université Grenoble Alpes, CNRS, Grenoble, France

    Lorraine Goeuriot

  2. Université Grenoble Alpes, CNRS, Grenoble, France

    Philippe Mulhem

  3. Université Grenoble Alpes, CNRS, Grenoble, France

    Georges Quénot

  4. Université Grenoble Alpes, CNRS, Grenoble, France

    Didier Schwab

  5. University of Padova, Padua, Italy

    Giorgio Maria Di Nunzio

  6. Sorbonne University, Paris, France

    Laure Soulier

  7. University of Stavanger, Stavanger, Norway

    Petra Galuščáková

  8. University of Essex, Colchester, UK

    Alba García Seco de Herrera

  9. University of Padova, Padua, Italy

    Guglielmo Faggioli

  10. University of Padova, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nentidis, A. et al. (2024). Overview of BioASQ 2024: The Twelfth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In: Goeuriot, L., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2024. Lecture Notes in Computer Science, vol 14959. Springer, Cham. https://doi.org/10.1007/978-3-031-71908-0_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-71908-0_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-71907-3

  • Online ISBN: 978-3-031-71908-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Keywords

Publish with us

Policies and ethics