This volume celebrates the twentieth anniversary of CLEF - the Cross-Language Evaluation Forum for the first ten years, and the Conference and Labs of the Evaluation Forum since - and traces its evolution over these first two decades. CLEF's main mission is to promote research, innovation and development of information retrieval (IR) systems by anticipating trends in information management in order to stimulate advances in the field of IR system experimentation and evaluation. The book is divided into six parts. Parts I and II provide background and context, with the first part explaining what is meant by experimental evaluation and the underlying theory, and describing how this has been interpreted in CLEF and in other internationally recognized evaluation initiatives. Part II presents research architectures and infrastructures that have been developed to manage experimental data and to provide evaluation services in CLEF and elsewhere. Parts III, IV and V represent the core of the book, presenting some of the most significant evaluation activities in CLEF, ranging from the early multilingual text processing exercises to the later, more sophisticated experiments on multimodal collections in diverse genres and media. In all cases, the focus is not only on describing "what has been achieved", but above all on "what has been learnt". The final part examines the impact CLEF has had on the research world and discusses current and future challenges, both academic and industrial, including the relevance of IR benchmarking in industrial settings. Mainly intended for researchers in academia and industry, it also offers useful insights and tips for practitioners in industry working on the evaluation and performance issues of IR tools, and graduate students specializing in information retrieval.

This book constitutes the refereed proceedings of the 9th International Conference of the CLEF Initiative, CLEF 2018, jointly organized by Avignon, Marseille and Toulon universities and held in Avignon, France, in September 2018. The conference has a clear focus on experimental information retrieval with special attention to the challenges of multimodality, multilinguality, and interactive search ranging from unstructured to semi structures and structured data. The 13 papers presented in this volume were carefully reviewed and selected from 39 submissions. Many papers tackle the medical ehealth and ehealth multimedia retrieval challenges, however there are many other topics of research such as document clustering, social biases in IR, social book search, personality profiling. Further this volume presents 9 "best of the labs" papers which were reviewed as a full paper submission with the same review criteria. The labs represented scientific challenges based on new data sets and real world problems in multimodal and multilingual information access. In addition to this, 10 benchmarking labs reported results of their yearlong activities in overview talks and lab sessions. The papers address all aspects of information access in any modularity and language and cover a broad range of topics in the field of multilingual and multimodal information access evaluation.

This volume LNCS 14163 constitutes the refereed proceedings of 14th International Conference of the CLEF Association, CLEF 2023, in Thessaloniki, Greece, during September 18–21, 2023.   The 10 full papers and one short paper included in this book were carefully reviewed and selected from 35 submissions. The conference focuses on authorship attribution, fake news detection and news tracking, noise-detection in automatically transferred relevance judgments, impact of online education on children’s conversational search behavior, analysis of multi-modal social media content, knowledge graphs for sensitivity identification, a fusion of deep learning and logic rules for sentiment analysis, medical concept normalization and domain-specific information extraction. In addition to this, the volume presents 7 “Best of the labs� papers which were reviewed as full paper submissions with the same review criteria. 13 lab overview papers were accepted and represent scientific challenges based on new datasets and real world problems in multimodal and multilingual information access.

The research domains of information retrieval and databases have traditionally adopted different approaches to information management. However, in recent years, there has been an increasing cross-fertilization among the two fields and now many research challenges are transversal to them. With this in mind, a winter school was organized in Bressanone, Italy, in February 2013, within the context of the EU-funded research project PROMISE (Participative Research Laboratory for Multimedia and Multilingual Information Systems Evaluation). PROMISE aimed at advancing the experimental evaluation of complex multimedia and multilingual information systems in order to support individuals, commercial entities and communities, who design, develop, employ and improve such complex systems. The overall goal of PROMISE was to deliver a unified environment collecting data, knowledge, tools and methodologies and to help the user community involved in experimental evaluation. This book constitutes the outcome of the PROMISE Winter School 2013 and contains 9 invited lectures from the research domains of information retrieval and databases plus short papers of the best student poster awards. A large variety of topics are covered, including databases, information retrieval, experimental evaluation, metrics and statistics, semantic search, keyword search in databases, semi-structured search, evaluation both in information retrieval and databases, crowdsourcing and social media.

This book constitutes the thoroughly refereed proceedings of the 9th Italian Research Conference on Digital Libraries, held in Rome, Italy, in January/February 2013. The 18 full papers presented together with an invited paper and a panel paper were selected from extended versions of the presentations given at the conference. The papers then went through an additional round of reviewing and revision after the event. The papers are organized in topical sections on information access; Digital Library (DL) architecture; DL projects; semantics and DLs; models and evaluation for DLs; DL applications; discussing DL perspectives.

The research domains information retrieval and information visualization have always been independent from each other. However, they have the potential to be mutually beneficial. With this in mind, a writer school was organized in Zinal, Switzerland, in January 2012, within the context of the EU-funded research project PROMISE (Participative Research Laboratory for Multimedia and Multilingual Information Systems Evaluation). PROMISE aims at advancing the experimental evaluation of complex multimedia and multilingual information systems in order to support individuals, commercial entities, and communities who design, develop, employ, and improve such complex systems. The overall goal of PROMISE is to deliver a unified environment collecting data, knowledge, tools, and methodologies, and to help the user community involved in experimental evaluation.

This book constitutes the outcome of the PROMISE Winter School 2012 and contains 11 invited lectures from the research domains information retrieval and information visualization. A large variety of subjects are covered, including hot topics such as crowdsourcing and social media.

This book constitutes the thoroughly refereed proceedings of the 8th Italian Research Conference on Digital Libraries, held in Bari, Italy, in February 2012. The 22 full papers, included together with 4 panel papers, were selected from extended versions of the presentations given at the conference, following an additional round of reviewing and revision after the event. The topics covered are as follows: legacy documents and cultural heritage; systems interoperability and data integration; formal and methodological foundations of digital libraries; semantic web and linked data for digital libraries; multilingual information access; digital library infrastructures; metadata creation and management; search engines for digital library systems; evaluation and log data; handling audio/visual and non-traditional objects; user interfaces and visualization; digital library quality; policies and copyright issues in digital libraries; scientific data curation, citation and scholarly publication, user behavior and modeling; and preservation and curation.

The ninth campaign of the Cross-Language Evaluation Forum (CLEF) for European languages was held from January to September 2008. There were seven main eval- tion tracks in CLEF 2008 plus two pilot tasks. The aim, as usual, was to test the p- formance of a wide range of multilingual information access (MLIA) systems or s- tem components. This year, 100 groups, mainly but not only from academia, parti- pated in the campaign. Most of the groups were from Europe but there was also a good contingent from North America and Asia plus a few participants from South America and Africa. Full details regarding the design of the tracks, the methodologies used for evaluation, and the results obtained by the participants can be found in the different sections of these proceedings. The results of the CLEF 2008 campaign were presented at a two-and-a-half day workshop held in Aarhus, Denmark, September 17-19, and attended by 150 resear- ers and system developers. The annual workshop, held in conjunction with the European Conference on Digital Libraries, plays an important role by providing the opportunity for all the groups that have participated in the evaluation campaign to get together comparing approaches and exchanging ideas. The schedule of the workshop was divided between plenary track overviews, and parallel, poster and breakout sessions presenting this year's experiments and discu- ing ideas for the future. There were several invited talks.

This book constitutes the proceedings of the 26th International Conference on Theory and Practice of Digital Libraries, TPDL 2022, which took place in Padua, Italy, in September 2022. The 18 full papers, 27 short papers and 15 accelerating innovation papers included in these proceedings were carefully reviewed and selected from 107 submissions. They focus on digital libraries and associated technical, practical, and social issues.

This book constitutes the refereed proceedings of the 13th International Conference of the CLEF Association, CLEF 2022, held in Bologna, Italy in September 2022.The conference has a clear focus on experimental information retrieval with special attention to the challenges of multimodality, multilinguality, and interactive search ranging from unstructured to semi structures and structured data. The 7 full papers presented together with 3 short papers in this volume were carefully reviewed and selected from 14 submissions. This year, the contributions addressed the following challenges: authorship attribution, fake news detection and news tracking, noise-detection in automatically transferred relevance judgments, impact of online education on children's conversational search behavior, analysis of multi-modal social media content, knowledge graphs for sensitivity identification, a fusion of deep learning and logic rules for sentiment analysis, medical concept normalization and domain-specific information extraction. In addition to this, the volume presents 7 "best of the labs" papers which were reviewed as full paper submissions with the same review criteria. 14 lab overview papers were accepted and represent scientific challenges based on new datasets and real world problems in multimodal and multilingual information access.

This book constitutes the refereed proceedings of the 12th International Conference of the CLEF Association, CLEF 2021, held virtually in September 2021.The conference has a clear focus on experimental information retrieval with special attention to the challenges of multimodality, multilinguality, and interactive search ranging from unstructured to semi structures and structured data. The 11 full papers presented in this volume were carefully reviewed and selected from 21 submissions. This year, the contributions addressed the following challenges: application of neural methods for entity recognition as well as misinformation detection in the health area, skills extraction in job-match databases, stock market prediction using financial news, and extraction of audio features for podcast retrieval. In addition to this, the volume presents 5 "best of the labs" papers which were reviewed as full paper submissions with the same review criteria. 12 lab overview papers were accepted and represent scientific challenges based on new data sets and real world problems in multimodal and multilingual information access.

This volume celebrates the twentieth anniversary of CLEF - the Cross-Language Evaluation Forum for the first ten years, and the Conference and Labs of the Evaluation Forum since - and traces its evolution over these first two decades. CLEF's main mission is to promote research, innovation and development of information retrieval (IR) systems by anticipating trends in information management in order to stimulate advances in the field of IR system experimentation and evaluation. The book is divided into six parts. Parts I and II provide background and context, with the first part explaining what is meant by experimental evaluation and the underlying theory, and describing how this has been interpreted in CLEF and in other internationally recognized evaluation initiatives. Part II presents research architectures and infrastructures that have been developed to manage experimental data and to provide evaluation services in CLEF and elsewhere. Parts III, IV and V represent the core of the book, presenting some of the most significant evaluation activities in CLEF, ranging from the early multilingual text processing exercises to the later, more sophisticated experiments on multimodal collections in diverse genres and media. In all cases, the focus is not only on describing "what has been achieved", but above all on "what has been learnt". The final part examines the impact CLEF has had on the research world and discusses current and future challenges, both academic and industrial, including the relevance of IR benchmarking in industrial settings. Mainly intended for researchers in academia and industry, it also offers useful insights and tips for practitioners in industry working on the evaluation and performance issues of IR tools, and graduate students specializing in information retrieval.

This book constitutes the refereed proceedings of the 11th International Conference of the CLEF Association, CLEF 2020, held in Thessaloniki, Greece, in September 2020.*The conference has a clear focus on experimental information retrieval with special attention to the challenges of multimodality, multilinguality, and interactive search ranging from unstructured to semi structures and structured data. The 5 full papers and 2 short papers presented in this volume were carefully reviewed and selected from 9 submissions. This year, the contributions addressed the following challenges: a large-scale evaluation of translation effects in academic search, advancement of assessor-driven aggregation methods for efficient relevance assessments, and development of a new test dataset. In addition to this, the volume presents 7 "best of the labs" papers which were reviewed as full paper submissions with the same review criteria. The 12 lab overview papers were accepted out of 15 submissions and represent scientific challenges based on new data sets and real world problems in multimodal and multilingual information access. * The conference was held virtually due to the COVID-19 pandemic.

This two-volume set LNCS 12035 and 12036 constitutes the refereed proceedings of the 42nd European Conference on IR Research, ECIR 2020, held in Lisbon, Portugal, in April 2020.* The 55 full papers presented together with 8 reproducibility papers, 46 short papers, 10 demonstration papers, 12 invited CLEF papers, 7 doctoral consortium papers, 4 workshop papers, and 3 tutorials were carefully reviewed and selected from 457 submissions. They were organized in topical sections named: Part I: deep learning I; entities; evaluation; recommendation; information extraction; deep learning II; retrieval; multimedia; deep learning III; queries; IR - general; question answering, prediction, and bias; and deep learning IV. Part II: reproducibility papers; short papers; demonstration papers; CLEF organizers lab track; doctoral consortium papers; workshops; and tutorials. *Due to the COVID-19 pandemic, this conference was held virtually.

This two-volume set LNCS 12035 and 12036 constitutes the refereed proceedings of the 42nd European Conference on IR Research, ECIR 2020, held in Lisbon, Portugal, in April 2020.* The 55 full papers presented together with 8 reproducibility papers, 46 short papers, 10 demonstration papers, 12 invited CLEF papers, 7 doctoral consortium papers, 4 workshop papers, and 3 tutorials were carefully reviewed and selected from 457 submissions. They were organized in topical sections named: Part I: deep learning I; entities; evaluation; recommendation; information extraction; deep learning II; retrieval; multimedia; deep learning III; queries; IR - general; question answering, prediction, and bias; and deep learning IV. Part II: reproducibility papers; short papers; demonstration papers; CLEF organizers lab track; doctoral consortium papers; workshops; and tutorials. *Due to the COVID-19 pandemic, this conference was held virtually.

This book constitutes the refereed proceedings of the 10th International Conference of the CLEF Association, CLEF 2019, held in Lugano, Switzerland, in September 2019.The conference has a clear focus on experimental information retrieval with special attention to the challenges of multimodality, multilinguality, and interactive search ranging from unstructured to semi structures and structured data. The 7 full papers and 8 short papers presented in this volume were carefully reviewed and selected from 30 submissions. This year, many contributions tackle the social networks with the detection of stances or early identification of depression signs on Twitter in a cross-lingual context. Further this volume presents 7 "best of the labs" papers which were reviewed as a full paper submission with the same review criteria. The labs represented scientific challenges based on new data sets and real world problems in multimodal and multilingual information access. In addition to this, 9 benchmarking labs reported results of their yearlong activities in overview talks and lab sessions.

This book constitutes the refereed proceedings of the 8th International Conference of the CLEF Initiative, CLEF 2017, held in Dublin, Ireland, in September 2017. The 7 full papers and 9 short papers presented together with 6 best of the labs papers were carefully reviewed and selected from 38 submissions. In addition, this volume contains the results of 10 benchmarking labs reporting their year long activities in overview talks and lab sessions. The papers address all aspects of information access in any modality and language and cover a broad range of topics in the field of multilingual and multimodal information access evaluation.

This book constitutes the refereed proceedings of the 7th International Conference of the CLEF Initiative, CLEF 2016, held in Toulouse, France, in September 2016. The 10 full papers and 8 short papers presented together with 5 best of the labs papers were carefully reviewed and selected from 36 submissions. In addition to these talks, this volume contains the results of 7 benchmarking labs reporting their year long activities in overview talks and lab sessions. The papers address all aspects of information access in any modality and language and cover a broad rangeof topics in the fields of multilingual and multimodal information access evaluation.

This book constitutes the refereed proceedings of the 38th European Conference on IR Research, ECIR 2016, held in Padua, Italy, in March 2016. The 42 full papers and 28 poster papers presented together with 3 keynote talks and 6 demonstration papers, were carefully reviewed and selected from 284 submissions. The volume contains the outcome of 4 workshops as well as 4 tutorial papers in addition. Being the premier European forum for the presentation of new research results in the field of Information Retrieval, ECIR features a wide range of topics such as: social context and news, machine learning, question answering, ranking, evaluation methodology, probalistic modeling, evaluation issues, multimedia and collaborative filtering, and many more.

This book constitutes the refereed proceedings of the 6th International Conference of the CLEF Initiative, CLEF 2015, held in Toulouse, France, in September 2015. The 31 full papers and 20 short papers presented were carefully reviewed and selected from 68 submissions. They cover a broad range of issues in the fields of multilingual and multimodal information access evaluation, also included are a set of labs and workshops designed to test different aspects of mono and cross-language information retrieval systems.