This book constitutes the refereed proceedings of the Second International Conference on Multilingual and Multimodal Information Access Evaluation, in continuation of the popular CLEF campaigns and workshops that have run for the last decade, CLEF 2011, held in Amsterdem, The Netherlands, in September 2011.
The 14 revised full papers presented together with 2 keynote talks were carefully reviewed and selected from numerous submissions. The papers accepted for the conference included research on evaluation methods and settings, natural language processing within different domains and languages, multimedia and reflections on CLEF. Two keynote speakers highlighted important developments in the field of evaluation: the role of users in evaluation and a framework for the use of crowdsourcing experiments in the setting of retrieval evaluation."

The ninth campaign of the Cross-Language Evaluation Forum (CLEF) for European languages was held from January to September 2008. There were seven main eval- tion tracks in CLEF 2008 plus two pilot tasks. The aim, as usual, was to test the p- formance of a wide range of multilingual information access (MLIA) systems or s- tem components. This year, 100 groups, mainly but not only from academia, parti- pated in the campaign. Most of the groups were from Europe but there was also a good contingent from North America and Asia plus a few participants from South America and Africa. Full details regarding the design of the tracks, the methodologies used for evaluation, and the results obtained by the participants can be found in the different sections of these proceedings. The results of the CLEF 2008 campaign were presented at a two-and-a-half day workshop held in Aarhus, Denmark, September 17-19, and attended by 150 resear- ers and system developers. The annual workshop, held in conjunction with the European Conference on Digital Libraries, plays an important role by providing the opportunity for all the groups that have participated in the evaluation campaign to get together comparing approaches and exchanging ideas. The schedule of the workshop was divided between plenary track overviews, and parallel, poster and breakout sessions presenting this year's experiments and discu- ing ideas for the future. There were several invited talks.

This book constitutes the refereed proceedings of the 10th European Conference on Research and Advanced Technology for Digital Libraries, ECDL 2006, held in Alicante, Spain in September 2006.

The 36 revised full papers presented together with the extended abstracts of 18 demo papers and 15 revised poster papers were carefully reviewed and selected from a total of 159 submissions. The papers are organized in topical sections on architectures, preservation, retrieval, applications, methodology, metadata, evaluation, user studies, modeling, audiovisual content, and language technologies.

This book constitutes the thoroughly refereed postproceedings of the 6th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, held in Vienna, Austria in September 2005.

The 111 revised papers presented together with an introduction were carefully reviewed and selected for inclusion in the book. The papers are organized in topical sections on multilingual textual document retrieval, cross-language and more, monolingual experiments, domain-specific information retrieval, interactive cross-language information retrieval, multiple language question answering, cross-language retrieval in image collections, cross-language speech retrieval, multilingual Web track, cross-language geographical retrieval, and evaluation issues.

The ?fth campaign of the Cross-Language Evaluation Forum (CLEF) for Eu- pean languages was held from January to September 2004. Participation in the CLEF campaigns has increased each year and CLEF 2004 was no exception: 55 groups submitted results for one or more of the di?erent tracks compared with 42 groups in the previous year. CLEF 2004 also marked a breaking point with respect to previous campaigns. The focus was no longer mainly concentrated on multilingual document retrieval as in previous years but was diversi?ed to include di?erent kinds of text retrieval across languages (e. g. , exact answers in the question-answering track) and retrieval on di?erent kinds of media (i. e. , not just plain text but collections containing image and speech as well). In ad- tion, increasing attention was given to issues that regard system usability and user satisfaction with tasks to measure the e?ectiveness of interactive systems or system components being included in both the cross-language question - swering and image retrieval tasks with the collaboration of the coordinators of the interactive track. The campaign culminated in a two-and-a-half-day workshop held in Bath, UK, 15-17 September, immediately following the 8th European Conference on Digital Libraries. The workshop was attended by nearly 100 researchers and s- tem developers.

The fourth campaign of the Cross-language Evaluation Forum (CLEF) for European languages was held from January to August 2003. Participation in this campaign showed a slight rise in the number of participants from the previous year, with 42 groups submitting results for one or more of the different tracks (compared with 37 in 2002), but a steep rise in the number of experiments attempted. A distinctive feature of CLEF 2003 was the number of new tracks and tasks that were offered as pilot experiments. The aim was to try out new ideas and to encourage the development of new evaluation methodologies, suited to the emerging requirements of both system developers and users with respect to today s digital collections and to encourage work on many European languages rather than just those most widely used. CLEF is thus gradually pushing its participants towards the ultimate goal: the development of truly multilingual systems capable of processing collections in diverse media. The campaign culminated in a two-day workshop held in Trondheim, Norway, 21 22 August, immediately following the 7th European Conference on Digital Libraries (ECDL 2003), and attended by more than 70 researchers and system developers. The objective of the workshop was to bring together the groups that had participated in the CLEF 2003 campaign so that they could report on the results of their experiments."

The third campaignof the Cross-LanguageEvaluation Forum (CLEF) for Eu- pean languages was held from January to September 2002. Participation in this campaignshowedaslightriseinthenumberofparticipantswith,37groupsfrom both academia and industry and a steep rise in the number of experiments they submitted for oneor moreof the ?ve o?cialtracks. The campaignculminated in atwo-dayworkshopheldinRome, Italy,19-20September, immediatelyfollowing the Sixth European Conference on Digital Libraries (ECDL 2002), attended by nearly 70 researchersand system developers. The objective of the workshop was to bring together the groups that had participated in CLEF 2002 so that they could report on the results of their experiments. Attendance at the workshop was thus limited to participants in the campaignplus severalinvited guests with recognized expertise in the multilingual information access ?eld. This volume contains thoroughly revised and expanded versions of the preliminary papers presented at the workshop accompanied by a complete run-down and detailed analysis of the results, and it thus provides an exhaustive record of the CLEF 2002 campaign. CLEF2002 wasconducted within the frameworkof a projectof the Infor- tion Society Technologies programme of the European Commission (IST-2000- 31002). The campaign was organized in collaboration with the US National - stitute ofStandardsandTechnology(NIST)andwiththe supportoftheDELOS Network of Excellence for Digital Libraries. The support of NIST and DELOS in the running of the evaluation campaign is gratefully acknowledged. We would also like to thank the other members of the Workshop Steering Committee for their assistance in the coordination of this even

This book constitutes the thoroughly refereed post-proceedings of the Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001, held in Darmstadt, Germany in September 2001.The 35 revised full papers presented together with two introductory survey articles and a comprehensive appendix were carefully improved during the round of reviewing and selections. The papers are organized in topical sections on systems evaluation experiments, mainly cross-language, monolingual experiments, interactive issues, and evaluation issues and results.

This book constitutes the refereed proceedings of the 8th International Conference of the CLEF Initiative, CLEF 2017, held in Dublin, Ireland, in September 2017. The 7 full papers and 9 short papers presented together with 6 best of the labs papers were carefully reviewed and selected from 38 submissions. In addition, this volume contains the results of 10 benchmarking labs reporting their year long activities in overview talks and lab sessions. The papers address all aspects of information access in any modality and language and cover a broad range of topics in the field of multilingual and multimodal information access evaluation.