Remember Me
Or use your Academic/Social account:


Or use your Academic/Social account:


You have just completed your registration at OpenAire.

Before you can login to the site, you will need to activate your account. An e-mail will be sent to you with the proper instructions.


Please note that this site is currently undergoing Beta testing.
Any new content you create is not guaranteed to be present to the final version of the site upon release.

Thank you for your patience,
OpenAire Dev Team.

Close This Message


Verify Password:
Verify E-mail:
*All Fields Are Required.
Please Verify You Are Human:
fbtwitterlinkedinvimeoflicker grey 14rssslideshare1
Hopfgartner, Frank; Brodt, Torben; Seiler, Jonas; Kille, Benjamin; Lommatzsch, Andreas; Larson, Martha; Turrin, Roberto; Serény, Andrá (2015)
Publisher: ACM Special Interest Group
Languages: English
Types: Article
The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms. The goal is to create an algorithm that is able to generate news items that users would click, respecting a strict time constraint. The lab challenges participants to compete in either a "living lab" (Task 1) or perform an evaluation that replays recorded streams (Task 2). In this report, we discuss the objectives and challenges of the NewsREEL lab, summarize last year's campaign and outline the main research challenges that can be addressed by participating in NewsREEL 2016.
  • The results below are discovered through our pilot algorithms. Let us know how we are doing!

    • [1] Xavier Amatriain, Pablo Castells, Arjen P. de Vries, Christian Posse, and Harald Steck, editors. Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE, RUE 2012, Dublin, Ireland, September 9, 2012, volume 910 of CEUR Workshop Proceedings. CEUR-WS.org, 2012.
    • [2] Krisztian Balog, Liadh Kelly, and Anne Schuth. Head rst: Living labs for ad-hoc search evaluation. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM '14, pages 1815{1818, New York, NY, USA, 2014. ACM.
    • [3] Torben Brodt and Frank Hopfgartner. Shedding Light on a Living Lab: The CLEF NewsREEL Open Recommendation Platform. In Proceedings of the Information Interaction in Context conference, IIiX'14, pages 223{226. Springer-Verlag, 2014.
    • [4] Jilin Chen, Rowan Nairn, Les Nelson, Michael Bernstein, and Ed Chi. Short and tweet: Experiments on recommending content from information streams. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '10, pages 1185{ 1194, New York, NY, USA, 2010. ACM.
    • [5] Ernesto Diaz-Aviles, Lucas Drumond, Lars Schmidt-Thieme, and Wolfgang Nejdl. Realtime top-n recommendation in social streams. In Sixth ACM Conference on Recommender Systems, RecSys '12, Dublin, Ireland, September 9-13, 2012, pages 59{66, 2012.
    • [6] Susan Dumais. Evaluating IR in situ. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, page 2, 2009.
    • [7] Nicola Ferro. CLEF 15th birthday: Past, present, and future. SIGIR Forum, 48(2):31{ 55, 2014.
    • [8] Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. O ine and online evaluation of news recommender systems at swissinfo.ch. In Proceedings of the 8th ACM Conference on Recommender Systems, RecSys '14, pages 169{176, New York, NY, USA, 2014. ACM.
    • [9] David Hawking. If SIGIR had an academic track, what would be in it? In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9-13, 2015, page 1077, 2015.
    • [10] Frank Hopfgartner, Allan Hanbury, Henning Mueller, Noriko Kando, Simon Mercer, Jayashree Kalpathy-Cramer, Martin Potthast, Tim Gollup, Anastasia Krithara, Jimmy Lin, Krisztian Balog, and Ivan Eggel. Report of the Evaluation-as-a-Service (EaaS) Expert Workshop. SIGIR Forum, 49(1):57{65, 2015.
    • [11] Frank Hopfgartner and Joemon M. Jose. Semantic user modelling for personal news video retrieval. In Advances in Multimedia Modeling, 16th International Multimedia Modeling Conference, MMM 2010, Chongqing, China, January 6-8, 2010. Proceedings, pages 336{346, 2010.
    • [12] Frank Hopfgartner, Benjamin Kille, Andreas Lommatzsch, Till Plumbaum, Torben Brodt, and Tobias Heintz. Benchmarking news recommendations in a living lab. In 5th International Conference of the CLEF Initiative, pages 250{267, 2014.
    • [13] Frank Hopfgartner, Jana Urban, Robert Villa, and Joemon M. Jose. Simulated testing of an adaptive multimedia information retrieval system. In International Workshop on Content-Based Multimedia Indexing, CBMI '07, Bordeaux, France, June 25-27, 2007, pages 328{335, 2007.
    • [14] Diane Kelly, Susan T. Dumais, and Jan O. Pedersen. Evaluation challenges and directions for information-seeking support systems. IEEE Computer, 42(3):60{66, 2009.
    • [15] Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. Optimised scheduling of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9-13, 2015, pages 453{462, 2015.
    • [16] Benjamin Kille, Frank Hopfgartner, Torben Brodt, and Tobias Heintz. The plista dataset. In NRS'13: Proceedings of the International Workshop and Challenge on News Recommender Systems, pages 14{21. ACM, 10 2013.
    • [17] Benjamin Kille, Andreas Lommatzsch, Roberto Turrin, Andras Sereny, Martha Larson, Torben Brodt, Jonas Seiler, and Frank Hopfgartner. Overview of CLEF newsreel 2015: News recommendation evaluation lab. In Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, September 8-11, 2015., 2015.
    • [18] Benjamin Kille, Andreas Lommatzsch, Roberto Turrin, Andras Sereny, Martha Larson, Torben Brodt, Jonas Seiler, and Frank Hopfgartner. Stream-Based Recommendations: Online and O ine Evaluation as a Service. In Proceedings of the Sixth International Conference of the CLEF Association, CLEF'15, pages 497{517, 2015.
    • [19] Ron Kohavi. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015, page 1, 2015.
    • [20] Jimmy Lin and Miles Efron. Evaluation as a service for information retrieval. SIGIR Forum, 47(2):8{14, 2013.
    • [21] Andreas Lommatzsch. Real-time news recommendation using context-aware ensembles. In Advances in Information Retrieval - 36th European Conference on IR Research, ECIR 2014, Amsterdam, The Netherlands, April 13-16, 2014. Proceedings, pages 51{62, 2014.
    • [22] Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. Content-based recommender systems: State of the art and trends. In Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor, editors, Recommender Systems Handbook, pages 73{105. Springer US, 2011.
    • [23] Anne Schuth, Katja Hofmann, and Filip Radlinski. Predicting search satisfaction metrics with interleaved comparisons. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9-13, 2015, pages 463{472, 2015.
    • [24] Ryen W. White, Ian Ruthven, Joemon M. Jose, and C. J. van Rijsbergen. Evaluating implicit feedback models using searcher simulations. ACM Trans. Inf. Syst., 23(3):325{ 361, 2005.
  • No related research data.
  • No similar publications.

Share - Bookmark

Funded by projects


Cite this article