Remember Me
Or use your Academic/Social account:


Or use your Academic/Social account:


You have just completed your registration at OpenAire.

Before you can login to the site, you will need to activate your account. An e-mail will be sent to you with the proper instructions.


Please note that this site is currently undergoing Beta testing.
Any new content you create is not guaranteed to be present to the final version of the site upon release.

Thank you for your patience,
OpenAire Dev Team.

Close This Message


Verify Password:
Verify E-mail:
*All Fields Are Required.
Please Verify You Are Human:
fbtwitterlinkedinvimeoflicker grey 14rssslideshare1
Clough, P.D. (2015)
Publisher: ACM
Languages: English
Types: Other
Evaluation of IR systems has typically focused on the system and specifically assessing the quality of a ranked list of results with respect to a query. However, IR functionality is typically just one component amongst many that are used to help support users' wider information seeking activities. Many systems that include a search box also provide features, such as faceted lists, subject hierarchies, visualizations and recommendations to help users find information. In this paper I discuss experiences gained from developing a system to support exploration and discovery in digital cultural heritage. In particular I focus on the development of system components to support search and navigation and how the different components were evaluated within the development life-cycle of the project. The importance of taking a holistic approach to evaluation, as well as utilising evaluation approaches from domains other than IR, is emphasized. In short, we need to be thinking outside the (search) box when it comes to evaluation in IR.
  • The results below are discovered through our pilot algorithms. Let us know how we are doing!

    • [1] G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. on Knowl. and Data Eng., 17(6):734-749, June 2005.
    • [2] E. Agirre, N. Aletras, P. D. Clough, S. Fernando, P. Goodale, M. M. Hall, A. Soroa, and M. Stevenson. PATHS: A system for accessing cultural heritage collections. In 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, Proceedings of the Conference System Demonstrations, 4-9 August 2013, Sofia, Bulgaria, pages 151-156, 2013.
    • [3] N. Aletras, M. Stevenson, and P. Clough. Computing similarity between items in a digital library of cultural heritage. J. Comput. Cult. Herit., 5(4):16:1-16:19, Jan. 2013.
    • [4] L. Azzopardi, K. Ja¨rvelin, J. Kamps, and M. D. Smucker. Report on the sigir 2010 workshop on the simulation of interaction. SIGIR Forum, 44(2):35-47, Jan. 2011.
    • [5] P. Bailey, N. Craswell, R. W. White, L. Chen, A. Satyanarayana, and S. Tahaghoghi. Evaluating whole-page relevance. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 767-768, New York, NY, USA, 2010. ACM.
    • [6] T. Ballard and A. Blaine. User search-limiting behavior in online catalogs: Comparing classic catalog use to search behavior in next-generation catalogs. New Library World, 112(5/6):261-273, 2011.
    • [7] P. Borlund. The iir evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research. An International Electronic Journal, 8(3), 2003.
    • [8] P. Borlund. User-Centred Evaluation of Information Retrieval Systems, pages 21-37. John Wiley and Sons, Ltd, 2009.
    • [9] J. Brooke. SUS: A quick and dirty usability scale. In P. W. Jordan, B. Weerdmeester, A. Thomas, and I. L. Mclelland, editors, Usability evaluation in industry. Taylor and Francis, London, 1996.
    • [10] J. P. Chin, V. A. Diehl, and K. L. Norman. Development of an instrument measuring user satisfaction of the human-computer interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '88, pages 213-218, New York, NY, USA, 1988. ACM.
    • [11] P. Clough, P. Goodale, M. M. Hall, and M. Stevenson. Supporting exploration and use of digital cultural heritage materials: the paths perspective. In I. Ruthven and G. Chowdhury, editors, Cultural Heritage Information Access and management. 2015.
    • [12] P. Clough, A. Otegi, and E. Agirre. Personalized page rank for making recommendations in digital cultural heritage collections. pages 49-52, 2014.
    • [13] P. Clough, A. Otegi, E. Agirre, and M. M. Hall. Implementing recommendations in the paths system. In u. Bolikowski, V. Casarosa, P. Goodale, N. Houssos, P. Manghi, and J. Schirrwagen, editors, Theory and Practice of Digital Libraries - TPDL 2013 Selected Workshops, volume 416 of Communications in Computer and Information Science, pages 169-173. Springer International Publishing, 2014.
    • [14] N. Ferro and D. Harman. Clef 2009: Grid@clef pilot track overview. In Proceedings of the 10th Cross-language Evaluation Forum Conference on Multilingual Information Access Evaluation: Text Retrieval Experiments, CLEF'09, pages 552-565, Berlin, Heidelberg, 2009. Springer-Verlag.
    • [15] P. Goodale, P. Clough, N. Ford, M. Hall, M. Stevenson, S. Fernando, N. Aletras, K. Fernie, P. Archer, and A. de Polo. User-centred design to support exploration and path creation in cultural heritage collections. In Proceedings of EuroHCIR2012, volume 909, pages 75-78, 2012.
    • [16] M. M. Hall and P. D. Clough. Exploring large digital library collections using a map-based visualisation. In Research and Advanced Technology for Digital Libraries, volume 8092 of Lecture Notes in Computer Science, pages 220-231, 2013.
    • [17] M. M. Hall, S. Fernando, P. Clough, A. Soroa, E. Agirre, and M. Stevenson. Evaluating hierarchical organisation structures for exploring digital libraries. Information Retrieval, 17(4):351-379, 2014.
    • [18] A. Hanbury and H. M´'uller. Automated component-level evaluation: Present and future. In M. Agosti, N. Ferro, C. Peters, M. de Rijke, and A. F. Smeaton, editors, CLEF, volume 6360 of Lecture Notes in Computer Science, pages 124-135. Springer, 2010.
    • [19] A. Hassan Awadallah, R. W. White, P. Pantel, S. T. Dumais, and Y.-M. Wang. Supporting complex search tasks. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM '14, pages 829-838, New York, NY, USA, 2014. ACM.
    • [20] M. A. Hearst. Search User Interfaces. Cambridge University Press, New York, NY, USA, 1st edition, 2009.
    • [21] J. L. Herlocker and J. A. Konstan. Content-independent task-focused recommendation. IEEE Internet Computing, 5(6):40-47, 2001.
    • [22] T. Isenberg, P. Isenberg, J. Chen, M. Sedlmair, and T. Moller. A systematic review on the practice of evaluating visualization. IEEE Transactions on Visualization and Computer Graphics, 19(12):2818-2827, 2013.
    • [23] K. Ja¨rvelin, P. Vakkari, P. Arvola, F. Baskaya, A. Ja¨rvelin, J. Keka¨l¨ainen, H. Keskustalo, S. Kumpulainen, M. Saastamoinen, R. Savolainen, and E. Sormunen. Task-based information interaction evaluation: The viewpoint of program theory. ACM Trans. Inf. Syst., 33(1):3:1-3:30, 2015.
    • [24] D. Kelly. Methods for evaluating interactive information retrieval systems with users. Found. Trends Inf. Retr., 3(1—2):1-224, Jan. 2009.
    • [25] D. Kelly, S. Dumais, and J. O. Pedersen. Evaluation Challenges and Directions for Information-Seeking Support Systems. Computer, 42(3):60-66, Mar. 2009.
    • [26] D. Lawrie, W. B. Croft, and A. Rosenberg. Finding topic words for hierarchical summarization. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '01, pages 349-357, New York, NY, USA, 2001. ACM.
    • [27] J. R. Lewis. Ibm computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int. J. Hum.-Comput. Interact., 7(1):57-78, Jan. 1995.
    • [28] T. Mandl. Evaluating gir: Geography-oriented or user-oriented? SIGSPATIAL Special, 3(2):42-45, July 2011.
    • [29] S. M. McNee, J. Riedl, and J. A. Konstan. Making recommendations better: An analytic model for human-recommender interaction. In CHI '06 Extended Abstracts on Human Factors in Computing Systems, CHI EA '06, pages 1103-1108, New York, NY, USA, 2006. ACM.
    • [30] G. J. Myers and C. Sandler. The Art of Software Testing. John Wiley and Sons, 2004.
    • [31] C. Plaisant. The challenge of information visualization evaluation. In Proceedings of the Working Conference on Advanced Visual Interfaces, AVI '04, pages 109-116, New York, NY, USA, 2004. ACM.
    • [32] C. J. V. Rijsbergen. Information Retrieval. Butterworth-Heinemann, Newton, MA, USA, 2nd edition, 1979.
    • [33] S. Robertson. On the history of evaluation in IR. J. Information Science, 34(4):439-456, 2008.
    • [34] M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 555-562, New York, NY, USA, 2010. ACM.
    • [35] T. Saracevic. Evaluation of evaluation in information retrieval. In SIGIR'95, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Seattle, Washington, USA, July 9-13, 1995 (Special Issue of the SIGIR Forum), pages 138-146, 1995.
    • [36] G. Singer, U. Norbisrath, and D. Lewandowski. Ordinary search engine users carrying out complex search tasks. Journal of Information Science, 39(3):346-358, 2013.
    • [37] E. M. Voorhees and D. K. Harman. TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press, 2005.
    • [38] M. L. Wilson, B. Kules, m. c. schraefel, and B. Shneiderman. From keyword search to exploration: Designing future search interfaces for the web. Found. Trends Web Sci., 2(1):1-97, Jan. 2010.
  • No related research data.
  • No similar publications.

Share - Bookmark

Funded by projects

  • EC | PATHS

Cite this article