LLM-as-a-Judge in Entity Retrieval: Assessing Explicit and Implicit Relevance

Mohammad Hossein Saliminabi, Negar Arabzadeh, Seyed Mohammad Hosseini, Dimitrios Androutsos, Morteza Zihayat, Ebrahim Bagheri
January 2025

Abstract

This paper explores the use of large language models as judges for assessing both explicit and implicit relevance in entity retrieval tasks, providing a novel evaluation framework for information retrieval systems.

Type
Conference paper
Publication
Proceedings of the 34th ACM International Conference on Information and Knowledge Management

This work investigates how large language models can serve as effective judges for evaluating relevance in entity retrieval systems.

Large Language Models Entity Retrieval Relevance Assessment Information Retrieval Evaluation
Morteza Zihayat
Morteza Zihayat
Principal Investigator

Dr. Morteza Zihayat is a Canada Research Chair (CRC) in Human-Centered AI and Associate Professor at Toronto Metropolitan University, Faculty of Engineering and Architectural Science. He also holds appointments as Adjunct Associate Professor at the University of Waterloo (Management Sciences) and IBM Faculty Fellow at IBM Centre for Advanced Studies. He is the Director of the Human-Centered Machine Intelligence Lab.