Datasets for Supervised Adversarial Attacks on Neural Rankers

Amir Khosrojerdi, Amin Bigdeli, Radin Hamidi Rad, Morteza Zihayat, Charles LA Clarke, Ebrahim Bagheri
January 2025

Abstract

We introduce a novel dataset for adversarial rank attacks against neural rankers, enabling systematic research on robustness. Unlike prior unsupervised or surrogate-based methods, our approach uses Retrieval-Augmented Generation (RAG) with a Large Language Model (LLM) to create high-quality adversarial examples that subtly alter rankings while maintaining coherence and relevance. Built via a self-refining LLM-Ranker feedback loop, the dataset includes two tiers: Gold and Diamond, based on attack strength, along with rich metadata, ranking labels, and quality metrics. Released with code and prompts, it supports training, evaluation, and benchmarking of robust ranking systems.

Type
Conference paper
Publication
Proceedings of the 34th ACM International Conference on Information and Knowledge Management

This work provides datasets and frameworks for studying adversarial attacks on neural ranking models.

Adversarial Attacks Neural Rankers Information Retrieval Security Robustness
Morteza Zihayat
Morteza Zihayat
Principal Investigator

Dr. Morteza Zihayat is a Canada Research Chair (CRC) in Human-Centered AI and Associate Professor at Toronto Metropolitan University, Faculty of Engineering and Architectural Science. He also holds appointments as Adjunct Associate Professor at the University of Waterloo (Management Sciences) and IBM Faculty Fellow at IBM Centre for Advanced Studies. He is the Director of the Human-Centered Machine Intelligence Lab.