Robust-EO
A benchmark for entity-oriented retrieval on TREC Robust04, built around LLM centrality filtering.
Entity annotations on retrieval corpora are noisy. Off-the-shelf entity linkers such as WAT will happily tag every proper noun in a document, which leaves a retrieval system reasoning about entities that are merely mentioned rather than actually what the document is about. Robust-EO is a diagnostic benchmark that asks how much of that noise can be removed, and whether removing it helps.
The core idea is centrality filtering: use an LLM to judge whether each linked entity is central, secondary or incidental to the document, then measure what happens downstream. We build five annotation variants — raw WAT, rho-filtered, centrality-filtered, both, and both plus constrained expansion — and evaluate each against human judgements and against retrieval effectiveness on Robust04.
The work is a research collaboration with Missouri University of Science and Technology, currently being written up as a short paper — unpublished as of now. My contribution spans the annotation pipeline, the LLM-as-annotator agreement analysis, and the retrieval-side evaluation.
Highlights
- Six research questions covering noise reduction, constrained expansion, label reliability, entity discriminativeness, and query-dependent vs query-independent centrality.
- LLM-as-annotator agreement measured against direct human entity judgements with Cohen's κ, precision and recall.
- Full annotation guidelines authored so that a second annotator can reproduce the labels.
- Retrieval evaluation with pytrec-eval over the standard Robust04 topic set.