Hello! I’m Nandan Thakur (नंदन ठाकुर / নন্দন ঠাকুর). I’m a (third) fourth-year PhD student at University of Waterloo working on building efficient embedding models and realistic evaluation benchmarks. I’m lucky to be advised by Professor Jimmy Lin. My PhD is partially supported by the David R. Cheriton Graduate Scholarship [link].

I have interned at Google, Vectara and Databricks. I’ve also collaborated with industry partners including Snowflake, Micrsoft and Huawei. Previously, I worked at UKP Lab in TU Darmstadt advised by Professor Iryna Gurevych and Nils Reimers, and KNOLSKAPE. I received my undergraduate degree from BITS Pilani KK Birla Goa Campus in 2018.

Visit Research to learn more about my work. You can reach me at nandan.thakur@uwaterloo.ca. I’d love to hear from you!

Curriculum Vitae: CV

Research

My research is focused on three aspects: (i) Constructing challenging & realistic benchmarks, focusing on evaluation and benchmarking (ii) Building efficient retrieval systems, whose quality and cost can be optimized and generalize to challenging domains, and (iii) Standardizing RAG evaluation, building a better foundation within the IR & NLP community.

To answer these questions, my research develops new benchmarks such as BEIR or MIRACL to enable realistic evaluations, and constructs efficient models with GPL & SWIM-IR. This accelerates RAG systems to help craft language model answers with reduced hallucinations and improved accuracy seen across domains and languages.

2025 (Recent Updates)

• [Apr 2025] My exciting internship work at Databricks on FreshStack to create realistic RAG benchmarks is available on arXiv!
• [Apr 2025] Our TREC 2024 RAG support (to be arxived soon) and Nuggets preprints have been accepted at SIGIR 2025!
• [Jan 2025] Gave a research talk on “Accelerating Multilingual RAG Systems” at Microsoft Research, Bangalore. [video].
• [Jan 2025] My work during my internship at Vectara on “MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems” is now accepted at NAACL 2025.
• [Jan 2025] Our contribution on including MIRACL in “MMTEB: Massive Multilingual Text Embedding Benchmark” is now accepted at ICLR 2025.

Papers

FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
N Thakur, J Lin, S Havens, M Carbin, O Khattab, A Drozdov
Preprint 2025 | paper | website

Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
R Pradeep, N Thakur, S Upadhyay, D Campos, N Craswell, J Lin
SIGIR 2025 (to appear) | paper

Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
R Pradeep*, N Thakur*, S Sharifymoghaddam, E Zhang, R Nguyen, D Campos, N Craswell, J Lin
ECIR 2025 (Findings) | paper

MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems
N Thakur, S Kazi, G Luo, J Lin, A Ahmad
NAACL 2025 | paper | code | website

MMTEB: Massive Multilingual Text Embedding Benchmark
K Enevoldsen, I Chung, ..., N Thakur, ..., N Muennighoff
ICLR 2025 | paper | website

A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
S Upadhyay, R Pradeep, N Thakur, D Campos, N Craswell, I Soboroff, H T Dang, J Lin
Preprint 2024 | paper

UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
S Upadhyay, R Pradeep, N Thakur, N Craswell, J Lin
Preprint 2024 | paper | code

“Knowing When You Don’t Know”: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation
N Thakur, L Bonifacio, X Zhang, O Ogundepo, E Kamalloo, D A Hermelo, ..., M Rezagholizadeh, J Lin
EMNLP 2024 (Findings) | paper | code

Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR
N Thakur, L Bonifacio, M Fröbe, A Bondarenko, E Kamalloo, M Potthast, M Hagen, J Lin
SIGIR 2024 (Repro) | paper | code

Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses
E Kamalloo, N Thakur, C Lassance, X Ma, JH Yang, J Lin
SIGIR 2024 (Resource) | paper

Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval
N Thakur, J Ni, G H Abrego, J Wieting, J Lin, D Cer
NAACL 2024 | paper | code

HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution
E Kamalloo, A Jafari, X Zhang, N Thakur, J Lin
Preprint 2023 | paper

Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval
J Lin, D Alfonso-Hermelo, V Jeronymo, E Kamalloo, C Lassance, ..., N Thakur, JH Yang, X Zhang
Preprint 2023 | paper

MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages
X Zhang*, N Thakur*, O Ogundepo, E Kamalloo, D A Hermelo, ..., M Rezagholizadeh, J Lin
TACL 2023 | paper | website

Evaluating Embedding APIs for Information Retrieval
E Kamalloo, X Zhang, O Ogundepo, N Thakur, D A Hermelo, M Rezagholizadeh, J Lin
ACL 2023 (Industry) | paper

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval
N Thakur, K Wang, I Gurevych, J Lin
SIGIR 2023 (Resource) | paper | code

Injecting Domain Adaptation with Learning-to-hash for Effective and Efficient Zero-shot Dense Retrieval
N Thakur, N Reimers, J Lin
ReNeuIR 2023 | paper | code

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval
K Wang, N Thakur, N Reimers, I Gurevych
NAACL 2022 | paper | code

BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
N Thakur, N Reimers, A Rücklé, A Srivastava, I Gurevych
NeurIPS 2021 (D&B) | paper | website | code

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks
N Thakur, N Reimers, J Daxenberger, I Gurevych
NAACL 2021 | paper | website

Old Updates