Hello! I’m Nandan Thakur (नंदन ठाकुर / নন্দন ঠাকুর). I’m a
(third) fourth-year PhD student at University of Waterloo working on building efficient embedding models and realistic evaluation benchmarks. I’m lucky to be advised by Professor Jimmy Lin. My PhD is partially supported by the David R. Cheriton Graduate Scholarship [link].
I have interned at Google, Vectara and Databricks. I’ve also collaborated with industry partners including Snowflake, Micrsoft and Huawei. Previously, I worked at UKP Lab in TU Darmstadt advised by Professor Iryna Gurevych and Nils Reimers, and KNOLSKAPE. I received my undergraduate degree from BITS Pilani KK Birla Goa Campus in 2018.
Visit Research to learn more about my work. You can reach me at nandan.thakur@uwaterloo.ca. I’d love to hear from you!
Curriculum Vitae: CV
PS: I will be in the academic job market in 2026!
Research
My research is focused on three aspects: (i) Constructing challenging & realistic benchmarks, focusing on evaluation and benchmarking (ii) Building efficient retrieval systems, whose quality and cost can be optimized and generalize to challenging domains, and (iii) Standardizing RAG evaluation, building a better foundation within the IR & NLP community.
To answer these questions, my research develops new benchmarks such as BEIR or MIRACL to enable realistic evaluations, and constructs efficient models with GPL & SWIM-IR. This accelerates RAG systems to help craft language model answers with reduced hallucinations and improved accuracy seen across domains and languages.
2025 (Recent Updates)
-
[Jan 2025]
Gave a research talk on “Accelerating Multilingual RAG Systems” at Microsoft Research, Bangalore. [video].
-
[Jan 2025]
My work during my internship at Vectara on “MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems” is now accepted at NAACL 2025.
-
[Jan 2025]
Our contribution on including MIRACL in “MMTEB: Massive Multilingual Text Embedding Benchmark” is now accepted at ICLR 2025.
Papers
MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems
N Thakur, S Kazi, G Luo, J Lin, A Ahmad
NAACL 2025 (to appear) | paper | code
MMTEB: Massive Multilingual Text Embedding Benchmark
K Enevoldsen, I Chung, ..., N Thakur, ..., N Muennighoff
ICLR 2025 (to appear) | paper
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
R Pradeep*, N Thakur*, S Sharifymoghaddam, E Zhang, R Nguyen, D Campos, N Craswell, J Lin
ECIR 2025 (Findings) (to appear) | paper
Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
R Pradeep, N Thakur, S Upadhyay, D Campos, N Craswell, J Lin
Preprint 2024 | paper
A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
S Upadhyay, R Pradeep, N Thakur, D Campos, N Craswell, I Soboroff, H T Dang, J Lin
Preprint 2024 | paper
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
S Upadhyay, R Pradeep, N Thakur, N Craswell, J Lin
Preprint 2024 | paper
“Knowing When You Don’t Know”: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation
N Thakur, L Bonifacio, X Zhang, O Ogundepo, E Kamalloo, D A Hermelo, ..., M Rezagholizadeh, J Lin
EMNLP 2024 (Findings) | paper
Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR
N Thakur, L Bonifacio, M Fröbe, A Bondarenko, E Kamalloo, M Potthast, M Hagen, J Lin
SIGIR 2024 (Repro) | paper
Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses
E Kamalloo, N Thakur, C Lassance, X Ma, JH Yang, J Lin
SIGIR 2024 (Resource) | paper
Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval
N Thakur, J Ni, G H Abrego, J Wieting, J Lin, D Cer
NAACL 2024 | paper
HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution
E Kamalloo, A Jafari, X Zhang, N Thakur, J Lin
Preprint 2023 | paper
Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval
J Lin, D Alfonso-Hermelo, V Jeronymo, E Kamalloo, C Lassance, ..., N Thakur, JH Yang, X Zhang
Preprint 2023 | paper
MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages
X Zhang*, N Thakur*, O Ogundepo, E Kamalloo, D A Hermelo, ..., M Rezagholizadeh, J Lin
TACL 2023 | paper
Evaluating Embedding APIs for Information Retrieval
E Kamalloo, X Zhang, O Ogundepo, N Thakur, D A Hermelo, M Rezagholizadeh, J Lin
ACL 2023 (Industry) | paper
SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval
N Thakur, K Wang, I Gurevych, J Lin
SIGIR 2023 (Resource) | paper
Injecting Domain Adaptation with Learning-to-hash for Effective and Efficient Zero-shot Dense Retrieval
N Thakur, N Reimers, J Lin
ReNeuIR 2023 | paper
GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval
K Wang, N Thakur, N Reimers, I Gurevych
NAACL 2022 | paper
BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
N Thakur, N Reimers, A Rücklé, A Srivastava, I Gurevych
NeurIPS 2021 (D&B) | paper
Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks
N Thakur, N Reimers, J Daxenberger, I Gurevych
NAACL 2021 | paper
Old Updates
2024
-
[Dec 2024]
My work on “Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track” has been accepted at ECIR 2025 (Resource).
-
[Sep 2024]
I started my Fall 2024 internship at Databricks in San Francisco, mentored by Omar Khattab and managed by Sam Havens and Michael Carbin.
-
[Aug 2024]
We have received over 40+ participants in the first year of the TREC 2024 RAG Track. One of the best participated tracks up to date!
-
[May 2024]
I have been awarded the David R. Cheriton Graduate Scholarship starting Fall 2024 for my scholastic excellence in my PhD! [Link]
-
[May 2024]
Collaboration with Snowflake AI towards building better BEIRv2 and TREC-RAG [blogpost].
-
[Apr 2024]
I will be attending in-person NAACL 2024 in Mexico City, Mexico between 16-20 June 2024 and SIGIR in Washington DC, USA between 14-18 July 2024. If interested, do reach out!
-
[Apr 2024]
Received a 3K USD grant from Google to attend the NAACL 2024 Conference in Mexico City, 2024.
-
[Apr 2024]
My work on “Systematic Evaluation of Neural Retrieval Models on the Touch{'e}~2020 Argument Retrieval Subset of BEIR” has been accepted at SIGIR 2024 (Reproduction).
-
[Apr 2024]
My work on “Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses” has been accepted at SIGIR 2024 (Resource).
-
[Mar 2024]
My Google internship work on “SWIM-IR: Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval” has been accepted at NAACL 2024.
-
[Feb 2024]
Started part time research collaboration on improving multilingual RAG systems with Vectara.
-
[Jan 2024]
Gave two research talks on “Heterogeneous Benchmarking of Information Retrieval” in IIT-D (Delhi) and IIIT-Delhi [presentation] [video].
2023
-
[Nov 2023]
TREC RAG 2024 has been accepted and will be conducted as a shared task in TREC 2024.
-
[Nov 2023]
My internship work at Google is out on Arxiv, dataset is released here.
-
[Jul 2023]
I will be attending the SIGIR 2023 virtual conference being held in Taipei, Taiwan! Say hi to me (virtually)!
-
[Jul 2023]
I will be attending the ACL 2023 in-person conference being held in Toronto, Canada! Say hi to me!
-
[Jun 2023]
The Domain Adaptation Paper has been accepted in ReNeuIR 2023 Workshop to be held jointly with SIGIR 2023!
-
[Jun 2023]
The SPRINT Toolkit Paper has been accepted in SIGIR 2023 Resource Track!
-
[May 2023]
The MIRACL Paper has been accepted in TACL 2023!
-
[May 2023]
The Evaluating Embedding API Paper has been accepted in ACL 2023 Industry Track!
2022
-
[Sep 2022]
The MIRACL Challenge was accepted in WSDM Cup 2023. The Challenge is now live and looking for participants.
-
[Aug 2022]
I started my Fall Internship at the Language Team in Google Research with Daniel Cer and Jianmo Ni.
2021
2020
-
[Nov 2020]
[Cancelled (COVID-19)] Selected to speak at PyCon Italia 2020: “Extract or Replace Keywords in sentences 28x times faster than Regex - FlashText”. Abstract YouTube Github
-
[Jul 2020]
ArgumenText won 4th place amongst 3000+ startups in Nordbayerischen Businessplan. Link
-
[Jul 2020]
I attended the Association for Computational Linguistics (ACL) 2020 virtual conference.