Rohit Vartak

Masdar City, Abu Dhabi

United Arab Emirates

I am a Research Assistant at MBZUAI (May 2026 –), where I am advised by Prof. Praneeth Vepakomma.

My current research broadly focuses on misalignment (sabotage) in LLMs and scalable oversight.

I am interested in the following directions:

Loss of Control in Multi-Agent Settings, What new threat vectors do multi-agent settings expose, and can current monitoring techniques keep up?
Social Dynamics in Agent Swarms, How do agent swarms organize themselves, what threats arise, and how do these systems compare to current paradigms (individual models or small swarms)?
Chain-of-Thought (CoT) Controllability & Obfuscation in LLMs, To what extent can LLMs control and conceal their CoT reasoning?

My previous work spans several areas, including:

Parameter-efficient fine-tuning, improving expressivity and adaptability of large models
Federated learning for LLMs, with an emphasis on communication-efficient training
(Multimodal) reasoning, studying how models process and reason over visual and textual inputs
LLM applications in healthcare, focusing on real-world user interaction with medical chatbots

Prior to this, I completed my M.S. in Computer Science at Duke University, where I worked with Prof. Bhuwan Dhingra on the mathematical reasoning capabilities of multimodal LLMs.

During this time, I also interned with Prof. Praneeth Vepakomma at MBZUAI (Summer 2025), working on efficient and scalable training methods for large models, and collaborated with Prof. Monica Agrawal on medical chatbots and their real-world usability.

Before this, I completed my B.Tech in Electrical Engineering and M.Tech in Artificial Intelligence and Data Science at IIT Bombay.

news

Apr 08, 2026	Successfully defended my M.S. degree (course-only track) at Duke University
Mar 04, 2026	Our paper “Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning” has been accepted at TMLR 2026.
Jan 20, 2026	Our paper “ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models” has been accepted at ICLR 2026.

selected publications

ICLR
ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models

Raghav Singhal^*, Kaustubh Ponkshe^*, Rohit Vartak^*, and 1 more author

ICLR 2026, Spotlight at ES-FOMO@ICML 2025 , 2025

Abs Bib PDF Code

Large Language Models have demonstrated strong performance across a wide range of tasks, but adapting them efficiently to new domains remains a key challenge. Parameter-Efficient Fine-Tuning (PEFT) methods address this by introducing lightweight, trainable modules while keeping most pre-trained weights fixed. The prevailing approach, LoRA, models updates using a low-rank decomposition, but its expressivity is inherently constrained by the rank. Recent methods like HiRA aim to increase expressivity by incorporating a Hadamard product with the frozen weights, but still rely on the structure of the pre-trained model. We introduce ABBA, a new PEFT architecture that reparameterizes the update as a Hadamard product of two independently learnable low-rank matrices. In contrast to prior work, ABBA fully decouples the update from the pre-trained weights, enabling both components to be optimized freely. This leads to significantly higher expressivity under the same parameter budget, a property we validate through matrix reconstruction experiments. Empirically, ABBA achieves state-of-the-art results on arithmetic and commonsense reasoning benchmarks, consistently outperforming existing PEFT methods by a significant margin across multiple models. Our code is publicly available at: [here](https://github.com/CERT-Lab/abba).
@article{singhal2025abba, title = {ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models}, author = {Singhal, Raghav and Ponkshe, Kaustubh and Vartak, Rohit and Vepakomma, Praneeth}, journal = {ICLR 2026}, year = {2025}, archiveprefix = {arXiv}, primaryclass = {cs.CL}, url = {https://arxiv.org/abs/2505.14238}, }
TMLR
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning

Raghav Singhal^*, Kaustubh Ponkshe^*, Rohit Vartak, and 2 more authors

TMLR, J2C Certification (Top 10% of accepted papers) , 2025

Abs Bib PDF Code

Low-Rank Adaptation (LoRA) has become ubiquitous for efficiently fine-tuning foundation models. However, federated fine-tuning using LoRA is challenging due to suboptimal updates arising from traditional federated averaging of individual adapters. Existing solutions either incur prohibitively high communication cost that scales linearly with the number of clients or suffer from performance degradation due to limited expressivity. We introduce Federated Silver Bullet (Fed-SB), a novel approach for federated fine-tuning of LLMs using LoRA-SB, a recently proposed low-rank adaptation method. LoRA-SB optimally aligns the optimization trajectory with the ideal low-rank full fine-tuning projection by learning a small square matrix (R) between adapters B and A, keeping other components fixed. Direct averaging of R guarantees exact updates, substantially reducing communication cost, which remains independent of the number of clients, and enables scalability. Fed-SB achieves state-of-the-art performance across commonsense reasoning, arithmetic reasoning, and language inference tasks while reducing communication costs by up to 230x. In private settings, Fed-SB further improves performance by (1) reducing trainable parameters, thereby lowering the noise required for differential privacy and (2) avoiding noise amplification introduced by other methods. Overall, Fed-SB offers a state-of-the-art, efficient, and scalable solution for both private and non-private federated fine-tuning. Our code is publicly available at: [here](https://github.com/CERT-Lab/fed-sb)
@article{singhal2025fedsb, title = {Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning}, author = {Singhal, Raghav and Ponkshe, Kaustubh and Vartak, Rohit and Varshney, Lav R. and Vepakomma, Praneeth}, journal = {TMLR}, year = {2025}, archiveprefix = {arXiv}, primaryclass = {cs.LG}, url = {https://arxiv.org/abs/2502.15436}, }