HiRA-SB

A Silver Bullet for HiRA (PEFT)

Overview

Hadamard-based high-rank adaptation (HiRA ) has shown empirical advantages over low-rank methods like LoRA, but it remains unclear whether HiRA can match the effectiveness of full fine-tuning under tight parameter constraints. In this work, we take a step toward bridging this gap with HiRA Silver Bullet (HiRA-SB), a parameter-efficient fine-tuning scheme that aims to replicate the dynamics of full fine-tuning within the Hadamard-adapted space. HiRA-SB explores a constrained parametrization and initialization scheme designed to preserve high-rank gradient directions while stabilizing training. This work is an attempt to understand whether full fine-tuning can be effectively simulated within Hadamard subspaces, and what structural constraints are necessary for that goal.

---

Brief overview of Methods we use

HiRA-LoRA

HiRA adopts the hadamard product to modulate the low-rank LoRA style update. We thus have the following update \begin{equation} \label{eq:hira-update} W = W_0 + W_0 \odot (BA) \;\; \text{where $\odot$ is the hadamard product} \end{equation}

LoRA-SB

LoRA-SB (cite) attempts to inialize such that the intial gradietn matches the full FT gradeint – explain

Attempt

To compute the appropriate initialization, we need to compute the gradients of HiRA update.

Preliminaries (some math)

We have the following gradient

\begin{equation} \frac{\partial (W_{0,ij} \cdot f(X){ij})}{\partial X{pq}} = \begin{cases} 0 & \text{if (i \ne p) and (j \ne q)}
W_{0, pq}\cdot\frac{\partial f(X){ij}}{\partial X{pq}} & \text{if (i=p) and (j=q)} \end{cases} \end{equation}

Simply put we compute the gradient elementwise.

Every project has a beautiful feature showcase page. It’s easy to include images in a flexible 3-column grid format. Make your photos 1/3, 2/3, or full width.

To give your project a background in the portfolio page, just add the img tag to the front matter like so:

---
layout: page
title: project
description: a project with a background image
img: /assets/img/12.jpg
---

Caption photos easily. On the left, a road goes through a tunnel. Middle, leaves artistically fall in a hipster photoshoot. Right, in another hipster photoshoot, a lumberjack grasps a handful of pine needles.

This image can also have a caption. It's like magic.

You can also put regular text between your rows of images. Say you wanted to write a little bit about your project before you posted the rest of the images.

Does math work?

\[f(x) = \frac{x}{y}\]

\begin{equation} \label{eq:cauchy-schwarz} \left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right) \end{equation}

Let us see if i can cite Eqn. ($\eqref{eq:cauchy-schwarz}$)

You describe how you toiled, sweated, bled for your project, and then… you reveal its glory in the next row of images.

You can also have artistically styled 2/3 + 1/3 images, like these.

The code is simple. Just wrap your images with <div class="col-sm"> and place them inside <div class="row"> (read more about the Bootstrap Grid system). To make images responsive, add img-fluid class to each; for rounded corners and shadows use rounded and z-depth-1 classes. Here’s the code for the last row of images above:

<div class="row justify-content-sm-center">
    <div class="col-sm-8 mt-3 mt-md-0">
        {% include figure.html path="assets/img/6.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
    </div>
    <div class="col-sm-4 mt-3 mt-md-0">
        {% include figure.html path="assets/img/11.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
    </div>
</div>