LoRA: a Low-Rank Update

Fine tuning changes a frozen weight matrix W zero by an update, delta W. Here the update is eight by eight: each dot above the floor is one of its entries.

Low rank adaptation writes the update as alpha over r, times B times A. B is tall and thin, with r columns, A is short and wide, with r rows, so the update has rank at most r.

Only B and A are trained: r times m plus n numbers, instead of m times n. For a projection of size four thousand ninety six with r equal to eight, that is two hundred fifty six times fewer.

Which rank r update comes closest to the one we want? By the Eckart Young theorem, the truncated SVD. The bars are the best rank r update, and the red gaps are what it misses. LoRA learns B and A by gradient descent instead of computing that SVD.

Your turn. Change the rank r. Find the smallest rank that captures ninety five percent of the update's energy.

When the update has low intrinsic rank, a thin B and a short A carry almost all of it, at a fraction of the cost.