Low-Rank
Adaptation

Teach a frozen model a new task by learning a small, low-rank change to its weights.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
Microsoft. ICLR 2022. Results on images from Zhang, Zhou & Liu 2022 and Zhu et al. 2024.

Presented by Ertugrul Senturk
October 2026

From VPT to LoRA

VPTlast time

…

→

frozen Vision Transformer

→

head

Learned vectors join the image patches, so every image carries a longer sequence through every layer.

LoRAtoday

…

→

frozen Vision Transformersmall trained change to its weights

→

head

The input stays as it is. A small change to the weights is learned, and after training it is added into them.

VPT trained 0.6% of the weights and beat full fine-tuning on 20 of 24 tasks. But its prompts make every image slower, and they can never be folded into the model.

LoRA leaves the input alone and changes the weights instead. It ends up storing about as little per task as VPT, while the model runs exactly as fast as before.

VPT: Jia et al., ECCV 2022 (0.6% of the weights with 50 prompts per layer and a 102-class head; 20 of 24 tasks, Table 1). LoRA: Hu et al., ICLR 2022, first posted in June 2021.

Every fine-tuned copy is as big as the model

The paper starts from the largest model of its time, GPT-3, with 175 billion weights. Each fine-tuned copy takes 350 GB, so a service with a hundred tasks would store 35 TB. LoRA keeps one shared model and only a small file per task, and the frozen weights also need no optimizer state, which cuts training memory from 1.2 TB to 350 GB.

Number of tasks1

Model

343 MB

345 MB

GPT-3 in 16-bit numbers: 350 GB per copy, 35 MB per LoRA file at rank 4 on the query and value weights (Hu et al. 2022, Sec. 4.2). ViT-B/16 in 32-bit: 343 MB per copy; its LoRA file is rank 8 on the same weights plus a 102-class head. Squares are sized by area.

Earlier cheap methods cost time or sequence length

Adapters insert small extra layers into every block. They have few weights, but an image or a sentence must pass through them one after another, so serving a single input takes 20 to 30% longer. With large batches the gap shrinks to 2 or 3%.

Prompt methods such as prefix tuning and VPT lengthen the input instead. That leaves less room for the real input and costs compute in every layer, and on GPT-3 prefix tuning got worse beyond 256 prompt tokens.

LoRA adds no layer and no token at test time.

Hu et al. 2022, Table 1: one forward pass of GPT-2 medium, mean of 100 trials on one NVIDIA Quadro RTX 8000; bars are drawn relative to the fine-tuned model in each setting. Prefix tuning: Sec. 5.5 and Table 15.

What a low-rank matrix is

The photo of a cat rebuilt from one piece

A grayscale photo is a matrix of 274 × 274 brightness values. Any matrix can be written as a sum of simple pieces, where each piece is one column of numbers times one row. The rank is the number of pieces.

LoRA bets that the change fine-tuning makes to a weight matrix is like this photo: a few pieces carry almost all of it.

Not from the paper: the best rank-r approximation (SVD) of an Oxford Pets photo, Jia et al. 2022, Fig. 10. The low-rank bet follows Li et al. 2018 and Aghajanyan et al. 2021.

One minute of linear algebra, because the whole paper rests on the word rank. Treat the photo as a matrix of brightness values. A rank-one matrix is a column times a row: every row of the image is the same pattern, just scaled, so you get bands. Step through with the arrow keys. Each step doubles the number of pieces. The pieces are ordered by how much they contribute, so the first few capture the big shapes and the later ones only fine detail. At rank 16 the cat is recognisable, from 12% of the numbers; at 64 it is hard to tell from the original, from 47%. Storing r pieces costs r times (rows plus columns) numbers instead of rows times columns. The drawing under the text shows the same thing as matrices, to scale: the r columns side by side are a tall matrix B, the r rows stacked are a wide matrix A, and B times A rebuilds the photo; at rank 1 they are a single line each, at all 274 they are as big as the photo itself. That is the trade LoRA makes for the weight change, with the same names B and A from now on. Why expect the change to be low-rank? Two earlier papers measured the "intrinsic dimension" of fine-tuning: you can freeze the model, train only a few hundred or thousand numbers, map them through a fixed random projection onto all the weights, and still get most of the accuracy. LoRA's assumption is the matrix version of that: each weight change has a low rank, so it can be written as B times A with a small r, chosen by hand.

LoRA adds a trainable bypass to a frozen weight

Paper figure: the input x goes through the pretrained weight W and, in parallel, through A and then B; the two outputs are added to give h. B starts at zero, A starts random.

Figure: Hu et al. 2022, Fig. 1, where orange is trained. Equation (3) and Sec. 4.1. The paper sets α to the first r it tries and does not tune it.

The paper's own figure; orange there is what we call red, the trained part. During training: x goes through the frozen W0 as before, and in parallel through A, which squeezes it down to r numbers, and B, which expands it back. The outputs are added. Two details matter. B starts at zero, so at step one the model is exactly the pretrained model and training starts from a known good point. And the bypass is scaled by alpha over r, so that changing the rank does not mean re-tuning the learning rate. Press the arrow key: after training, B times A is just a d × d matrix, so we add it into W0 once and throw A and B away. The model is again a single matrix per layer, same size, same speed. To switch task you subtract this task's BA and add another task's, a cheap operation. That is the property VPT did not have.

LoRA on a 3 × 3 weight matrix

Made-up numbers with rank r = 1 and α/r = 1, so every product can be checked by eye. In ViT-B the same happens with 768 × 768 weights and r = 8.

Made-up numbers, rank 1, so every product can be checked by eye. First the frozen path: the pretrained layer computes W times x, which gives 5, 7 and 8. Next step: LoRA adds a second path. A is one row of three numbers, so A times x is a single number, 1 + 2 + 0 = 3; the input is squeezed down to r numbers. B is a column of three, so B times 3 gives 3, 0 and 6, back to full size. The two paths are added: 8, 7, 14. During training only these six red numbers would change. Next step: after training we multiply B by A once. A column times a row is a full 3 × 3 matrix, the same shape as W, so we can add it to W; the red cells are the ones that changed. Last step: the merged W′ times x gives 8, 7, 14 again, exactly the training-time output, because W x + B(A x) equals (W + BA) x. So A and B are deleted and the layer is one ordinary matrix. The point people often miss: B and A are never multiplied with W. They are multiplied with each other, and the product is added to W.

Where LoRA goes in a Vision Transformer

One of the 12 blocks of ViT-B/16. Placement as in Hu et al. 2022, Sec. 4.2, and Houlsby et al. 2019. Zhang et al. 2022, Fig. 1a, draws LoRA, the adapter and VPT in one block.

Where exactly the pairs go. This is one of the twelve blocks of a ViT, frozen in blue. The attention part has four weight matrices: query, key, value and output; the MLP has two. The paper only adapts attention weights, by default the query and value, and leaves everything else frozen, including the MLP. The red pairs sit beside Wq and Wv in every block, 24 pairs in ViT-B. Press the arrow key for an adapter, for contrast: a small trained two-layer network after the MLP, an extra layer every input has to go through, which is where the latency from the earlier chart comes from. LoRA instead sits beside weights that already exist, so after merging it disappears. VPT, the third option, changes nothing inside the block; the next slide compares it with LoRA step by step. NOAH, the vision paper we will see, searches over all three at once.

VPT and LoRA side by side

One layer of ViT-B/16 drawn for each. VPT-deep with 50 prompts per layer (Jia et al. 2022); LoRA with rank 8 on the query and value weights (Hu et al. 2022). Both train the same 102-class head.

This is the slide that puts the two methods next to each other, in three steps. What is learned: VPT learns extra input vectors, 50 per layer, and the same ones are used for every image of the task; they start random, they are not cut from any image. LoRA learns a correction to the weights, the pair B and A beside the query and value weights; it is also one pair for the whole task, not per image, and B starts at zero. Next step, training: in both cases every training image goes through the frozen model, the loss is computed, and gradients flow back. In VPT they end in the prompts and the head; in LoRA in B, A and the head. In both, the pretrained weights never change. Next step, test time, which is where the two differ. VPT's prompts are inputs, so they have to be attached to every test image, and every layer processes 247 tokens instead of 197, 27% more compute. LoRA's correction is a matrix of the same shape as W, so it is added into W once, and the layer is an ordinary layer again, exactly the original cost. The next slide puts numbers on both differences: what each stores per task, and what each image costs.

How much LoRA trains and stores per task

2 · N · m · d · r = 2 × 12 × 2 × 768 × 8 = 294,912numbers in the pairs, plus 78,438 for a 102-class head

Numbers trained and stored per task

Full fine-tuning

all 85.8 million weights, retrained for this task

343 MB100%

VPT-deep, 50 prompts

frozen and shared by every task

2.2 MB0.63%

LoRA

frozen and shared by every task

1.5 MB0.44%

Compute per image at test time

Plain ViT

197 tokens

1.00×

VPT-deep, 50 prompts

197 tokens

1.27×+50 tokens

LoRA, merged

197 tokens

1.00×nothing extra

Rank r

Weights

ViT-B/16: N = 12 blocks, d = 768, m adapted weights per block, 32-bit numbers. VPT-deep as last time. Compute counts multiply-adds per layer for n tokens (12nd² + 2n²d).

The numbers behind the previous slide, with the same accounting as last time. Each pair has 2 × 768 × r numbers; with query and value in 12 blocks and rank 8 that is 294,912, plus the same 102-class head as before: 0.44% of the model, a 1.5 MB file. VPT-deep with 50 prompts was 0.63% and 2.2 MB, so storage is about the same. The difference is the second chart: VPT made every image 27% more expensive, LoRA after merging costs exactly what the plain ViT costs. Try rank 64 on all four attention weights: the file grows to 19 MB, 5.6% of the model, but the compute bar does not move. If someone asks about training compute: the pairs still have to be computed during training, and the backward pass still goes through every layer, so training is not much cheaper per image than full fine-tuning; what drops is optimizer memory.

LoRA's saving grows with the width of the model

Full change ΔW

d × d = d²

589,824 numbers on ViT-B

B and A together

d × r + r × d = 2rd

12,288 numbers on ViT-B

A full change to one d × d weight has d² numbers, while B and A together have 2rd. Since d is the square root of d², LoRA's cost grows like the square root of the full size, times the small constant 2r.

So the wider the model, the smaller LoRA's share. Together with adapting only the query and value weights, this is why GPT-3's task file is 10,000 times smaller than a copy.

One d × d attention weight, rank r = 8 for every model. Widths: ViT-B/16 768, ViT-L/16 1,024, ViT-H/14 1,280 (Dosovitskiy et al. 2021); GPT-3 12,288 (Hu et al. 2022, Sec. 1). 10,000 times: Hu et al. 2022, Sec. 4.2.

LoRA on a ViT in a few lines

vit = timm.create_model(BACKBONE, pretrained=True, num_classes=C)vit.requires_grad_(False); vit.head.requires_grad_(True) class LoRAqv(nn.Module):              # wraps qkv = Linear(768, 2304)    def __init__(self, qkv, r=8, alpha=8):        super().__init__()        d = qkv.in_features        self.qkv, self.s = qkv, alpha / r    # frozen W0, scale α/r        self.Aq = nn.Parameter(torch.randn(r, d) / d ** 0.5)        self.Av = nn.Parameter(torch.randn(r, d) / d ** 0.5)        self.Bq = nn.Parameter(torch.zeros(d, r))   # B = 0        self.Bv = nn.Parameter(torch.zeros(d, r))    def forward(self, x):        q, k, v = self.qkv(x).chunk(3, dim=-1)      # W0 x        q = q + self.s * x @ self.Aq.T @ self.Bq.T  # + B A x        v = v + self.s * x @ self.Av.T @ self.Bv.T        return torch.cat([q, k, v], dim=-1) for blk in vit.blocks: blk.attn.qkv = LoRAqv(blk.attn.qkv)

B starts at zero and A at small random values, so the first forward pass is exactly the pretrained model.

The two new choices are the rank r and which weights get a pair. On ViT-B, Zhang et al. use rank 8 with AdamW at a learning rate of 0.001 for 100 epochs.

After training, each pair is added into its qkv weight once, and the model is a plain timm ViT again.

My sketch for timm's ViT-B/16: 294,912 trained numbers in the pairs, and identical outputs before and after merging. Official code: loralib, github.com/microsoft/LoRA. Settings: Hu et al. 2022, Sec. 4.1; Zhang et al. 2022, App. A.2.

Results in the paper: language models

What happens when more weights are trained

Hu et al. 2022, Table 15 (the points of Fig. 2): GPT-3 175B, validation accuracy; LoRA with equal rank on query and value, from rank 1 to 64. Full fine-tuning trains all 175 billion weights.

Which weight matrices to adapt

Hu et al. 2022, Table 5: GPT-3 175B with the same 18 million trained weights in every LoRA row, validation accuracy (noise ±0.5 on WikiSQL, ±0.1 on MNLI). Full fine-tuning: Table 4.

How small the rank can be

Hu et al. 2022, Table 6: GPT-3 175B, validation accuracy at five ranks; noise ±0.5 on WikiSQL and ±0.1 on MNLI. On GPT-2 the best rank was between 4 and 16 (App. H.2).

Why a small rank is enough

Paper figure: heat maps of subspace similarity between the directions learned with rank 8 (rows i) and rank 64 (columns j), for the query and value changes. Only the first row is bright.

The authors trained rank 8 and rank 64 separately and compared the directions each learned. Bright means the two runs found the same direction. Only the first direction is clearly shared, with similarity above 0.5, and the rest overlap little, so most of the extra rank holds noise.

21.67

is how strongly the pretrained weight W_q acts along its own top 4 directions.

0.32

is how strongly it acts along the 4 directions of the learned change ΔW_q: weak, but more than the 0.02 of random directions.

6.91

is the strength of the change along those directions, about 21 times what the pretrained weight had there.

The change does not repeat what the model already does. It strongly amplifies directions the model learned in pretraining but used weakly.

Hu et al. 2022, Fig. 3 (layer 48 of 96, GPT-3; values are normalized subspace similarity, 0 to 1) and Table 7 (rank 4, Frobenius norms).

Section 7 asks why this works. First analysis: train rank 8 and rank 64 on the same task, take the directions each one learned, and measure how much the top i directions of one lie inside the top j of the other. In the heat maps rows are rank-8 directions, columns rank-64 directions; colour is overlap from 0 to 1. Only the first row lights up: both runs agree on one direction and disagree on the rest, so the useful part of the change is about one-dimensional and the rest is noise that training accumulates. That explains rank 1 working. Press the arrow key for the second analysis, Table 7. Project the pretrained weight onto the four directions of the learned change: 0.32, small, though larger than for random directions, 0.02. Its own top directions are 21.67, so the change is not about what the model already does strongly. The change itself has strength 6.91 in those directions, an amplification of about 21. The authors' reading: LoRA amplifies features that pretraining learned but did not emphasise, and the task needs. This is measured on GPT-3; nobody in these papers repeats it for ViT.

LoRA on images

The paper itself has no image experiments. Vision groups tested it next, first on the same 19 VTAB-1k tasks as VPT, then on medical images.

Both use a ViT-B/16, so the numbers sit next to last time's.

Rendered 3D objects to count, from CLEVR

LoRA, VPT and adapters on the same ViT

Zhang et al. 2022, Table 1: ViT-B/16 pretrained on ImageNet-21k, VTAB-1k test accuracy. LoRA (rank 8) and the adapter were trained by Zhang et al.; the other rows are Jia et al. 2022, Table 1. Group means computed from the per-task values; "all 19" is the mean of the three groups.

Same backbone and benchmark as the VPT talk: 1,000 training images per task, three groups. The adapter here is NOAH's re-run with the best recipe, which is much stronger than the adapter in the VPT paper's table. Natural: all three cheap methods are two to four points above full fine-tuning, LoRA narrowly best, 79.5. Specialized: LoRA 84.6 and the adapter 84.1 are slightly above full fine-tuning at 83.4; VPT-deep is a point below. Structured: the big gap again, all three beat full fine-tuning on all eight tasks, LoRA the most, 59.8 against 47.6. All 19: LoRA 74.6, adapter 73.9, VPT-deep 72.0, full 69.0. VPT-deep beats full fine-tuning on one more task, 16 against 15, but by smaller margins. Two caveats if asked: NOAH's table prints 87.2 for full fine-tuning on Flowers where the VPT paper has 97.2, so I used the VPT paper's numbers for those rows; and NOAH prints 74.5 for LoRA's average, while its own per-task numbers give 74.6.

Neither LoRA nor VPT wins every task

LoRA is ahead on 15 of the 19 tasks, by up to 14 points on counting objects in CLEVR. VPT-deep is ahead on 4, and on CIFAR-100 by 12 points.

Zhang et al. conclude that no single method fits every task. Their method, NOAH, searches for a mix of LoRA, adapter and prompts per task, and reaches 75.5 on average.

Zhang et al. 2022, Table 1: per-task test accuracy of LoRA minus VPT-deep on VTAB-1k, ViT-B/16. Right of the line, LoRA is ahead.

LoRA on medical images

MeLo, ISBI 2024, from ShanghaiTech and Shanghai Jiao Tong: LoRA with rank 4 on the query and value weights of a ViT-B pretrained on ImageNet, applied to four medical tasks: tuberculosis on chest X-rays, four blood cell types, malignancy in mammograms, and 14 thoracic diseases. They train 0.14 million weights, 0.17%. AUC first: within a point of full fine-tuning on three tasks, 3.6 points lower on blood cells. Accuracy, second step: ahead on tuberculosis and blood cells, slightly behind on the other two. The title says "better than fine-tuning", but these are single runs on small test sets, about 80 mammograms for INbreast, so I would read it as "about the same". Third step is their real point, deployment, with a giant CLIP ViT: 25 images from each task in random order. Reloading the right fine-tuned copy for each image takes 218 s for 100 images; keeping all four copies loaded is fast but needs 7 billion weights in GPU memory; one model with four LoRA pairs takes 10.6 s at the memory of a single model.

Where LoRA falls short

The rank and the weights are chosen by hand

The paper picks the query and value weights and a small rank by trying settings, and leaves a principled choice to future work. On VTAB-1k the best method even changes from task to task.

Merged weights serve one task at a time

Once BA is added into W, every input in a batch must belong to that task. Mixing tasks in one batch means keeping the pairs separate, which brings back a small extra cost; VPT's prompts mix freely.

A small rank assumes a nearby task

The authors expect a small rank to fail when the new task is far from pretraining, such as a different language. The low-rank analysis was only done on GPT-3, not on vision models.

Training still runs through the whole model

LoRA saves the optimizer state of the frozen weights, but the gradient still passes back through every layer. On ViT-B most memory is activations: freezing the backbone for VPT only cut it from 11.7 to 10.3 GB.

Hu et al. 2022, Sec. 4.2 (batching), footnote 6 (a different language), Sec. 7 and 8 (analysis on GPT-3, future work). Zhang et al. 2022, Fig. 1b. Jia et al. 2022, Table 12 (ViT-B/16, batch 64).

Four honest limits. First, choices by hand: which matrices, which rank. The paper's guidance comes from GPT-3, and NOAH's per-task spread shows the best choice varies. Second, a deployment detail people miss: merging is what makes LoRA free at test time, but a merged model serves one task. If a server mixes requests of different tasks in one batch, you have to keep the pairs unmerged and pay a small extra computation, while VPT's prompts are just inputs and can differ per sample. Third, the low-rank assumption: the authors themselves say a very different task, like a new language, may need full rank; the evidence for low rank is from GPT-3 only. Fourth, training memory: the 3× saving on GPT-3 came from optimizer state, which dominates for huge models. On ViT-B, activations dominate, and those are still stored for the whole network, as with VPT. My own measurement of this will be part of the experiments next.

My experiments

I trained LoRA on the same three medical datasets as my VPT runs, with the same base model, data and training recipe.

So it sits next to VPT-deep, a linear probe and full fine-tuning. First the results, then the models running live.

I tested LoRA on the same three medical datasets

Each method was trained twice: on 1,000 images, as in VTAB-1k, and on the full training set.

Dermoscopy image of a melanoma — melanoma

Skin lesions

Dermoscopy, 7 classes

1,000 or all 7,007 training images

Microscopy image of a neutrophil — neutrophil

Microscopy image of a platelet — platelet

Blood cells

Microscopy, 8 cell types

1,000 or all 11,959 training images

Chest X-rays

Radiographs, pneumonia or normal

1,000 or all 4,708 training images

head

ViT-B/16 frozen

Linear probe

Trains only a new head.

head

ViT-B/16 frozen

VPT-deep

Trains a head and 50 prompts per layer.

head

ViT-B/16 frozen

+

LoRA

Trains a head and rank-8 pairs beside W_q and W_v.

head

ViT-B/16 retrained

Full fine-tuning

Retrains all 85.8 million weights.

MedMNIST+ at 224 × 224 (Yang et al. 2023); skin lesions from HAM10000 (Tschandl et al. 2018). RTX 5070 Ti laptop GPU. SGD for linear probe and VPT, AdamW for LoRA and full fine-tuning; learning rate picked on validation.

How LoRA compares on my three datasets

My runs: ViT-B/16 (ImageNet-21k), MedMNIST+ 224 × 224, balanced accuracy on the test set (the mean per-class recall), bars start at 0. Each method was trained once with 1,000 images and once with the full training set; there are no points in between.

What each method costs on skin lesions

Three steps, one cost each, all measured on my laptop for skin lesions trained on the full training split; every number is read from the training logs and the benchmark file. Storage: both LoRA and VPT keep one copy of the base model and add a small file per task, while full fine-tuning needs a whole 343 MB model per task. Training: LoRA and VPT both still send the error signal back through all twelve frozen layers, so neither is cheap like the linear probe; the memory saving compared with full fine-tuning is the optimizer state of the frozen weights. Inference: this is where the two differ. VPT makes every layer process 50 extra tokens. LoRA with its pairs kept separate adds a little extra work, and once merged into the weights it runs exactly as fast as the plain model, which is the paper's main practical claim.

Live demo: one image, three models

Pick a test image below,
or drop your own here

Linear probe

shared frozen model + head

LoRA

shared frozen model + pairs

Full fine-tuning

its own retrained model

All three start from the same ViT-B/16, pretrained on ImageNet-21k. Linear probe and LoRA share one frozen copy of it; full fine-tuning runs its own retrained copy.

Not connectedResearch demo, not a diagnostic tool

The same demo as last time, with LoRA in the middle column. Every click runs the image through all three models right here in the browser, with onnxruntime-web. Linear probe and LoRA use the very same frozen copy of the base model in memory: for LoRA the rank-8 pairs of the chosen task are fed in beside the query and value weights of every block; for the linear probe the pairs are zero. That is the unmerged form, so one copy serves every task. Merged, LoRA would run at exactly the speed of the plain model, but then it would be one model per task again. Full fine-tuning retrained every weight, so it is a separate model per task; it is the same model as in the VPT demo. The percentage under each name is that model's balanced accuracy on the whole test set. Now switch to 1,000 training images: for linear probe and LoRA only a small head and a 1.2 MB file change, so it is instant. Full fine-tuning has to download and load a second whole model, 172 MB in 16-bit. The thumbnails are the same random test images as last time, not picked to look good, so a wrong answer can happen; say so. Research demo, not a diagnostic tool.

Summary

0.44%

of ViT-B's weights are trained with rank 8 on the query and value weights and a head: a 1.5 MB file per task.

1.00×

the test-time cost of the original model, because the learned change is added into the weights after training.

74.6

mean VTAB-1k accuracy on ViT-B/16, against 72.0 for VPT-deep and 69.0 for full fine-tuning.

r = 1

was already enough on GPT-3 for the query and value weights: the change fine-tuning needs is low-rank.

–

average balanced accuracy of LoRA on my three medical datasets.

ViT-B/16 with a 102-class head, 32-bit numbers. Zhang et al. 2022, Table 1, mean of the three VTAB-1k groups; VPT-deep and full fine-tuning from Jia et al. 2022. Hu et al. 2022, Table 6. Medical datasets: my runs, balanced accuracy on the test set.

References

This paper

Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang & Chen 2022. LoRA: low-rank adaptation of large language models. ICLR. arXiv:2106.09685.

LoRA on images

Zhang, Zhou & Liu 2022. Neural prompt search (NOAH). arXiv:2206.04673.
Zhu et al. 2024. MeLo: low-rank adaptation is better than fine-tuning for medical image diagnosis. ISBI. arXiv:2311.08236.

Ideas LoRA builds on

Li, Farkhoor, Liu & Yosinski 2018. Measuring the intrinsic dimension of objective landscapes. ICLR.
Aghajanyan, Zettlemoyer & Gupta 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. ACL.

Methods compared

Houlsby et al. 2019. Parameter-efficient transfer learning for NLP (adapter). ICML.
Lin, Madotto & Fung 2020. Exploring versatile generative language model via parameter-efficient transfer learning. Findings of EMNLP.
Rücklé et al. 2021. AdapterDrop: on the efficiency of adapters in Transformers. EMNLP.
Li & Liang 2021. Prefix-tuning: optimizing continuous prompts for generation. ACL.
Ben Zaken, Goldberg & Ravfogel 2022. BitFit. ACL.
Jia et al. 2022. Visual prompt tuning. ECCV.

Models

Dosovitskiy et al. 2021. An image is worth 16×16 words: Transformers for image recognition at scale (ViT). ICLR.
Brown et al. 2020. Language models are few-shot learners (GPT-3). NeurIPS.
Radford et al. 2019. Language models are unsupervised multitask learners (GPT-2). OpenAI report.
Liu et al. 2019. RoBERTa: a robustly optimized BERT pretraining approach. arXiv:1907.11692.
Radford et al. 2021. Learning transferable visual models from natural language supervision (CLIP). ICML.

Datasets

Zhai et al. 2019. A large-scale study of representation learning with the visual task adaptation benchmark (VTAB). arXiv:1910.04867.
Zhong, Xiong & Socher 2017. Seq2SQL (WikiSQL). arXiv:1709.00103.
Williams, Nangia & Bowman 2018. A broad-coverage challenge corpus for sentence understanding through inference (MNLI). NAACL.
Wang et al. 2019. GLUE: a multi-task benchmark and analysis platform for natural language understanding. ICLR.
Parkhi, Vedaldi, Zisserman & Jawahar 2012. Cats and dogs (Oxford Pets). CVPR.
Jaeger et al. 2014; Moreira et al. 2012; Wang et al. 2017. Shenzhen chest X-ray, INbreast and ChestX-ray14, used by MeLo.

Low-RankAdaptation

From VPT to LoRA

Every fine-tuned copy is as big as the model

Earlier cheap methods cost time or sequence length

What a low-rank matrix is

LoRA adds a trainable bypass to a frozen weight

LoRA on a 3 × 3 weight matrix

Where LoRA goes in a Vision Transformer

VPT and LoRA side by side

How much LoRA trains and stores per task

Numbers trained and stored per task

Compute per image at test time

LoRA's saving grows with the width of the model

LoRA on a ViT in a few lines

Results in the paper: language models

What happens when more weights are trained

Which weight matrices to adapt

How small the rank can be

Why a small rank is enough

LoRA on images

LoRA, VPT and adapters on the same ViT

Neither LoRA nor VPT wins every task

LoRA on medical images

Where LoRA falls short

The rank and the weights are chosen by hand

Merged weights serve one task at a time

A small rank assumes a nearby task

Training still runs through the whole model

My experiments

I tested LoRA on the same three medical datasets

Skin lesions

Blood cells

Chest X-rays

Linear probe

VPT-deep

LoRA

Full fine-tuning

How LoRA compares on my three datasets

What each method costs on skin lesions

Live demo: one image, three models

Linear probe

LoRA

Full fine-tuning

Summary

References

This paper

LoRA on images

Ideas LoRA builds on

Methods compared

Models

Datasets

Low-Rank
Adaptation