Visual Prompt
Tuning

Teach a frozen Vision Transformer a new task by learning only a few extra input vectors.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim
Cornell University, Meta AI and University of Copenhagen. ECCV 2022.

Presented by Ertugrul Senturk
October 2026

Pretrain one big model once, then adapt it to each task

ImageNet-21k14 million photos in 21,000 everyday categories. Pretraining takes weeks and is done once, by Google.

ViT-B/16A Vision Transformer with 85.8 million weights that has learned general visual features.

Fine-grained categories200 bird species, 196 car models

Satellite and microscopeland use from above, tissue slides

Counting and distancehow many objects, how far away

Medical imagesmy own experiments, shown later

Each new task is small: most tasks in the paper have only 1,000 training images, far too few to train from scratch. So we start from the pretrained model and adapt it.

The paper asks how to adapt it best: accurately, and cheaply enough to store for every task. Next, the standard answer and what it costs, then the cheaper methods tried before VPT.

Dataset examples from Jia et al. 2022, Fig. 10. ViT-B/16: Dosovitskiy et al. 2021. The pretrained checkpoint used here and in the paper is Google's ImageNet-21k ViT-B/16.

Full fine-tuning stores a new copy of the model per task

Full fine-tuning retrains every weight of the pretrained model on the new task, and it is usually the most accurate option. The cost is that each task keeps its own retrained copy, so storage grows with every task and with the size of the model.

Number of tasks1

Model size

343 MB

345 MB

32-bit weights: ViT-B/16 85.8M, ViT-L/16 303M, ViT-H/14 632M. A VPT file is 50 prompts at every layer plus a 102-class head. Squares are sized by area.

Cheaper methods existed, but none matched full fine-tuning

Bias came closest and even beat full fine-tuning on 8 of the 24 tasks, but on average no cheaper method reached it. VPT takes a third route: it leaves the model alone and changes the input.

Mean test accuracy over the paper's 24 tasks, ViT-B/16 (Jia et al. 2022, Table 1, mean of the four group averages). The right column is the share of weights trained for each task.

Where the word "prompt" comes from

2020GPT-3

Classifythesentiment:prompt: words a person chose

thefilmwasgreatinput

→

frozen language model

→

positive

2021Prompt tuning

prompt: learned vectors, no words

thefilmwasgreatinput

→

frozen language model

→

positive

In 2020, a prompt was text. GPT-3, a frozen model with 175 billion weights, was only trained to continue text, yet it does a new task when you type a description of the task in front of the input. Not a single weight changes. But words are discrete, so the only way to improve a prompt is trial and error.

In 2021, prompt tuning (Lester et al.) replaced the words with learned vectors that correspond to no word. They are trained by gradient descent on the task's examples while the model stays frozen, and they are all you store per task. Prefix-tuning (Li & Liang) does the same at every layer.

The sentiment example is illustrative. Brown et al. 2020 (GPT-3); Lester, Al-Rfou, Constant 2021 (prompt tuning, T5); Li & Liang 2021 (prefix-tuning).

A visual prompt is the same idea applied to image patches

LanguageLester et al. 2021

thefilmwasgreat

→

frozen language model

→

positive

learned prompt: random at first, trained for the task, same for every input

↑“the film was great”

VisionJia et al. 2022

…

→

frozen Vision Transformer

→

melanoma

learned prompt: random at first, trained for the task, same for every image

↑

The dermoscopy image with its grid of 16 by 16 pixel patches

all 196 patches
of the image

The grey vectors come from the input, as before. The red ones do not: they start as random numbers, are trained on the task's examples while the model stays frozen, and are then put in front of every input of that task. A visual prompt is therefore not text and not a pattern painted on the pixels, only learned numbers in the model's input space.

In language, prompt tuning matched full fine-tuning only once the model had about 10 billion weights. VPT beats full fine-tuning on 20 of 24 tasks with an 86-million-weight ViT-B.

Lester et al. 2021 (T5); Jia et al. 2022, Table 1. Not to be confused with CoOp (Zhou et al. 2021, text prompts for CLIP) or Bahng et al. 2022 (a pixel pattern).

The one-sentence version: VPT is prompt tuning for a Vision Transformer. Both rows learn vectors in the model's input space and leave the model alone; the only difference is where the other vectors come from, words or image patches. Say plainly what it is not: not the text prompts for CLIP that CoOp learns, and not a learned pattern painted on the pixels as in Bahng et al.; the paper tests the pixel version too and it is clearly weaker. Then the surprise: in language, learned prompts needed about 10 billion weights to match full fine-tuning; VPT beats it on 20 of 24 tasks with 86 million. The authors themselves suggest vision prompts may behave differently from language prompts. Point at the labels under each row: the grey vectors come from the input, the red ones never do; they start random, are trained on the task, and the same set goes in front of every image of that task.

How a Vision Transformer reads an image

A fixed grid cuts the image into 196 patches of 16 × 16 pixels, and each patch, read row by row, becomes a vector of 768 numbers. One extra vector, [CLS], goes in front to collect a summary of the whole image: the 12 layers mix all 197 vectors, and the head reads only [CLS] to name the class.

The layers only see a sequence of vectors, so we can add a few more.

ViT-B/16: 12 layers, vectors of 768 numbers, 85.8 million weights. Dosovitskiy et al., ICLR 2021. Image: a melanoma from DermaMNIST, 224 × 224 pixels.

Spend a minute here, it pays off. Walk the picture from left to right and up: a fixed grid cuts the image into 14 by 14 patches, whatever the image shows; nothing is selected, all 196 go in, read row by row from patch 1 at the top left to 196 at the bottom right. Each patch, 16 × 16 × 3 = 768 pixel values, goes through one linear layer to become a grey vector of 768 numbers (the two 768s are a coincidence of ViT-B), with a position vector added so the model knows where it came from. In front sits one extra learned vector, [CLS], which carries no image content and collects a summary as it goes up. The 12 Transformer layers, blue because they will stay frozen in VPT, let every vector look at every other one. At the top, only the [CLS] vector goes to the head, the one task-specific part, which outputs melanoma. The point for VPT: the Transformer does not care how long the sequence is or where a vector came from, so we can add our own vectors.

VPT trains a few extra input vectors and the head

Figures: Jia et al. 2022, Fig. 2. Equations (6) and (8) of the paper: x class token, E patch vectors, P prompts, Z the prompt positions after a layer, L_i layer i.

Why a few extra tokens can steer a frozen network

In every layer, each token takes in information from all the other tokens; this is attention. With prompts in the sequence, [CLS] and every patch also take in information from the prompts.

So training the prompts changes what the frozen network computes, without changing a single weight.

Mechanism: Sec. 3 of Jia et al. 2022; Appendix B relates it to prefix-tuning ("VPT-prefix"). Line widths and the red share of [CLS] are illustrative, not measured.

How much VPT trains and stores per task

N · p · d = 12 × 50 × 768 = 460,800prompt numbers, plus 78,438 for a 102-class head

Numbers trained and stored per task

Full fine-tuning

all 85.8 million weights, retrained for this task

343 MB100%

VPT-deep

the same 85.8 million weights, frozen and shared by every task

2.2 MB0.63%

Compute per image

Plain ViT

197 tokens

1.00×

VPT-deep

197 tokens

1.27×+50 prompt tokens

Prompt length p

More prompts keep the file small, but every image carries them through all 12 layers.

ViT-B/16, N = 12 layers, d = 768, 32-bit numbers. Compute counts multiply-adds per layer for n tokens (12nd² + 2n²d). The paper measures about 2× test latency at p = 200 (Jia et al. 2022, Table 12).

VPT in code: a few lines and one new hyperparameter

# VPT-deep on a frozen ViT-B/16, simplified from demo/vpt/vpt_model.pyvit = timm.create_model(BACKBONE, pretrained=True)                # ImageNet-21k ViT-B/16vit.requires_grad_(False)                                         # backbone frozenprompts = nn.Parameter(torch.empty(12, p, 768).uniform_(-b, b))   # trained: 12 layers × p × 768head = nn.Linear(768, num_classes)                                # trained: the classifier def forward(img):    x = vit._pos_embed(vit.patch_embed(img))                      # [CLS] + 196 patch vectors    for i, layer in enumerate(vit.blocks):        rest = x[:, 1:] if i == 0 else x[:, 1 + p:]               # drop last layer's prompts        P = prompts[i].expand(len(x), -1, -1)                     # this layer's prompts        x = layer(torch.cat([x[:, :1], P, rest], dim=1))          # frozen layer on [CLS, P, rest]    return head(vit.norm(x)[:, 0])                                # classify from [CLS]

The prompts start random, with Xavier-uniform values as in the official code. The paper tried smarter starts, such as class prototypes, and random did as well or better.

Training is plain SGD with momentum for 100 epochs on a cosine schedule. Learning rate and weight decay are picked on a validation split, for every method alike.

The one new hyperparameter is the prompt length p, chosen per task from {1, 5, 10, 50, 100, 200} on the validation split.

Recipe: Jia et al. 2022, Sec. 4.1 and Appendix A. Code is my re-implementation, which reproduces the paper's Oxford Flowers result (99.1% vs the paper's 99.0%).

The paper tests VPT on 24 very different tasks

Frangipani flowers, from Oxford Flowers — Oxford Flowers

A blue car, from Stanford Cars — Stanford Cars

A Boston terrier, from Stanford Dogs — Stanford Dogs

FGVC, 5 tasks

Fine-grained categories, such as 200 bird species, with thousands of training images.

A Persian cat, from Oxford Pets — Oxford Pets

VTAB Natural, 7 tasks

Ordinary photos of objects, textures, animals and scenes, close to the pretraining data.

Farmland seen from a satellite, from EuroSAT — EuroSAT

An aerial photo of a building, from Resisc45 — Resisc45

A stained tissue slide, from Patch Camelyon — Patch Camelyon

A retina scan, from Retinopathy — Retinopathy

VTAB Specialized, 4 tasks

Images from special sensors: satellites, tissue slides under a microscope, and retina cameras.

Rendered 3D objects to count, from CLEVR — CLEVR count

A white shape on black, from dSprites — dSprites location

A street seen from a car, from KITTI — KITTI distance

A toy plane at an angle, from SmallNORB — SmallNORB azimuth

VTAB Structured, 8 tasks

Geometry instead of categories, such as counting objects or judging distance.

All 24 tasks use the same frozen ViT-B/16, pretrained on ImageNet-21k. The 19 VTAB tasks have only 1,000 training images each.

Images: Jia et al. 2022, Fig. 10, four example tasks per group. VTAB-1k: Zhai et al. 2019.

VPT-deep beats full fine-tuning on 20 of 24 tasks

Jia et al. 2022, Table 1 and Tables 13–14: ViT-B/16 pretrained on ImageNet-21k, mean test accuracy within each group. For all 24 tasks, the mean of the four group averages.

Step through the four groups with the arrow keys, then the summary. The chart keeps the families from the cheaper-methods slide: full fine-tuning on top with the dashed line at its score, VPT in red, the older cheap methods in grey. The right column counts the tasks on which a method beats full fine-tuning. FGVC: everyone is strong, VPT-deep edges ahead, 89.1 vs 88.5, and loses only Stanford Cars. Natural: only 1,000 training images, so full fine-tuning overfits; VPT-deep wins 6 of 7, losing SVHN, and even VPT-shallow is above full. Specialized: the one group VPT-deep loses on average, 82.4 vs 83.4, losing Resisc45 and Retinopathy; medical images come back in my own results. Structured: the biggest gap, 55.0 vs 47.6, all 8 tasks won; the linear probe collapses to 26.8, so the frozen features alone do not hold this information, and the prompts make the frozen network compute it. All 24: VPT-deep is the only cheap method above full fine-tuning, 76.3 vs 73.9, with 20 wins, and its total storage for 24 tasks is 1.18 backbones against 24.02.

VPT leads with little data and stays ahead with more

VPT-deep against

Jia et al. 2022, Fig. 3: ViT-B/16, mean test accuracy over the five FGVC tasks, each method trained on a share of every task's training images. The paper gives these values only as a figure; they were read from its vector drawing in the PDF, and the 100% points match Table 1.

This is the paper's Fig. 3, redrawn one comparison at a time. Each point is the average test accuracy over the five FGVC tasks when the method is trained on 10% up to 100% of every task's training images; the axis is logarithmic, as in the paper. Start with full fine-tuning: with 10% of the images it falls to 54.6 against 68.5 for VPT-deep, because it has 86 million weights to fit with very few examples. With all the data the two almost meet, 89.1 against 88.5, but VPT-deep is never below. Then step through the baselines. The linear probe is the classic low-data trick: good at 10%, overtaken by full fine-tuning from 30% on. The adapter is overtaken from 40% on. Bias is the strongest rival and beats full fine-tuning up to 80% of the data, but VPT-deep is above it at every point. So VPT is not just a low-data trick. VPT-shallow is left out to keep the chart readable; it always sits between the linear probe and VPT-deep. If asked: the paper publishes these numbers only as a figure; I read them from the figure's vector data in the PDF, and the 100% points match Table 1 to within 0.01.

The result holds for larger backbones too

VTAB group

Jia et al. 2022, Fig. 4: VTAB-1k, mean test accuracy per group, backbones pretrained on ImageNet-21k. The paper gives these values only as a figure; they were read from its vector drawing in the PDF, and the ViT-B points match Table 1.

The paper's Fig. 4, redrawn one group at a time: the same VTAB-1k tasks with three backbones, ViT-Base, ViT-Large and ViT-Huge, all pretrained on ImageNet-21k; the horizontal axis is the number of weights. Natural: VPT-deep wins with every backbone, and the gap grows from 2.6 points to about 7, because full fine-tuning gets worse as the model grows, from 75.9 to 70.9, with only 1,000 training images per task. Specialized: a tie, within one point everywhere, the same picture as on ViT-Base. Structured: VPT-deep leads by 6 to 7 points with every backbone, and the linear probe stays near 26, so the frozen features alone do not solve these tasks at any size. Why this matters: the storage argument grows with the model. A fully fine-tuned ViT-Huge is about 2.5 GB per task, while 50 prompts at each of its 32 layers take about 8 MB. It does not hold for every backbone, though, as we will see shortly.

Four places to put the prompt

Numbers from Jia et al. 2022, Fig. 5 (ViT-B/16, prompts in the first layer only); "All four" is the mean of the three groups. Diagrams redrawn.

The question: where should the learned numbers enter the network? Two places in principle: in the embedding space, as vectors next to the patch vectors, or in pixel space, as part of the image before the embedding layer. Prepend is the default we have seen: extra tokens after the embedding. Add: the prompts are added onto patch vectors, so the sequence is not longer; it loses in all three groups, 8 points on Structured, and the deep version loses too (43.2 against 55.0 on Structured). Prepend-pixel: learned pixel patches next to the image patches, then the frozen embedding layer; loses everywhere, 5 points on Natural. Concat-channel: a learned fourth channel on the image, the embedding layer trained to read it; it collapses, 30 points lower on Natural and below even a linear probe there (68.9). Summary: separate tokens in the embedding space are the right place. All numbers are VPT-shallow because that is the only setting where the paper tests all four options.

How many prompts, and at which layers

First, the number of prompts, the one new hyperparameter. Each panel is one VTAB group: the red line is VPT-deep with 1 to 200 prompts at every layer, the dashed line is full fine-tuning, the grey line the best adapter. The best number is different in every group, 10 on Natural, 5 on Specialized, 200 on Structured, and per task it varies even more, from 5 prompts for Oxford Flowers to 200 for Stanford Cars (Table 14). That is why the paper picks p per task on the validation set. Even a single prompt per layer beats the adapter everywhere, and on Natural it already beats full fine-tuning. Press the arrow key for which layers. The red line adds prompts from the input upwards: layer 1 alone is VPT-shallow, all twelve layers is VPT-deep. The grey line adds them from the output downwards. More prompted layers generally help, and the input side wins: on Structured, prompts in layers 1 to 3 give 50.6, prompts in layers 9 to 12 only 37.6. The authors point out that this is the opposite of what was seen with text prompts. The paper has no table for these two experiments, so the values are read off the plotted points of its figures; the end points match VPT-shallow and VPT-deep in Table 13.

With prompts, the same frozen network separates the classes

Linear probe

frozen network, no prompts

VPT-deep

same frozen network, with prompts

Full fine-tuning

every weight retrained

Each dot is one test image, coloured by its true class; the number is test accuracy. With the frozen features alone, the classes overlap. With prompts, the same frozen network produces clearly separated groups, so a simple linear head can tell them apart.

t-SNE maps of the final [CLS] vector, test set, from Jia et al. 2022, Fig. 9. Accuracies from Fig. 9 and Table 13.

This is the payoff of the "why it can work" slide: same weights, different input, completely different output. Each map is a 2-D t-SNE picture of the 768-number [CLS] vector the head receives. House numbers: the frozen features are one mixed blob, 36.6%; with prompts the ten digits form islands, 78.1%. Full fine-tuning is still better here, 87.4%, so be honest about that. Press the arrow key for satellite images: the frozen features are already half sorted, 87.5%, and VPT-deep gives clean clusters, 96.1%, slightly above full fine-tuning at 95.7%. Press again for counting objects in CLEVR: the frozen features know nothing about the count, 34.3%; VPT-deep reaches 68.5%, well above full fine-tuning at 56.3%, and arranges the images along a band that follows the object count. If asked: VPT-shallow sits in between (74.5, 92.0, 50.5); it is in the paper's figure.

Where VPT falls short

Be honest here; it is what a reading group is for. Three settings where the main result does not carry over. First, the pretraining. Everything so far used a ViT-B pretrained with labels on ImageNet-21k. The paper repeats VTAB-1k with the same architecture pretrained without labels on ImageNet-1k, by MoCo v3 (contrastive) and by MAE (masked autoencoder). The chart shows the mean of the three VTAB group averages. With labels, VPT-deep leads full fine-tuning, 72.0 against 69.0. With MoCo v3 it is 65.2 against 69.6, and Bias (69.2), Partial-1 (68.3) and the adapter (68.2) are all ahead of it. With MAE it collapses to 41.1, closer to the linear probe at 32.1 than to full fine-tuning at 64.3, while Partial-1, which retrains only the last block, reaches 61.5. The authors only say these self-supervised ViTs are fundamentally different and leave the reason open. It connects to the MAE paper from last time: MAE features are known to probe badly, and a few input tokens apparently cannot reorganise them the way retraining the last block can. Press the arrow key for Swin, a hierarchical Transformer that attends within local windows and merges patches in deeper layers; the prompts sit in the windows and are dropped at patch merging. VPT still beats the other cheap methods (Bias, linear probe, Partial, MLP-3) in every group, but full fine-tuning is ahead of VPT-deep in all three, by 6.3 points on Structured. Oddly, VPT-shallow does slightly better than full fine-tuning on Natural, 79.9 against 79.1; it is left out of the chart to keep it simple. Press again for segmentation: ADE20K, 150 classes per pixel, SETR on a ViT-L pretrained on ImageNet-21k; mIoU is the mean overlap between predicted and true regions. The decoder head is trained in every method, which is why even "head only" trains 13 million weights. VPT-deep gets 42.1 against 48.3, a little below Bias at 43.4; prompts plus biases give 44.0, within 1.5 of a fully fine-tuned ResNet-101 with four times the trained weights. This is also where LoRA will come in: it edits weights, so it does not depend on the input space being "steerable".

VPT saves storage, not training memory or inference time

Easy to miss in the main paper: the saving is disk, not GPU. Three arrow-key steps, same rows each time: full fine-tuning in dark with the dashed line at its value, the cheap baselines in grey, VPT-deep in red with 1, 50, 100 and 200 prompts per layer. Storage: a task keeps only what it trained, so VPT-deep is 0.35 MB with one prompt per layer and 7.7 MB with 200, against 343 MB for a full copy; bias and adapter are as small, so storage is what all these methods buy, and it is VPT's real win. Training memory: no saving. The prompts in the first layer only get a gradient if it travels back through all twelve frozen layers, so their activations are kept, as in full fine-tuning; with 200 prompts the longer sequence makes it 25.8 GB against 11.7. Only the linear probe, 0.9 GB, avoids this. Training time per image is not shown: one prompt trains faster than full fine-tuning, 214 against 359 ms, but 200 prompts are as slow, 360 ms. Inference time: every test image carries the prompts through all twelve layers. Fifty prompts take the sequence from 197 to 247 tokens and the time from 70 to 82 ms; 200 prompts double it, 141 ms. The paper notes this is far from the 4x one would expect from quadratic attention. The prompts cannot be folded into the weights, so the cost stays; the paper's prefix variant, which puts the prompts only into the keys and values, recovers some of it (Fig. 19). LoRA, the second paper, merges its update into the weights and costs nothing at test time. If asked: the table has only 1 and 200 prompts; the 50 and 100 values are read off Fig. 18's plotted points, and the 1 and 200 points of that figure match the table. If someone asks "why not just use this for everything", this is the answer.

Low-Rank
Adaptation

Teach a frozen model a new task by learning a small, low-rank change to its weights.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
Microsoft. ICLR 2022. Results on images from Zhang, Zhou & Liu 2022 and Zhu et al. 2024.

Presented by Ertugrul Senturk
October 2026

From VPT to LoRA

VPTJia et al. 2022

…

→

frozen Vision Transformer

→

head

Learned vectors join the image patches, so every image carries a longer sequence through every layer.

LoRAHu et al. 2022

…

→

frozen Vision Transformersmall trained change to its weights

→

head

The input stays as it is. A small change to the weights is learned, and after training it is added into them.

VPT trained 0.6% of the weights and beat full fine-tuning on 20 of 24 tasks. But its prompts make every image slower, and they can never be folded into the model.

LoRA leaves the input alone and changes the weights instead. It ends up storing about as little per task as VPT, while the model runs exactly as fast as before.

VPT: Jia et al., ECCV 2022 (0.6% of the weights with 50 prompts per layer and a 102-class head; 20 of 24 tasks, Table 1). LoRA: Hu et al., ICLR 2022, first posted in June 2021.

Every fine-tuned copy is as big as the model

The paper starts from the largest model of its time, GPT-3, with 175 billion weights. Each fine-tuned copy takes 350 GB, so a service with a hundred tasks would store 35 TB. LoRA keeps one shared model and only a small file per task, and the frozen weights also need no optimizer state, which cuts training memory from 1.2 TB to 350 GB.

Number of tasks1

Model

343 MB

345 MB

GPT-3 in 16-bit numbers: 350 GB per copy, 35 MB per LoRA file at rank 4 on the query and value weights (Hu et al. 2022, Sec. 4.2). ViT-B/16 in 32-bit: 343 MB per copy; its LoRA file is rank 8 on the same weights plus a 102-class head. Squares are sized by area.

Earlier cheap methods cost time or sequence length

Adapters insert small extra layers into every block. They have few weights, but an image or a sentence must pass through them one after another, so serving a single input takes 20 to 30% longer. With large batches the gap shrinks to 2 or 3%.

Prompt methods such as prefix tuning and VPT lengthen the input instead. That leaves less room for the real input and costs compute in every layer, and on GPT-3 prefix tuning got worse beyond 256 prompt tokens.

LoRA adds no layer and no token at test time.

Hu et al. 2022, Table 1: one forward pass of GPT-2 medium, mean of 100 trials on one NVIDIA Quadro RTX 8000; bars are drawn relative to the fine-tuned model in each setting. Prefix tuning: Sec. 5.5 and Table 15.

What a low-rank matrix is

The photo of a cat rebuilt from one piece

A grayscale photo is a matrix of 274 × 274 brightness values. Any matrix can be written as a sum of simple pieces, where each piece is one column of numbers times one row. The rank is the number of pieces.

LoRA bets that the change fine-tuning makes to a weight matrix is like this photo: a few pieces carry almost all of it.

Not from the paper: the best rank-r approximation (SVD) of an Oxford Pets photo, Jia et al. 2022, Fig. 10. The low-rank bet follows Li et al. 2018 and Aghajanyan et al. 2021.

One minute of linear algebra, because the whole paper rests on the word rank. Treat the photo as a matrix of brightness values. A rank-one matrix is a column times a row: every row of the image is the same pattern, just scaled, so you get bands. Step through with the arrow keys. Each step doubles the number of pieces. The pieces are ordered by how much they contribute, so the first few capture the big shapes and the later ones only fine detail. At rank 16 the cat is recognisable, from 12% of the numbers; at 64 it is hard to tell from the original, from 47%. Storing r pieces costs r times (rows plus columns) numbers instead of rows times columns. The drawing under the text shows the same thing as matrices, to scale: the r columns side by side are a tall matrix B, the r rows stacked are a wide matrix A, and B times A rebuilds the photo; at rank 1 they are a single line each, at all 274 they are as big as the photo itself. That is the trade LoRA makes for the weight change, with the same names B and A from now on. Why expect the change to be low-rank? Two earlier papers measured the "intrinsic dimension" of fine-tuning: you can freeze the model, train only a few hundred or thousand numbers, map them through a fixed random projection onto all the weights, and still get most of the accuracy. LoRA's assumption is the matrix version of that: each weight change has a low rank, so it can be written as B times A with a small r, chosen by hand.

LoRA adds a trainable bypass to a frozen weight

Paper figure: the input x goes through the pretrained weight W and, in parallel, through A and then B; the two outputs are added to give h. B starts at zero, A starts random.

Figure: Hu et al. 2022, Fig. 1, where orange is trained. Equation (3) and Sec. 4.1. The paper sets α to the first r it tries and does not tune it.

The paper's own figure; orange there is what we call red, the trained part. During training: x goes through the frozen W0 as before, and in parallel through A, which squeezes it down to r numbers, and B, which expands it back. The outputs are added. Two details matter. B starts at zero, so at step one the model is exactly the pretrained model and training starts from a known good point. And the bypass is scaled by alpha over r, so that changing the rank does not mean re-tuning the learning rate. Press the arrow key: after training, B times A is just a d × d matrix, so we add it into W0 once and throw A and B away. The model is again a single matrix per layer, same size, same speed. To switch task you subtract this task's BA and add another task's, a cheap operation. That is the property VPT did not have.

LoRA on a 3 × 3 weight matrix

Made-up numbers with rank r = 1 and α/r = 1, so every product can be checked by eye. In ViT-B the same happens with 768 × 768 weights and r = 8.

Made-up numbers, rank 1, so every product can be checked by eye. First the frozen path: the pretrained layer computes W times x, which gives 5, 7 and 8. Next step: LoRA adds a second path. A is one row of three numbers, so A times x is a single number, 1 + 2 + 0 = 3; the input is squeezed down to r numbers. B is a column of three, so B times 3 gives 3, 0 and 6, back to full size. The two paths are added: 8, 7, 14. During training only these six red numbers would change. Next step: after training we multiply B by A once. A column times a row is a full 3 × 3 matrix, the same shape as W, so we can add it to W; the red cells are the ones that changed. Last step: the merged W′ times x gives 8, 7, 14 again, exactly the training-time output, because W x + B(A x) equals (W + BA) x. So A and B are deleted and the layer is one ordinary matrix. The point people often miss: B and A are never multiplied with W. They are multiplied with each other, and the product is added to W.

Where LoRA goes in a Vision Transformer

One of the 12 blocks of ViT-B/16. Placement as in Hu et al. 2022, Sec. 4.2, and Houlsby et al. 2019. Zhang et al. 2022, Fig. 1a, draws LoRA, the adapter and VPT in one block.

Where exactly the pairs go. This is one of the twelve blocks of a ViT, frozen in blue. The attention part has four weight matrices: query, key, value and output; the MLP has two. The paper only adapts attention weights, by default the query and value, and leaves everything else frozen, including the MLP. The red pairs sit beside Wq and Wv in every block, 24 pairs in ViT-B. Press the arrow key for an adapter, for contrast: a small trained two-layer network after the MLP, an extra layer every input has to go through, which is where the latency from the earlier chart comes from. LoRA instead sits beside weights that already exist, so after merging it disappears. VPT, the third option, changes nothing inside the block; the next slide compares it with LoRA step by step. NOAH, the vision paper we will see, searches over all three at once.

VPT and LoRA side by side

One layer of ViT-B/16 drawn for each. VPT-deep with 50 prompts per layer (Jia et al. 2022); LoRA with rank 8 on the query and value weights (Hu et al. 2022). Both train the same 102-class head.

This is the slide that puts the two methods next to each other, in three steps. What is learned: VPT learns extra input vectors, 50 per layer, and the same ones are used for every image of the task; they start random, they are not cut from any image. LoRA learns a correction to the weights, the pair B and A beside the query and value weights; it is also one pair for the whole task, not per image, and B starts at zero. Next step, training: in both cases every training image goes through the frozen model, the loss is computed, and gradients flow back. In VPT they end in the prompts and the head; in LoRA in B, A and the head. In both, the pretrained weights never change. Next step, test time, which is where the two differ. VPT's prompts are inputs, so they have to be attached to every test image, and every layer processes 247 tokens instead of 197, 27% more compute. LoRA's correction is a matrix of the same shape as W, so it is added into W once, and the layer is an ordinary layer again, exactly the original cost. The next slide puts numbers on both differences: what each stores per task, and what each image costs.

How much LoRA trains and stores per task

2 · N · m · d · r = 2 × 12 × 2 × 768 × 8 = 294,912numbers in the pairs, plus 78,438 for a 102-class head

Numbers trained and stored per task

Full fine-tuning

all 85.8 million weights, retrained for this task

343 MB100%

VPT-deep, 50 prompts

frozen and shared by every task

2.2 MB0.63%

LoRA

frozen and shared by every task

1.5 MB0.44%

Compute per image at test time

Plain ViT

197 tokens

1.00×

VPT-deep, 50 prompts

197 tokens

1.27×+50 tokens

LoRA, merged

197 tokens

1.00×nothing extra

Rank r

Weights

ViT-B/16: N = 12 blocks, d = 768, m adapted weights per block, 32-bit numbers. VPT-deep as before. Compute counts multiply-adds per layer for n tokens (12nd² + 2n²d).

The numbers behind the previous slide, with the same accounting as for VPT. Each pair has 2 × 768 × r numbers; with query and value in 12 blocks and rank 8 that is 294,912, plus the same 102-class head as before: 0.44% of the model, a 1.5 MB file. VPT-deep with 50 prompts was 0.63% and 2.2 MB, so storage is about the same. The difference is the second chart: VPT made every image 27% more expensive, LoRA after merging costs exactly what the plain ViT costs. Try rank 64 on all four attention weights: the file grows to 19 MB, 5.6% of the model, but the compute bar does not move. If someone asks about training compute: the pairs still have to be computed during training, and the backward pass still goes through every layer, so training is not much cheaper per image than full fine-tuning; what drops is optimizer memory.

LoRA's saving grows with the width of the model

Full change ΔW

d × d = d²

589,824 numbers on ViT-B

B and A together

d × r + r × d = 2rd

12,288 numbers on ViT-B

A full change to one d × d weight has d² numbers, while B and A together have 2rd. Since d is the square root of d², LoRA's cost grows like the square root of the full size, times the small constant 2r.

So the wider the model, the smaller LoRA's share. Together with adapting only the query and value weights, this is why GPT-3's task file is 10,000 times smaller than a copy.

One d × d attention weight, rank r = 8 for every model. Widths: ViT-B/16 768, ViT-L/16 1,024, ViT-H/14 1,280 (Dosovitskiy et al. 2021); GPT-3 12,288 (Hu et al. 2022, Sec. 1). 10,000 times: Hu et al. 2022, Sec. 4.2.

LoRA on a ViT in a few lines

vit = timm.create_model(BACKBONE, pretrained=True, num_classes=C)vit.requires_grad_(False); vit.head.requires_grad_(True) class LoRAqv(nn.Module):              # wraps qkv = Linear(768, 2304)    def __init__(self, qkv, r=8, alpha=8):        super().__init__()        d = qkv.in_features        self.qkv, self.s = qkv, alpha / r    # frozen W0, scale α/r        self.Aq = nn.Parameter(torch.randn(r, d) / d ** 0.5)        self.Av = nn.Parameter(torch.randn(r, d) / d ** 0.5)        self.Bq = nn.Parameter(torch.zeros(d, r))   # B = 0        self.Bv = nn.Parameter(torch.zeros(d, r))    def forward(self, x):        q, k, v = self.qkv(x).chunk(3, dim=-1)      # W0 x        q = q + self.s * x @ self.Aq.T @ self.Bq.T  # + B A x        v = v + self.s * x @ self.Av.T @ self.Bv.T        return torch.cat([q, k, v], dim=-1) for blk in vit.blocks: blk.attn.qkv = LoRAqv(blk.attn.qkv)

B starts at zero and A at small random values, so the first forward pass is exactly the pretrained model.

The two new choices are the rank r and which weights get a pair. On ViT-B, Zhang et al. use rank 8 with AdamW at a learning rate of 0.001 for 100 epochs.

After training, each pair is added into its qkv weight once, and the model is a plain timm ViT again.

My sketch for timm's ViT-B/16: 294,912 trained numbers in the pairs, and identical outputs before and after merging. Official code: loralib, github.com/microsoft/LoRA. Settings: Hu et al. 2022, Sec. 4.1; Zhang et al. 2022, App. A.2.

Results in the paper: language models

What happens when more weights are trained

Hu et al. 2022, Table 15 (the points of Fig. 2): GPT-3 175B, validation accuracy; LoRA with equal rank on query and value, from rank 1 to 64. Full fine-tuning trains all 175 billion weights.

Which weight matrices to adapt

Hu et al. 2022, Table 5: GPT-3 175B with the same 18 million trained weights in every LoRA row, validation accuracy (noise ±0.5 on WikiSQL, ±0.1 on MNLI). Full fine-tuning: Table 4.

How small the rank can be

Hu et al. 2022, Table 6: GPT-3 175B, validation accuracy at five ranks; noise ±0.5 on WikiSQL and ±0.1 on MNLI. On GPT-2 the best rank was between 4 and 16 (App. H.2).

Why a small rank is enough

Paper figure: heat maps of subspace similarity between the directions learned with rank 8 (rows i) and rank 64 (columns j), for the query and value changes. Only the first row is bright.

The authors trained rank 8 and rank 64 separately and compared the directions each learned. Bright means the two runs found the same direction. Only the first direction is clearly shared, with similarity above 0.5, and the rest overlap little, so most of the extra rank holds noise.

21.67

is how strongly the pretrained weight W_q acts along its own top 4 directions.

0.32

is how strongly it acts along the 4 directions of the learned change ΔW_q: weak, but more than the 0.02 of random directions.

6.91

is the strength of the change along those directions, about 21 times what the pretrained weight had there.

The change does not repeat what the model already does. It strongly amplifies directions the model learned in pretraining but used weakly.

Hu et al. 2022, Fig. 3 (layer 48 of 96, GPT-3; values are normalized subspace similarity, 0 to 1) and Table 7 (rank 4, Frobenius norms).

Section 7 asks why this works. First analysis: train rank 8 and rank 64 on the same task, take the directions each one learned, and measure how much the top i directions of one lie inside the top j of the other. In the heat maps rows are rank-8 directions, columns rank-64 directions; colour is overlap from 0 to 1. Only the first row lights up: both runs agree on one direction and disagree on the rest, so the useful part of the change is about one-dimensional and the rest is noise that training accumulates. That explains rank 1 working. Press the arrow key for the second analysis, Table 7. Project the pretrained weight onto the four directions of the learned change: 0.32, small, though larger than for random directions, 0.02. Its own top directions are 21.67, so the change is not about what the model already does strongly. The change itself has strength 6.91 in those directions, an amplification of about 21. The authors' reading: LoRA amplifies features that pretraining learned but did not emphasise, and the task needs. This is measured on GPT-3; nobody in these papers repeats it for ViT.

LoRA on images

The paper itself has no image experiments. Vision groups tested it next, first on the same 19 VTAB-1k tasks as VPT, then on medical images.

Both use a ViT-B/16, so the numbers sit next to the VPT results.

LoRA, VPT and adapters on the same ViT

Zhang et al. 2022, Table 1: ViT-B/16 pretrained on ImageNet-21k, VTAB-1k test accuracy. LoRA (rank 8) and the adapter were trained by Zhang et al.; the other rows are Jia et al. 2022, Table 1. Group means computed from the per-task values; "all 19" is the mean of the three groups.

Same backbone and benchmark as the VPT paper: 1,000 training images per task, three groups. The adapter here is NOAH's re-run with the best recipe, which is much stronger than the adapter in the VPT paper's table. Natural: all three cheap methods are two to four points above full fine-tuning, LoRA narrowly best, 79.5. Specialized: LoRA 84.6 and the adapter 84.1 are slightly above full fine-tuning at 83.4; VPT-deep is a point below. Structured: the big gap again, all three beat full fine-tuning on all eight tasks, LoRA the most, 59.8 against 47.6. All 19: LoRA 74.6, adapter 73.9, VPT-deep 72.0, full 69.0. VPT-deep beats full fine-tuning on one more task, 16 against 15, but by smaller margins. Two caveats if asked: NOAH's table prints 87.2 for full fine-tuning on Flowers where the VPT paper has 97.2, so I used the VPT paper's numbers for those rows; and NOAH prints 74.5 for LoRA's average, while its own per-task numbers give 74.6.

Neither LoRA nor VPT wins every task

LoRA is ahead on 15 of the 19 tasks, by up to 14 points on counting objects in CLEVR. VPT-deep is ahead on 4, and on CIFAR-100 by 12 points.

Zhang et al. conclude that no single method fits every task. Their method, NOAH, searches for a mix of LoRA, adapter and prompts per task, and reaches 75.5 on average.

Zhang et al. 2022, Table 1: per-task test accuracy of LoRA minus VPT-deep on VTAB-1k, ViT-B/16. Right of the line, LoRA is ahead.

LoRA on medical images

MeLo, ISBI 2024, from ShanghaiTech and Shanghai Jiao Tong: LoRA with rank 4 on the query and value weights of a ViT-B pretrained on ImageNet, applied to four medical tasks: tuberculosis on chest X-rays, four blood cell types, malignancy in mammograms, and 14 thoracic diseases. They train 0.14 million weights, 0.17%. AUC first: within a point of full fine-tuning on three tasks, 3.6 points lower on blood cells. Accuracy, second step: ahead on tuberculosis and blood cells, slightly behind on the other two. The title says "better than fine-tuning", but these are single runs on small test sets, about 80 mammograms for INbreast, so I would read it as "about the same". Third step is their real point, deployment, with a giant CLIP ViT: 25 images from each task in random order. Reloading the right fine-tuned copy for each image takes 218 s for 100 images; keeping all four copies loaded is fast but needs 7 billion weights in GPU memory; one model with four LoRA pairs takes 10.6 s at the memory of a single model.

Where LoRA falls short

The rank and the weights are chosen by hand

The paper picks the query and value weights and a small rank by trying settings, and leaves a principled choice to future work. On VTAB-1k the best method even changes from task to task.

Merged weights serve one task at a time

Once BA is added into W, every input in a batch must belong to that task. Mixing tasks in one batch means keeping the pairs separate, which brings back a small extra cost; VPT's prompts mix freely.

A small rank assumes a nearby task

The authors expect a small rank to fail when the new task is far from pretraining, such as a different language. The low-rank analysis was only done on GPT-3, not on vision models.

Training still runs through the whole model

LoRA saves the optimizer state of the frozen weights, but the gradient still passes back through every layer. On ViT-B most memory is activations: freezing the backbone for VPT only cut it from 11.7 to 10.3 GB.

Hu et al. 2022, Sec. 4.2 (batching), footnote 6 (a different language), Sec. 7 and 8 (analysis on GPT-3, future work). Zhang et al. 2022, Fig. 1b. Jia et al. 2022, Table 12 (ViT-B/16, batch 64).

Four honest limits. First, choices by hand: which matrices, which rank. The paper's guidance comes from GPT-3, and NOAH's per-task spread shows the best choice varies. Second, a deployment detail people miss: merging is what makes LoRA free at test time, but a merged model serves one task. If a server mixes requests of different tasks in one batch, you have to keep the pairs unmerged and pay a small extra computation, while VPT's prompts are just inputs and can differ per sample. Third, the low-rank assumption: the authors themselves say a very different task, like a new language, may need full rank; the evidence for low rank is from GPT-3 only. Fourth, training memory: the 3× saving on GPT-3 came from optimizer state, which dominates for huge models. On ViT-B, activations dominate, and those are still stored for the whole network, as with VPT. My own measurement of this will be part of the experiments next.

My experiments

I re-implemented VPT and LoRA and trained both on three medical imaging datasets from MedMNIST. A linear probe and full fine-tuning ran next to them, all four starting from the same ViT-B/16.

First the results, then all four models running live in this browser.

I tested VPT and LoRA on three medical datasets

Each method was trained twice: on 1,000 images, as in VTAB-1k, and on the full training set.

Dermoscopy image of a melanoma — melanoma

Skin lesions

Dermoscopy, 7 classes

1,000 or all 7,007 training images

Microscopy image of a neutrophil — neutrophil

Microscopy image of a platelet — platelet

Blood cells

Microscopy, 8 cell types

1,000 or all 11,959 training images

Chest X-rays

Radiographs, pneumonia or normal

1,000 or all 4,708 training images

head

ViT-B/16 frozen

Linear probe

Trains only a new head.

head

ViT-B/16 frozen

VPT-deep

Trains a head and 50 prompts per layer.

head

ViT-B/16 frozen

+

LoRA

Trains a head and rank-8 pairs beside W_q and W_v.

head

ViT-B/16 retrained

Full fine-tuning

Retrains all 85.8 million weights.

MedMNIST+ at 224 × 224 (Yang et al. 2023); skin lesions from HAM10000 (Tschandl et al. 2018). RTX 5070 Ti laptop GPU. SGD for linear probe and VPT, AdamW for LoRA and full fine-tuning; learning rate picked on validation.

Three public datasets from MedMNIST+, at 224 × 224 pixels, all far from ImageNet photos. Skin lesions are heavily imbalanced, two thirds are nevi, so I report balanced accuracy, the mean per-class recall: always answering nevus would score 67% plain accuracy. Each task was trained twice, on 1,000 random training images, the size of VTAB-1k, and on the whole training split; the bars show how small 1,000 is next to each full set. Next step: the four methods. All start from the same ViT-B/16 ImageNet-21k checkpoint. The linear probe, VPT and LoRA keep it frozen and train only the red parts: a head, prompts in every layer, or low-rank pairs beside the query and value weights. Full fine-tuning retrains every weight, so each task needs its own 343 MB copy. Each method uses its paper's optimizer, the learning rate was picked on the validation split, and the reported model is the one with the best validation balanced accuracy, the same rule for all four.

VPT and LoRA on my three datasets

My runs: ViT-B/16 (ImageNet-21k), MedMNIST+ 224 × 224, balanced accuracy on the test set (the mean per-class recall), bars start at 0. Each method was trained once with 1,000 images and once with the full training set; there are no points in between.

What each method costs on skin lesions

Three steps, one cost each, all measured on my laptop for skin lesions trained on the full training split; every number is read from the training logs and the benchmark file. Storage: both LoRA and VPT keep one copy of the base model and add a small file per task, while full fine-tuning needs a whole 343 MB model per task. Training: LoRA and VPT both still send the error signal back through all twelve frozen layers, so neither is cheap like the linear probe; the memory saving compared with full fine-tuning is the optimizer state of the frozen weights. Inference: this is where the two differ. VPT makes every layer process 50 extra tokens. LoRA with its pairs kept separate adds a little extra work, and once merged into the weights it runs exactly as fast as the plain model, which is the paper's main practical claim.

Live demo: one image, four models

Pick a test image below,
or drop your own here

Linear probe

shared model + head

VPT-deep

shared model + prompts

LoRA

shared model + pairs

Full fine-tuning

own retrained model

All four start from the same ViT-B/16, pretrained on ImageNet-21k. Linear probe, VPT-deep and LoRA share one frozen copy of it; full fine-tuning runs its own retrained copy.

Not connectedResearch demo, not a diagnostic tool

Every click runs the image through all four models right here in the browser, with onnxruntime-web. Linear probe, VPT-deep and LoRA all use the very same frozen copy of the base model in memory: VPT-deep feeds its 50 prompts per layer in beside the patches, LoRA feeds its rank-8 pairs in beside the query and value weights, and the linear probe uses neither. That is the unmerged form of LoRA, so one copy serves every task; merged, it would run at exactly the speed of the plain model, but it would be one model per task again. Full fine-tuning retrained every weight, so it is a separate model per task. Compare the times: VPT-deep is the slowest of the three shared models, because every layer carries 50 extra tokens. The percentage under each name is that model's balanced accuracy on the whole test set. Now switch to 1,000 training images: for linear probe, VPT-deep and LoRA only a small head and a file of a megabyte or two change, so it is instant; full fine-tuning has to download and load a second whole model, 172 MB in 16-bit. The thumbnails are random test images, not picked to look good, so a wrong answer can happen; say so. Research demo, not a diagnostic tool.

Summary

20of 24

tasks where VPT-deep beat full fine-tuning, training 0.6% of a ViT-B pretrained with labels.

99.4%

less storage per task than a fully fine-tuned copy: 2.2 MB for VPT instead of 343 MB, and 1.5 MB for LoRA.

1.00×

the test-time cost of the original model with LoRA, because its change is added into the weights. VPT's prompts cost 1.27×.

74.6

mean VTAB-1k accuracy of LoRA on ViT-B/16, against 72.0 for VPT-deep and 69.0 for full fine-tuning.

–

average balanced accuracy of LoRA on my three medical datasets.

Jia et al. 2022, Table 1 (24 tasks, ViT-B/16) and Table 12; 0.6% is VPT-deep with 50 prompts per layer and a 102-class head, LoRA is rank 8 on the query and value weights with the same head. Zhang et al. 2022, Table 1, mean of the three VTAB-1k groups. Medical datasets: my runs, balanced accuracy on the test set.

References

The two papers

Jia, Tang, Chen, Cardie, Belongie, Hariharan & Lim 2022. Visual prompt tuning. ECCV. arXiv:2203.12119.
Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang & Chen 2022. LoRA: low-rank adaptation of large language models. ICLR. arXiv:2106.09685.

Prompting before VPT

Lester, Al-Rfou & Constant 2021. The power of scale for parameter-efficient prompt tuning. EMNLP.
Li & Liang 2021. Prefix-tuning: optimizing continuous prompts for generation. ACL.
Zhou, Yang, Loy & Liu 2022. Learning to prompt for vision-language models (CoOp). IJCV. arXiv:2109.01134.
Bahng, Jahanian, Sankaranarayanan & Isola 2022. Exploring visual prompts for adapting large-scale models. arXiv:2203.17274.

Ideas LoRA builds on

Li, Farkhoor, Liu & Yosinski 2018. Measuring the intrinsic dimension of objective landscapes. ICLR.
Aghajanyan, Zettlemoyer & Gupta 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. ACL.

Other parameter-efficient methods

Houlsby et al. 2019. Parameter-efficient transfer learning for NLP (adapter). ICML.
Lin, Madotto & Fung 2020. Exploring versatile generative language model via parameter-efficient transfer learning. Findings of EMNLP.
Rücklé et al. 2021. AdapterDrop: on the efficiency of adapters in Transformers. EMNLP.
Ben Zaken, Goldberg & Ravfogel 2022. BitFit: simple parameter-efficient fine-tuning for Transformer-based masked language-models (Bias). ACL.
Zhang, Sax, Zamir, Guibas & Malik 2020. Side-tuning: a baseline for network adaptation via additive side networks (Sidetune). ECCV.

LoRA on images

Zhang, Zhou & Liu 2022. Neural prompt search (NOAH). arXiv:2206.04673.
Zhu et al. 2024. MeLo: low-rank adaptation is better than fine-tuning for medical image diagnosis. ISBI. arXiv:2311.08236.

References, continued

Models

Dosovitskiy et al. 2021. An image is worth 16×16 words: Transformers for image recognition at scale (ViT). ICLR.
Liu et al. 2021. Swin Transformer: hierarchical vision Transformer using shifted windows. ICCV.
Chen, Xie & He 2021. An empirical study of training self-supervised vision Transformers (MoCo v3). ICCV.
He et al. 2022. Masked autoencoders are scalable vision learners (MAE). CVPR.
Zheng et al. 2021. Rethinking semantic segmentation from a sequence-to-sequence perspective with Transformers (SETR). CVPR.
Chen et al. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation (DeepLab v3+). ECCV.
Radford et al. 2021. Learning transferable visual models from natural language supervision (CLIP). ICML.
Brown et al. 2020. Language models are few-shot learners (GPT-3). NeurIPS.
Radford et al. 2019. Language models are unsupervised multitask learners (GPT-2). OpenAI report.
Liu et al. 2019. RoBERTa: a robustly optimized BERT pretraining approach. arXiv:1907.11692.

Datasets and benchmarks

Zhai et al. 2019. A large-scale study of representation learning with the visual task adaptation benchmark (VTAB). arXiv:1910.04867.
Parkhi, Vedaldi, Zisserman & Jawahar 2012. Cats and dogs (Oxford Pets). CVPR.
Zhou et al. 2019. Semantic understanding of scenes through the ADE20K dataset. IJCV.
Zhong, Xiong & Socher 2017. Seq2SQL: generating structured queries from natural language using reinforcement learning (WikiSQL). arXiv:1709.00103.
Williams, Nangia & Bowman 2018. A broad-coverage challenge corpus for sentence understanding through inference (MNLI). NAACL.
Wang et al. 2019. GLUE: a multi-task benchmark and analysis platform for natural language understanding. ICLR.
Jaeger et al. 2014; Moreira et al. 2012; Wang et al. 2017. Shenzhen chest X-ray, INbreast and ChestX-ray14, used by MeLo.
Yang et al. 2023. MedMNIST v2: a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data.
Tschandl, Rosendahl & Kittler 2018. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data.

Visual PromptTuning

Pretrain one big model once, then adapt it to each task

Full fine-tuning stores a new copy of the model per task

Cheaper methods existed, but none matched full fine-tuning

Where the word "prompt" comes from

A visual prompt is the same idea applied to image patches

How a Vision Transformer reads an image

VPT trains a few extra input vectors and the head

Why a few extra tokens can steer a frozen network

How much VPT trains and stores per task

Numbers trained and stored per task

Compute per image

VPT in code: a few lines and one new hyperparameter

The paper tests VPT on 24 very different tasks

FGVC, 5 tasks

VTAB Natural, 7 tasks

VTAB Specialized, 4 tasks

VTAB Structured, 8 tasks

VPT-deep beats full fine-tuning on 20 of 24 tasks

VPT leads with little data and stays ahead with more

The result holds for larger backbones too

Four places to put the prompt

How many prompts, and at which layers

With prompts, the same frozen network separates the classes

Where VPT falls short

VPT saves storage, not training memory or inference time

Low-RankAdaptation

From VPT to LoRA

Every fine-tuned copy is as big as the model

Earlier cheap methods cost time or sequence length

What a low-rank matrix is

LoRA adds a trainable bypass to a frozen weight

LoRA on a 3 × 3 weight matrix

Where LoRA goes in a Vision Transformer

VPT and LoRA side by side

How much LoRA trains and stores per task

Numbers trained and stored per task

Compute per image at test time

LoRA's saving grows with the width of the model

LoRA on a ViT in a few lines

Results in the paper: language models

What happens when more weights are trained

Which weight matrices to adapt

How small the rank can be

Why a small rank is enough

LoRA on images

LoRA, VPT and adapters on the same ViT

Neither LoRA nor VPT wins every task

LoRA on medical images

Where LoRA falls short

The rank and the weights are chosen by hand

Merged weights serve one task at a time

A small rank assumes a nearby task

Training still runs through the whole model

My experiments

I tested VPT and LoRA on three medical datasets

Skin lesions

Blood cells

Chest X-rays

Linear probe

VPT-deep

LoRA

Full fine-tuning

VPT and LoRA on my three datasets

What each method costs on skin lesions

Live demo: one image, four models

Linear probe

VPT-deep

LoRA

Full fine-tuning

Summary

References

The two papers

Prompting before VPT

Ideas LoRA builds on

Other parameter-efficient methods

LoRA on images

References, continued

Models

Datasets and benchmarks

Visual Prompt
Tuning

Low-Rank
Adaptation