Visual Prompt
Tuning

Teach a frozen Vision Transformer a new task by learning only a few extra input vectors.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim
Cornell University, Meta AI and University of Copenhagen. ECCV 2022.

Presented by Ertugrul Senturk
October 2026

Pretrain one big model once, then adapt it to each task

ImageNet-21k14 million photos in 21,000 everyday categories. Pretraining takes weeks and is done once, by Google.

ViT-B/16A Vision Transformer with 85.8 million weights that has learned general visual features.

Fine-grained categories200 bird species, 196 car models

Satellite and microscopeland use from above, tissue slides

Counting and distancehow many objects, how far away

Medical imagesmy own experiments, shown later

Each new task is small: most tasks in the paper have only 1,000 training images, far too few to train from scratch. So we start from the pretrained model and adapt it.

The paper asks how to adapt it best: accurately, and cheaply enough to store for every task. Next, the standard answer and what it costs, then the cheaper methods tried before VPT.

Dataset examples from Jia et al. 2022, Fig. 10. ViT-B/16: Dosovitskiy et al. 2021. The pretrained checkpoint used here and in the paper is Google's ImageNet-21k ViT-B/16.

Full fine-tuning stores a new copy of the model per task

Full fine-tuning retrains every weight of the pretrained model on the new task, and it is usually the most accurate option. The cost is that each task keeps its own retrained copy, so storage grows with every task and with the size of the model.

Number of tasks1

Model size

343 MB

345 MB

32-bit weights: ViT-B/16 85.8M, ViT-L/16 303M, ViT-H/14 632M. A VPT file is 50 prompts at every layer plus a 102-class head. Squares are sized by area.

Cheaper methods existed, but none matched full fine-tuning

Bias came closest and even beat full fine-tuning on 8 of the 24 tasks, but on average no cheaper method reached it. VPT takes a third route: it leaves the model alone and changes the input.

Mean test accuracy over the paper's 24 tasks, ViT-B/16 (Jia et al. 2022, Table 1, mean of the four group averages). The right column is the share of weights trained for each task.

Where the word "prompt" comes from

2020GPT-3

Classifythesentiment:prompt: words a person chose

thefilmwasgreatinput

→

frozen language model

→

positive

2021Prompt tuning

prompt: learned vectors, no words

thefilmwasgreatinput

→

frozen language model

→

positive

In 2020, a prompt was text. GPT-3, a frozen model with 175 billion weights, was only trained to continue text, yet it does a new task when you type a description of the task in front of the input. Not a single weight changes. But words are discrete, so the only way to improve a prompt is trial and error.

In 2021, prompt tuning (Lester et al.) replaced the words with learned vectors that correspond to no word. They are trained by gradient descent on the task's examples while the model stays frozen, and they are all you store per task. Prefix-tuning (Li & Liang) does the same at every layer.

The sentiment example is illustrative. Brown et al. 2020 (GPT-3); Lester, Al-Rfou, Constant 2021 (prompt tuning, T5); Li & Liang 2021 (prefix-tuning).

How a Vision Transformer reads an image

A fixed grid cuts the image into 196 patches of 16 × 16 pixels, and each patch, read row by row, becomes a vector of 768 numbers. One extra vector, [CLS], goes in front to collect a summary of the whole image: the 12 layers mix all 197 vectors, and the head reads only [CLS] to name the class.

The layers only see a sequence of vectors, so we can add a few more.

ViT-B/16: 12 layers, vectors of 768 numbers, 85.8 million weights. Dosovitskiy et al., ICLR 2021. Image: a melanoma from DermaMNIST, 224 × 224 pixels.

Spend a minute here, it pays off. Walk the picture from left to right and up: a fixed grid cuts the image into 14 by 14 patches, whatever the image shows; nothing is selected, all 196 go in, read row by row from patch 1 at the top left to 196 at the bottom right. Each patch, 16 × 16 × 3 = 768 pixel values, goes through one linear layer to become a grey vector of 768 numbers (the two 768s are a coincidence of ViT-B), with a position vector added so the model knows where it came from. In front sits one extra learned vector, [CLS], which carries no image content and collects a summary as it goes up. The 12 Transformer layers, blue because they will stay frozen in VPT, let every vector look at every other one. At the top, only the [CLS] vector goes to the head, the one task-specific part, which outputs melanoma. The point for VPT: the Transformer does not care how long the sequence is or where a vector came from, so we can add our own vectors.

A visual prompt is the same idea applied to image patches

LanguageLester et al. 2021

thefilmwasgreat

→

frozen language model

→

positive

learned prompt: random at first, trained for the task, same for every input

↑“the film was great”

VisionJia et al. 2022

…

→

frozen Vision Transformer

→

melanoma

learned prompt: random at first, trained for the task, same for every image

↑

The dermoscopy image with its grid of 16 by 16 pixel patches

all 196 patches
of the image

The grey vectors come from the input, as before. The red ones do not: they start as random numbers, are trained on the task's examples while the model stays frozen, and are then put in front of every input of that task. A visual prompt is therefore not text and not a pattern painted on the pixels, only learned numbers in the model's input space.

In language, prompt tuning matched full fine-tuning only once the model had about 10 billion weights. VPT beats full fine-tuning on 20 of 24 tasks with an 86-million-weight ViT-B.

Lester et al. 2021 (T5); Jia et al. 2022, Table 1. Not to be confused with CoOp (Zhou et al. 2021, text prompts for CLIP) or Bahng et al. 2022 (a pixel pattern).

The one-sentence version: VPT is prompt tuning for a Vision Transformer. Both rows learn vectors in the model's input space and leave the model alone; the only difference is where the other vectors come from, words or image patches. Say plainly what it is not: not the text prompts for CLIP that CoOp learns, and not a learned pattern painted on the pixels as in Bahng et al.; the paper tests the pixel version too and it is clearly weaker. Then the surprise: in language, learned prompts needed about 10 billion weights to match full fine-tuning; VPT beats it on 20 of 24 tasks with 86 million. The authors themselves suggest vision prompts may behave differently from language prompts. Point at the labels under each row: the grey vectors come from the input, the red ones never do; they start random, are trained on the task, and the same set goes in front of every image of that task.

VPT trains a few extra input vectors and the head

Figures: Jia et al. 2022, Fig. 2. Equations (6) and (8) of the paper: x class token, E patch vectors, P prompts, Z the prompt positions after a layer, L_i layer i.

Why a few extra tokens can steer a frozen network

In every layer, each token takes in information from all the other tokens; this is attention. With prompts in the sequence, [CLS] and every patch also take in information from the prompts.

So training the prompts changes what the frozen network computes, without changing a single weight.

Mechanism: Sec. 3 of Jia et al. 2022; Appendix B relates it to prefix-tuning ("VPT-prefix"). Line widths and the red share of [CLS] are illustrative, not measured.

How much VPT trains and stores per task

N · p · d = 12 × 50 × 768 = 460,800prompt numbers, plus 78,438 for a 102-class head

Numbers trained and stored per task

Full fine-tuning

all 85.8 million weights, retrained for this task

343 MB100%

VPT-deep

the same 85.8 million weights, frozen and shared by every task

2.2 MB0.63%

Compute per image

Plain ViT

197 tokens

1.00×

VPT-deep

197 tokens

1.27×+50 prompt tokens

Prompt length p

More prompts keep the file small, but every image carries them through all 12 layers.

ViT-B/16, N = 12 layers, d = 768, 32-bit numbers. Compute counts multiply-adds per layer for n tokens (12nd² + 2n²d). The paper measures about 2× test latency at p = 200 (Jia et al. 2022, Table 12).

VPT in code: a few lines and one new hyperparameter

# VPT-deep on a frozen ViT-B/16, simplified from demo/vpt/vpt_model.pyvit = timm.create_model(BACKBONE, pretrained=True)                # ImageNet-21k ViT-B/16vit.requires_grad_(False)                                         # backbone frozenprompts = nn.Parameter(torch.empty(12, p, 768).uniform_(-b, b))   # trained: 12 layers × p × 768head = nn.Linear(768, num_classes)                                # trained: the classifier def forward(img):    x = vit._pos_embed(vit.patch_embed(img))                      # [CLS] + 196 patch vectors    for i, layer in enumerate(vit.blocks):        rest = x[:, 1:] if i == 0 else x[:, 1 + p:]               # drop last layer's prompts        P = prompts[i].expand(len(x), -1, -1)                     # this layer's prompts        x = layer(torch.cat([x[:, :1], P, rest], dim=1))          # frozen layer on [CLS, P, rest]    return head(vit.norm(x)[:, 0])                                # classify from [CLS]

The prompts start random, with Xavier-uniform values as in the official code. The paper tried smarter starts, such as class prototypes, and random did as well or better.

Training is plain SGD with momentum for 100 epochs on a cosine schedule. Learning rate and weight decay are picked on a validation split, for every method alike.

The one new hyperparameter is the prompt length p, chosen per task from {1, 5, 10, 50, 100, 200} on the validation split.

Recipe: Jia et al. 2022, Sec. 4.1 and Appendix A. Code is my re-implementation, which reproduces the paper's Oxford Flowers result (99.1% vs the paper's 99.0%).

The paper tests VPT on 24 very different tasks

Frangipani flowers, from Oxford Flowers — Oxford Flowers

A blue car, from Stanford Cars — Stanford Cars

A Boston terrier, from Stanford Dogs — Stanford Dogs

FGVC, 5 tasks

Fine-grained categories, such as 200 bird species, with thousands of training images.

A Persian cat, from Oxford Pets — Oxford Pets

VTAB Natural, 7 tasks

Ordinary photos of objects, textures, animals and scenes, close to the pretraining data.

Farmland seen from a satellite, from EuroSAT — EuroSAT

An aerial photo of a building, from Resisc45 — Resisc45

A stained tissue slide, from Patch Camelyon — Patch Camelyon

A retina scan, from Retinopathy — Retinopathy

VTAB Specialized, 4 tasks

Images from special sensors: satellites, tissue slides under a microscope, and retina cameras.

Rendered 3D objects to count, from CLEVR — CLEVR count

A white shape on black, from dSprites — dSprites location

A street seen from a car, from KITTI — KITTI distance

A toy plane at an angle, from SmallNORB — SmallNORB azimuth

VTAB Structured, 8 tasks

Geometry instead of categories, such as counting objects or judging distance.

All 24 tasks use the same frozen ViT-B/16, pretrained on ImageNet-21k. The 19 VTAB tasks have only 1,000 training images each.

Images: Jia et al. 2022, Fig. 10, four example tasks per group. VTAB-1k: Zhai et al. 2019.

VPT-deep beats full fine-tuning on 20 of 24 tasks

Jia et al. 2022, Table 1 and Tables 13–14: ViT-B/16 pretrained on ImageNet-21k, mean test accuracy within each group. For all 24 tasks, the mean of the four group averages.

Step through the four groups with the arrow keys, then the summary. The chart keeps the families from the cheaper-methods slide: full fine-tuning on top with the dashed line at its score, VPT in red, the older cheap methods in grey. The right column counts the tasks on which a method beats full fine-tuning. FGVC: everyone is strong, VPT-deep edges ahead, 89.1 vs 88.5, and loses only Stanford Cars. Natural: only 1,000 training images, so full fine-tuning overfits; VPT-deep wins 6 of 7, losing SVHN, and even VPT-shallow is above full. Specialized: the one group VPT-deep loses on average, 82.4 vs 83.4, losing Resisc45 and Retinopathy; medical images come back in my own results. Structured: the biggest gap, 55.0 vs 47.6, all 8 tasks won; the linear probe collapses to 26.8, so the frozen features alone do not hold this information, and the prompts make the frozen network compute it. All 24: VPT-deep is the only cheap method above full fine-tuning, 76.3 vs 73.9, with 20 wins, and its total storage for 24 tasks is 1.18 backbones against 24.02.

VPT leads with little data and stays ahead with more

VPT-deep against

Jia et al. 2022, Fig. 3: ViT-B/16, mean test accuracy over the five FGVC tasks, each method trained on a share of every task's training images. The paper gives these values only as a figure; they were read from its vector drawing in the PDF, and the 100% points match Table 1.

This is the paper's Fig. 3, redrawn one comparison at a time. Each point is the average test accuracy over the five FGVC tasks when the method is trained on 10% up to 100% of every task's training images; the axis is logarithmic, as in the paper. Start with full fine-tuning: with 10% of the images it falls to 54.6 against 68.5 for VPT-deep, because it has 86 million weights to fit with very few examples. With all the data the two almost meet, 89.1 against 88.5, but VPT-deep is never below. Then step through the baselines. The linear probe is the classic low-data trick: good at 10%, overtaken by full fine-tuning from 30% on. The adapter is overtaken from 40% on. Bias is the strongest rival and beats full fine-tuning up to 80% of the data, but VPT-deep is above it at every point. So VPT is not just a low-data trick. VPT-shallow is left out to keep the chart readable; it always sits between the linear probe and VPT-deep. If asked: the paper publishes these numbers only as a figure; I read them from the figure's vector data in the PDF, and the 100% points match Table 1 to within 0.01.

The result holds for larger backbones too

VTAB group

Jia et al. 2022, Fig. 4: VTAB-1k, mean test accuracy per group, backbones pretrained on ImageNet-21k. The paper gives these values only as a figure; they were read from its vector drawing in the PDF, and the ViT-B points match Table 1.

The paper's Fig. 4, redrawn one group at a time: the same VTAB-1k tasks with three backbones, ViT-Base, ViT-Large and ViT-Huge, all pretrained on ImageNet-21k; the horizontal axis is the number of weights. Natural: VPT-deep wins with every backbone, and the gap grows from 2.6 points to about 7, because full fine-tuning gets worse as the model grows, from 75.9 to 70.9, with only 1,000 training images per task. Specialized: a tie, within one point everywhere, the same picture as on ViT-Base. Structured: VPT-deep leads by 6 to 7 points with every backbone, and the linear probe stays near 26, so the frozen features alone do not solve these tasks at any size. Why this matters: the storage argument grows with the model. A fully fine-tuned ViT-Huge is about 2.5 GB per task, while 50 prompts at each of its 32 layers take about 8 MB. It does not hold for every backbone, though, as we will see shortly.

Four places to put the prompt

Numbers from Jia et al. 2022, Fig. 5 (ViT-B/16, prompts in the first layer only); "All four" is the mean of the three groups. Diagrams redrawn.

The question: where should the learned numbers enter the network? Two places in principle: in the embedding space, as vectors next to the patch vectors, or in pixel space, as part of the image before the embedding layer. Prepend is the default we have seen: extra tokens after the embedding. Add: the prompts are added onto patch vectors, so the sequence is not longer; it loses in all three groups, 8 points on Structured, and the deep version loses too (43.2 against 55.0 on Structured). Prepend-pixel: learned pixel patches next to the image patches, then the frozen embedding layer; loses everywhere, 5 points on Natural. Concat-channel: a learned fourth channel on the image, the embedding layer trained to read it; it collapses, 30 points lower on Natural and below even a linear probe there (68.9). Summary: separate tokens in the embedding space are the right place. All numbers are VPT-shallow because that is the only setting where the paper tests all four options.

How many prompts, and at which layers

First, the number of prompts, the one new hyperparameter. Each panel is one VTAB group: the red line is VPT-deep with 1 to 200 prompts at every layer, the dashed line is full fine-tuning, the grey line the best adapter. The best number is different in every group, 10 on Natural, 5 on Specialized, 200 on Structured, and per task it varies even more, from 5 prompts for Oxford Flowers to 200 for Stanford Cars (Table 14). That is why the paper picks p per task on the validation set. Even a single prompt per layer beats the adapter everywhere, and on Natural it already beats full fine-tuning. Press the arrow key for which layers. The red line adds prompts from the input upwards: layer 1 alone is VPT-shallow, all twelve layers is VPT-deep. The grey line adds them from the output downwards. More prompted layers generally help, and the input side wins: on Structured, prompts in layers 1 to 3 give 50.6, prompts in layers 9 to 12 only 37.6. The authors point out that this is the opposite of what was seen with text prompts. The paper has no table for these two experiments, so the values are read off the plotted points of its figures; the end points match VPT-shallow and VPT-deep in Table 13.

With prompts, the same frozen network separates the classes

Linear probe

frozen network, no prompts

VPT-deep

same frozen network, with prompts

Full fine-tuning

every weight retrained

Each dot is one test image, coloured by its true class; the number is test accuracy. With the frozen features alone, the classes overlap. With prompts, the same frozen network produces clearly separated groups, so a simple linear head can tell them apart.

t-SNE maps of the final [CLS] vector, test set, from Jia et al. 2022, Fig. 9. Accuracies from Fig. 9 and Table 13.

This is the payoff of the "why it can work" slide: same weights, different input, completely different output. Each map is a 2-D t-SNE picture of the 768-number [CLS] vector the head receives. House numbers: the frozen features are one mixed blob, 36.6%; with prompts the ten digits form islands, 78.1%. Full fine-tuning is still better here, 87.4%, so be honest about that. Press the arrow key for satellite images: the frozen features are already half sorted, 87.5%, and VPT-deep gives clean clusters, 96.1%, slightly above full fine-tuning at 95.7%. Press again for counting objects in CLEVR: the frozen features know nothing about the count, 34.3%; VPT-deep reaches 68.5%, well above full fine-tuning at 56.3%, and arranges the images along a band that follows the object count. If asked: VPT-shallow sits in between (74.5, 92.0, 50.5); it is in the paper's figure.

Where VPT falls short

Be honest here; it is what a reading group is for. Three settings where the main result does not carry over. First, the pretraining. Everything so far used a ViT-B pretrained with labels on ImageNet-21k. The paper repeats VTAB-1k with the same architecture pretrained without labels on ImageNet-1k, by MoCo v3 (contrastive) and by MAE (masked autoencoder). The chart shows the mean of the three VTAB group averages. With labels, VPT-deep leads full fine-tuning, 72.0 against 69.0. With MoCo v3 it is 65.2 against 69.6, and Bias (69.2), Partial-1 (68.3) and the adapter (68.2) are all ahead of it. With MAE it collapses to 41.1, closer to the linear probe at 32.1 than to full fine-tuning at 64.3, while Partial-1, which retrains only the last block, reaches 61.5. The authors only say these self-supervised ViTs are fundamentally different and leave the reason open. It connects to the MAE paper from last time: MAE features are known to probe badly, and a few input tokens apparently cannot reorganise them the way retraining the last block can. Press the arrow key for Swin, a hierarchical Transformer that attends within local windows and merges patches in deeper layers; the prompts sit in the windows and are dropped at patch merging. VPT still beats the other cheap methods (Bias, linear probe, Partial, MLP-3) in every group, but full fine-tuning is ahead of VPT-deep in all three, by 6.3 points on Structured. Oddly, VPT-shallow does slightly better than full fine-tuning on Natural, 79.9 against 79.1; it is left out of the chart to keep it simple. Press again for segmentation: ADE20K, 150 classes per pixel, SETR on a ViT-L pretrained on ImageNet-21k; mIoU is the mean overlap between predicted and true regions. The decoder head is trained in every method, which is why even "head only" trains 13 million weights. VPT-deep gets 42.1 against 48.3, a little below Bias at 43.4; prompts plus biases give 44.0, within 1.5 of a fully fine-tuned ResNet-101 with four times the trained weights. This is also where LoRA will come in: it edits weights, so it does not depend on the input space being "steerable".

VPT saves storage, not training memory or inference time

Easy to miss in the main paper: the saving is disk, not GPU. Three arrow-key steps, same rows each time: full fine-tuning in dark with the dashed line at its value, the cheap baselines in grey, VPT-deep in red with 1, 50, 100 and 200 prompts per layer. Storage: a task keeps only what it trained, so VPT-deep is 0.35 MB with one prompt per layer and 7.7 MB with 200, against 343 MB for a full copy; bias and adapter are as small, so storage is what all these methods buy, and it is VPT's real win. Training memory: no saving. The prompts in the first layer only get a gradient if it travels back through all twelve frozen layers, so their activations are kept, as in full fine-tuning; with 200 prompts the longer sequence makes it 25.8 GB against 11.7. Only the linear probe, 0.9 GB, avoids this. Training time per image is not shown: one prompt trains faster than full fine-tuning, 214 against 359 ms, but 200 prompts are as slow, 360 ms. Inference time: every test image carries the prompts through all twelve layers. Fifty prompts take the sequence from 197 to 247 tokens and the time from 70 to 82 ms; 200 prompts double it, 141 ms. The paper notes this is far from the 4x one would expect from quadratic attention. The prompts cannot be folded into the weights, so the cost stays; the paper's prefix variant, which puts the prompts only into the keys and values, recovers some of it (Fig. 19). LoRA, next time, merges its update into the weights and costs nothing at test time. If asked: the table has only 1 and 200 prompts; the 50 and 100 values are read off Fig. 18's plotted points, and the 1 and 200 points of that figure match the table. If someone asks "why not just use this for everything", this is the answer.

My experiments

I re-implemented VPT and trained it on three medical imaging datasets from MedMNIST. A linear probe and full fine-tuning ran next to it, all three starting from the paper's ViT-B/16.

First the results, then all three models running live in this browser.

I tested VPT on three medical datasets

Each one was trained twice: on 1,000 images, as in the paper's VTAB-1k, and on its full training set.

Dermoscopy image of a melanoma — melanoma

Skin lesions

Dermoscopy, 7 classes

1,000 or all 7,007 training images

Microscopy image of a neutrophil — neutrophil

Microscopy image of a platelet — platelet

Blood cells

Microscopy, 8 cell types

1,000 or all 11,959 training images

Chest X-rays

Radiographs, pneumonia or normal

1,000 or all 4,708 training images

head

ViT-B/16 frozen

Linear probe

Trains only a new head.

head

ViT-B/16 frozen

VPT-deep

Trains a head and 50 prompts per layer.

head

ViT-B/16 retrained

Full fine-tuning

Retrains all 85.8 million weights.

MedMNIST+ at 224 × 224 (Yang et al. 2023); skin lesions from HAM10000 (Tschandl et al. 2018). Trained on an RTX 5070 Ti laptop GPU with the paper's recipe: SGD for linear probe and VPT, AdamW for full fine-tuning, learning rate picked on validation.

Three public datasets from MedMNIST+, at 224 × 224 pixels, all far from ImageNet photos. Skin lesions are heavily imbalanced, two thirds are nevi, so I report balanced accuracy, the mean per-class recall: always answering "nevus" would score 67% plain accuracy. Each task was trained twice, on 1,000 random training images, the size of the paper's VTAB-1k, and on the whole training split; the bars show how small 1,000 is next to each full set. Next step: the three methods. All start from the same ViT-B/16 ImageNet-21k checkpoint the paper uses. The linear probe and VPT keep it frozen and train only the red parts; full fine-tuning retrains every weight, so each task needs its own 343 MB copy. Learning rates were picked on the validation split, and the reported model is the one with the best validation balanced accuracy, the same rule for all three methods.

VPT always beat the linear probe and came close to full fine-tuning

My runs: ViT-B/16 (ImageNet-21k), MedMNIST+ 224 × 224, balanced accuracy on the test set (the mean per-class recall: on skin lesions, always answering "nevus" scores 67% accuracy but only 14% balanced accuracy), bars start at 0. Each method was trained once with 1,000 images and once with the full training set; there are no points in between.

Step through the three datasets with the arrow keys. Each tab has only two measured settings, 1,000 training images and the full training set, so they are drawn as two separate groups of bars, not as a line. The score is balanced accuracy on the test set, the mean recall over classes, because skin lesions are so imbalanced. Skin lesions: VPT is 5 points above the linear probe with 1,000 images and 13 points above with all 7,007, because the prompts can reshape the features and a linear head cannot; full fine-tuning stays 2 to 3 points ahead. Blood cells: everything is above 96, and VPT and full fine-tuning are within a few tenths of a point of each other. Chest X-rays: with 1,000 images VPT barely beats the linear probe and trails full fine-tuning by almost 4 points; with all 4,708 it comes out on top. So across all six settings VPT never loses to the linear probe and is never more than 4 points behind full fine-tuning. Unlike the paper, the low-data setting did not favour VPT here; medical images behave like the paper's Specialized group.

What each method costs on skin lesions

Three steps, one cost each, all measured on my laptop for skin lesions trained on the full training split; every number on the slide is read from the training logs and the benchmark file. Storage first: this is the paper's argument, and it holds. Linear probe and VPT keep one copy of the base model and add a small file per task, kilobytes or about 2 MB, while full fine-tuning needs a whole new 343 MB model per task; the accuracy panel is the price, a few points below full fine-tuning but far above the linear probe. Training: VPT is cheaper than full fine-tuning but much more expensive than the linear probe, because the error signal has to travel back through all twelve frozen layers to reach the prompts, so the activations must be kept. That is the same pattern as the paper's Table 12 on an A100. Inference: VPT is the only method that is slower than the original model, because every layer processes 50 extra tokens, and unlike a weight update the prompts cannot be merged into the model after training. Linear probe and full fine-tuning run the same network, so their small difference is noise. Accuracy for all three tasks and both data sizes is on the previous slide.

Live demo: one image, three models

Pick a test image below,
or drop your own here

Linear probe

shared frozen model + head

VPT-deep

shared frozen model + prompts

Full fine-tuning

its own retrained model

All three start from the same ViT-B/16, pretrained on ImageNet-21k. Linear probe and VPT-deep share one frozen copy of it; full fine-tuning runs its own retrained copy.

Not connectedResearch demo, not a diagnostic tool

Every click runs the image through all three models right here in the browser, with onnxruntime-web. Linear probe and VPT-deep use the very same frozen copy of the base model in memory; full fine-tuning retrained every weight, so it is a separate model per task. The percentage under each name is that model's balanced accuracy on the whole test set. Now switch to 1,000 training images: most accuracies drop, by far the most on skin lesions. For linear probe and VPT only a small head and a 1.9 MB prompt file change, so it is instant. Full fine-tuning has to download and load a second whole model, 172 MB here because the browser copies use 16-bit weights (343 MB at 32-bit). That is the storage argument, live. The thumbnails are random test images, not picked to look good, so a wrong answer can happen; say so. Research demo, not a diagnostic tool.

Summary

0.6%

of the weights are trained: a few vectors among the patch tokens and a small head, while the Transformer stays frozen.

20of 24

tasks where VPT-deep beat full fine-tuning, on a ViT-B pretrained with labels.

99.4%

less storage per task than a fully fine-tuned copy: 2.2 MB instead of 343 MB. Training memory and inference time are not saved.

–

average balanced accuracy on my three medical datasets.

Jia et al. 2022, Table 1 (24 tasks, ViT-B/16) and Table 12 (memory, test time); 0.6% is VPT-deep with 50 prompts and a 102-class head. Medical datasets: my runs, balanced accuracy on the test set.

This is the slide to leave up while summing up and taking questions: four points, each with the one number to remember. First, what a visual prompt is. Not text and not pixels, but learned vectors among the patch tokens of a frozen ViT; with 50 prompts at every layer and the head that is about 0.6% of the weights. Second, the headline: on a ViT-B pretrained with labels, VPT-deep beat full fine-tuning on 20 of the 24 tasks, and all 24 tasks together cost 1.18 models of storage instead of 24. Third, the limits: it saves storage only, 99.4% less per task than a fully fine-tuned copy. Training memory is about the same as full fine-tuning, inference is slower because the sequence is longer, and on self-supervised backbones it loses, 23 points behind on MAE. That is the opening for LoRA next time. Fourth, my own runs: averaged over the three medical datasets and both training-set sizes, VPT-deep is about a point behind full fine-tuning and clearly ahead of the linear probe, as the demo showed.

References

This paper

Jia, Tang, Chen, Cardie, Belongie, Hariharan & Lim 2022. Visual prompt tuning. ECCV. arXiv:2203.12119.

Prompting before VPT

Brown et al. 2020. Language models are few-shot learners (GPT-3). NeurIPS.
Lester, Al-Rfou & Constant 2021. The power of scale for parameter-efficient prompt tuning. EMNLP.
Li & Liang 2021. Prefix-tuning: optimizing continuous prompts for generation. ACL.
Zhou et al. 2021. Learning to prompt for vision-language models (CoOp). arXiv:2109.01134; IJCV 2022.
Bahng et al. 2022. Exploring visual prompts for adapting large-scale models. arXiv:2203.17274.

Parameter-efficient methods

Houlsby et al. 2019. Parameter-efficient transfer learning for NLP (Adapter). ICML.
Ben Zaken, Goldberg & Ravfogel 2022. BitFit: simple parameter-efficient fine-tuning for Transformer-based masked language-models (Bias). ACL.
Zhang et al. 2020. Side-tuning: a baseline for network adaptation via additive side networks (Sidetune). ECCV.
Hu et al. 2022. LoRA: low-rank adaptation of large language models. ICLR.

Models

Dosovitskiy et al. 2021. An image is worth 16×16 words: Transformers for image recognition at scale (ViT). ICLR.
Liu et al. 2021. Swin Transformer: hierarchical vision Transformer using shifted windows. ICCV.
Chen, Xie & He 2021. An empirical study of training self-supervised vision Transformers (MoCo v3). ICCV.
He et al. 2022. Masked autoencoders are scalable vision learners (MAE). CVPR.
Zheng et al. 2021. Rethinking semantic segmentation from a sequence-to-sequence perspective with Transformers (SETR). CVPR.
Chen et al. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation (DeepLab v3+). ECCV.

Datasets

Zhai et al. 2019. A large-scale study of representation learning with the visual task adaptation benchmark (VTAB). arXiv:1910.04867.
Zhou et al. 2019. Semantic understanding of scenes through the ADE20K dataset. IJCV.
Yang et al. 2023. MedMNIST v2: a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data.
Tschandl, Rosendahl & Kittler 2018. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data.

Visual PromptTuning

Pretrain one big model once, then adapt it to each task

Full fine-tuning stores a new copy of the model per task

Cheaper methods existed, but none matched full fine-tuning

Where the word "prompt" comes from

How a Vision Transformer reads an image

A visual prompt is the same idea applied to image patches

VPT trains a few extra input vectors and the head

Why a few extra tokens can steer a frozen network

How much VPT trains and stores per task

Numbers trained and stored per task

Compute per image

VPT in code: a few lines and one new hyperparameter

The paper tests VPT on 24 very different tasks

FGVC, 5 tasks

VTAB Natural, 7 tasks

VTAB Specialized, 4 tasks

VTAB Structured, 8 tasks

VPT-deep beats full fine-tuning on 20 of 24 tasks

VPT leads with little data and stays ahead with more

The result holds for larger backbones too

Four places to put the prompt

How many prompts, and at which layers

With prompts, the same frozen network separates the classes

Where VPT falls short

VPT saves storage, not training memory or inference time

My experiments

I tested VPT on three medical datasets

Skin lesions

Blood cells

Chest X-rays

Linear probe

VPT-deep

Full fine-tuning

VPT always beat the linear probe and came close to full fine-tuning

What each method costs on skin lesions

Live demo: one image, three models

Linear probe

VPT-deep

Full fine-tuning

Summary

References

This paper

Prompting before VPT

Parameter-efficient methods

Models

Datasets

Visual Prompt
Tuning