Labels are the bottleneck

Each square is 1.28 million labeled images. Teal: ImageNet-1K, public. Gray: JFT-300M, Google-internal, used to pretrain ViT.

ViT-Large is big enough to memorize ImageNet. Trained from scratch on it, it overfits and lands below a ResNet. It only works well after pretraining on JFT-300M, which nobody else has. The question is whether ImageNet alone can be enough, without any extra labels.

Hide part of the data, predict it

No one writes a label for a hidden word. Predicting it turns out to require understanding the sentence, and that understanding transfers to every downstream task. GPT and BERT scale to hundreds of billions of parameters this way.

The image is split into a 14 × 14 grid, 196 patches. 147 of them are removed at random. The encoder, a normal ViT, gets only the 49 that remain.

The decoder gets the encoder output plus a mask token at each of the 147 missing positions, and predicts the missing pixels. The pixel error is the only training signal.

After pretraining the decoder is dropped. The encoder is used on full images for recognition.

Two choices make this work: the encoder never sees a mask token, which keeps it cheap, and the masking ratio is high, which keeps the task hard.

The decoder's last layer outputs 16 × 16 × 3 values per token, which reshape into an image. The loss is the mean squared error against the original, computed on the masked patches only. Computing it on all patches costs about half a point of accuracy.

Normalizing each target patch by its own mean and standard deviation gives the best results in the paper.

Reconstructions are blurry, since squared error rewards the average, but they are structurally right. They stay plausible on COCO images the model never saw, which is the first sign that the encoder learned something general.

All at 75 percent: two ImageNet validation images (Figure 10) and a COCO image (Figure 3). Visible patches look worse in the output because no loss is applied to them; the authors show the raw output on purpose.

random 75%
84.9 · 73.5

block 50%
83.9 · 72.3

grid 75%
84.0 · 66.0

Figure 6 and Table 1f, numbers are fine-tuning · linear probing. Block masking at 75%: 82.8 · 63.9.

Block masking removes large regions. The task gets harder, the loss higher, the output blurrier, and at 75 percent it breaks down. Grid masking keeps one patch in four, which is easy: sharp reconstructions, worse features. Random sampling is the only one that tolerates a high ratio, and the high ratio is where the speed comes from.

Table 1e.

MAE barely needs augmentation. With no augmentation at all it loses less than a point in fine-tuning. Contrastive methods depend on it: SimCLR loses 28 points with cropping only, and with no augmentation its two views are identical and there is nothing to learn. In MAE, a new random mask every iteration already gives a new training sample from the same image.

The simplest choice is the pixels themselves, and that is what MAE does. The alternative, used by BEiT, a masked image method published a few months earlier, is to predict a code instead of pixels. A separate network first maps every patch to one of 8192 codes, a kind of visual vocabulary. The model then guesses the code of each hidden patch, the way BERT guesses a word. The argument was that codes are more meaningful than pixels.

On MAE, codes turn out slightly better than raw pixels, no better than normalized pixels, and worse for linear probing. Predicting a compressed version of the patch is worse than predicting the full patch, so the fine detail is useful too.

The code network also has to be pretrained on 250 million extra images and adds 40 percent to training compute. Pixels are enough.

Table 1d. Normalizing per patch means subtracting the patch's own mean and dividing by its standard deviation before comparing.

Detection is box AP, segmentation is mean IoU; higher is better for both. Base and Large are the ViT sizes. Standard detection and segmentation heads, all methods fine-tuned the same way. Tables 4 and 5.

The last two rows use the ImageNet-fine-tuned model as is, with no extra training. Tables 6 and 13.

Does the ImageNet gain carry over to other tasks? Yes, and it grows with model size. On detection, MAE is 2.4 points above training with labels for the base model and 4.0 for the large one. The contrastive method gives no gain at all here, even though it looked better under a linear probe. Segmentation shows the same pattern.

On fine-grained species classification the huge model beats the previous record by more than five points.

The robustness rows are the most striking. On the adversarial set, the huge MAE model scores 68.2 where the same model trained with labels scores 33.1.

What MAE does: hide 75 percent of the patches, run only the visible ones through the encoder, let a small decoder fill in the pixels, then throw the decoder away and keep the encoder.

Why it matters: it is simple and fast, it needs almost no augmentation, and it gets better as the model gets bigger and trains longer, which contrastive methods on ViT did not. With nothing but ImageNet, a huge ViT reaches 87.8 percent, and the gains carry over to detection and segmentation.

One more lesson: linear probing can mislead. MAE looks weak under a linear probe and strong as soon as a single block is tuned.

The patches we hide are not objects, and the pixels we predict mean nothing on their own. The encoder still ends up understanding the image, because there is no other way to fill in three quarters of it.

Paraphrasing the discussion section of the paper.