BEiT: BERT Pre-Training of Image Transformers, arxiv

PaddlePaddle training/validation code and pretrained models for BEiT.

The official and 3rd party pytorch implementation are here.

This implementation is developed by PaddleViT.

BEiT Model Overview

Update

• Update (2021-10-19): Bug fix and weights links are updated.
• Update (2021-09-27): Code is released and ported weights are uploaded.

Models Zoo

Model	Acc@1	Acc@5	#Params	FLOPs	Image Size	Crop_pct	Interpolation	Link
beit_base_patch16_224	85.21	97.66	87M	12.7G	224	0.9	bicubic	google/baidu(fshn)
beit_base_patch16_384	86.81	98.14	87M	37.3G	384	1.0	bicubic	google/baidu(arvc)
beit_large_patch16_224	87.48	98.30	304M	45.0G	224	0.9	bicubic	google/baidu(2ya2)
beit_large_patch16_384	88.40	98.60	304M	131.7G	384	1.0	bicubic	google/baidu(qtrn)
beit_large_patch16_512	88.60	98.66	304M	234.0G	512	1.0	bicubic	google/baidu(567v)

*The results are evaluated on ImageNet2012 validation set.

*These models have been fine-tuned (ImageNet 22k -> 1k)

Note: BEiT weights are ported from here

Notebooks

We provide a few notebooks in aistudio to help you get started:

*(coming soon)*

Requirements

• Python>=3.6
• yaml>=0.2.5
• PaddlePaddle>=2.1.0
• yacs>=0.1.8

Data

ImageNet2012 dataset is used in the following folder structure:

│imagenet/
├──train/
│  ├── n01440764
│  │   ├── n01440764_10026.JPEG
│  │   ├── n01440764_10027.JPEG
│  │   ├── ......
│  ├── ......
├──val/
│  ├── n01440764
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── ......

Usage

To use the model with pretrained weights, download the .pdparam weight file and change related file paths in the following python scripts. The model config files are located in ./configs/.

For example, assume the downloaded weight file is stored in ./beit_base_patch16_224_ft22kto1k.pdparams, to use the beit_base_patch16_224_ft22kto1k model in python:

from config import get_config
from beit import build_beit as build_model
# config files in ./configs/
config = get_config('./configs/beit_base_patch16_224.yaml')
# build model
model = build_model(config)
# load pretrained weights, .pdparams is NOT needed
model_state_dict = paddle.load('./beit_base_patch16_224_ft22kto1k')
model.set_dict(model_state_dict)

Evaluation

To evaluate BEiT model performance on ImageNet2012 with a single GPU, run the following script using command line:

sh run_eval.sh

or

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
    -cfg='./configs/beit_base_patch16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./beit_base_patch16_224_ft22kto1k'

Run evaluation using multi-GPUs:

sh run_eval_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg='./configs/beit_base_patch16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./beit_base_patch16_224_ft22kto1k'

Training

To train the BEiT Transformer model on ImageNet2012 with single GPUs, run the following script using command line:

sh run_train.sh

or

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
  -cfg='./configs/beit_base_patch16_224.yaml' \
  -dataset='imagenet2012' \
  -batch_size=32 \
  -data_path='/dataset/imagenet' \

Run training using multi-GPUs:

sh run_train_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg='./configs/beit_base_patch16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \ 

Visualization Attention Map

(coming soon)

Reference

@article{beit,
      title={{BEiT}: {BERT} Pre-Training of Image Transformers}, 
      author={Hangbo Bao and Li Dong and Furu Wei},
      year={2021},
      eprint={2106.08254},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}