Skip to content

Latest commit

 

History

History
145 lines (104 loc) · 6.97 KB

tutorial.training-example.rst

File metadata and controls

145 lines (104 loc) · 6.97 KB

Example on Finetuning BLIP on COCO-Captioning

To finetune BLIP model on the coco caption dataset, first refer to :ref:`prep coco` to prepare the dataset if you have not done so.

To finetune the model, we have prepared a run script for you, which can run as follows:

bash run_scripts/blip/train/train_caption_coco_large.sh

This will finetune the pre-trained BLIP large model into a new model that can be used for captioning.

Deep Dive

Now let's take a closer look at the script and see what it does.

python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis/projects/blip/train/caption_coco_large_ft.yaml

As can be seen, the script simply calls the train.py with PyTorch distributed training enabled. The --cfg-path argument specifies the runtime config file to use. The config file is a YAML file that specifies the training parameters, shown as follows:

.. literalinclude:: ../lavis/projects/blip/train/caption_coco_large_ft.yaml
    :language: yaml
    :linenos:
The runtime config file is divided into 3 sections:
  • model: specifies the model architecture and type to use.
  • data: specifies the dataset to use.
  • run: specifies the runner arguments, such as tasks, optimizer, learning rate scheduler, etc.

We describe each section in detail below.

Model configurations

.. literalinclude:: ../lavis/projects/blip/train/caption_coco_large_ft.yaml
    :language: yaml
    :linenos:
    :lines: 6-10

The arch argument specifies the model architecture to use. In this case, we use the blip_caption architecture. You can find available architectures by inspecting the model_zoo. Once the architecture is specified, the runner will look for the model class registered with the name and try to instantiate a model instance. In this case BlipCaption is the model registered with the name blip_caption.

The registry maintains a mapping from the name string to the model class. This allows the runner to find the model class dynamically based on the name string from the config file. The following segment in lavis/models/blip_models/blip_caption.py shows how BlipCaption is registered with the name string blip_caption:

.. literalinclude:: ../lavis/models/blip_models/blip_caption.py
    :language: python
    :linenos:
    :lines: 20-38

One same model architecture may be pre-trained or finetuned on different datasets or have different model configurations. For example, BlipCaption have:

  • base_coco: pre-trained base BLIP model adapated for COCO captioning finetuning.
  • large_coco: pre-trained large BLIP model adapated for COCO captioning finetuning.

Therefore, we also need to specify model_type. Here we use large_coco. And we set load_finetuned to False to indicate that we are finetuning the model from the pre-trained weights. If load_finetuned set to True as by default, the model will load finetuned weights on coco captioning.

Given the model architecture and type, the library will then look for the default model config for large_coco in lavis/models/blip_models/blip_caption.py. As can be seen in the above code snippet, the corresponding config path is stored in BlipCaption.PRETRAINED_MODEL_CONFIG_DICT. Then the library will load lavis/configs/models/blip_caption_large_coco.yaml as the configuration to build the model.

Priority of Configs: Note that the priority of the run config is higher than the default model config, meaning that arguments in the run config will override the default model config. For example, in the default model config, load_finetuned is set to True by default, while in the run config, we set it to False and finetuning from the pre-trained weights only.

Dataset configurations

The second section of the config file specifies the dataset(s) to use.

.. literalinclude:: ../lavis/projects/blip/train/caption_coco_large_ft.yaml
    :language: yaml
    :linenos:
    :lines: 12-24

We associate each dataset with a vis_processor and a text_processor, responsible for processing the visual and textual input respectively. Here we again use the registry mechanism to dynamically load the processor class based on the name string. For example, blip_image_train is the name string for the BlipImageTrainProcessor class, which is registered in lavis/processors/blip_processors.py.

Similarly, the dataset name string is also registered in the registry, pointing to a dataset builder COCOCapBuilder class. By default, the builder will load the default dataset configuration as in DATASET_CONFIG_DICT. You may also add new dataset types by adding new entries to the dictionary.

The dataset configuration used here is:

.. literalinclude:: ../lavis/configs/datasets/coco/defaults_cap.yaml
    :language: yaml
    :linenos:
    :lines: 6-28

In this configuration file, we specify the dataset name and mainly its building information. The build information is divided into two parts: annotation and images. The annotation files will be automatically downloaded upon loading the dataset for the first time. The images part specifies the image root directory. This is a relative path to the cache directory, which is cache by default. If you have a local copy of the dataset, you can specify the path to the local copy by overwriting the images part in the runtime config file. For example, you may alter the run config as below to use your local dataset copy:

datasets:
    coco_caption: # name of the dataset builder
        vis_processor:
            train:
            name: "blip_image_train"
            eval:
            name: "blip_image_eval"
        text_processor:
            train:
            name: "blip_caption"
            prompt: "a picture of "
            eval:
            name: "blip_caption"
        images:
            YOUR_LOCAL_IMAGE_ROOT_DIR

LAVIS supports using multiple datasets for training. See an example in lavis/projects/blip/train/pretrain_14m.yaml.

Runner configurations

The last section of the config file specifies the arguments for the runner, shown below:

.. literalinclude:: ../lavis/projects/blip/train/caption_coco_large_ft.yaml
    :language: yaml
    :linenos:
    :lines: 26-56
Here we specify runner-related arguments, including
  • task-specific arguments, such as task, max_len, min_len, etc.
  • learning rate schedulers, optimizer;
  • distributed training settings;
  • logging and checkpointing settings.

Available Configurations

See :ref:`config` for the full list of available configurations and their descriptions.