Transformers_multilabel_distilbert.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Transformers_multilabel_distilbert.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "mount_file_id": "15ZXLCtJmBmByqWLkJsyuRuidGunSQt3c",
      "authorship_tag": "ABX9TyMYFn2H5CH0K8Hce4nsXrwt",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kT5-oqMPB6vp",
        "colab_type": "text"
      },
      "source": [
        "# Fine Tuning DistilBERT for MultiLabel Text Classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4R39UTxNKTk",
        "colab_type": "text"
      },
      "source": [
        "### Introduction\n",
        "\n",
        "In this tutorial we will be fine tuning a transformer model for the **Multilabel text classification** problem. \n",
        "This is one of the most common business problems where a given piece of text/sentence/document needs to be classified into one or more of categories out of the given list. For example a movie can be categorized into 1 or more genres.\n",
        "\n",
        "#### Flow of the notebook\n",
        "\n",
        "The notebook will be divided into seperate sections to provide a organized walk through for the process used. This process can be modified for individual use cases. The sections are:\n",
        "\n",
        "1. [Importing Python Libraries and preparing the environment](#section01)\n",
        "2. [Importing and Pre-Processing the domain data](#section02)\n",
        "3. [Preparing the Dataset and Dataloader](#section03)\n",
        "4. [Creating the Neural Network for Fine Tuning](#section04)\n",
        "5. [Fine Tuning the Model](#section05)\n",
        "6. [Validating the Model Performance](#section06)\n",
        "7. [Saving the model and artifacts for Inference in Future](#section07)\n",
        "\n",
        "#### Technical Details\n",
        "\n",
        "This script leverages on multiple tools designed by other teams. Details of the tools used below. Please ensure that these elements are present in your setup to successfully implement this script.\n",
        "\n",
        " - Data: \n",
        "\t - We are using the Jigsaw toxic data from [Kaggle](https://www.kaggle.com/)\n",
        "     - This is competion provide the souce dataset [Toxic Comment Competition](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)\n",
        "\t - We are referring only to the first csv file from the data dump: `train.csv`\n",
        "\t - There are rows of data.  Where each row has the following data-point: \n",
        "\t\t - Comment Text\n",
        "\t\t - `toxic`\n",
        "\t\t - `severe_toxic`\n",
        "\t\t - `obscene`\n",
        "\t\t - `threat`\n",
        "\t\t - `insult`\n",
        "\t\t - `identity_hate`\n",
        "\n",
        "Each comment can be marked for multiple categories. If the comment is `toxic` and `obscene`, then for both those headers the value will be `1` and for the others it will be `0`.\n",
        "\n",
        "\n",
        " - Language Model Used:\n",
        "\t - DistilBERT is a smaller transformer model as compared to BERT or Roberta. It is created by process of distillation applied to Bert.  \n",
        "\t - [Blog-Post](https://medium.com/huggingface/distilbert-8cf3380435b5)\n",
        "\t - [Research Paper](https://arxiv.org/pdf/1910.01108)\n",
        "     - [Documentation for python](https://huggingface.co/transformers/model_doc/distilbert.html)\n",
        "\n",
        "\n",
        " - Hardware Requirements:\n",
        "\t - Python 3.6 and above\n",
        "\t - Pytorch, Transformers and All the stock Python ML Libraries\n",
        "\t - GPU enabled setup \n",
        "\n",
        "\n",
        " - Script Objective:\n",
        "\t - The objective of this script is to fine tune DistilBERT to be able to label a comment  into the following categories:\n",
        "\t\t - `toxic`\n",
        "\t\t - `severe_toxic`\n",
        "\t\t - `obscene`\n",
        "\t\t - `threat`\n",
        "\t\t - `insult`\n",
        "\t\t - `identity_hate`\n",
        "\n",
        "---\n",
        "***NOTE***\n",
        "- *It is to be noted that the overall mechanisms for a multiclass and multilabel problems are similar, except for few differences namely:*\n",
        "\t- *Loss function is designed to evaluate all the probability of categories individually rather than as compared to other categories. Hence the use of `BCE` rather than `Cross Entropy` when defining loss.*\n",
        "\t- *Sigmoid of the outputs calcuated to rather than Softmax. Again for the reasons defined in the previous point*\n",
        "\t- *The [loss metrics](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hamming_loss.html) and **Hamming Score**  are used for direct comparison of expected vs predicted*\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-FA0wthINIsH",
        "colab_type": "text"
      },
      "source": [
        "<a id='section01'></a>\n",
        "### Importing Python Libraries and preparing the environment\n",
        "\n",
        "At this step we will be importing the libraries and modules needed to run our script. Libraries are:\n",
        "* warnings\n",
        "* Numpy\n",
        "* Pandas\n",
        "* tqdm\n",
        "* scikit-learn metrics\n",
        "* Pytorch\n",
        "* Pytorch Utils for Dataset and Dataloader\n",
        "* Transformers\n",
        "* DistilBERT Model and Tokenizer\n",
        "* logging\n",
        "\n",
        "Followed by that we will preapre the device for CUDA execeution. This configuration is needed if you want to leverage on onboard GPU. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XTkCDy7NPuOx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! pip install transformers==3.0.2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zHxRRzqpBf76",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Importing stock ml libraries\n",
        "import warnings\n",
        "warnings.simplefilter('ignore')\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from tqdm import tqdm\n",
        "from sklearn import metrics\n",
        "import transformers\n",
        "import torch\n",
        "from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler\n",
        "from transformers import DistilBertTokenizer, DistilBertModel\n",
        "import logging\n",
        "logging.basicConfig(level=logging.ERROR)"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B7N-SkxWC7zT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# # Setting up the device for GPU usage\n",
        "\n",
        "from torch import cuda\n",
        "device = 'cuda' if cuda.is_available() else 'cpu'"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tr6XeiZW3YbT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def hamming_score(y_true, y_pred, normalize=True, sample_weight=None):\n",
        "    acc_list = []\n",
        "    for i in range(y_true.shape[0]):\n",
        "        set_true = set( np.where(y_true[i])[0] )\n",
        "        set_pred = set( np.where(y_pred[i])[0] )\n",
        "        tmp_a = None\n",
        "        if len(set_true) == 0 and len(set_pred) == 0:\n",
        "            tmp_a = 1\n",
        "        else:\n",
        "            tmp_a = len(set_true.intersection(set_pred))/\\\n",
        "                    float( len(set_true.union(set_pred)) )\n",
        "        acc_list.append(tmp_a)\n",
        "    return np.mean(acc_list)"
      ],
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6RJt2gnlNKT3",
        "colab_type": "text"
      },
      "source": [
        "<a id='section02'></a>\n",
        "### Importing and Pre-Processing the domain data\n",
        "\n",
        "We will be working with the data and preparing for fine tuning purposes. \n",
        "*Assuming that the `train.csv` is already downloaded, unzipped and saved in your `data` folder*\n",
        "\n",
        "* First step will be to remove the **id** column from the data.\n",
        "* A new dataframe is made and input text is stored in the **text** column.\n",
        "* The values of all the categories and coverting it into a list.\n",
        "* The list is appened as a new column names as **labels**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9w86kD1qC_kb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = pd.read_csv('train.csv')"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "goM0DYRyvDP_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data.drop(['id'], inplace=True, axis=1)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Svs-AUlrvaMA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "new_df = pd.DataFrame()\n",
        "new_df['text'] = data['comment_text']\n",
        "new_df['labels'] = data.iloc[:, 1:].values.tolist()"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lfj9oQc1v6Dc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "9a3a5d3f-c8cb-4b7b-9a79-5274329aba05"
      },
      "source": [
        "new_df.head()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>labels</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Explanation\\nWhy the edits made under my usern...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>D'aww! He matches this background colour I'm s...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Hey man, I'm really not trying to edit war. It...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>\"\\nMore\\nI can't make any real suggestions on ...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>You, sir, are my hero. Any chance you remember...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text              labels\n",
              "0  Explanation\\nWhy the edits made under my usern...  [0, 0, 0, 0, 0, 0]\n",
              "1  D'aww! He matches this background colour I'm s...  [0, 0, 0, 0, 0, 0]\n",
              "2  Hey man, I'm really not trying to edit war. It...  [0, 0, 0, 0, 0, 0]\n",
              "3  \"\\nMore\\nI can't make any real suggestions on ...  [0, 0, 0, 0, 0, 0]\n",
              "4  You, sir, are my hero. Any chance you remember...  [0, 0, 0, 0, 0, 0]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yobavIDmNKT7",
        "colab_type": "text"
      },
      "source": [
        "<a id='section03'></a>\n",
        "### Preparing the Dataset and Dataloader\n",
        "\n",
        "We will start with defining few key variables that will be used later during the training/fine tuning stage.\n",
        "Followed by creation of MultiLabelDataset class - This defines how the text is pre-processed before sending it to the neural network. We will also define the Dataloader that will feed  the data in batches to the neural network for suitable training and processing. \n",
        "Dataset and Dataloader are constructs of the PyTorch library for defining and controlling the data pre-processing and its passage to neural network. For further reading into Dataset and Dataloader read the [docs at PyTorch](https://pytorch.org/docs/stable/data.html)\n",
        "\n",
        "#### *MultiLabelDataset* Dataset Class\n",
        "- This class is defined to accept the `tokenizer`, `dataframe` and `max_length` as input and generate tokenized output and tags that is used by the BERT model for training. \n",
        "- We are using the DistilBERT tokenizer to tokenize the data in the `text` column of the dataframe.\n",
        "- The tokenizer uses the `encode_plus` method to perform tokenization and generate the necessary outputs, namely: `ids`, `attention_mask`, `token_type_ids`\n",
        "\n",
        "- To read further into the tokenizer, [refer to this document](https://huggingface.co/transformers/model_doc/distilbert.html#distilberttokenizer)\n",
        "- `targets` is the list of categories labled as `0` or `1` in the dataframe. \n",
        "- The *MultiLabelDataset* class is used to create 2 datasets, for training and for validation.\n",
        "- *Training Dataset* is used to fine tune the model: **80% of the original data**\n",
        "- *Validation Dataset* is used to evaluate the performance of the model. The model has not seen this data during training. \n",
        "\n",
        "#### Dataloader\n",
        "- Dataloader is used to for creating training and validation dataloader that load data to the neural network in a defined manner. This is needed because all the data from the dataset cannot be loaded to the memory at once, hence the amount of dataloaded to the memory and then passed to the neural network needs to be controlled.\n",
        "- This control is achieved using the parameters such as `batch_size` and `max_len`.\n",
        "- Training and Validation dataloaders are used in the training and validation part of the flow respectively"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VpF4ZAaxC_OJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Sections of config\n",
        "\n",
        "# Defining some key variables that will be used later on in the training\n",
        "MAX_LEN = 128\n",
        "TRAIN_BATCH_SIZE = 4\n",
        "VALID_BATCH_SIZE = 4\n",
        "EPOCHS = 1\n",
        "LEARNING_RATE = 1e-05\n",
        "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased', truncation=True, do_lower_case=True)"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fC7zA4nYDGX3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MultiLabelDataset(Dataset):\n",
        "\n",
        "    def __init__(self, dataframe, tokenizer, max_len):\n",
        "        self.tokenizer = tokenizer\n",
        "        self.data = dataframe\n",
        "        self.text = dataframe.text\n",
        "        self.targets = self.data.labels\n",
        "        self.max_len = max_len\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.text)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        text = str(self.text[index])\n",
        "        text = \" \".join(text.split())\n",
        "\n",
        "        inputs = self.tokenizer.encode_plus(\n",
        "            text,\n",
        "            None,\n",
        "            add_special_tokens=True,\n",
        "            max_length=self.max_len,\n",
        "            pad_to_max_length=True,\n",
        "            return_token_type_ids=True\n",
        "        )\n",
        "        ids = inputs['input_ids']\n",
        "        mask = inputs['attention_mask']\n",
        "        token_type_ids = inputs[\"token_type_ids\"]\n",
        "\n",
        "\n",
        "        return {\n",
        "            'ids': torch.tensor(ids, dtype=torch.long),\n",
        "            'mask': torch.tensor(mask, dtype=torch.long),\n",
        "            'token_type_ids': torch.tensor(token_type_ids, dtype=torch.long),\n",
        "            'targets': torch.tensor(self.targets[index], dtype=torch.float)\n",
        "        }"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qOZWsLfGDOBG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "8ebb8e94-4f9f-4db6-89f4-20a30217cc15"
      },
      "source": [
        "# Creating the dataset and dataloader for the neural network\n",
        "\n",
        "train_size = 0.8\n",
        "train_data=new_df.sample(frac=train_size,random_state=200)\n",
        "test_data=new_df.drop(train_data.index).reset_index(drop=True)\n",
        "train_data = train_data.reset_index(drop=True)\n",
        "\n",
        "\n",
        "print(\"FULL Dataset: {}\".format(new_df.shape))\n",
        "print(\"TRAIN Dataset: {}\".format(train_data.shape))\n",
        "print(\"TEST Dataset: {}\".format(test_data.shape))\n",
        "\n",
        "training_set = MultiLabelDataset(train_data, tokenizer, MAX_LEN)\n",
        "testing_set = MultiLabelDataset(test_data, tokenizer, MAX_LEN)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "FULL Dataset: (159571, 2)\n",
            "TRAIN Dataset: (127657, 2)\n",
            "TEST Dataset: (31914, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "StbPlIyKDP9E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_params = {'batch_size': TRAIN_BATCH_SIZE,\n",
        "                'shuffle': True,\n",
        "                'num_workers': 0\n",
        "                }\n",
        "\n",
        "test_params = {'batch_size': VALID_BATCH_SIZE,\n",
        "                'shuffle': True,\n",
        "                'num_workers': 0\n",
        "                }\n",
        "\n",
        "training_loader = DataLoader(training_set, **train_params)\n",
        "testing_loader = DataLoader(testing_set, **test_params)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yU4TWUBtNKUN",
        "colab_type": "text"
      },
      "source": [
        "<a id='section04'></a>\n",
        "### Creating the Neural Network for Fine Tuning\n",
        "\n",
        "#### Neural Network\n",
        " - We will be creating a neural network with the `DistilBERTClass`. \n",
        " - This network will have the `DistilBERT` model.  Follwed by a `Droput` and `Linear Layer`. They are added for the purpose of **Regulariaztion** and **Classification** respectively. \n",
        " - In the forward loop, there are 2 output from the `DistilBERTClass` layer.\n",
        " - The second output `output_1` or called the `pooled output` is passed to the `Drop Out layer` and the subsequent output is given to the `Linear layer`. \n",
        " - Keep note the number of dimensions for `Linear Layer` is **6** because that is the total number of categories in which we are looking to classify our model.\n",
        " - The data will be fed to the `DistilBERTClass` as defined in the dataset. \n",
        " - Final layer outputs is what will be used to calcuate the loss and to determine the accuracy of models prediction. \n",
        " - We will initiate an instance of the network called `model`. This instance will be used for training and then to save the final trained model for future inference. \n",
        " \n",
        "#### Loss Function and Optimizer\n",
        " - The Loss is defined in the next cell as `loss_fn`.\n",
        " - As defined above, the loss function used will be a combination of Binary Cross Entropy which is implemented as [BCELogits Loss](https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss) in PyTorch\n",
        " - `Optimizer` is defined in the next cell.\n",
        " - `Optimizer` is used to update the weights of the neural network to improve its performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FeftvDhjDSPp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "4ba915de-3a1e-4650-d253-2b43857f2d99"
      },
      "source": [
        "# Creating the customized model, by adding a drop out and a dense layer on top of distil bert to get the final output for the model. \n",
        "\n",
        "class DistilBERTClass(torch.nn.Module):\n",
        "    def __init__(self):\n",
        "        super(DistilBERTClass, self).__init__()\n",
        "        self.l1 = DistilBertModel.from_pretrained(\"distilbert-base-uncased\")\n",
        "        self.pre_classifier = torch.nn.Linear(768, 768)\n",
        "        self.dropout = torch.nn.Dropout(0.1)\n",
        "        self.classifier = torch.nn.Linear(768, 6)\n",
        "\n",
        "    def forward(self, input_ids, attention_mask, token_type_ids):\n",
        "        output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)\n",
        "        hidden_state = output_1[0]\n",
        "        pooler = hidden_state[:, 0]\n",
        "        pooler = self.pre_classifier(pooler)\n",
        "        pooler = torch.nn.Tanh()(pooler)\n",
        "        pooler = self.dropout(pooler)\n",
        "        output = self.classifier(pooler)\n",
        "        return output\n",
        "\n",
        "model = DistilBERTClass()\n",
        "model.to(device)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DistilBERTClass(\n",
              "  (l1): DistilBertModel(\n",
              "    (embeddings): Embeddings(\n",
              "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
              "      (position_embeddings): Embedding(512, 768)\n",
              "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "      (dropout): Dropout(p=0.1, inplace=False)\n",
              "    )\n",
              "    (transformer): Transformer(\n",
              "      (layer): ModuleList(\n",
              "        (0): TransformerBlock(\n",
              "          (attention): MultiHeadSelfAttention(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "          )\n",
              "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (ffn): FFN(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          )\n",
              "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "        )\n",
              "        (1): TransformerBlock(\n",
              "          (attention): MultiHeadSelfAttention(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "          )\n",
              "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (ffn): FFN(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          )\n",
              "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "        )\n",
              "        (2): TransformerBlock(\n",
              "          (attention): MultiHeadSelfAttention(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "          )\n",
              "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (ffn): FFN(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          )\n",
              "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "        )\n",
              "        (3): TransformerBlock(\n",
              "          (attention): MultiHeadSelfAttention(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "          )\n",
              "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (ffn): FFN(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          )\n",
              "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "        )\n",
              "        (4): TransformerBlock(\n",
              "          (attention): MultiHeadSelfAttention(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "          )\n",
              "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (ffn): FFN(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          )\n",
              "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "        )\n",
              "        (5): TransformerBlock(\n",
              "          (attention): MultiHeadSelfAttention(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
              "          )\n",
              "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (ffn): FFN(\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          )\n",
              "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (pre_classifier): Linear(in_features=768, out_features=768, bias=True)\n",
              "  (dropout): Dropout(p=0.1, inplace=False)\n",
              "  (classifier): Linear(in_features=768, out_features=6, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MZ_wI0YwDVJZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def loss_fn(outputs, targets):\n",
        "    return torch.nn.BCEWithLogitsLoss()(outputs, targets)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oO49FuR9DXsW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = torch.optim.Adam(params =  model.parameters(), lr=LEARNING_RATE)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_frCoix1NKUZ",
        "colab_type": "text"
      },
      "source": [
        "<a id='section05'></a>\n",
        "### Fine Tuning the Model\n",
        "\n",
        "After all the effort of loading and preparing the data and datasets, creating the model and defining its loss and optimizer. This is probably the easier steps in the process. \n",
        "\n",
        "Here we define a training function that trains the model on the training dataset created above, specified number of times (EPOCH), An epoch defines how many times the complete data will be passed through the network. \n",
        "\n",
        "Following events happen in this function to fine tune the neural network:\n",
        "- The dataloader passes data to the model based on the batch size. \n",
        "- Subsequent output from the model and the actual category are compared to calculate the loss. \n",
        "- Loss value is used to optimize the weights of the neurons in the network.\n",
        "- After every 5000 steps the loss value is printed in the console.\n",
        "\n",
        "As you can see just in 1 epoch by the final step the model was working with a miniscule loss of 0.05 i.e. the network output is extremely close to the actual output."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fb9-Yr9YDZqo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train(epoch):\n",
        "    model.train()\n",
        "    for _,data in tqdm(enumerate(training_loader, 0)):\n",
        "        ids = data['ids'].to(device, dtype = torch.long)\n",
        "        mask = data['mask'].to(device, dtype = torch.long)\n",
        "        token_type_ids = data['token_type_ids'].to(device, dtype = torch.long)\n",
        "        targets = data['targets'].to(device, dtype = torch.float)\n",
        "\n",
        "        outputs = model(ids, mask, token_type_ids)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "        loss = loss_fn(outputs, targets)\n",
        "        if _%5000==0:\n",
        "            print(f'Epoch: {epoch}, Loss:  {loss.item()}')\n",
        "        \n",
        "        loss.backward()\n",
        "        optimizer.step()"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Reta6H84DcJq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "dd66c704-9c33-4815-828c-b552e7b5cf5e"
      },
      "source": [
        "for epoch in range(EPOCHS):\n",
        "    train(epoch)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1it [00:00,  9.65it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, Loss:  0.6973826289176941\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "5003it [05:32, 14.64it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, Loss:  0.0016139214858412743\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "10003it [11:14, 14.59it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, Loss:  0.031105097383260727\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "15003it [16:56, 14.58it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, Loss:  0.0019174569752067327\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "20003it [22:38, 14.60it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, Loss:  0.0015925116604194045\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "25003it [28:20, 14.58it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, Loss:  0.08796875923871994\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "30003it [34:03, 14.58it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, Loss:  0.05103427171707153\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "31915it [36:14, 14.68it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFv7mNcuNKUh",
        "colab_type": "text"
      },
      "source": [
        "<a id='section06'></a>\n",
        "### Validating the Model\n",
        "\n",
        "During the validation stage we pass the unseen data(Testing Dataset) to the model. This step determines how good the model performs on the unseen data. \n",
        "\n",
        "This unseen data is the 20% of `train.csv` which was seperated during the Dataset creation stage. \n",
        "During the validation stage the weights of the model are not updated. Only the final output is compared to the actual value. This comparison is then used to calcuate the accuracy of the model. \n",
        "\n",
        "As defined above to get a measure of our models performance we are using the following metrics. \n",
        "- Hamming Score\n",
        "- Hamming Loss\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aPqZKQ1BDfLW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def validation(testing_loader):\n",
        "    model.eval()\n",
        "    fin_targets=[]\n",
        "    fin_outputs=[]\n",
        "    with torch.no_grad():\n",
        "        for _, data in tqdm(enumerate(testing_loader, 0)):\n",
        "            ids = data['ids'].to(device, dtype = torch.long)\n",
        "            mask = data['mask'].to(device, dtype = torch.long)\n",
        "            token_type_ids = data['token_type_ids'].to(device, dtype = torch.long)\n",
        "            targets = data['targets'].to(device, dtype = torch.float)\n",
        "            outputs = model(ids, mask, token_type_ids)\n",
        "            fin_targets.extend(targets.cpu().detach().numpy().tolist())\n",
        "            fin_outputs.extend(torch.sigmoid(outputs).cpu().detach().numpy().tolist())\n",
        "    return fin_outputs, fin_targets"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cQmJd7RvB4A-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ce9b6e23-94d3-4e11-bce0-2ad3ae5ab38e"
      },
      "source": [
        "outputs, targets = validation(testing_loader)\n",
        "\n",
        "final_outputs = np.array(outputs) >=0.5"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7979it [02:39, 50.13it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qekc58XPDhIp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "950ef928-9171-46a1-f36d-69183bc369f3"
      },
      "source": [
        "val_hamming_loss = metrics.hamming_loss(targets, final_outputs)\n",
        "val_hamming_score = hamming_score(np.array(targets), np.array(final_outputs))\n",
        "\n",
        "print(f\"Hamming Score = {val_hamming_score}\")\n",
        "print(f\"Hamming Loss = {val_hamming_loss}\")"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Hamming Score = 0.9452533893171231\n",
            "Hamming Loss = 0.017207704037935284\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nA59gbi0NItL",
        "colab_type": "text"
      },
      "source": [
        "<a id='section07'></a>\n",
        "### Saving the Trained Model for inference\n",
        "\n",
        "This is the final step in the process of fine tuning the model. \n",
        "\n",
        "The model and its vocabulary are saved locally. These files are then used in the future to make inference on new inputs of news headlines."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9yzLQmsdNItM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Saving the files for inference\n",
        "\n",
        "output_model_file = './models/pytorch_distilbert_news.bin'\n",
        "output_vocab_file = './models/vocab_distilbert_news.bin'\n",
        "\n",
        "torch.save(model, output_model_file)\n",
        "tokenizer.save_vocabulary(output_vocab_file)\n",
        "\n",
        "print('Saved')"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}