transformers_doc/quicktour.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformers installation\n",
    "! pip install transformers datasets\n",
    "# To install from source instead of the last release, comment the command above and uncomment the following one.\n",
    "# ! pip install git+https://github.com/huggingface/transformers.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quick tour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a quick look at the 🤗 Transformers library features. The library downloads pretrained models for Natural\n",
    "Language Understanding (NLU) tasks, such as analyzing the sentiment of a text, and Natural Language Generation (NLG),\n",
    "such as completing a prompt with new text or translating in another language.\n",
    "\n",
    "First we will see how to easily leverage the pipeline API to quickly use those pretrained models at inference. Then, we\n",
    "will dig a little bit more and see how the library gives you access to those models and helps you preprocess your data.\n",
    "\n",
    "<Tip>\n",
    "\n",
    "All code examples presented in the documentation have a switch on the top left for Pytorch versus TensorFlow. If\n",
    "not, the code is expected to work for both backends without any change needed.\n",
    "\n",
    "</Tip>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started on a task with a pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to use a pretrained model on a given task is to use [pipeline()](https://huggingface.co/docs/transformers/master/en/main_classes/pipelines#transformers.pipeline)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/tiZFewofSLM?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#@title\n",
    "from IPython.display import HTML\n",
    "\n",
    "HTML('<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/tiZFewofSLM?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤗 Transformers provides the following tasks out of the box:\n",
    "\n",
    "- Sentiment analysis: is a text positive or negative?\n",
    "- Text generation (in English): provide a prompt and the model will generate what follows.\n",
    "- Name entity recognition (NER): in an input sentence, label each word with the entity it represents (person, place, etc.)\n",
    "- Question answering: provide the model with some context and a question, extract the answer from the context.\n",
    "- Filling masked text: given a text with masked words (e.g., replaced by `[MASK]`), fill the blanks.\n",
    "- Summarization: generate a summary of a long text.\n",
    "- Translation: translate a text in another language.\n",
    "- Feature extraction: return a tensor representation of the text.\n",
    "\n",
    "Let's see how this work for sentiment analysis (the other tasks are all covered in the [task summary](https://huggingface.co/docs/transformers/master/en/task_summary)):\n",
    "\n",
    "Install the following dependencies (if not already installed):\n",
    "\n",
    "```bash\n",
    "pip install torch\n",
    "```\n",
    "\n",
    "```bash\n",
    "pip install tensorflow\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "classifier = pipeline('sentiment-analysis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When typing this command for the first time, a pretrained model and its tokenizer are downloaded and cached. We will\n",
    "look at both later on, but as an introduction the tokenizer's job is to preprocess the text for the model, which is\n",
    "then responsible for making predictions. The pipeline groups all of that together, and post-process the predictions to\n",
    "make them readable. For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'POSITIVE', 'score': 0.9998}]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier('We are very happy to show you the 🤗 Transformers library.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's encouraging! You can use it on a list of sentences, which will be preprocessed then fed to the model, returning\n",
    "a list of dictionaries like this one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label: POSITIVE, with score: 0.9998\n",
       "label: NEGATIVE, with score: 0.5309"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = classifier([\"We are very happy to show you the 🤗 Transformers library.\",\n",
    "           \"We hope you don't hate it.\"])\n",
    "for result in results:\n",
    "    print(f\"label: {result['label']}, with score: {round(result['score'], 4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use with a large dataset, look at [iterating over a pipeline](https://huggingface.co/docs/transformers/master/en/main_classes/pipelines)\n",
    "\n",
    "You can see the second sentence has been classified as negative (it needs to be positive or negative) but its score is\n",
    "fairly neutral.\n",
    "\n",
    "By default, the model downloaded for this pipeline is called \"distilbert-base-uncased-finetuned-sst-2-english\". We can\n",
    "look at its [model page](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english) to get more\n",
    "information about it. It uses the [DistilBERT architecture](https://huggingface.co/docs/transformers/master/en/model_doc/distilbert) and has been fine-tuned on a\n",
    "dataset called SST-2 for the sentiment analysis task.\n",
    "\n",
    "Let's say we want to use another model; for instance, one that has been trained on French data. We can search through\n",
    "the [model hub](https://huggingface.co/models) that gathers models pretrained on a lot of data by research labs, but\n",
    "also community models (usually fine-tuned versions of those big models on a specific dataset). Applying the tags\n",
    "\"French\" and \"text-classification\" gives back a suggestion \"nlptown/bert-base-multilingual-uncased-sentiment\". Let's\n",
    "see how we can use it.\n",
    "\n",
    "You can directly pass the name of the model to use to [pipeline()](https://huggingface.co/docs/transformers/master/en/main_classes/pipelines#transformers.pipeline):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = pipeline('sentiment-analysis', model=\"nlptown/bert-base-multilingual-uncased-sentiment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This classifier can now deal with texts in English, French, but also Dutch, German, Italian and Spanish! You can also\n",
    "replace that name by a local folder where you have saved a pretrained model (see below). You can also pass a model\n",
    "object and its associated tokenizer.\n",
    "\n",
    "We will need two classes for this. The first is [AutoTokenizer](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoTokenizer), which we will use to download the tokenizer associated to the model we picked and instantiate it. The second is [AutoModelForSequenceClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or [TFAutoModelForSequenceClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.TFAutoModelForSequenceClassification) if you are using TensorFlow), which we will use to download the model itself. Note that if we were using the library on an other task, the class of the model would change. The [task summary](https://huggingface.co/docs/transformers/master/en/task_summary) tutorial summarizes which class is used for which task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, TFAutoModelForSequenceClassification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to download the models and tokenizer we found previously, we just have to use the\n",
    "`from_pretrained()` method (feel free to replace `model_name` by\n",
    "any other model from the model hub):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "classifier = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n",
    "# This model only exists in PyTorch, so we use the _from_pt_ flag to import that model in TensorFlow.\n",
    "model = TFAutoModelForSequenceClassification.from_pretrained(model_name, from_pt=True)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "classifier = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't find a model that has been pretrained on some data similar to yours, you will need to fine-tune a\n",
    "pretrained model on your data. We provide [example scripts](https://huggingface.co/docs/transformers/master/en/examples) to do so. Once you're done, don't forget\n",
    "to share your fine-tuned model on the hub with the community, using [this tutorial](https://huggingface.co/docs/transformers/master/en/model_sharing).\n",
    "\n",
    "<a id='pretrained-model'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Under the hood: pretrained models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now see what happens beneath the hood when using those pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/AhChOFRegn4?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#@title\n",
    "from IPython.display import HTML\n",
    "\n",
    "HTML('<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/AhChOFRegn4?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw, the model and tokenizer are created using the `from_pretrained` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "model_name = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
    "pt_model = AutoModelForSequenceClassification.from_pretrained(model_name)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, TFAutoModelForSequenceClassification\n",
    "model_name = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
    "tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We mentioned the tokenizer is responsible for the preprocessing of your texts. First, it will split a given text in\n",
    "words (or part of words, punctuation symbols, etc.) usually called _tokens_. There are multiple rules that can govern\n",
    "that process (you can learn more about them in the [tokenizer summary](https://huggingface.co/docs/transformers/master/en/tokenizer_summary)), which is why we need\n",
    "to instantiate the tokenizer using the name of the model, to make sure we use the same rules as when the model was\n",
    "pretrained.\n",
    "\n",
    "The second step is to convert those _tokens_ into numbers, to be able to build a tensor out of them and feed them to\n",
    "the model. To do this, the tokenizer has a _vocab_, which is the part we download when we instantiate it with the\n",
    "`from_pretrained` method, since we need to use the same _vocab_ as when the model was pretrained.\n",
    "\n",
    "To apply these steps on a given text, we can just feed it to our tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tokenizer(\"We are very happy to show you the 🤗 Transformers library.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns a dictionary string to list of ints. It contains the [ids of the tokens](https://huggingface.co/docs/transformers/master/en/glossary#input-ids), as\n",
    "mentioned before, but also additional arguments that will be useful to the model. Here for instance, we also have an\n",
    "[attention mask](https://huggingface.co/docs/transformers/master/en/glossary#attention-mask) that the model will use to have a better understanding of the sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 2057, 2024, 2200, 3407, 2000, 2265, 2017, 1996, 100, 19081, 3075, 1012, 102],\n",
       " 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass a list of sentences directly to your tokenizer. If your goal is to send them through your model as a\n",
    "batch, you probably want to pad them all to the same length, truncate them to the maximum length the model can accept\n",
    "and get tensors back. You can specify all of that to the tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pt_batch = tokenizer(\n",
    "    [\"We are very happy to show you the 🤗 Transformers library.\", \"We hope you don't hate it.\"],\n",
    "    padding=True,\n",
    "    truncation=True,\n",
    "    max_length=512,\n",
    "    return_tensors=\"pt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_batch = tokenizer(\n",
    "    [\"We are very happy to show you the 🤗 Transformers library.\", \"We hope you don't hate it.\"],\n",
    "    padding=True,\n",
    "    truncation=True,\n",
    "    max_length=512,\n",
    "    return_tensors=\"tf\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The padding is automatically applied on the side expected by the model (in this case, on the right), with the padding\n",
    "token the model was pretrained with. The attention mask is also adapted to take the padding into account:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "input_ids: [[101, 2057, 2024, 2200, 3407, 2000, 2265, 2017, 1996, 100, 19081, 3075, 1012, 102], [101, 2057, 3246, 2017, 2123, 1005, 1056, 5223, 2009, 1012, 102, 0, 0, 0]]\n",
       "attention_mask: [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for key, value in pt_batch.items():\n",
    "    print(f\"{key}: {value.numpy().tolist()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "input_ids: [[101, 2057, 2024, 2200, 3407, 2000, 2265, 2017, 1996, 100, 19081, 3075, 1012, 102], [101, 2057, 3246, 2017, 2123, 1005, 1056, 5223, 2009, 1012, 102, 0, 0, 0]]\n",
       "attention_mask: [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for key, value in tf_batch.items():\n",
    "    print(f\"{key}: {value.numpy().tolist()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can learn more about tokenizers [here](https://huggingface.co/docs/transformers/master/en/preprocessing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once your input has been preprocessed by the tokenizer, you can send it directly to the model. As we mentioned, it will\n",
    "contain all the relevant information the model needs. If you're using a TensorFlow model, you can pass the dictionary\n",
    "keys directly to tensors, for a PyTorch model, you need to unpack the dictionary by adding `**`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pt_outputs = pt_model(**pt_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_outputs = tf_model(tf_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In 🤗 Transformers, all outputs are objects that contain the model's final activations along with other metadata. These\n",
    "objects are described in greater detail [here](https://huggingface.co/docs/transformers/master/en/main_classes/output). For now, let's inspect the output ourselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequenceClassifierOutput(loss=None, logits=tensor([[-4.0833,  4.3364],\n",
       "        [ 0.0818, -0.0418]], grad_fn=&amp;lt;AddmmBackward>), hidden_states=None, attentions=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(pt_outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TFSequenceClassifierOutput(loss=None, logits=&amp;lt;tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[-4.0833 ,  4.3364  ],\n",
       "       [ 0.0818, -0.0418]], dtype=float32)>, hidden_states=None, attentions=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(tf_outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the output object has a `logits` attribute. You can use this to access the model's final activations.\n",
    "\n",
    "<Tip>\n",
    "\n",
    "All 🤗 Transformers models (PyTorch or TensorFlow) return the activations of the model *before* the final activation\n",
    "function (like SoftMax) since this final activation function is often fused with the loss.\n",
    "\n",
    "</Tip>\n",
    "\n",
    "Let's apply the SoftMax activation to get predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "pt_predictions = nn.functional.softmax(pt_outputs.logits, dim=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf_predictions = tf.nn.softmax(tf_outputs.logits, axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see we get the numbers from before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2.2043e-04, 9.9978e-01],\n",
       "        [5.3086e-01, 4.6914e-01]], grad_fn=&amp;lt;SoftmaxBackward>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(pt_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tf.Tensor(\n",
       "[[2.2043e-04 9.9978e-01]\n",
       " [5.3086e-01 4.6914e-01]], shape=(2, 2), dtype=float32)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(tf_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you provide the model with labels in addition to inputs, the model output object will also contain a `loss`\n",
    "attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequenceClassifierOutput(loss=tensor(0.3167, grad_fn=&amp;lt;NllLossBackward>), logits=tensor([[-4.0833,  4.3364],\n",
       "        [ 0.0818, -0.0418]], grad_fn=&amp;lt;AddmmBackward>), hidden_states=None, attentions=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "pt_outputs = pt_model(**pt_batch, labels = torch.tensor([1, 0]))\n",
    "print(pt_outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TFSequenceClassifierOutput(loss=&amp;lt;tf.Tensor: shape=(2,), dtype=float32, numpy=array([2.2051e-04, 6.3326e-01], dtype=float32)>, logits=&amp;lt;tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[-4.0833 ,  4.3364  ],\n",
       "       [ 0.0818, -0.0418]], dtype=float32)>, hidden_states=None, attentions=None)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "tf_outputs = tf_model(tf_batch, labels = tf.constant([1, 0]))\n",
    "print(tf_outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models are standard [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) or [tf.keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) so you can use them in your usual training loop. 🤗 Transformers also provides a [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) class to help with your training in PyTorch (taking care of things such as distributed training, mixed precision, etc.) whereas you can leverage the `fit()` method in Keras. See the [training tutorial](https://huggingface.co/docs/transformers/master/en/training) for more details.\n",
    "\n",
    "<Tip>\n",
    "\n",
    "Pytorch model outputs are special dataclasses so that you can get autocompletion for their attributes in an IDE.\n",
    "They also behave like a tuple or a dictionary (e.g., you can index with an integer, a slice or a string) in which\n",
    "case the attributes not set (that have `None` values) are ignored.\n",
    "\n",
    "</Tip>\n",
    "\n",
    "Once your model is fine-tuned, you can save it with its tokenizer in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pt_save_directory = './pt_save_pretrained'\n",
    "tokenizer.save_pretrained(pt_save_directory)\n",
    "pt_model.save_pretrained(pt_save_directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_save_directory = './tf_save_pretrained'\n",
    "tokenizer.save_pretrained(tf_save_directory)\n",
    "tf_model.save_pretrained(tf_save_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then load this model back using the `AutoModel.from_pretrained()` method by passing the\n",
    "directory name instead of the model name. One cool feature of 🤗 Transformers is that you can easily switch between\n",
    "PyTorch and TensorFlow: any model saved as before can be loaded back either in PyTorch or TensorFlow.\n",
    "\n",
    "\n",
    "If you would like to load your saved model in the other framework, first make sure it is installed:\n",
    "\n",
    "```bash\n",
    "pip install tensorflow\n",
    "```\n",
    "\n",
    "```bash\n",
    "pip install torch\n",
    "```\n",
    "\n",
    "Then, use the corresponding Auto class to load it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TFAutoModel\n",
    "tokenizer = AutoTokenizer.from_pretrained(pt_save_directory)\n",
    "tf_model = TFAutoModel.from_pretrained(pt_save_directory, from_pt=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModel\n",
    "tokenizer = AutoTokenizer.from_pretrained(tf_save_directory)\n",
    "pt_model = AutoModel.from_pretrained(tf_save_directory, from_tf=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, you can also ask the model to return all hidden states and all attention weights if you need them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pt_outputs = pt_model(**pt_batch, output_hidden_states=True, output_attentions=True)\n",
    "all_hidden_states  = pt_outputs.hidden_states \n",
    "all_attentions = pt_outputs.attentions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_outputs = tf_model(tf_batch, output_hidden_states=True, output_attentions=True)\n",
    "all_hidden_states =  tf_outputs.hidden_states\n",
    "all_attentions = tf_outputs.attentions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing the code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [AutoModel](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModel) and [AutoTokenizer](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoTokenizer) classes are just shortcuts that will automatically work with any\n",
    "pretrained model. Behind the scenes, the library has one model class per combination of architecture plus class, so the\n",
    "code is easy to access and tweak if you need to.\n",
    "\n",
    "In our previous example, the model was called \"distilbert-base-uncased-finetuned-sst-2-english\", which means it's using\n",
    "the [DistilBERT](https://huggingface.co/docs/transformers/master/en/model_doc/distilbert) architecture. As [AutoModelForSequenceClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or [TFAutoModelForSequenceClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.TFAutoModelForSequenceClassification) if you are using TensorFlow) was used, the model automatically created is then a [DistilBertForSequenceClassification](https://huggingface.co/docs/transformers/master/en/model_doc/distilbert#transformers.DistilBertForSequenceClassification). You can look at its documentation for all details relevant to that specific model, or browse the source code. This is how you would\n",
    "directly instantiate model and tokenizer without the auto magic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertTokenizer, DistilBertForSequenceClassification\n",
    "model_name = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
    "model = DistilBertForSequenceClassification.from_pretrained(model_name)\n",
    "tokenizer = DistilBertTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertTokenizer, TFDistilBertForSequenceClassification\n",
    "model_name = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
    "model = TFDistilBertForSequenceClassification.from_pretrained(model_name)\n",
    "tokenizer = DistilBertTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Customizing the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to change how the model itself is built, you can define a custom configuration class. Each architecture\n",
    "comes with its own relevant configuration. For example, [DistilBertConfig](https://huggingface.co/docs/transformers/master/en/model_doc/distilbert#transformers.DistilBertConfig) allows you to specify\n",
    "parameters such as the hidden dimension, dropout rate, etc for DistilBERT. If you do core modifications, like changing\n",
    "the hidden size, you won't be able to use a pretrained model anymore and will need to train from scratch. You would\n",
    "then instantiate the model directly from this configuration.\n",
    "\n",
    "Below, we load a predefined vocabulary for a tokenizer with the\n",
    "[from_pretrained()](https://huggingface.co/docs/transformers/master/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.from_pretrained) method. However, unlike the tokenizer, we wish to initialize\n",
    "the model from scratch. Therefore, we instantiate the model from a configuration instead of using the\n",
    "[DistilBertForSequenceClassification.from_pretrained()](https://huggingface.co/docs/transformers/master/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertConfig, DistilBertTokenizer, DistilBertForSequenceClassification\n",
    "config = DistilBertConfig(n_heads=8, dim=512, hidden_dim=4*512)\n",
    "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n",
    "model = DistilBertForSequenceClassification(config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertConfig, DistilBertTokenizer, TFDistilBertForSequenceClassification\n",
    "config = DistilBertConfig(n_heads=8, dim=512, hidden_dim=4*512)\n",
    "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n",
    "model = TFDistilBertForSequenceClassification(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For something that only changes the head of the model (for instance, the number of labels), you can still use a\n",
    "pretrained model for the body. For instance, let's define a classifier for 10 different labels using a pretrained body.\n",
    "Instead of creating a new configuration with all the default values just to change the number of labels, we can instead\n",
    "pass any argument a configuration would take to the `from_pretrained` method and it will update the default\n",
    "configuration appropriately:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertConfig, DistilBertTokenizer, DistilBertForSequenceClassification\n",
    "model_name = \"distilbert-base-uncased\"\n",
    "model = DistilBertForSequenceClassification.from_pretrained(model_name, num_labels=10)\n",
    "tokenizer = DistilBertTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertConfig, DistilBertTokenizer, TFDistilBertForSequenceClassification\n",
    "model_name = \"distilbert-base-uncased\"\n",
    "model = TFDistilBertForSequenceClassification.from_pretrained(model_name, num_labels=10)\n",
    "tokenizer = DistilBertTokenizer.from_pretrained(model_name)"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 4
}