transformers_doc/preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformers installation\n",
    "! pip install transformers datasets\n",
    "# To install from source instead of the last release, comment the command above and uncomment the following one.\n",
    "# ! pip install git+https://github.com/huggingface/transformers.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll explore how to preprocess your data using 🤗 Transformers. The main tool for this is what we\n",
    "call a [tokenizer](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/tokenizer). You can build one using the tokenizer class associated to the model\n",
    "you would like to use, or directly with the [AutoTokenizer](https://huggingface.co/docs/transformers/v4.16.2/en/model_doc/auto#transformers.AutoTokenizer) class.\n",
    "\n",
    "As we saw in the [quick tour](https://huggingface.co/docs/transformers/v4.16.2/en/quicktour), the tokenizer will first split a given text in words (or part of\n",
    "words, punctuation symbols, etc.) usually called _tokens_. Then it will convert those _tokens_ into numbers, to be able\n",
    "to build a tensor out of them and feed them to the model. It will also add any additional inputs the model might expect\n",
    "to work properly.\n",
    "\n",
    "<Tip>\n",
    "\n",
    "If you plan on using a pretrained model, it's important to use the associated pretrained tokenizer: it will split\n",
    "the text you give it in tokens the same way for the pretraining corpus, and it will use the same correspondence\n",
    "token to index (that we usually call a _vocab_) as during pretraining.\n",
    "\n",
    "</Tip>\n",
    "\n",
    "To automatically download the vocab used during pretraining or fine-tuning a given model, you can use the\n",
    "[AutoTokenizer.from_pretrained()](https://huggingface.co/docs/transformers/v4.16.2/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Base use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/Yffk5aydLzg?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#@title\n",
    "from IPython.display import HTML\n",
    "\n",
    "HTML('<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/Yffk5aydLzg?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [PreTrainedTokenizer](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) has many methods, but the only one you need to remember for preprocessing\n",
    "is its `__call__`: you just need to feed your sentence to your tokenizer object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102], \n",
       " 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \n",
       " 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_input = tokenizer(\"Hello, I'm a single sentence!\")\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns a dictionary string to list of ints. The [input_ids](https://huggingface.co/docs/transformers/v4.16.2/en/glossary#input-ids) are the indices corresponding\n",
    "to each token in our sentence. We will see below what the [attention_mask](https://huggingface.co/docs/transformers/v4.16.2/en/glossary#attention-mask) is used for and\n",
    "in [the next section](#preprocessing-pairs-of-sentences) the goal of [token_type_ids](https://huggingface.co/docs/transformers/v4.16.2/en/glossary#token-type-ids).\n",
    "\n",
    "The tokenizer can decode a list of token ids in a proper sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"[CLS] Hello, I'm a single sentence! [SEP]\""
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(encoded_input[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the tokenizer automatically added some special tokens that the model expects. Not all models need\n",
    "special tokens; for instance, if we had used _gpt2-medium_ instead of _bert-base-cased_ to create our tokenizer, we\n",
    "would have seen the same sentence as the original one here. You can disable this behavior (which is only advised if you\n",
    "have added those special tokens yourself) by passing `add_special_tokens=False`.\n",
    "\n",
    "If you have several sentences you want to process, you can do this efficiently by sending them as a list to the\n",
    "tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102],\n",
       "               [101, 1262, 1330, 5650, 102],\n",
       "               [101, 1262, 1103, 1304, 1304, 1314, 1141, 102]],\n",
       " 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "                    [0, 0, 0, 0, 0],\n",
       "                    [0, 0, 0, 0, 0, 0, 0, 0]],\n",
       " 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "                    [1, 1, 1, 1, 1],\n",
       "                    [1, 1, 1, 1, 1, 1, 1, 1]]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_sentences = [\"Hello I'm a single sentence\", \"And another sentence\", \"And the very very last one\"]\n",
    "encoded_inputs = tokenizer(batch_sentences)\n",
    "print(encoded_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get back a dictionary once again, this time with values being lists of lists of ints.\n",
    "\n",
    "If the purpose of sending several sentences at a time to the tokenizer is to build a batch to feed the model, you will\n",
    "probably want:\n",
    "\n",
    "- To pad each sentence to the maximum length there is in your batch.\n",
    "- To truncate each sentence to the maximum length the model can accept (if applicable).\n",
    "- To return tensors.\n",
    "\n",
    "You can do all of this by using the following options when feeding your list of sentences to the tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[ 101, 8667,  146,  112,  182,  170, 1423, 5650,  102],\n",
       "                      [ 101, 1262, 1330, 5650,  102,    0,    0,    0,    0],\n",
       "                      [ 101, 1262, 1103, 1304, 1304, 1314, 1141,  102,    0]]),\n",
       " 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "                           [0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "                           [0, 0, 0, 0, 0, 0, 0, 0, 0]]), \n",
       " 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "                           [1, 1, 1, 1, 1, 0, 0, 0, 0],\n",
       "                           [1, 1, 1, 1, 1, 1, 1, 1, 0]])}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors=\"pt\")\n",
    "print(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tf.Tensor([[ 101, 8667,  146,  112,  182,  170, 1423, 5650,  102],\n",
       "                      [ 101, 1262, 1330, 5650,  102,    0,    0,    0,    0],\n",
       "                      [ 101, 1262, 1103, 1304, 1304, 1314, 1141,  102,    0]]),\n",
       " 'token_type_ids': tf.Tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "                           [0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "                           [0, 0, 0, 0, 0, 0, 0, 0, 0]]), \n",
       " 'attention_mask': tf.Tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "                           [1, 1, 1, 1, 1, 0, 0, 0, 0],\n",
       "                           [1, 1, 1, 1, 1, 1, 1, 1, 0]])}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors=\"tf\")\n",
    "print(batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It returns a dictionary with string keys and tensor values. We can now see what the [attention_mask](https://huggingface.co/docs/transformers/v4.16.2/en/glossary#attention-mask) is all about: it points out which tokens the model should pay attention to and which ones\n",
    "it should not (because they represent padding in this case).\n",
    "\n",
    "\n",
    "Note that if your model does not have a maximum length associated to it, the command above will throw a warning. You\n",
    "can safely ignore it. You can also pass `verbose=False` to stop the tokenizer from throwing those kinds of warnings.\n",
    "\n",
    "<a id='sentence-pairs'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing pairs of sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/0u3ioSwev3s?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#@title\n",
    "from IPython.display import HTML\n",
    "\n",
    "HTML('<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/0u3ioSwev3s?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you need to feed a pair of sentences to your model. For instance, if you want to classify if two sentences in\n",
    "a pair are similar, or for question-answering models, which take a context and a question. For BERT models, the input\n",
    "is then represented like this: `[CLS] Sequence A [SEP] Sequence B [SEP]`\n",
    "\n",
    "You can encode a pair of sentences in the format expected by your model by supplying the two sentences as two arguments\n",
    "(not a list since a list of two sentences will be interpreted as a batch of two single sentences, as we saw before).\n",
    "This will once again return a dict string to list of ints:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 1731, 1385, 1132, 1128, 136, 102, 146, 112, 182, 127, 1201, 1385, 102], \n",
       " 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1], \n",
       " 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_input = tokenizer(\"How old are you?\", \"I'm 6 years old\")\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows us what the [token_type_ids](https://huggingface.co/docs/transformers/v4.16.2/en/glossary#token-type-ids) are for: they indicate to the model which part of\n",
    "the inputs correspond to the first sentence and which part corresponds to the second sentence. Note that\n",
    "_token_type_ids_ are not required or handled by all models. By default, a tokenizer will only return the inputs that\n",
    "its associated model expects. You can force the return (or the non-return) of any of those special arguments by using\n",
    "`return_input_ids` or `return_token_type_ids`.\n",
    "\n",
    "If we decode the token ids we obtained, we will see that the special tokens have been properly added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"[CLS] How old are you? [SEP] I'm 6 years old [SEP]\""
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(encoded_input[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a list of pairs of sequences you want to process, you should feed them as two lists to your tokenizer: the\n",
    "list of first sentences and the list of second sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102, 146, 112, 182, 170, 5650, 1115, 2947, 1114, 1103, 1148, 5650, 102], \n",
       "               [101, 1262, 1330, 5650, 102, 1262, 146, 1431, 1129, 12544, 1114, 1103, 1248, 5650, 102], \n",
       "               [101, 1262, 1103, 1304, 1304, 1314, 1141, 102, 1262, 146, 1301, 1114, 1103, 1304, 1314, 1141, 102]], \n",
       "'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \n",
       "                   [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \n",
       "                   [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]], \n",
       "'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \n",
       "                   [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \n",
       "                   [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_sentences = [\"Hello I'm a single sentence\", \"And another sentence\", \"And the very very last one\"]\n",
    "batch_of_second_sentences = [\n",
    "    \"I'm a sentence that goes with the first sentence\",\n",
    "    \"And I should be encoded with the second sentence\",\n",
    "    \"And I go with the very last one\",\n",
    "]\n",
    "encoded_inputs = tokenizer(batch_sentences, batch_of_second_sentences)\n",
    "print(encoded_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, it returns a dictionary where each value is a list of lists of ints.\n",
    "\n",
    "To double-check what is fed to the model, we can decode each list in _input_ids_ one by one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[CLS] Hello I'm a single sentence [SEP] I'm a sentence that goes with the first sentence [SEP]\n",
       "[CLS] And another sentence [SEP] And I should be encoded with the second sentence [SEP]\n",
       "[CLS] And the very very last one [SEP] And I go with the very last one [SEP]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for ids in encoded_inputs[\"input_ids\"]:\n",
    "    print(tokenizer.decode(ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, you can automatically pad your inputs to the maximum sentence length in the batch, truncate to the maximum\n",
    "length the model can accept and return tensors directly with the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = tokenizer(batch_sentences, batch_of_second_sentences, padding=True, truncation=True, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = tokenizer(batch_sentences, batch_of_second_sentences, padding=True, truncation=True, return_tensors=\"tf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Everything you always wanted to know about padding and truncation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have seen the commands that will work for most cases (pad your batch to the length of the maximum sentence and\n",
    "truncate to the maximum length the model can accept). However, the API supports more strategies if you need them. The\n",
    "three arguments you need to know for this are `padding`, `truncation` and `max_length`.\n",
    "\n",
    "- `padding` controls the padding. It can be a boolean or a string which should be:\n",
    "\n",
    "  - `True` or `'longest'` to pad to the longest sequence in the batch (doing no padding if you only provide\n",
    "    a single sequence).\n",
    "  - `'max_length'` to pad to a length specified by the `max_length` argument or the maximum length accepted\n",
    "    by the model if no `max_length` is provided (`max_length=None`). If you only provide a single sequence,\n",
    "    padding will still be applied to it.\n",
    "  - `False` or `'do_not_pad'` to not pad the sequences. As we have seen before, this is the default\n",
    "    behavior.\n",
    "\n",
    "- `truncation` controls the truncation. It can be a boolean or a string which should be:\n",
    "\n",
    "  - `True` or `'longest_first'` truncate to a maximum length specified by the `max_length` argument or\n",
    "    the maximum length accepted by the model if no `max_length` is provided (`max_length=None`). This will\n",
    "    truncate token by token, removing a token from the longest sequence in the pair until the proper length is\n",
    "    reached.\n",
    "  - `'only_second'` truncate to a maximum length specified by the `max_length` argument or the maximum\n",
    "    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate\n",
    "    the second sentence of a pair if a pair of sequence (or a batch of pairs of sequences) is provided.\n",
    "  - `'only_first'` truncate to a maximum length specified by the `max_length` argument or the maximum\n",
    "    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate\n",
    "    the first sentence of a pair if a pair of sequence (or a batch of pairs of sequences) is provided.\n",
    "  - `False` or `'do_not_truncate'` to not truncate the sequences. As we have seen before, this is the\n",
    "    default behavior.\n",
    "\n",
    "- `max_length` to control the length of the padding/truncation. It can be an integer or `None`, in which case\n",
    "  it will default to the maximum length the model can accept. If the model has no specific maximum input length,\n",
    "  truncation/padding to `max_length` is deactivated.\n",
    "\n",
    "Here is a table summarizing the recommend way to setup padding and truncation. If you use pair of inputs sequence in\n",
    "any of the following examples, you can replace `truncation=True` by a `STRATEGY` selected in\n",
    "`['only_first', 'only_second', 'longest_first']`, i.e. `truncation='only_second'` or `truncation= 'longest_first'` to control how both sequence in the pair are truncated as detailed before.\n",
    "\n",
    "| Truncation                           | Padding                           | Instruction                                                                                 |\n",
    "|--------------------------------------|-----------------------------------|---------------------------------------------------------------------------------------------|\n",
    "| no truncation                        | no padding                        | `tokenizer(batch_sentences)`                                                           |\n",
    "|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True)` or                                          |\n",
    "|                                      |                                   | `tokenizer(batch_sentences, padding='longest')`                                        |\n",
    "|                                      | padding to max model input length | `tokenizer(batch_sentences, padding='max_length')`                                     |\n",
    "|                                      | padding to specific length        | `tokenizer(batch_sentences, padding='max_length', max_length=42)`                      |\n",
    "| truncation to max model input length | no padding                        | `tokenizer(batch_sentences, truncation=True)` or                                       |\n",
    "|                                      |                                   | `tokenizer(batch_sentences, truncation=STRATEGY)`                                      |\n",
    "|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True, truncation=True)` or                         |\n",
    "|                                      |                                   | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY)`                        |\n",
    "|                                      | padding to max model input length | `tokenizer(batch_sentences, padding='max_length', truncation=True)` or                 |\n",
    "|                                      |                                   | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY)`                |\n",
    "|                                      | padding to specific length        | Not possible                                                                                |\n",
    "| truncation to specific length        | no padding                        | `tokenizer(batch_sentences, truncation=True, max_length=42)` or                        |\n",
    "|                                      |                                   | `tokenizer(batch_sentences, truncation=STRATEGY, max_length=42)`                       |\n",
    "|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True, truncation=True, max_length=42)` or          |\n",
    "|                                      |                                   | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY, max_length=42)`         |\n",
    "|                                      | padding to max model input length | Not possible                                                                                |\n",
    "|                                      | padding to specific length        | `tokenizer(batch_sentences, padding='max_length', truncation=True, max_length=42)` or  |\n",
    "|                                      |                                   | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY, max_length=42)` |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-tokenized inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizer also accept pre-tokenized inputs. This is particularly useful when you want to compute labels and extract\n",
    "predictions in [named entity recognition (NER)](https://en.wikipedia.org/wiki/Named-entity_recognition) or\n",
    "[part-of-speech tagging (POS tagging)](https://en.wikipedia.org/wiki/Part-of-speech_tagging).\n",
    "\n",
    "<Tip warning={true}>\n",
    "\n",
    "Pre-tokenized does not mean your inputs are already tokenized (you wouldn't need to pass them through the tokenizer\n",
    "if that was the case) but just split into words (which is often the first step in subword tokenization algorithms\n",
    "like BPE).\n",
    "\n",
    "</Tip>\n",
    "\n",
    "If you want to use pre-tokenized inputs, just set `is_split_into_words=True` when passing your inputs to the\n",
    "tokenizer. For instance, we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 8667, 146, 112, 182, 170, 1423, 5650, 102],\n",
       " 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], \n",
       " 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_input = tokenizer([\"Hello\", \"I'm\", \"a\", \"single\", \"sentence\"], is_split_into_words=True)\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the tokenizer still adds the ids of special tokens (if applicable) unless you pass\n",
    "`add_special_tokens=False`.\n",
    "\n",
    "This works exactly as before for batch of sentences or batch of pairs of sentences. You can encode a batch of sentences\n",
    "like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_sentences = [\n",
    "    [\"Hello\", \"I'm\", \"a\", \"single\", \"sentence\"],\n",
    "    [\"And\", \"another\", \"sentence\"],\n",
    "    [\"And\", \"the\", \"very\", \"very\", \"last\", \"one\"],\n",
    "]\n",
    "encoded_inputs = tokenizer(batch_sentences, is_split_into_words=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or a batch of pair sentences like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_of_second_sentences = [\n",
    "    [\"I'm\", \"a\", \"sentence\", \"that\", \"goes\", \"with\", \"the\", \"first\", \"sentence\"],\n",
    "    [\"And\", \"I\", \"should\", \"be\", \"encoded\", \"with\", \"the\", \"second\", \"sentence\"],\n",
    "    [\"And\", \"I\", \"go\", \"with\", \"the\", \"very\", \"last\", \"one\"],\n",
    "]\n",
    "encoded_inputs = tokenizer(batch_sentences, batch_of_second_sentences, is_split_into_words=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you can add padding, truncation as well as directly return tensors like before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = tokenizer(\n",
    "    batch_sentences,\n",
    "    batch_of_second_sentences,\n",
    "    is_split_into_words=True,\n",
    "    padding=True,\n",
    "    truncation=True,\n",
    "    return_tensors=\"pt\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = tokenizer(\n",
    "    batch_sentences,\n",
    "    batch_of_second_sentences,\n",
    "    is_split_into_words=True,\n",
    "    padding=True,\n",
    "    truncation=True,\n",
    "    return_tensors=\"tf\",\n",
    ")"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 4
}