|
126 | 126 | "source": [
|
127 | 127 | "The next step is to tokenize the text into a readable format by the model. It is important to load the same tokenizer a\n",
|
128 | 128 | "model was trained with to ensure appropriately tokenized words. Load the DistilBERT tokenizer with the\n",
|
129 |
| - "[AutoTokenizer](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoTokenizer) because we will eventually train a classifier using a pretrained [DistilBERT](https://huggingface.co/distilbert-base-uncased) model:" |
| 129 | + "[AutoTokenizer](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.AutoTokenizer) because we will eventually train a classifier using a pretrained [DistilBERT](https://huggingface.co/distilbert-base-uncased) model:" |
130 | 130 | ]
|
131 | 131 | },
|
132 | 132 | {
|
|
207 | 207 | "cell_type": "markdown",
|
208 | 208 | "metadata": {},
|
209 | 209 | "source": [
|
210 |
| - "Now load your model with the [AutoModelForSequenceClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForSequenceClassification) class along with the number of expected labels:" |
| 210 | + "Now load your model with the [AutoModelForSequenceClassification](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.AutoModelForSequenceClassification) class along with the number of expected labels:" |
211 | 211 | ]
|
212 | 212 | },
|
213 | 213 | {
|
|
227 | 227 | "source": [
|
228 | 228 | "At this point, only three steps remain:\n",
|
229 | 229 | "\n",
|
230 |
| - "1. Define your training hyperparameters in [TrainingArguments](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.TrainingArguments).\n", |
231 |
| - "2. Pass the training arguments to a [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) along with the model, dataset, tokenizer, and data collator.\n", |
| 230 | + "1. Define your training hyperparameters in [TrainingArguments](https://huggingface.co/docs/transformers/doc-build-test/en/main_classes/trainer#transformers.TrainingArguments).\n", |
| 231 | + "2. Pass the training arguments to a [Trainer](https://huggingface.co/docs/transformers/doc-build-test/en/main_classes/trainer#transformers.Trainer) along with the model, dataset, tokenizer, and data collator.\n", |
232 | 232 | "3. Call `Trainer.train()` to fine-tune your model."
|
233 | 233 | ]
|
234 | 234 | },
|
|
274 | 274 | "source": [
|
275 | 275 | "Fine-tuning with TensorFlow is just as easy, with only a few differences.\n",
|
276 | 276 | "\n",
|
277 |
| - "Start by batching the processed examples together with dynamic padding using the [DataCollatorWithPadding](https://huggingface.co/docs/transformers/master/en/main_classes/data_collator#transformers.DataCollatorWithPadding) function.\n", |
| 277 | + "Start by batching the processed examples together with dynamic padding using the [DataCollatorWithPadding](https://huggingface.co/docs/transformers/doc-build-test/en/main_classes/data_collator#transformers.DataCollatorWithPadding) function.\n", |
278 | 278 | "Make sure you set `return_tensors=\"tf\"` to return `tf.Tensor` outputs instead of PyTorch tensors!"
|
279 | 279 | ]
|
280 | 280 | },
|
|
345 | 345 | "cell_type": "markdown",
|
346 | 346 | "metadata": {},
|
347 | 347 | "source": [
|
348 |
| - "Load your model with the [TFAutoModelForSequenceClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.TFAutoModelForSequenceClassification) class along with the number of expected labels:" |
| 348 | + "Load your model with the [TFAutoModelForSequenceClassification](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.TFAutoModelForSequenceClassification) class along with the number of expected labels:" |
349 | 349 | ]
|
350 | 350 | },
|
351 | 351 | {
|
|
548 | 548 | "cell_type": "markdown",
|
549 | 549 | "metadata": {},
|
550 | 550 | "source": [
|
551 |
| - "Now you need to tokenize the text. Load the DistilBERT tokenizer with an [AutoTokenizer](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoTokenizer):" |
| 551 | + "Now you need to tokenize the text. Load the DistilBERT tokenizer with an [AutoTokenizer](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.AutoTokenizer):" |
552 | 552 | ]
|
553 | 553 | },
|
554 | 554 | {
|
|
680 | 680 | "cell_type": "markdown",
|
681 | 681 | "metadata": {},
|
682 | 682 | "source": [
|
683 |
| - "Load your model with the [AutoModelForTokenClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForTokenClassification) class along with the number of expected labels:" |
| 683 | + "Load your model with the [AutoModelForTokenClassification](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.AutoModelForTokenClassification) class along with the number of expected labels:" |
684 | 684 | ]
|
685 | 685 | },
|
686 | 686 | {
|
|
698 | 698 | "cell_type": "markdown",
|
699 | 699 | "metadata": {},
|
700 | 700 | "source": [
|
701 |
| - "Gather your training arguments in [TrainingArguments](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.TrainingArguments):" |
| 701 | + "Gather your training arguments in [TrainingArguments](https://huggingface.co/docs/transformers/doc-build-test/en/main_classes/trainer#transformers.TrainingArguments):" |
702 | 702 | ]
|
703 | 703 | },
|
704 | 704 | {
|
|
722 | 722 | "cell_type": "markdown",
|
723 | 723 | "metadata": {},
|
724 | 724 | "source": [
|
725 |
| - "Collect your model, training arguments, dataset, data collator, and tokenizer in [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer):" |
| 725 | + "Collect your model, training arguments, dataset, data collator, and tokenizer in [Trainer](https://huggingface.co/docs/transformers/doc-build-test/en/main_classes/trainer#transformers.Trainer):" |
726 | 726 | ]
|
727 | 727 | },
|
728 | 728 | {
|
|
814 | 814 | "cell_type": "markdown",
|
815 | 815 | "metadata": {},
|
816 | 816 | "source": [
|
817 |
| - "Load the model with the [TFAutoModelForTokenClassification](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.TFAutoModelForTokenClassification) class along with the number of expected labels:" |
| 817 | + "Load the model with the [TFAutoModelForTokenClassification](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.TFAutoModelForTokenClassification) class along with the number of expected labels:" |
818 | 818 | ]
|
819 | 819 | },
|
820 | 820 | {
|
|
990 | 990 | "cell_type": "markdown",
|
991 | 991 | "metadata": {},
|
992 | 992 | "source": [
|
993 |
| - "Load the DistilBERT tokenizer with an [AutoTokenizer](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoTokenizer):" |
| 993 | + "Load the DistilBERT tokenizer with an [AutoTokenizer](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.AutoTokenizer):" |
994 | 994 | ]
|
995 | 995 | },
|
996 | 996 | {
|
|
1123 | 1123 | "cell_type": "markdown",
|
1124 | 1124 | "metadata": {},
|
1125 | 1125 | "source": [
|
1126 |
| - "Load your model with the [AutoModelForQuestionAnswering](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForQuestionAnswering) class:" |
| 1126 | + "Load your model with the [AutoModelForQuestionAnswering](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.AutoModelForQuestionAnswering) class:" |
1127 | 1127 | ]
|
1128 | 1128 | },
|
1129 | 1129 | {
|
|
1141 | 1141 | "cell_type": "markdown",
|
1142 | 1142 | "metadata": {},
|
1143 | 1143 | "source": [
|
1144 |
| - "Gather your training arguments in [TrainingArguments](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.TrainingArguments):" |
| 1144 | + "Gather your training arguments in [TrainingArguments](https://huggingface.co/docs/transformers/doc-build-test/en/main_classes/trainer#transformers.TrainingArguments):" |
1145 | 1145 | ]
|
1146 | 1146 | },
|
1147 | 1147 | {
|
|
1165 | 1165 | "cell_type": "markdown",
|
1166 | 1166 | "metadata": {},
|
1167 | 1167 | "source": [
|
1168 |
| - "Collect your model, training arguments, dataset, data collator, and tokenizer in [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer):" |
| 1168 | + "Collect your model, training arguments, dataset, data collator, and tokenizer in [Trainer](https://huggingface.co/docs/transformers/doc-build-test/en/main_classes/trainer#transformers.Trainer):" |
1169 | 1169 | ]
|
1170 | 1170 | },
|
1171 | 1171 | {
|
|
1284 | 1284 | "cell_type": "markdown",
|
1285 | 1285 | "metadata": {},
|
1286 | 1286 | "source": [
|
1287 |
| - "Load your model with the [TFAutoModelForQuestionAnswering](https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.TFAutoModelForQuestionAnswering) class:" |
| 1287 | + "Load your model with the [TFAutoModelForQuestionAnswering](https://huggingface.co/docs/transformers/doc-build-test/en/model_doc/auto#transformers.TFAutoModelForQuestionAnswering) class:" |
1288 | 1288 | ]
|
1289 | 1289 | },
|
1290 | 1290 | {
|
|
0 commit comments