Skip to content

Commit 30fc249

Browse files
sguggersgugger
committedDec 23, 2020
Fix script to ignore copyright and update
1 parent a332869 commit 30fc249

28 files changed

+43
-501
lines changed
 

‎transformers_doc/benchmarks.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/custom_datasets.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/multilingual.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/perplexity.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/preprocessing.ipynb

+5-23
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},
@@ -44,10 +27,10 @@
4427
"call a [tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). You can build one using the tokenizer class associated to the model\n",
4528
"you would like to use, or directly with the `AutoTokenizer` class.\n",
4629
"\n",
47-
"As we saw in the [quicktour](https://huggingface.co/transformers/quicktour.html), the tokenizer will first split a given text in words (or part of words,\n",
48-
"punctuation symbols, etc.) usually called *tokens*. Then it will convert those *tokens* into numbers, to be able to\n",
49-
"build a tensor out of them and feed them to the model. It will also add any additional inputs the model might expect to\n",
50-
"work properly."
30+
"As we saw in the [quick tour](https://huggingface.co/transformers/quicktour.html), the tokenizer will first split a given text in words (or part of\n",
31+
"words, punctuation symbols, etc.) usually called *tokens*. Then it will convert those *tokens* into numbers, to be able\n",
32+
"to build a tensor out of them and feed them to the model. It will also add any additional inputs the model might expect\n",
33+
"to work properly."
5134
]
5235
},
5336
{
@@ -276,7 +259,7 @@
276259
"\n",
277260
"\n",
278261
"Note that if your model does not have a maximum length associated to it, the command above will throw a warning. You\n",
279-
"can safely ignore it. You can also pass `verbose=False` to stop the tokenizer to throw those kinds of warnings."
262+
"can safely ignore it. You can also pass `verbose=False` to stop the tokenizer from throwing those kinds of warnings."
280263
]
281264
},
282265
{
@@ -477,7 +460,6 @@
477460
"metadata": {},
478461
"source": [
479462
"We have seen the commands that will work for most cases (pad your batch to the length of the maximum sentence and\n",
480-
"\n",
481463
"truncate to the maximum length the mode can accept). However, the API supports more strategies if you need them. The\n",
482464
"three arguments you need to know for this are `padding`, `truncation` and `max_length`.\n",
483465
"\n",

‎transformers_doc/pytorch/benchmarks.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/pytorch/custom_datasets.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/pytorch/multilingual.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/pytorch/perplexity.ipynb

-17
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},

‎transformers_doc/pytorch/preprocessing.ipynb

+5-23
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},
@@ -44,10 +27,10 @@
4427
"call a [tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). You can build one using the tokenizer class associated to the model\n",
4528
"you would like to use, or directly with the `AutoTokenizer` class.\n",
4629
"\n",
47-
"As we saw in the [quicktour](https://huggingface.co/transformers/quicktour.html), the tokenizer will first split a given text in words (or part of words,\n",
48-
"punctuation symbols, etc.) usually called *tokens*. Then it will convert those *tokens* into numbers, to be able to\n",
49-
"build a tensor out of them and feed them to the model. It will also add any additional inputs the model might expect to\n",
50-
"work properly."
30+
"As we saw in the [quick tour](https://huggingface.co/transformers/quicktour.html), the tokenizer will first split a given text in words (or part of\n",
31+
"words, punctuation symbols, etc.) usually called *tokens*. Then it will convert those *tokens* into numbers, to be able\n",
32+
"to build a tensor out of them and feed them to the model. It will also add any additional inputs the model might expect\n",
33+
"to work properly."
5134
]
5235
},
5336
{
@@ -245,7 +228,7 @@
245228
"\n",
246229
"\n",
247230
"Note that if your model does not have a maximum length associated to it, the command above will throw a warning. You\n",
248-
"can safely ignore it. You can also pass `verbose=False` to stop the tokenizer to throw those kinds of warnings."
231+
"can safely ignore it. You can also pass `verbose=False` to stop the tokenizer from throwing those kinds of warnings."
249232
]
250233
},
251234
{
@@ -435,7 +418,6 @@
435418
"metadata": {},
436419
"source": [
437420
"We have seen the commands that will work for most cases (pad your batch to the length of the maximum sentence and\n",
438-
"\n",
439421
"truncate to the maximum length the mode can accept). However, the API supports more strategies if you need them. The\n",
440422
"three arguments you need to know for this are `padding`, `truncation` and `max_length`.\n",
441423
"\n",

‎transformers_doc/pytorch/quicktour.ipynb

+1-18
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},
@@ -287,7 +270,7 @@
287270
"source": [
288271
"We mentioned the tokenizer is responsible for the preprocessing of your texts. First, it will split a given text in\n",
289272
"words (or part of words, punctuation symbols, etc.) usually called *tokens*. There are multiple rules that can govern\n",
290-
"that process (you can learn more about them in the [tokenizer summary](https://huggingface.co/transformers/tokenizer_summary.html), which is why we need\n",
273+
"that process (you can learn more about them in the [tokenizer summary](https://huggingface.co/transformers/tokenizer_summary.html)), which is why we need\n",
291274
"to instantiate the tokenizer using the name of the model, to make sure we use the same rules as when the model was\n",
292275
"pretrained.\n",
293276
"\n",

‎transformers_doc/pytorch/task_summary.ipynb

+2-19
Original file line numberDiff line numberDiff line change
@@ -12,23 +12,6 @@
1212
"# ! pip install git+https://github.com/huggingface/transformers.git\n"
1313
]
1414
},
15-
{
16-
"cell_type": "markdown",
17-
"metadata": {},
18-
"source": [
19-
".. \n",
20-
" Copyright 2020 The HuggingFace Team. All rights reserved.\n",
21-
"\n",
22-
" Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\n",
23-
" the License. You may obtain a copy of the License at\n",
24-
"\n",
25-
" http://www.apache.org/licenses/LICENSE-2.0\n",
26-
"\n",
27-
" Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on\n",
28-
" an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\n",
29-
" specific language governing permissions and limitations under the License."
30-
]
31-
},
3215
{
3316
"cell_type": "markdown",
3417
"metadata": {},
@@ -364,7 +347,7 @@
364347
"Masked language modeling is the task of masking tokens in a sequence with a masking token, and prompting the model to\n",
365348
"fill that mask with an appropriate token. This allows the model to attend to both the right context (tokens on the\n",
366349
"right of the mask) and the left context (tokens on the left of the mask). Such a training creates a strong basis for\n",
367-
"downstream tasks, requiring bi-directional context such as SQuAD (question answering, see [Lewis, Lui, Goyal et al.](https://arxiv.org/abs/1910.13461), part 4.2).\n",
350+
"downstream tasks requiring bi-directional context, such as SQuAD (question answering, see [Lewis, Lui, Goyal et al.](https://arxiv.org/abs/1910.13461), part 4.2).\n",
368351
"\n",
369352
"Here is an example of using pipelines to replace a mask from a sequence:"
370353
]
@@ -782,7 +765,7 @@
782765
"cell_type": "markdown",
783766
"metadata": {},
784767
"source": [
785-
"Note, how the tokens of the sequence \"Hugging Face\" have been identified as an organisation, and \"New York City\",\n",
768+
"Note how the tokens of the sequence \"Hugging Face\" have been identified as an organisation, and \"New York City\",\n",
786769
"\"DUMBO\" and \"Manhattan Bridge\" have been identified as locations.\n",
787770
"\n",
788771
"Here is an example of doing named entity recognition, using a model and a tokenizer. The process is the following:\n",

0 commit comments

Comments
 (0)
Please sign in to comment.