Commits

History for

Commits on Jun 1, 2021

Add regression tests for slow sentencepiece tokenizers. (#11737 )
PhilipMay
authored

Commits on May 13, 2021

Enable option for subword regularization in more tokenizers. (#11417 )
PhilipMay
authored

Commits on Dec 7, 2020

Copyright (#8970 )
sgugger
authored

Commits on Oct 18, 2020

[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659 )

thomwolf
and
sgugger
authored

Commits on Oct 8, 2020

Adding Fast tokenizers for SentencePiece based tokenizers - Breaking: remove Transfo-XL fast tokenizer (#7141 )

thomwolf
and
sgugger
authored

Commits on Jul 1, 2020

Move tests/utils.py -> transformers/testing_utils.py (#5350 )
sshleifer
authored

Commits on May 19, 2020

[cleanup] test_tokenization_common.py (#4390 )
sshleifer
authored

Commits on Jan 15, 2020

💄 super
julien-c
committed

Commits on Jan 6, 2020

GPU text generation: mMoved the encoded_prompt to correct device

alberduris
authored and
LysandreJik
committed
Moved the encoded_prompts to correct device

alberduris
authored and
LysandreJik
committed

Loading