Commits

History for

Commits on Jun 1, 2021

Add regression tests for slow sentencepiece tokenizers. (#11737 )
PhilipMay
authored

Commits on May 13, 2021

Enable option for subword regularization in more tokenizers. (#11417 )
PhilipMay
authored

Commits on Feb 2, 2021

ALBERT Tokenizer integration test (#9943 )
LysandreJik
authored

Commits on Oct 18, 2020

[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659 )

thomwolf
and
sgugger
authored

Commits on Oct 8, 2020

Adding Fast tokenizers for SentencePiece based tokenizers - Breaking: remove Transfo-XL fast tokenizer (#7141 )

thomwolf
and
sgugger
authored

Commits on Jun 15, 2020

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510 )

authored

Commits on May 19, 2020

[cleanup] test_tokenization_common.py (#4390 )
sshleifer
authored

Commits on Jan 15, 2020

💄 super
julien-c
committed

Commits on Jan 6, 2020

GPU text generation: mMoved the encoded_prompt to correct device

alberduris
authored and
LysandreJik
committed
Moved the encoded_prompts to correct device

alberduris
authored and
LysandreJik
committed

Loading