Commits

Breadcrumbs

History for

onv4.19.4

User selector

All users

Datepicker

All time

Commit History

Commits on Apr 13, 2022

• Fix #16660 (tokenizers setters of ids of special tokens) (#16661)

  Show description for 9f8bfe7
  davidleonfdez

  authored
  9f8bfe7

  Copy full SHA for 9f8bfe7

Commits on Apr 4, 2022

• add a test checking the format of convert_tokens_to_string's output (#16540)

  Show description for be9474b
  SaulLu

  authored
  be9474b

  Copy full SHA for be9474b

Commits on Mar 23, 2022

• Make Transformers use cache files when hf.co is down (#16362)

  Show description for c595b6e
  sgugger

  authored
  c595b6e

  Copy full SHA for c595b6e

Commits on Feb 15, 2022

• Allow custom code for Processors (#15649)

  Show description for 45f5658
  sgugger

  authored
  45f5658

  Copy full SHA for 45f5658

Commits on Feb 2, 2022

• fix set truncation attribute in __init__ of PreTrainedTokenizerBase (#15456)

  Show description for 39b5d1a
  

  3 peopleauthored
  39b5d1a

  Copy full SHA for 39b5d1a

• Save code of registered custom models (#15379)

  Show description for 44b21f1
  

  3 peopleauthored
  44b21f1

  Copy full SHA for 44b21f1

Commits on Feb 1, 2022

• fix the tokenizer_config.json file for the slow tokenizer when a fast version is available (#15319)

  Show description for 7b8bdd8
  SaulLu

  authored
  7b8bdd8

  Copy full SHA for 7b8bdd8

• replace assert with exception for padding_side arg in PreTrainedTokenizerBase __init__ (#15454)

  Show description for 6d585fe
  SaulLu

  authored
  6d585fe

  Copy full SHA for 6d585fe

Commits on Jan 27, 2022

• improve saving strategy of sentencepiece tokenizer (#15328)

  Show description for ade7371
  

  SaulLu

  and

  sgugger

  authored
  ade7371

  Copy full SHA for ade7371

Commits on Jan 6, 2022

• Remove old asserts. (#15012)

  Narsil

  authored
  d2183a4

  Copy full SHA for d2183a4

Commits on Jan 3, 2022

• Improve truncation_side (#14947)

  Show description for d33dc79
  

  Narsil

  and

  NielsRogge

  authored
  d33dc79

  Copy full SHA for d33dc79

Commits on Dec 30, 2021

• Fixing a pathological case for slow tokenizers (#14981)

  Show description for d7d60df
  Narsil

  authored
  d7d60df

  Copy full SHA for d7d60df

Commits on Dec 3, 2021

• Improve tokenizer tests (#13594)

  Show description for 66ea739
  qqaatw

  authored
  66ea739

  Copy full SHA for 66ea739

Commits on Nov 10, 2021

• Fix list index out of range when padding nested empty lists (#13876)

  Show description for 9e37c5c
  qqaatw

  authored
  9e37c5c

  Copy full SHA for 9e37c5c

Commits on Nov 8, 2021

• Expand dynamic supported objects to configs and tokenizers (#14296)

  Show description for dfb00bf
  sgugger

  authored
  dfb00bf

  Copy full SHA for dfb00bf

Commits on Nov 2, 2021

• Update Transformers to huggingface_hub >= 0.1.0 (#14251)

  Show description for 558f854
  sgugger

  authored
  558f854

  Copy full SHA for 558f854

Commits on Oct 11, 2021

• Honor existing attention mask in tokenzier.pad (#13926)

  Show description for 4a18337
  sgugger

  authored
  4a18337

  Copy full SHA for 4a18337

Commits on Oct 8, 2021

• Adding support for tokens being suffixes or part of each other. (#13918)

  Show description for d70919e
  Narsil

  authored
  d70919e

  Copy full SHA for d70919e

Commits on Oct 5, 2021

• Fixing 1-length special tokens cut. (#13862)

  Narsil

  authored
  7079a99

  Copy full SHA for 7079a99

Commits on Sep 17, 2021

• Fix special tokens not correctly tokenized (#13489)

  Show description for da8beaa
  qqaatw

  authored
  da8beaa

  Copy full SHA for da8beaa

Commits on Sep 9, 2021

• [Tentative] Moving slow tokenizer to the Trie world. (#13220)

  Show description for 3dd538c
  

  3 peopleauthored
  3dd538c

  Copy full SHA for 3dd538c

Commits on Sep 2, 2021

• Correct order of overflowing_tokens for slow tokenizer (#13179)

  Show description for b91e65a
  Apoorvgarg-creator

  authored
  b91e65a

  Copy full SHA for b91e65a

Commits on Sep 1, 2021

• Fix tokenizer saving during training with Trainer (#12806)

  Show description for c4d78f0
  

  SaulLu

  and

  sgugger

  authored
  c4d78f0

  Copy full SHA for c4d78f0

Commits on Aug 23, 2021

• Change how "additional_special_tokens" argument in the ".from_pretrained" method of the tokenizer is taken into account (#13056)

  Show description for 7223844
  SaulLu

  authored
  7223844

  Copy full SHA for 7223844

Commits on Jul 17, 2021

• Add tokenizers class mismatch detection between cls and checkpoint (#12619)

  Show description for 08d609b
  

  europeanplaice

  and

  SaulLu

  authored
  08d609b

  Copy full SHA for 08d609b

Commits on Jul 16, 2021

• Preserve list type of additional_special_tokens in special_token_map (#12759)

  Show description for 6e87010
  

  SaulLu

  and

  sgugger

  authored
  6e87010

  Copy full SHA for 6e87010

Commits on Jul 1, 2021

• Add test for a WordLevel tokenizer model (#12437)

  Show description for 3aa37b9
  SaulLu

  authored
  3aa37b9

  Copy full SHA for 3aa37b9

Commits on Jun 29, 2021

• Easily train a new fast tokenizer from a given one (#12361)

  Show description for dc42e77
  

  sgugger

  and

  SaulLu

  authored
  dc42e77

  Copy full SHA for dc42e77

Commits on Jun 23, 2021

• Clean push to hub API (#12187)

  Show description for 53c60ba
  

  3 peopleauthored
  53c60ba

  Copy full SHA for 53c60ba

Commits on Jun 14, 2021

• Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810)

  Show description for 476ba67
  

  3 peopleauthored
  476ba67

  Copy full SHA for 476ba67

Commits on Jun 7, 2021

• fix deberta 2 tokenizer integration test (#12017)

  PhilipMay

  authored
  3857f2b

  Copy full SHA for 3857f2b

Commits on Jun 1, 2021

• Add regression tests for slow sentencepiece tokenizers. (#11737)

  Show description for fcad801
  PhilipMay

  authored
  fcad801

  Copy full SHA for fcad801

Commits on May 13, 2021

• Enable option for subword regularization in more tokenizers. (#11417)

  Show description for 37ed3ab
  PhilipMay

  authored
  37ed3ab

  Copy full SHA for 37ed3ab

Commits on May 4, 2021

• Enable added tokens (#11325)

  Show description for 09b0bcf
  LysandreJik

  authored
  09b0bcf

  Copy full SHA for 09b0bcf

Commits on Apr 26, 2021

• Give each test a different repo name (#11453)

  sgugger

  authored
  7959d83

  Copy full SHA for 7959d83

Pagination

Previous

Next