You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
🍟 [SFT] Handles the dataset if it has been preprocessed (#2863)
* return dataset if it's preprocessed
* add is_processed flag variable
* add test
* move test_sft_trainer_directly_with_pretokenized_data to Tester2
* Update sft_trainer.py
* no need for padding and truncation
* minor reorganization
* Update trl/trainer/sft_trainer.py
* let the collator pad
* style
* fix tests
---------
Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
Co-authored-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co>
0 commit comments