Add QLoRA Train and Merge16bit Test

[jeromeku]

QLoRA Train and Merge Tests

Overview

Tests that performing QLoRA training and merging weights to 16-bits post-training maintains same behavior as trained model.

Details

The tests train a QLoRA model on a single prompt dataset

QUESTION = "What day was I born?"
ANSWER = "January 1, 2058"
USER_MESSAGE = {"role": "user", "content": QUESTION}
ASSISTANT_MESSAGE = {"role": "assistant", "content": ANSWER}

Given that the answer is impossible to answer accurately without finetuning, we can only expect the model to answer the question correctly if the model has been trained on the question.

To check this behavior, we check the model's response to the question before and after training and after merging, checking that the model's response contains the answer after training and merging but not before training.

Results

For the unsloth test, the model's behavior is as expected:

• before training, the model's response does not contain the answer
• after training, the model's response contains the answer
• after merging, the model's response contains the answer

For the huggingface test, the model's behavior is as expected:

• before training, the model's response does not contains the answer
• after training, the model's response contains the answer
• after using peft's merge_and_unload, the model's response does not contain the answer since it requantizes base layers after merging adapter weights
• after using a custom merge function which dequantizes and merges without requantizing, the model's response contains the answer.

Usage

Run unsloth test:

python tests/qlora/test_unsloth_qlora_train_and_merge.py

Run huggingface test:

python tests/qlora/test_hf_qlora_train_and_merge.py

The scripts should output training params, training logs, as well as model responses before and after training and after merging (only prints model responses if answer is not contained in response).

[danielhanchen]

@jeromeku Very very good! Could you also copy paste the generic Apache 2 license header to every file :)

[jeromeku]

@danielhanchen

done!

[danielhanchen]

Thanks!!