LLaMA-Factory-310P3/examples/train_full/llama3_full_sft_ds3.yaml

### model
model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct

### method
stage: sft
do_train: true
finetuning_type: full
deepspeed: examples/deepspeed/ds_z3_config.json

### dataset
dataset: identity,alpaca_en_demo
template: llama3
cutoff_len: 1024
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/llama3-8b/full/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 2
learning_rate: 1.0e-5
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500
update examples 2024-05-17 01:02:00 +08:00			`### model`
update example docs 2024-05-06 22:51:02 +08:00			`model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct`

update examples 2024-05-17 01:02:00 +08:00			`### method`
update example docs 2024-05-06 22:51:02 +08:00			`stage: sft`
			`do_train: true`
			`finetuning_type: full`
			`deepspeed: examples/deepspeed/ds_z3_config.json`

update examples 2024-05-17 01:02:00 +08:00			`### dataset`
improve KTO impl., replace datasets 2024-05-18 03:44:56 +08:00			`dataset: identity,alpaca_en_demo`
update example docs 2024-05-06 22:51:02 +08:00			`template: llama3`
			`cutoff_len: 1024`
			`max_samples: 1000`
			`overwrite_cache: true`
			`preprocessing_num_workers: 16`

update examples 2024-05-17 01:02:00 +08:00			`### output`
update example docs 2024-05-06 22:51:02 +08:00			`output_dir: saves/llama3-8b/full/sft`
			`logging_steps: 10`
			`save_steps: 500`
			`plot_loss: true`
			`overwrite_output_dir: true`

update examples 2024-05-17 01:02:00 +08:00			`### train`
update example docs 2024-05-06 22:51:02 +08:00			`per_device_train_batch_size: 1`
			`gradient_accumulation_steps: 2`
fix #4944 2024-07-24 16:42:51 +08:00			`learning_rate: 1.0e-5`
update example docs 2024-05-06 22:51:02 +08:00			`num_train_epochs: 3.0`
			`lr_scheduler_type: cosine`
fix #4005 #4013 2024-06-03 19:12:29 +08:00			`warmup_ratio: 0.1`
update examples 2024-06-28 01:17:07 +08:00			`bf16: true`
Update llama3_full_sft_ds3.yaml 2024-06-13 03:16:20 +08:00			`ddp_timeout: 180000000`
update example docs 2024-05-06 22:51:02 +08:00
update examples 2024-05-17 01:02:00 +08:00			`### eval`
update examples 2024-05-13 20:39:36 +08:00			`val_size: 0.1`
update example docs 2024-05-06 22:51:02 +08:00			`per_device_eval_batch_size: 1`
update train hparams 2024-06-06 01:49:20 +08:00			`eval_strategy: steps`
update example docs 2024-05-06 22:51:02 +08:00			`eval_steps: 500`