fix chatglm2 tokenizer

README.md

@@ -22,7 +22,7 @@
 
 [23/07/31] Now we support **dataset streaming**. Try `--streaming` and `--max_steps 10000` arguments to load your dataset in streaming mode.
 
-[23/07/29] We release two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos ([LLaMA-2](https://huggingface.co/hiyouga/Llama-2-Chinese-13b-chat) / [Baichuan](https://huggingface.co/hiyouga/baichuan-13b-sft)) for details.
+[23/07/29] We release two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos ([LLaMA-2](https://huggingface.co/hiyouga/Llama-2-Chinese-13b-chat) / [Baichuan](https://huggingface.co/hiyouga/Baichuan-13B-sft)) for details.
 
 [23/07/19] Now we support training the **LLaMA-2** models in this repo. Try `--model_name_or_path meta-llama/Llama-2-7b-hf` argument to use the LLaMA-2 model. Remember to use `--template llama2` argument when you are using the LLaMA-2-chat model.
 
@@ -36,7 +36,7 @@
 
 [23/07/05] Now we support training the **Falcon-7B/40B** models in this repo. Try `--model_name_or_path tiiuae/falcon-7b` and `--lora_target query_key_value` arguments to use the Falcon model.
 
-[23/06/29] We provide a **reproducible example** of training a chat model using instruction-following datasets, see this [Hugging Face Repo](https://huggingface.co/hiyouga/baichuan-7b-sft) for details.
+[23/06/29] We provide a **reproducible example** of training a chat model using instruction-following datasets, see [Baichuan-7B-sft](https://huggingface.co/hiyouga/Baichuan-7B-sft) for details.
 
 [23/06/22] Now we align the [demo API](src/api_demo.py) with the [OpenAI's](https://platform.openai.com/docs/api-reference/chat) format where you can insert the fine-tuned model in **arbitrary ChatGPT-based applications**.
 

README_zh.md

@@ -22,7 +22,7 @@
 
 [23/07/31] 现在我们支持了**数据流式加载**。请尝试使用 `--streaming` 和 `--max_steps 10000` 参数来流式加载数据集。
 
-[23/07/29] 我们在 Hugging Face 发布了两个 13B 指令微调模型。详细内容请查阅我们的 Hugging Face 项目（[LLaMA-2](https://huggingface.co/hiyouga/Llama-2-Chinese-13b-chat) / [Baichuan](https://huggingface.co/hiyouga/baichuan-13b-sft)）。
+[23/07/29] 我们在 Hugging Face 发布了两个 13B 指令微调模型。详细内容请查阅我们的 Hugging Face 项目（[LLaMA-2](https://huggingface.co/hiyouga/Llama-2-Chinese-13b-chat) / [Baichuan](https://huggingface.co/hiyouga/Baichuan-13B-sft)）。
 
 [23/07/19] 现在我们支持了 **LLaMA-2** 模型的训练。请尝试使用 `--model_name_or_path meta-llama/Llama-2-7b-hf` 参数。使用 LLaMA-2-chat 模型时请添加 `--template llama2` 参数。
 
@@ -36,7 +36,7 @@
 
 [23/07/05] 现在我们支持了 **Falcon-7B/40B** 模型的训练。请尝试使用 `--model_name_or_path tiiuae/falcon-7b` 和 `--lora_target query_key_value` 参数。
 
-[23/06/29] 我们提供了一个**可复现的**指令模型微调示例，详细内容请查阅 [Hugging Face 项目](https://huggingface.co/hiyouga/baichuan-7b-sft)。
+[23/06/29] 我们提供了一个**可复现的**指令模型微调示例，详细内容请查阅 [Baichuan-7B-sft](https://huggingface.co/hiyouga/Baichuan-7B-sft)。
 
 [23/06/22] 我们对齐了[示例 API](src/api_demo.py) 与 [OpenAI API](https://platform.openai.com/docs/api-reference/chat) 的格式，您可以将微调模型接入**任意基于 ChatGPT 的应用**中。
 

src/llmtuner/tuner/core/loader.py

@@ -72,6 +72,10 @@ def load_model_and_tokenizer(
         **config_kwargs
     )
 
+    # Fix tokenizer (for ChatGLM2)
+    if "PreTrainedTokenizerBase" not in str(tokenizer._pad.__func__):
+        tokenizer._pad = MethodType(PreTrainedTokenizerBase._pad, tokenizer)
+
     if finetuning_args.finetuning_type == "full" and model_args.checkpoint_dir is not None:
         model_to_load = model_args.checkpoint_dir[0]
     else:

tests/llamafy_baichuan2.py

@@ -1,7 +1,6 @@
 # coding=utf-8
 # Converts the Baichuan2-7B model in the same format as LLaMA2-7B.
-# Usage: python llamafy_baichuan2.py --baichuan2_json baichuan2.index.json --llama2_json llama2.index.json
-#                                    --input_dir baichuan2_original --output_dir baichuan2_llamafied
+# Usage: python llamafy_baichuan2.py --llama2_json llama2.index.json --input_dir input --output_dir output
 # Inspired by: https://huggingface.co/fireballoon/baichuan-llama-7b/blob/main/convert_baichuan_to_llama.py
 # Converted model: https://huggingface.co/hiyouga/Baichuan2-7B-Base-LLaMAfied
 
@@ -17,20 +16,20 @@ SHARD_B = "pytorch_model-00002-of-00002.bin"
 
 
 def llamafy_baichuan2(
-    baichuan2_json: str,
     llama2_json: str,
     input_dir: str,
     output_dir: str
 ):
-    weight_shard_a = torch.load(os.path.join(input_dir, SHARD_A), map_location="cpu")
-    weight_shard_b = torch.load(os.path.join(input_dir, SHARD_B), map_location="cpu")
-
     baichuan2_state_dict = OrderedDict()
-    baichuan2_state_dict.update(weight_shard_a)
-    baichuan2_state_dict.update(weight_shard_b)
+    for filepath in os.listdir(input_dir):
+        if os.path.isfile(os.path.join(input_dir, filepath)) and filepath.endswith(".bin"):
+            shard_weight = torch.load(os.path.join(input_dir, filepath), map_location="cpu")
+            baichuan2_state_dict.update(shard_weight)
 
     llama2_state_dict = OrderedDict()
+    total_size = 0
     for key, value in baichuan2_state_dict.items():
+        total_size += 2 * value.numel() # half precision
         if "W_pack" in key:
             llama2_state_dict[key.replace("W_pack", "q_proj")] = value[:4096, :]
             llama2_state_dict[key.replace("W_pack", "k_proj")] = value[4096:2*4096, :]
@@ -40,13 +39,11 @@ def llamafy_baichuan2(
         else:
             llama2_state_dict[key] = value
 
-    with open(os.path.join(input_dir, baichuan2_json), "r", encoding="utf-8") as f:
-        baichuan2_index = json.load(f)
     with open(os.path.join(input_dir, llama2_json), "r", encoding="utf-8") as f:
         llama2_index = json.load(f)
 
     merged_index = OrderedDict()
-    merged_index["metadata"] = baichuan2_index["metadata"]
+    merged_index["metadata"] = {"total_size": total_size}
     merged_index["weight_map"] = llama2_index["weight_map"]
 
     state_dict_a, state_dict_b = OrderedDict(), OrderedDict()
@@ -60,7 +57,7 @@ def llamafy_baichuan2(
     torch.save(state_dict_a, os.path.join(output_dir, SHARD_A))
     torch.save(state_dict_b, os.path.join(output_dir, SHARD_B))
     with open(os.path.join(output_dir, "pytorch_model.bin.index.json"), "w", encoding="utf-8") as f:
-        json.dump(merged_index, f)
+        json.dump(merged_index, f, indent=2)
     print("Completed!")