patch modelscope

2023-12-01 22:53:15 +08:00 · 2023-12-01 22:53:15 +08:00 · bd42c229b0
parent 3a64506031
commit bd42c229b0
7 changed files with 312 additions and 222 deletions
--- a/README.md
+++ b/README.md
@ -44,17 +44,23 @@ Compared to ChatGLM's [P-Tuning](https://github.com/THUDM/ChatGLM2-6B/tree/main/

 ![benchmark](assets/benchmark.svg)

+<details><summary>Definitions</summary>
+
 - **Training Speed**: the number of training samples processed per second during the training. (bs=4, cutoff_len=1024)
 - **Rouge Score**: Rouge-2 score on the development set of the [advertising text generation](https://aclanthology.org/D19-1321.pdf) task. (bs=4, cutoff_len=1024)
 - **GPU Memory**: Peak GPU memory usage in 4-bit quantized training. (bs=1, cutoff_len=1024)
 - We adopt `pre_seq_len=128` for ChatGLM's P-Tuning and `lora_rank=32` for LLaMA-Factory's LoRA tuning.

+</details>
+
 ## Changelog

-[23/12/01] We supported **[ModelScope Hub](https://www.modelscope.cn/models)** to accelerate model downloading. Add environment variable `USE_MODELSCOPE_HUB=1` to your command line, then you can use the model-id of ModelScope Hub.
+[23/12/01] We supported downloading pre-trained models from the **[ModelScope Hub](https://modelscope.cn/models)** for Chinese mainland users. See [this tutorial](#use-modelscope-models-optional) for usage.

 [23/10/21] We supported **[NEFTune](https://arxiv.org/abs/2310.05914)** trick for fine-tuning. Try `--neft_alpha` argument to activate NEFTune, e.g., `--neft_alpha 5`.

+<details><summary>Full Changelog</summary>
+
 [23/09/27] We supported **$S^2$-Attn** proposed by [LongLoRA](https://github.com/dvlab-research/LongLoRA) for the LLaMA models. Try `--shift_attn` argument to enable shift short attention.

 [23/09/23] We integrated MMLU, C-Eval and CMMLU benchmarks in this repo. See [this example](#evaluation) to evaluate your models.
@ -79,6 +85,8 @@ Compared to ChatGLM's [P-Tuning](https://github.com/THUDM/ChatGLM2-6B/tree/main/

 [23/06/03] We supported quantized training and inference (aka **[QLoRA](https://github.com/artidoro/qlora)**). Try `--quantization_bit 4/8` argument to work with quantized models.

+</details>
+
 ## Supported Models

 | Model                                                    | Model size                  | Default module    | Template  |
@ -231,31 +239,26 @@ If you want to enable the quantized LoRA (QLoRA) on the Windows platform, you wi
 pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.39.1-py3-none-win_amd64.whl
 ```

-### Use ModelScope Models
+### Use ModelScope Models (optional)

-If you have trouble with downloading models from HuggingFace, we have supported ModelScope Hub. To use LLaMA-Factory together with ModelScope, please add a environment variable:
+If you have trouble with downloading models from Hugging Face, you can use LLaMA-Factory together with ModelScope in the following manner.

-```shell
-export USE_MODELSCOPE_HUB=1
+```bash
+export USE_MODELSCOPE_HUB=1 # `set USE_MODELSCOPE_HUB=1` for Windows
 ```

-> [!NOTE]
->
-> Please use integers only. 0 or not set for using HuggingFace hub. Other values will be treated as use ModelScope hub.
+Then you can train the corresponding model by specifying a model ID of the ModelScope Hub. (find a full list of model IDs at [ModelScope Hub](https://modelscope.cn/models))

-Then you can use LLaMA-Factory with ModelScope model-ids:
-
-```shell
+```bash
 python src/train_bash.py \
-    --model_name_or_path ZhipuAI/chatglm3-6b \
-    ... other arguments
-# You can find all model ids in this link: https://www.modelscope.cn/models
+    --model_name_or_path modelscope/Llama-2-7b-ms \
+    ... # arguments (same as above)
 ```

-Web demo also supports ModelScope, after setting the environment variable please run with this command:
+LLaMA Board also supports using the models on the ModelScope Hub.

-```shell
-CUDA_VISIBLE_DEVICES=0 python src/train_web.py
+```bash
+CUDA_VISIBLE_DEVICES=0 USE_MODELSCOPE_HUB=1 python src/train_web.py
 ```

 ### Train on a single GPU
--- a/README_zh.md
+++ b/README_zh.md
@ -44,17 +44,23 @@ https://github.com/hiyouga/LLaMA-Factory/assets/16256802/6ba60acc-e2e2-4bec-b846

 ![benchmark](assets/benchmark.svg)

+<details><summary>变量定义</summary>
+
 - **Training Speed**: 训练阶段每秒处理的样本数量。（批处理大小=4，截断长度=1024）
 - **Rouge Score**: [广告文案生成](https://aclanthology.org/D19-1321.pdf)任务验证集上的 Rouge-2 分数。（批处理大小=4，截断长度=1024）
 - **GPU Memory**: 4 比特量化训练的 GPU 显存峰值。（批处理大小=1，截断长度=1024）
 - 我们在 ChatGLM 的 P-Tuning 中采用 `pre_seq_len=128`，在 LLaMA-Factory 的 LoRA 微调中采用 `lora_rank=32`。

+</details>
+
 ## 更新日志

-[23/12/01] 我们支持了 **[魔搭ModelHub](https://www.modelscope.cn/models)** 进行模型下载加速。在启动命令前环境变量中增加 `USE_MODELSCOPE_HUB=1` 即可开启。
+[23/12/01] 我们支持了从 **[魔搭社区](https://modelscope.cn/models)** 下载预训练模型。详细用法请参照 [此教程](#使用魔搭社区可跳过)。

 [23/10/21] 我们支持了 **[NEFTune](https://arxiv.org/abs/2310.05914)** 训练技巧。请使用 `--neft_alpha` 参数启用 NEFTune，例如 `--neft_alpha 5`。

+<details><summary>展开日志</summary>
+
 [23/09/27] 我们针对 LLaMA 模型支持了 [LongLoRA](https://github.com/dvlab-research/LongLoRA) 提出的 **$S^2$-Attn**。请使用 `--shift_attn` 参数以启用该功能。

 [23/09/23] 我们在项目中集成了 MMLU、C-Eval 和 CMMLU 评估集。使用方法请参阅[此示例](#模型评估)。
@ -79,6 +85,8 @@ https://github.com/hiyouga/LLaMA-Factory/assets/16256802/6ba60acc-e2e2-4bec-b846

 [23/06/03] 我们实现了 4 比特的 LoRA 训练（也称 **[QLoRA](https://github.com/artidoro/qlora)**）。请使用 `--quantization_bit 4` 参数进行 4 比特量化微调。

+</details>
+
 ## 模型

 | 模型名                                                   | 模型大小                     | 默认模块           | Template  |
@ -231,31 +239,26 @@ pip install -r requirements.txt
 pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.39.1-py3-none-win_amd64.whl
 ```

-### 使用魔搭的模型
+### 使用魔搭社区（可跳过）

-如果下载HuggingFace模型存在问题，我们已经支持了魔搭的ModelHub，只需要添加一个环境变量：
+如果您在 Hugging Face 模型的下载中遇到了问题，可以通过下述方法使用魔搭社区。

-```shell
-export USE_MODELSCOPE_HUB=1
+```bash
+export USE_MODELSCOPE_HUB=1 # Windows 使用 `set USE_MODELSCOPE_HUB=1`
 ```

-> [!NOTE]
->
-> 该环境变量仅支持整数，0或者不设置代表使用HuggingFace，其他值代表使用ModelScope
+接着即可通过指定模型名称来训练对应的模型。（在[魔搭社区](https://modelscope.cn/models)查看所有可用的模型）

-之后就可以在命令行中指定魔搭的模型id：
-
-```shell
+```bash
 python src/train_bash.py \
-    --model_name_or_path ZhipuAI/chatglm3-6b \
-    ... other arguments
-# 在这个链接中可以看到所有可用模型: https://www.modelscope.cn/models
+    --model_name_or_path modelscope/Llama-2-7b-ms \
+    ... # 参数同上
 ```

-Web demo目前也支持了魔搭, 在设置环境变量后即可使用：
+LLaMA Board 同样支持魔搭社区的模型下载。

-```shell
-CUDA_VISIBLE_DEVICES=0 python src/train_web.py
+```bash
+CUDA_VISIBLE_DEVICES=0 USE_MODELSCOPE_HUB=1 python src/train_web.py
 ```

 ### 单 GPU 训练
--- a/src/llmtuner/extras/constants.py
+++ b/src/llmtuner/extras/constants.py
@ -1,6 +1,7 @@
-import os
+from enum import Enum
 from collections import defaultdict, OrderedDict
-from typing import Dict, Optional, Union
+from typing import Dict, Optional
+

 CHOICES = ["A", "B", "C", "D"]

@ -20,8 +21,6 @@ SUBJECTS = ["Average", "STEM", "Social Sciences", "Humanities", "Other"]

 SUPPORTED_MODELS = OrderedDict()

-ALL_OFFICIAL_MODELS = OrderedDict()
-
 TRAINING_STAGES = {
    "Supervised Fine-Tuning": "sft",
    "Reward Modeling": "rm",
@ -30,9 +29,13 @@ TRAINING_STAGES = {
    "Pre-Training": "pt"
 }

+class DownloadSource(str, Enum):
+    DEFAULT = "hf"
+    MODELSCOPE = "ms"
+

 def register_model_group(
-    models: Dict[str, Union[str, Dict[str, str]]],
+    models: Dict[str, Dict[DownloadSource, str]],
    module: Optional[str] = None,
    template: Optional[str] = None
 ) -> None:
@ -42,14 +45,7 @@ def register_model_group(
            prefix = name.split("-")[0]
        else:
            assert prefix == name.split("-")[0], "prefix should be identical."
-
-        ALL_OFFICIAL_MODELS[name] = [path] if isinstance(path, str) else list(path.values())
-        if not int(os.environ.get('USE_MODELSCOPE_HUB', '0')):
-            # If path is a string, we treat it as a huggingface model-id by default.
-            SUPPORTED_MODELS[name] = path["hf"] if isinstance(path, dict) else path
-        elif isinstance(path, dict) and "ms" in path:
-            # Use ModelScope modelhub
-            SUPPORTED_MODELS[name] = path["ms"]
+        SUPPORTED_MODELS[name] = path
    if module is not None:
        DEFAULT_MODULE[prefix] = module
    if template is not None:
@ -59,16 +55,16 @@ def register_model_group(
 register_model_group(
    models={
        "Baichuan-7B-Base": {
-            "hf": "baichuan-inc/Baichuan-7B",
-            "ms": "baichuan-inc/baichuan-7B",
+            DownloadSource.DEFAULT: "baichuan-inc/Baichuan-7B",
+            DownloadSource.MODELSCOPE: "baichuan-inc/baichuan-7B"
        },
        "Baichuan-13B-Base": {
-            "hf": "baichuan-inc/Baichuan-13B-Base",
-            "ms": "baichuan-inc/Baichuan-13B-Base",
+            DownloadSource.DEFAULT: "baichuan-inc/Baichuan-13B-Base",
+            DownloadSource.MODELSCOPE: "baichuan-inc/Baichuan-13B-Base"
        },
        "Baichuan-13B-Chat": {
-            "hf": "baichuan-inc/Baichuan-13B-Chat",
-            "ms": "baichuan-inc/Baichuan-13B-Base",
+            DownloadSource.DEFAULT: "baichuan-inc/Baichuan-13B-Chat",
+            DownloadSource.MODELSCOPE: "baichuan-inc/Baichuan-13B-Chat"
        }
    },
    module="W_pack",
@ -79,20 +75,20 @@ register_model_group(
 register_model_group(
    models={
        "Baichuan2-7B-Base": {
-            "hf": "baichuan-inc/Baichuan2-7B-Base",
-            "ms": "baichuan-inc/Baichuan2-7B-Base",
+            DownloadSource.DEFAULT: "baichuan-inc/Baichuan2-7B-Base",
+            DownloadSource.MODELSCOPE: "baichuan-inc/Baichuan2-7B-Base"
        },
        "Baichuan2-13B-Base": {
-            "hf": "baichuan-inc/Baichuan2-13B-Base",
-            "ms": "baichuan-inc/Baichuan2-13B-Base",
+            DownloadSource.DEFAULT: "baichuan-inc/Baichuan2-13B-Base",
+            DownloadSource.MODELSCOPE: "baichuan-inc/Baichuan2-13B-Base"
        },
        "Baichuan2-7B-Chat": {
-            "hf": "baichuan-inc/Baichuan2-7B-Chat",
-            "ms": "baichuan-inc/Baichuan2-7B-Chat",
+            DownloadSource.DEFAULT: "baichuan-inc/Baichuan2-7B-Chat",
+            DownloadSource.MODELSCOPE: "baichuan-inc/Baichuan2-7B-Chat"
        },
        "Baichuan2-13B-Chat": {
-            "hf": "baichuan-inc/Baichuan2-13B-Chat",
-            "ms": "baichuan-inc/Baichuan2-13B-Chat",
+            DownloadSource.DEFAULT: "baichuan-inc/Baichuan2-13B-Chat",
+            DownloadSource.MODELSCOPE: "baichuan-inc/Baichuan2-13B-Chat"
        }
    },
    module="W_pack",
@ -103,16 +99,16 @@ register_model_group(
 register_model_group(
    models={
        "BLOOM-560M": {
-            "hf": "bigscience/bloom-560m",
-            "ms": "AI-ModelScope/bloom-560m",
+            DownloadSource.DEFAULT: "bigscience/bloom-560m",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/bloom-560m"
        },
        "BLOOM-3B": {
-            "hf": "bigscience/bloom-3b",
-            "ms": "AI-ModelScope/bloom-3b",
+            DownloadSource.DEFAULT: "bigscience/bloom-3b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/bloom-3b"
        },
        "BLOOM-7B1": {
-            "hf": "bigscience/bloom-7b1",
-            "ms": "AI-ModelScope/bloom-7b1",
+            DownloadSource.DEFAULT: "bigscience/bloom-7b1",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/bloom-7b1"
        }
    },
    module="query_key_value"
@ -122,16 +118,16 @@ register_model_group(
 register_model_group(
    models={
        "BLOOMZ-560M": {
-            "hf": "bigscience/bloomz-560m",
-            "ms": "AI-ModelScope/bloomz-560m",
+            DownloadSource.DEFAULT: "bigscience/bloomz-560m",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/bloomz-560m"
        },
        "BLOOMZ-3B": {
-            "hf": "bigscience/bloomz-3b",
-            "ms": "AI-ModelScope/bloomz-3b",
+            DownloadSource.DEFAULT: "bigscience/bloomz-3b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/bloomz-3b"
        },
        "BLOOMZ-7B1-mt": {
-            "hf": "bigscience/bloomz-7b1-mt",
-            "ms": "AI-ModelScope/bloomz-7b1-mt",
+            DownloadSource.DEFAULT: "bigscience/bloomz-7b1-mt",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/bloomz-7b1-mt"
        }
    },
    module="query_key_value"
@ -141,12 +137,12 @@ register_model_group(
 register_model_group(
    models={
        "BlueLM-7B-Base": {
-            "hf": "vivo-ai/BlueLM-7B-Base",
-            "ms": "vivo-ai/BlueLM-7B-Base",
+            DownloadSource.DEFAULT: "vivo-ai/BlueLM-7B-Base",
+            DownloadSource.MODELSCOPE: "vivo-ai/BlueLM-7B-Base"
        },
        "BlueLM-7B-Chat": {
-            "hf": "vivo-ai/BlueLM-7B-Chat",
-            "ms": "vivo-ai/BlueLM-7B-Chat",
+            DownloadSource.DEFAULT: "vivo-ai/BlueLM-7B-Chat",
+            DownloadSource.MODELSCOPE: "vivo-ai/BlueLM-7B-Chat"
        }
    },
    template="bluelm"
@ -156,8 +152,8 @@ register_model_group(
 register_model_group(
    models={
        "ChatGLM2-6B-Chat": {
-            "hf": "THUDM/chatglm2-6b",
-            "ms": "ZhipuAI/chatglm2-6b",
+            DownloadSource.DEFAULT: "THUDM/chatglm2-6b",
+            DownloadSource.MODELSCOPE: "ZhipuAI/chatglm2-6b"
        }
    },
    module="query_key_value",
@ -168,12 +164,12 @@ register_model_group(
 register_model_group(
    models={
        "ChatGLM3-6B-Base": {
-            "hf": "THUDM/chatglm3-6b-base",
-            "ms": "ZhipuAI/chatglm3-6b-base",
+            DownloadSource.DEFAULT: "THUDM/chatglm3-6b-base",
+            DownloadSource.MODELSCOPE: "ZhipuAI/chatglm3-6b-base"
        },
        "ChatGLM3-6B-Chat": {
-            "hf": "THUDM/chatglm3-6b",
-            "ms": "ZhipuAI/chatglm3-6b",
+            DownloadSource.DEFAULT: "THUDM/chatglm3-6b",
+            DownloadSource.MODELSCOPE: "ZhipuAI/chatglm3-6b"
        }
    },
    module="query_key_value",
@ -184,59 +180,105 @@ register_model_group(
 register_model_group(
    models={
        "ChineseLLaMA2-1.3B": {
-            "hf": "hfl/chinese-llama-2-1.3b",
-            "ms": "AI-ModelScope/chinese-llama-2-1.3b",
+            DownloadSource.DEFAULT: "hfl/chinese-llama-2-1.3b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/chinese-llama-2-1.3b"
        },
        "ChineseLLaMA2-7B": {
-            "hf": "hfl/chinese-llama-2-7b",
-            "ms": "AI-ModelScope/chinese-llama-2-7b",
+            DownloadSource.DEFAULT: "hfl/chinese-llama-2-7b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/chinese-llama-2-7b"
        },
        "ChineseLLaMA2-13B": {
-            "hf": "hfl/chinese-llama-2-13b",
-            "ms": "AI-ModelScope/chinese-llama-2-13b",
+            DownloadSource.DEFAULT: "hfl/chinese-llama-2-13b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/chinese-llama-2-13b"
        },
        "ChineseLLaMA2-1.3B-Chat": {
-            "hf": "hfl/chinese-alpaca-2-1.3b",
-            "ms": "AI-ModelScope/chinese-alpaca-2-1.3b",
+            DownloadSource.DEFAULT: "hfl/chinese-alpaca-2-1.3b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/chinese-alpaca-2-1.3b"
        },
        "ChineseLLaMA2-7B-Chat": {
-            "hf": "hfl/chinese-alpaca-2-7b",
-            "ms": "AI-ModelScope/chinese-alpaca-2-7b",
+            DownloadSource.DEFAULT: "hfl/chinese-alpaca-2-7b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/chinese-alpaca-2-7b"
        },
        "ChineseLLaMA2-13B-Chat": {
-            "hf": "hfl/chinese-alpaca-2-13b",
-            "ms": "AI-ModelScope/chinese-alpaca-2-13b",
+            DownloadSource.DEFAULT: "hfl/chinese-alpaca-2-13b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/chinese-alpaca-2-13b"
        }
    },
    template="llama2_zh"
 )


+register_model_group(
+    models={
+        "DeepseekLLM-7B-Base": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-llm-7b-base",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-llm-7b-base"
+        },
+        "DeepseekLLM-67B-Base": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-llm-67b-base",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-llm-67b-base"
+        },
+        "DeepseekLLM-7B-Chat": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-llm-7b-chat",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-llm-7b-chat"
+        },
+        "DeepseekLLM-67B-Chat": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-llm-67b-chat",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-llm-67b-chat"
+        }
+    },
+    template="deepseek"
+)
+
+
+register_model_group(
+    models={
+        "DeepseekCoder-6.7B-Base": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-coder-6.7b-base",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-coder-6.7b-base"
+        },
+        "DeepseekCoder-33B-Base": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-coder-33b-base",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-coder-33b-base"
+        },
+        "DeepseekCoder-6.7B-Chat": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-coder-6.7b-instruct",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-coder-6.7b-instruct"
+        },
+        "DeepseekCoder-33B-Chat": {
+            DownloadSource.DEFAULT: "deepseek-ai/deepseek-coder-33b-instruct",
+            DownloadSource.MODELSCOPE: "deepseek-ai/deepseek-coder-33b-instruct"
+        }
+    },
+    template="deepseekcoder"
+)
+
+
 register_model_group(
    models={
        "Falcon-7B": {
-            "hf": "tiiuae/falcon-7b",
-            "ms": "AI-ModelScope/falcon-7b",
+            DownloadSource.DEFAULT: "tiiuae/falcon-7b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/falcon-7b"
        },
        "Falcon-40B": {
-            "hf": "tiiuae/falcon-40b",
-            "ms": "AI-ModelScope/falcon-40b",
+            DownloadSource.DEFAULT: "tiiuae/falcon-40b",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/falcon-40b"
        },
        "Falcon-180B": {
-            "hf": "tiiuae/falcon-180B",
-            "ms": "AI-ModelScope/falcon-180B",
+            DownloadSource.DEFAULT: "tiiuae/falcon-180b",
+            DownloadSource.MODELSCOPE: "modelscope/falcon-180B"
        },
        "Falcon-7B-Chat": {
-            "hf": "tiiuae/falcon-7b-instruct",
-            "ms": "AI-ModelScope/falcon-7b-instruct",
+            DownloadSource.DEFAULT: "tiiuae/falcon-7b-instruct",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/falcon-7b-instruct"
        },
        "Falcon-40B-Chat": {
-            "hf": "tiiuae/falcon-40b-instruct",
-            "ms": "AI-ModelScope/falcon-40b-instruct",
+            DownloadSource.DEFAULT: "tiiuae/falcon-40b-instruct",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/falcon-40b-instruct"
        },
        "Falcon-180B-Chat": {
-            "hf": "tiiuae/falcon-180B-chat",
-            "ms": "AI-ModelScope/falcon-180B-chat",
+            DownloadSource.DEFAULT: "tiiuae/falcon-180b-chat",
+            DownloadSource.MODELSCOPE: "modelscope/falcon-180B-chat"
        }
    },
    module="query_key_value",
@ -247,20 +289,20 @@ register_model_group(
 register_model_group(
    models={
        "InternLM-7B": {
-            "hf": "internlm/internlm-7b",
-            "ms": "Shanghai_AI_Laboratory/internlm-7b",
+            DownloadSource.DEFAULT: "internlm/internlm-7b",
+            DownloadSource.MODELSCOPE: "Shanghai_AI_Laboratory/internlm-7b"
        },
        "InternLM-20B": {
-            "hf": "internlm/internlm-20b",
-            "ms": "Shanghai_AI_Laboratory/internlm-20b",
+            DownloadSource.DEFAULT: "internlm/internlm-20b",
+            DownloadSource.MODELSCOPE: "Shanghai_AI_Laboratory/internlm-20b"
        },
        "InternLM-7B-Chat": {
-            "hf": "internlm/internlm-chat-7b",
-            "ms": "Shanghai_AI_Laboratory/internlm-chat-7b",
+            DownloadSource.DEFAULT: "internlm/internlm-chat-7b",
+            DownloadSource.MODELSCOPE: "Shanghai_AI_Laboratory/internlm-chat-7b"
        },
        "InternLM-20B-Chat": {
-            "hf": "internlm/internlm-chat-20b",
-            "ms": "Shanghai_AI_Laboratory/internlm-chat-20b",
+            DownloadSource.DEFAULT: "internlm/internlm-chat-20b",
+            DownloadSource.MODELSCOPE: "Shanghai_AI_Laboratory/internlm-chat-20b"
        }
    },
    template="intern"
@ -270,8 +312,8 @@ register_model_group(
 register_model_group(
    models={
        "LingoWhale-8B": {
-            "hf": "deeplang-ai/LingoWhale-8B",
-            "ms": "DeepLang/LingoWhale-8B",
+            DownloadSource.DEFAULT: "deeplang-ai/LingoWhale-8B",
+            DownloadSource.MODELSCOPE: "DeepLang/LingoWhale-8B"
        }
    },
    module="qkv_proj"
@ -281,20 +323,20 @@ register_model_group(
 register_model_group(
    models={
        "LLaMA-7B": {
-            "hf": "huggyllama/llama-7b",
-            "ms": "skyline2006/llama-7b",
+            DownloadSource.DEFAULT: "huggyllama/llama-7b",
+            DownloadSource.MODELSCOPE: "skyline2006/llama-7b"
        },
        "LLaMA-13B": {
-            "hf": "huggyllama/llama-13b",
-            "ms": "skyline2006/llama-13b",
+            DownloadSource.DEFAULT: "huggyllama/llama-13b",
+            DownloadSource.MODELSCOPE: "skyline2006/llama-13b"
        },
        "LLaMA-30B": {
-            "hf": "huggyllama/llama-30b",
-            "ms": "skyline2006/llama-30b",
+            DownloadSource.DEFAULT: "huggyllama/llama-30b",
+            DownloadSource.MODELSCOPE: "skyline2006/llama-30b"
        },
        "LLaMA-65B": {
-            "hf": "huggyllama/llama-65b",
-            "ms": "skyline2006/llama-65b",
+            DownloadSource.DEFAULT: "huggyllama/llama-65b",
+            DownloadSource.MODELSCOPE: "skyline2006/llama-65b"
        }
    }
 )
@ -303,28 +345,28 @@ register_model_group(
 register_model_group(
    models={
        "LLaMA2-7B": {
-            "hf": "meta-llama/Llama-2-7b-hf",
-            "ms": "modelscope/Llama-2-7b-ms",
+            DownloadSource.DEFAULT: "meta-llama/Llama-2-7b-hf",
+            DownloadSource.MODELSCOPE: "modelscope/Llama-2-7b-ms"
        },
        "LLaMA2-13B": {
-            "hf": "meta-llama/Llama-2-13b-hf",
-            "ms": "modelscope/Llama-2-13b-ms",
+            DownloadSource.DEFAULT: "meta-llama/Llama-2-13b-hf",
+            DownloadSource.MODELSCOPE: "modelscope/Llama-2-13b-ms"
        },
        "LLaMA2-70B": {
-            "hf": "meta-llama/Llama-2-70b-hf",
-            "ms": "modelscope/Llama-2-70b-ms",
+            DownloadSource.DEFAULT: "meta-llama/Llama-2-70b-hf",
+            DownloadSource.MODELSCOPE: "modelscope/Llama-2-70b-ms"
        },
        "LLaMA2-7B-Chat": {
-            "hf": "meta-llama/Llama-2-7b-chat-hf",
-            "ms": "modelscope/Llama-2-7b-chat-ms",
+            DownloadSource.DEFAULT: "meta-llama/Llama-2-7b-chat-hf",
+            DownloadSource.MODELSCOPE: "modelscope/Llama-2-7b-chat-ms"
        },
        "LLaMA2-13B-Chat": {
-            "hf": "meta-llama/Llama-2-13b-chat-hf",
-            "ms": "modelscope/Llama-2-13b-chat-ms",
+            DownloadSource.DEFAULT: "meta-llama/Llama-2-13b-chat-hf",
+            DownloadSource.MODELSCOPE: "modelscope/Llama-2-13b-chat-ms"
        },
        "LLaMA2-70B-Chat": {
-            "hf": "meta-llama/Llama-2-70b-chat-hf",
-            "ms": "modelscope/Llama-2-70b-chat-ms",
+            DownloadSource.DEFAULT: "meta-llama/Llama-2-70b-chat-hf",
+            DownloadSource.MODELSCOPE: "modelscope/Llama-2-70b-chat-ms"
        }
    },
    template="llama2"
@ -334,12 +376,12 @@ register_model_group(
 register_model_group(
    models={
        "Mistral-7B": {
-            "hf": "mistralai/Mistral-7B-v0.1",
-            "ms": "AI-ModelScope/Mistral-7B-v0.1",
+            DownloadSource.DEFAULT: "mistralai/Mistral-7B-v0.1",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/Mistral-7B-v0.1"
        },
        "Mistral-7B-Chat": {
-            "hf": "mistralai/Mistral-7B-Instruct-v0.1",
-            "ms": "AI-ModelScope/Mistral-7B-Instruct-v0.1",
+            DownloadSource.DEFAULT: "mistralai/Mistral-7B-Instruct-v0.1",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/Mistral-7B-Instruct-v0.1"
        }
    },
    template="mistral"
@ -349,8 +391,8 @@ register_model_group(
 register_model_group(
    models={
        "OpenChat3.5-7B-Chat": {
-            "hf": "openchat/openchat_3.5",
-            "ms": "myxiongmodel/openchat_3.5",
+            DownloadSource.DEFAULT: "openchat/openchat_3.5",
+            DownloadSource.MODELSCOPE: "myxiongmodel/openchat_3.5"
        }
    },
    template="openchat"
@ -360,8 +402,8 @@ register_model_group(
 register_model_group(
    models={
        "Phi1.5-1.3B": {
-            "hf": "microsoft/phi-1_5",
-            "ms": "allspace/PHI_1-5",
+            DownloadSource.DEFAULT: "microsoft/phi-1_5",
+            DownloadSource.MODELSCOPE: "allspace/PHI_1-5"
        }
    },
    module="Wqkv"
@ -370,37 +412,69 @@ register_model_group(

 register_model_group(
    models={
+        "Qwen-1.8B": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-1_8B",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-1_8B"
+        },
        "Qwen-7B": {
-            "hf": "Qwen/Qwen-7B",
-            "ms": "qwen/Qwen-7B",
+            DownloadSource.DEFAULT: "Qwen/Qwen-7B",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-7B"
        },
        "Qwen-14B": {
-            "hf": "Qwen/Qwen-14B",
-            "ms": "qwen/Qwen-14B",
+            DownloadSource.DEFAULT: "Qwen/Qwen-14B",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-14B"
+        },
+        "Qwen-72B": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-72B",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-72B"
+        },
+        "Qwen-1.8B-Chat": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-1_8B-Chat",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-1_8B-Chat"
        },
        "Qwen-7B-Chat": {
-            "hf": "Qwen/Qwen-7B-Chat",
-            "ms": "qwen/Qwen-7B-Chat",
+            DownloadSource.DEFAULT: "Qwen/Qwen-7B-Chat",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-7B-Chat"
        },
        "Qwen-14B-Chat": {
-            "hf": "Qwen/Qwen-14B-Chat",
-            "ms": "qwen/Qwen-14B-Chat",
+            DownloadSource.DEFAULT: "Qwen/Qwen-14B-Chat",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-14B-Chat"
+        },
+        "Qwen-72B-Chat": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-72B-Chat",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-72B-Chat"
+        },
+        "Qwen-1.8B-int8-Chat": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-1_8B-Chat-Int8",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-1_8B-Chat-Int8"
+        },
+        "Qwen-1.8B-int4-Chat": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-1_8B-Chat-Int4",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-1_8B-Chat-Int4"
        },
        "Qwen-7B-int8-Chat": {
-            "hf": "Qwen/Qwen-7B-Chat-Int8",
-            "ms": "qwen/Qwen-7B-Chat-Int8",
+            DownloadSource.DEFAULT: "Qwen/Qwen-7B-Chat-Int8",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-7B-Chat-Int8"
        },
        "Qwen-7B-int4-Chat": {
-            "hf": "Qwen/Qwen-7B-Chat-Int4",
-            "ms": "qwen/Qwen-7B-Chat-Int4",
+            DownloadSource.DEFAULT: "Qwen/Qwen-7B-Chat-Int4",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-7B-Chat-Int4"
        },
        "Qwen-14B-int8-Chat": {
-            "hf": "Qwen/Qwen-14B-Chat-Int8",
-            "ms": "qwen/Qwen-14B-Chat-Int8",
+            DownloadSource.DEFAULT: "Qwen/Qwen-14B-Chat-Int8",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-14B-Chat-Int8"
        },
        "Qwen-14B-int4-Chat": {
-            "hf": "Qwen/Qwen-14B-Chat-Int4",
-            "ms": "qwen/Qwen-14B-Chat-Int4",
+            DownloadSource.DEFAULT: "Qwen/Qwen-14B-Chat-Int4",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-14B-Chat-Int4"
+        },
+        "Qwen-72B-int8-Chat": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-72B-Chat-Int8",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-72B-Chat-Int8"
+        },
+        "Qwen-72B-int4-Chat": {
+            DownloadSource.DEFAULT: "Qwen/Qwen-72B-Chat-Int4",
+            DownloadSource.MODELSCOPE: "qwen/Qwen-72B-Chat-Int4"
        }
    },
    module="c_attn",
@ -411,8 +485,8 @@ register_model_group(
 register_model_group(
    models={
        "Skywork-13B-Base": {
-            "hf": "Skywork/Skywork-13B-base",
-            "ms": "skywork/Skywork-13B-base",
+            DownloadSource.DEFAULT: "Skywork/Skywork-13B-base",
+            DownloadSource.MODELSCOPE: "skywork/Skywork-13B-base"
        }
    }
 )
@ -421,12 +495,12 @@ register_model_group(
 register_model_group(
    models={
        "Vicuna1.5-7B-Chat": {
-            "hf": "lmsys/vicuna-7b-v1.5",
-            "ms": "AI-ModelScope/vicuna-7b-v1.5",
+            DownloadSource.DEFAULT: "lmsys/vicuna-7b-v1.5",
+            DownloadSource.MODELSCOPE: "Xorbits/vicuna-7b-v1.5"
        },
        "Vicuna1.5-13B-Chat": {
-            "hf": "lmsys/vicuna-13b-v1.5",
-            "ms": "Xorbits/vicuna-13b-v1.5",
+            DownloadSource.DEFAULT: "lmsys/vicuna-13b-v1.5",
+            DownloadSource.MODELSCOPE: "Xorbits/vicuna-13b-v1.5"
        }
    },
    template="vicuna"
@ -436,24 +510,24 @@ register_model_group(
 register_model_group(
    models={
        "XVERSE-7B": {
-            "hf": "xverse/XVERSE-7B",
-            "ms": "xverse/XVERSE-7B",
+            DownloadSource.DEFAULT: "xverse/XVERSE-7B",
+            DownloadSource.MODELSCOPE: "xverse/XVERSE-7B"
        },
        "XVERSE-13B": {
-            "hf": "xverse/XVERSE-13B",
-            "ms": "xverse/XVERSE-13B",
+            DownloadSource.DEFAULT: "xverse/XVERSE-13B",
+            DownloadSource.MODELSCOPE: "xverse/XVERSE-13B"
        },
        "XVERSE-65B": {
-            "hf": "xverse/XVERSE-65B",
-            "ms": "xverse/XVERSE-65B",
+            DownloadSource.DEFAULT: "xverse/XVERSE-65B",
+            DownloadSource.MODELSCOPE: "xverse/XVERSE-65B"
        },
        "XVERSE-7B-Chat": {
-            "hf": "xverse/XVERSE-7B-Chat",
-            "ms": "xverse/XVERSE-7B-Chat",
+            DownloadSource.DEFAULT: "xverse/XVERSE-7B-Chat",
+            DownloadSource.MODELSCOPE: "xverse/XVERSE-7B-Chat"
        },
        "XVERSE-13B-Chat": {
-            "hf": "xverse/XVERSE-13B-Chat",
-            "ms": "xverse/XVERSE-13B-Chat",
+            DownloadSource.DEFAULT: "xverse/XVERSE-13B-Chat",
+            DownloadSource.MODELSCOPE: "xverse/XVERSE-13B-Chat"
        }
    },
    template="xverse"
@ -463,12 +537,12 @@ register_model_group(
 register_model_group(
    models={
        "Yayi-7B": {
-            "hf": "wenge-research/yayi-7b-llama2",
-            "ms": "AI-ModelScope/yayi-7b-llama2",
+            DownloadSource.DEFAULT: "wenge-research/yayi-7b-llama2",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/yayi-7b-llama2"
        },
        "Yayi-13B": {
-            "hf": "wenge-research/yayi-13b-llama2",
-            "ms": "AI-ModelScope/yayi-13b-llama2",
+            DownloadSource.DEFAULT: "wenge-research/yayi-13b-llama2",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/yayi-13b-llama2"
        }
    },
    template="yayi"
@ -478,20 +552,20 @@ register_model_group(
 register_model_group(
    models={
        "Yi-6B": {
-            "hf": "01-ai/Yi-6B",
-            "ms": "01ai/Yi-6B",
+            DownloadSource.DEFAULT: "01-ai/Yi-6B",
+            DownloadSource.MODELSCOPE: "01ai/Yi-6B"
        },
        "Yi-34B": {
-            "hf": "01-ai/Yi-34B",
-            "ms": "01ai/Yi-34B",
+            DownloadSource.DEFAULT: "01-ai/Yi-34B",
+            DownloadSource.MODELSCOPE: "01ai/Yi-34B"
        },
        "Yi-34B-Chat": {
-            "hf": "01-ai/Yi-34B-Chat",
-            "ms": "01ai/Yi-34B-Chat",
+            DownloadSource.DEFAULT: "01-ai/Yi-34B-Chat",
+            DownloadSource.MODELSCOPE: "01ai/Yi-34B-Chat"
        },
        "Yi-34B-int8-Chat": {
-            "hf": "01-ai/Yi-34B-Chat-8bits",
-            "ms": "01ai/Yi-34B-Chat-8bits",
+            DownloadSource.DEFAULT: "01-ai/Yi-34B-Chat-8bits",
+            DownloadSource.MODELSCOPE: "01ai/Yi-34B-Chat-8bits"
        }
    },
    template="yi"
@ -501,12 +575,12 @@ register_model_group(
 register_model_group(
    models={
        "Zephyr-7B-Alpha-Chat": {
-            "hf": "HuggingFaceH4/zephyr-7b-alpha",
-            "ms": "AI-ModelScope/zephyr-7b-alpha",
+            DownloadSource.DEFAULT: "HuggingFaceH4/zephyr-7b-alpha",
+            DownloadSource.MODELSCOPE: "AI-ModelScope/zephyr-7b-alpha"
        },
        "Zephyr-7B-Beta-Chat": {
-            "hf": "HuggingFaceH4/zephyr-7b-beta",
-            "ms": "modelscope/zephyr-7b-beta",
+            DownloadSource.DEFAULT: "HuggingFaceH4/zephyr-7b-beta",
+            DownloadSource.MODELSCOPE: "modelscope/zephyr-7b-beta"
        }
    },
    template="zephyr"
--- a/src/llmtuner/extras/misc.py
+++ b/src/llmtuner/extras/misc.py
@ -23,6 +23,7 @@ except ImportError:

 if TYPE_CHECKING:
    from transformers import HfArgumentParser
+    from llmtuner.hparams import ModelArguments


 class AverageMeter:
@ -117,3 +118,23 @@ def torch_gc() -> None:
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
        torch.cuda.ipc_collect()
+
+
+def try_download_model_from_ms(model_args: "ModelArguments") -> None:
+    if not use_modelscope() or os.path.exists(model_args.model_name_or_path):
+        return
+
+    try:
+        from modelscope import snapshot_download # type: ignore
+        revision = "master" if model_args.model_revision == "main" else model_args.model_revision
+        model_args.model_name_or_path = snapshot_download(
+            model_args.model_name_or_path,
+            revision=revision,
+            cache_dir=model_args.cache_dir
+        )
+    except ImportError:
+        raise ImportError("Please install modelscope via `pip install modelscope -U`")
+
+
+def use_modelscope() -> bool:
+    return bool(int(os.environ.get("USE_MODELSCOPE_HUB", "0")))
--- a/src/llmtuner/hparams/model_args.py
+++ b/src/llmtuner/hparams/model_args.py
@ -8,8 +8,8 @@ class ModelArguments:
    Arguments pertaining to which model/config/tokenizer we are going to fine-tune.
    """
    model_name_or_path: str = field(
-        metadata={"help": "Path to pretrained model or model identifier "
-                          "from huggingface.co/models or modelscope.cn/models."}
+        metadata={"help": "Path to pretrained model or model identifier from \
+                  huggingface.co/models or modelscope.cn/models."}
    )
    cache_dir: Optional[str] = field(
        default=None,
--- a/src/llmtuner/model/loader.py
+++ b/src/llmtuner/model/loader.py
@ -1,6 +1,4 @@
 import math
-import os
-
 import torch
 from types import MethodType
 from typing import TYPE_CHECKING, Literal, Optional, Tuple
@ -23,8 +21,8 @@ try:
 except ImportError: # https://github.com/huggingface/transformers/releases/tag/v4.33.1
    from transformers.deepspeed import is_deepspeed_zero3_enabled

-from llmtuner.extras.logging import reset_logging, get_logger
-from llmtuner.extras.misc import count_parameters, get_current_device, infer_optim_dtype
+from llmtuner.extras.logging import get_logger
+from llmtuner.extras.misc import count_parameters, get_current_device, infer_optim_dtype, try_download_model_from_ms
 from llmtuner.extras.packages import is_flash_attn2_available
 from llmtuner.extras.patches import llama_patch as LlamaPatches
 from llmtuner.hparams import FinetuningArguments
@ -58,6 +56,8 @@ def load_model_and_tokenizer(
    Support both training and inference.
    """

+    try_download_model_from_ms(model_args)
+
    config_kwargs = {
        "trust_remote_code": True,
        "cache_dir": model_args.cache_dir,
@ -65,8 +65,6 @@ def load_model_and_tokenizer(
        "token": model_args.hf_hub_token
    }

-    try_download_model_from_ms(model_args)
-
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.model_name_or_path,
        use_fast=model_args.use_fast_tokenizer,
@ -232,16 +230,3 @@ def load_model_and_tokenizer(
        logger.info("This IS expected that the trainable params is 0 if you are using model for inference only.")

    return model, tokenizer
-
-
-def try_download_model_from_ms(model_args):
-    if int(os.environ.get('USE_MODELSCOPE_HUB', '0')) and not os.path.exists(model_args.model_name_or_path):
-        try:
-            from modelscope import snapshot_download
-            revision = model_args.model_revision
-            if revision == 'main':
-                revision = 'master'
-            model_args.model_name_or_path = snapshot_download(model_args.model_name_or_path, revision)
-        except ImportError as e:
-            raise ImportError(f'You are using `USE_MODELSCOPE_HUB=1` but you have no modelscope sdk installed. '
-                              f'Please install it by `pip install modelscope -U`') from e
--- a/src/llmtuner/webui/common.py
+++ b/src/llmtuner/webui/common.py
@ -11,18 +11,17 @@ from transformers.utils import (
    ADAPTER_SAFE_WEIGHTS_NAME
 )

-
 from llmtuner.extras.constants import (
    DEFAULT_MODULE,
    DEFAULT_TEMPLATE,
    SUPPORTED_MODELS,
-    ALL_OFFICIAL_MODELS,
-    TRAINING_STAGES
+    TRAINING_STAGES,
+    DownloadSource
 )
+from llmtuner.extras.misc import use_modelscope
 from llmtuner.hparams.data_args import DATA_CONFIG


-
 DEFAULT_CACHE_DIR = "cache"
 DEFAULT_DATA_DIR = "data"
 DEFAULT_SAVE_DIR = "saves"
@ -66,10 +65,15 @@ def save_config(lang: str, model_name: Optional[str] = None, model_path: Optiona

 def get_model_path(model_name: str) -> str:
    user_config = load_config()
-    cached_path = user_config["path_dict"].get(model_name, None)
-    if cached_path in ALL_OFFICIAL_MODELS.get(model_name, []):
-        cached_path = None
-    return cached_path or SUPPORTED_MODELS.get(model_name, "")
+    path_dict: Dict[DownloadSource, str] = SUPPORTED_MODELS.get(model_name, [])
+    model_path = user_config["path_dict"].get(model_name, None) or path_dict.get(DownloadSource.DEFAULT, "")
+    if (
+        use_modelscope()
+        and path_dict.get(DownloadSource.MODELSCOPE)
+        and model_path == path_dict.get(DownloadSource.DEFAULT)
+    ): # replace path
+        model_path = path_dict.get(DownloadSource.MODELSCOPE)
+    return model_path


 def get_prefix(model_name: str) -> str: