p04798526
/
LLaMA-Factory-310P3
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update vllm example

This commit is contained in:
hiyouga 2024-04-02 22:45:20 +08:00
parent 66b0fe4e96
commit 49a2dfaf90
2 changed files with 6 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@ -369,10 +369,11 @@ Use `python src/train_bash.py -h` to display arguments description.
### Deploy with OpenAI-style API and vLLM
```bash
CUDA_VISIBLE_DEVICES=0 API_PORT=8000 python src/api_demo.py \
CUDA_VISIBLE_DEVICES=0,1 API_PORT=8000 python src/api_demo.py \
--model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
--template mistral \
--infer_backend vllm
--infer_backend vllm \
--vllm_enforce_eager
```
### Use ModelScope Hub

5
README_zh.md
View File

@ -369,10 +369,11 @@ docker compose -f ./docker-compose.yml up -d
### 使用 OpenAI 风格 API 和 vLLM 部署
```bash
CUDA_VISIBLE_DEVICES=0 API_PORT=8000 python src/api_demo.py \
CUDA_VISIBLE_DEVICES=0,1 API_PORT=8000 python src/api_demo.py \
--model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
--template mistral \
--infer_backend vllm
--infer_backend vllm \
--vllm_enforce_eager
```
### 使用魔搭社区