From 30f2164c0deb21bc3e3c58c4f931854a01c0af92 Mon Sep 17 00:00:00 2001 From: "chaoyu@qiyuanlab.com" Date: Tue, 16 Jul 2024 18:51:28 +0800 Subject: [PATCH] =?UTF-8?q?=E4=BF=AE=E6=94=B9=E4=B8=80=E4=BA=9B=E6=A0=BC?= =?UTF-8?q?=E5=BC=8F=E9=97=AE=E9=A2=98?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- quick_start_clean/readmes/quick_start.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/quick_start_clean/readmes/quick_start.md b/quick_start_clean/readmes/quick_start.md index c67c9cd..5c7a17a 100644 --- a/quick_start_clean/readmes/quick_start.md +++ b/quick_start_clean/readmes/quick_start.md @@ -158,7 +158,9 @@ python convert_json2index.py \ ``` 脚本运行成功时，会有如下显示：（不需要用hadoop所以不用管hadoop: not found的警告信息） -![alt text](./055bf7ce-faab-403b-a7ee-896279bee11f.png) + + +![alt text](https://www.osredm.com/jiuyuan/CPM-9G-8B/tree/FM_9G/quick_start_clean/readmes/055bf7ce-faab-403b-a7ee-896279bee11f.png) 转完后，在index的目录下会生成四个文件：data.jsonl（原先的jsonl数据）、index、index.h5、meta.json（记录数据集信息，包含 "language", "nlines", "nbytes", "length_distribute", "avg_token_per_line", "hdfs_path", "data_sample"字段）。 这里有一个meta.json的例子：