jiuyuan
/
CPM-9G-8B
5
0
Fork 54
Code Issues Pull Requests Projects Releases Wiki Activity

修改一些格式问题

This commit is contained in:
chaoyu@qiyuanlab.com 2024-07-16 18:52:23 +08:00
parent 30f2164c0d
commit e0bb66c34d
1 changed files with 1 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
quick_start_clean/readmes/quick_start.md
View File

@ -159,8 +159,7 @@ python convert_json2index.py \
脚本运行成功时，会有如下显示：（不需要用hadoop所以不用管hadoop: not found的警告信息）
<!-- ![alt text](./055bf7ce-faab-403b-a7ee-896279bee11f.png) -->
![alt text](https://www.osredm.com/jiuyuan/CPM-9G-8B/tree/FM_9G/quick_start_clean/readmes/055bf7ce-faab-403b-a7ee-896279bee11f.png)
![脚本运行成功后的显示](./055bf7ce-faab-403b-a7ee-896279bee11f.png)
转完后，在index的目录下会生成四个文件：data.jsonl（原先的jsonl数据）、index、index.h5、meta.json（记录数据集信息，包含 "language", "nlines", "nbytes", "length_distribute", "avg_token_per_line", "hdfs_path", "data_sample"字段）。
这里有一个meta.json的例子：