p83651209
/
CPM-9G-8B
forked from jiuyuan/CPM-9G-8B
1
0
Fork 1
Code Issues Pull Requests Packages Projects Releases Wiki Activity

修改一些格式问题

This commit is contained in:
chaoyu@qiyuanlab.com 2024-07-16 18:46:28 +08:00
parent e667e27f2f
commit a7e6f8bb98
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
quick_start_clean/readmes/quick_start.md
View File

@ -158,7 +158,7 @@ python convert_json2index.py \
```
脚本运行成功时，会有如下显示：（不需要用hadoop所以不用管hadoop: not found的警告信息）
![alt text](055bf7ce-faab-403b-a7ee-896279bee11f.png)
![alt text](./055bf7ce-faab-403b-a7ee-896279bee11f.png)
转完后，在index的目录下会生成四个文件：data.jsonl（原先的jsonl数据）、index、index.h5、meta.json（记录数据集信息，包含 "language", "nlines", "nbytes", "length_distribute", "avg_token_per_line", "hdfs_path", "data_sample"字段）。
这里有一个meta.json的例子：