jiuyuan
/
CPM-9G-8B
5
0
Fork 54
Code Issues Pull Requests Projects Releases Wiki Activity

修改一些格式问题

This commit is contained in:
chaoyu@qiyuanlab.com 2024-07-16 18:51:28 +08:00
parent a7e6f8bb98
commit 30f2164c0d
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
quick_start_clean/readmes/quick_start.md
View File

@ -158,7 +158,9 @@ python convert_json2index.py \
```
脚本运行成功时，会有如下显示：（不需要用hadoop所以不用管hadoop: not found的警告信息）
![alt text](./055bf7ce-faab-403b-a7ee-896279bee11f.png)
<!-- ![alt text](./055bf7ce-faab-403b-a7ee-896279bee11f.png) -->
![alt text](https://www.osredm.com/jiuyuan/CPM-9G-8B/tree/FM_9G/quick_start_clean/readmes/055bf7ce-faab-403b-a7ee-896279bee11f.png)
转完后，在index的目录下会生成四个文件：data.jsonl（原先的jsonl数据）、index、index.h5、meta.json（记录数据集信息，包含 "language", "nlines", "nbytes", "length_distribute", "avg_token_per_line", "hdfs_path", "data_sample"字段）。
这里有一个meta.json的例子：