From 818726e9bcdedfbd330ea7a60e02ee5b03aed459 Mon Sep 17 00:00:00 2001 From: hiyouga Date: Sat, 9 Mar 2024 04:11:55 +0800 Subject: [PATCH] add GaLore results --- README.md | 4 ++++ README_zh.md | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/README.md b/README.md index afd78d4c..a287ac55 100644 --- a/README.md +++ b/README.md @@ -276,12 +276,16 @@ huggingface-cli login | ------ | ---- | ----- | ----- | ----- | ------ | ------ | | Full | AMP | 120GB | 240GB | 600GB | 1200GB | 900GB | | Full | 16 | 60GB | 120GB | 300GB | 600GB | 400GB | +| GaLore | 16 | 28GB | 60GB | 150GB | 300GB | 200GB | | Freeze | 16 | 20GB | 40GB | 80GB | 200GB | 160GB | | LoRA | 16 | 16GB | 32GB | 64GB | 160GB | 120GB | | QLoRA | 8 | 10GB | 20GB | 40GB | 80GB | 60GB | | QLoRA | 4 | 6GB | 12GB | 24GB | 48GB | 30GB | | QLoRA | 2 | 4GB | 8GB | 16GB | 24GB | 18GB | +> [!NOTE] +> We report the GaLore results without per-layer weight updates. + ## Getting Started ### Data Preparation (optional) diff --git a/README_zh.md b/README_zh.md index 697f0239..d92ab254 100644 --- a/README_zh.md +++ b/README_zh.md @@ -276,12 +276,16 @@ huggingface-cli login | ------- | ---- | ----- | ----- | ----- | ------ | ------ | | 全参数 | AMP | 120GB | 240GB | 600GB | 1200GB | 900GB | | 全参数 | 16 | 60GB | 120GB | 300GB | 600GB | 400GB | +| GaLore | 16 | 28GB | 60GB | 150GB | 300GB | 200GB | | 部分参数 | 16 | 20GB | 40GB | 80GB | 200GB | 160GB | | LoRA | 16 | 16GB | 32GB | 64GB | 160GB | 120GB | | QLoRA | 8 | 10GB | 20GB | 40GB | 80GB | 60GB | | QLoRA | 4 | 6GB | 12GB | 24GB | 48GB | 30GB | | QLoRA | 2 | 4GB | 8GB | 16GB | 24GB | 18GB | +> [!NOTE] +> 上述 GaLore 的结果中不包含逐层权重更新。 + ## 如何使用 ### 数据准备（可跳过）