p04798526
/
LLaMA-Factory-310P3
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add patch_mixtral_replace_moe_impl for full training Mitral using DeepSpeed Zero3. 

Signed-off-by: ldwang <ftgreat@gmail.com>
This commit is contained in:
ldwang 2024-01-24 15:25:31 +08:00
parent 18923b1402
commit c284665425
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
src/llmtuner/model/patcher.py
View File

@ -269,6 +269,7 @@ def patch_config(
def patch_mixtral_replace_moe_impl() -> None: def patch_mixtral_replace_moe_impl() -> None:
import torch.nn.functional as F
def mlp_forward(self, hidden_states): def mlp_forward(self, hidden_states):
current_hidden_states = self.act_fn(self.w1(hidden_states)) * self.w3(hidden_states) current_hidden_states = self.act_fn(self.w1(hidden_states)) * self.w3(hidden_states)
current_hidden_states = self.w2(current_hidden_states) current_hidden_states = self.w2(current_hidden_states)