You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
According to the paper, the lora parameter count for GPT-2 medium is 0.35M, but since the hidden dimension is 1024, and the model have 24 layers, with a rank=4, the correct parameter count should be 1024 * 2[for B,A] * 4[for rank=4] * 2[for Q,V] * 24[num. of layers] = 393216 = 0.39M.
I also checked the code for the above calculation.
Activity
HaiminZhang commentedon Aug 9, 2024
I think your counting is correct. Have you figured it out how 0.35M is counted?