Skip to content

Parameter count on GPT-2 medium #172

Open

Description

@Heimine

According to the paper, the lora parameter count for GPT-2 medium is 0.35M, but since the hidden dimension is 1024, and the model have 24 layers, with a rank=4, the correct parameter count should be 1024 * 2[for B,A] * 4[for rank=4] * 2[for Q,V] * 24[num. of layers] = 393216 = 0.39M.

I also checked the code for the above calculation.

Do I miss something here?

Thank you very much!

Activity

  1. HaiminZhang commented on Aug 9, 2024

    @HaiminZhang

    I think your counting is correct. Have you figured it out how 0.35M is counted?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions