Skip to content

perf: cibench workflow - #179

Merged
seanmonstar merged 7 commits into
seanmonstar:masterfrom
AaronO:perf/cibench
Sep 1, 2024
Merged

seanmonstar merged 7 commits into
seanmonstar:masterfrom
AaronO:perf/cibench

Conversation

@AaronO

@AaronO AaronO commented Sep 1, 2024 •

Copy link
Copy Markdown
Contributor

First pass at adding benchmarks in CI.
Covers matrix of key archs and SIMD extensions.
Outputs delta against main with critcmp (with 5% noise threshold).

Future improvements

  • Run classifier on results to indicate regression/improvement/neutral (aggregate or on specific arch)
  • Post summary (or details if regression) as PR comment (didn't add cause without summary would be noisy)
  • Could save main baselines as assets (and just ref/download them) instead of recomputing them for each PR
    (comes at a cost of statefulness and some adhoc complexity but would reduce CI time cost)

Notes

  • This adds CI (compute) time, but don't think it adds wall time since there are longer subtasks
  • CI is still more noisy than controlled local env, but probably good enough for substantial perf changes and sanity-checking

@AaronO
AaronO marked this pull request as ready for review September 1, 2024 05:08

@seanmonstar seanmonstar left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@seanmonstar
seanmonstar merged commit 38878a0 into seanmonstar:master Sep 1, 2024
@AaronO
AaronO deleted the perf/cibench branch September 1, 2024 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants