Skip to content

Use fixed V8 hash and random seeds in the benchmarks - #22323

Merged
calixteman merged 1 commit into
mozilla:masterfrom
calixteman:codspeed_fixed_seeds
Oct 24, 2026
Merged

calixteman merged 1 commit into
mozilla:masterfrom
calixteman:codspeed_fixed_seeds

Conversation

@calixteman

Copy link
Copy Markdown
Contributor

The walltime results vary between runs of the same code by up to ~10% for some benchmarks, while being stable within a run. Fixing the seeds roughly halved the number of benchmarks differing by more than 5% between runs.

The walltime results vary between runs of the same code by up to ~10% for
some benchmarks, while being stable within a run. Fixing the seeds roughly
halved the number of benchmarks differing by more than 5% between runs.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.47%. Comparing base (5b4187c) to head (f0a22da).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #22323   +/-   ##
=======================================
  Coverage   90.47%   90.47%           
=======================================
  Files         277      277           
  Lines       68313    68400   +87     
=======================================
+ Hits        61803    61882   +79     
- Misses       6510     6518    +8     
Flag Coverage Δ
unittestcli 58.63% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Oct 23, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 1.44%

⚡ 4 improved benchmarks
❌ 6 regressed benchmarks
✅ 78 untouched benchmarks
⏩ 86 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ Lexer.getObj: paths (bug1743245.pdf) 23.9 ms 27.7 ms -13.58%
❌ XFAFactory: parse, bind and layout (xfa_issue14315.pdf) 9.2 ms 9.9 ms -7.14%
❌ Pattern.parseShading: function-based, PostScript (function_based_shading.pdf) 43.7 ms 46.5 ms -5.92%
❌ MetadataParser: XMP (issue11144_reduced.pdf) 4.4 ms 4.6 ms -5.56%
❌ EvaluatorPreprocessor.read: paths (bug1743245.pdf) 58.6 ms 61.9 ms -5.4%
❌ EvaluatorPreprocessor.read: text (issue12963.pdf) 35.1 ms 37 ms -5.1%
⚡ Parser: parse content stream objects 18.9 ms 17.1 ms +10.11%
⚡ PartialEvaluator.buildPath: lines (bug1743245.pdf) 15.9 ms 14.6 ms +8.67%
⚡ PartialEvaluator.buildPath: curves (issue12705.pdf) 1.2 ms 1.1 ms +6.51%
⚡ compileType3Glyph (issue918.pdf) 9.7 ms 9.2 ms +6.07%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing calixteman:codspeed_fixed_seeds (f0a22da) with master (67667f1)

Open in CodSpeed

Footnotes

  1. 86 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@calixteman
calixteman merged commit a30101a into mozilla:master Oct 24, 2026
12 of 13 checks passed
@calixteman
calixteman deleted the codspeed_fixed_seeds branch October 24, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants