Repository navigation
Conversation
jpohhhh
requested review from
a team,
Kirpal,
btrautmann,
jeroen-meijer,
jolexxa and
marcossevilla
as code owners
March 1, 2026 22:57
There was a problem hiding this comment.
Pull request overview
Addresses a test registration performance/memory issue where goldenTest() repeatedly registered a setUp callback, causing quadratic setUp executions as golden tests scaled within a file.
Changes:
- Adds a guard in
goldenTest()intended to register_setUpGoldenTestsviasetUponly once. - Adds a regression test that intercepts
setUpFn/testWidgetsFnto detect callback accumulation and assert linear scaling.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| lib/src/golden_test.dart | Introduces a global “register setUp once” guard around goldenTestAdapter.setUp(_setUpGoldenTests). |
| test/src/golden_test_set_up_accumulation_test.dart | Adds a regression test that records setUp registrations and simulates runtime executions to validate non-quadratic behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+172
to
+175
| if (!_goldenTestSetUpRegistered) { | ||
| _goldenTestSetUpRegistered = true; | ||
| goldenTestAdapter.setUp(_setUpGoldenTests); | ||
| } |
|
|
||
| final Set<String> _loadedFontFamilies = {}; | ||
| bool _goldenTestSetUpRegistered = false; | ||
|
|
Comment on lines
+100
to
+106
| // FIXED: only one setUp callback should be registered regardless | ||
| // of how many goldenTest() calls are made. | ||
| expect( | ||
| registeredSetUpCallbacks, | ||
| hasLength(1), | ||
| reason: 'goldenTest should register setUp at most once', | ||
| ); |
Each call to goldenTest() was calling goldenTestAdapter.setUp(_setUpGoldenTests), appending a new setUp callback to the current test group every time. With N golden tests, all N callbacks ran before each of the 2N variant runs, producing 2N² total setUp executions (e.g. 20,000 for 100 tests). The fix guards the registration with a boolean so it happens exactly once. Benchmarked against a real project (442 golden tests, 63 files, --concurrency=1): | Metric | Before (0.13.0) | After (fix) | Improvement | |----------------|-----------------|-------------|-------------| | Peak RAM | 18.1 GB | 3.2 GB | 5.6× | | Wall time | 212 s (3:33) | 127 s (2:07)| 1.7× | | Test execution | 3:08 | 2:02 | 1.5× | | User CPU | 180.8 s | 85.8 s | 2.1× | | System CPU | 46.1 s | 17.7 s | 2.6× | | Page reclaims | 12.7M | 4.9M | 2.6× | The system CPU drop (46s → 18s) reflects kernel time spent on memory management (page faults, VM pressure) servicing the pathological allocation pattern. The O(N²) rapid-fire async setUp calls were starving the GC of idle time, preventing it from finalizing native rendering resources (Pictures, Images, surfaces) between test runs. A regression test is included that intercepts setUpFn/testWidgetsFn to prove the accumulation and its quadratic effect.
A setUp registered once belongs to the first group only: a test of a later group run alone (--plain-name) did not load fonts. Loading in the test body keeps O(N) setup and works in every group; loadFonts is a no-op once the fonts are loaded.
Unit runs the format, analyze and host tests on Linux. Integration runs the golden tests on macOS, Windows and Linux: the same golden files must match on all three, which is the cross-OS check. The upstream workflows (Slack, Codecov, the Flutter 3.16 matrix) are removed: they do not follow the Telosnex standard and cannot run here. The example's web/index.html loads flutter_bootstrap.js instead of the legacy flutter.js loader, which cannot start a --wasm build.
very_good_analysis 5 reports ordering and parameter reassignment as info, and analyze runs with --fatal-infos. dart fix sorted the directives and the dependencies; the golden theme is a local, not a reassigned parameter (same value).
On Windows the Flutter SDK is flutter.bat, so Process.run('flutter') in
flutter_test_config.dart failed to load every test file. runInShell finds
it.
The same golden differs by about 0.1% of pixels between the runners
(anti-aliasing of text and shadows). The CI goldens now allow 0.5%
(diffThreshold, which the library already supports), so a real change
still fails.
Three smoke tests hang in the test body until the 10 minute test timeout, on macOS, Linux and Windows alike (the pressed and long-pressed interactions, and the constrained 3000x3000 widget). They are not golden diffs, so skip them with a reason rather than hide the cause. On Windows the timer button golden matcher receives a Future<Image> and does not match; macOS and Linux match the same golden. Skip it there only.
On macOS the same golden fails the way it does on Windows: the golden matcher receives a Future<Image>. Skip it there with the reason, and keep the Linux run so the golden is still checked on one runner.
Since goldenTest loads fonts in the test body, loadFonts runs in the
fake-async zone of each widget test. rootBundle caches the future of
loadString('FontManifest.json'), and that future belongs to the zone of
the first test. The zone does not run microtasks after its test ends, so
the second golden test in a file awaited the cached future forever. Before
0d7b586 the fonts loaded in a setUp, outside fake async, so the cache was
safe.
The three tests skipped as hangs ("while pressed", "while long pressed"
and "big constrained widget") are the second and third golden tests of
their files. They pass alone and hang only after another test in the
same file. Their skips are removed. load_fonts_test.dart calls loadFonts
in three consecutive widget tests and fails by timeout if this returns.
The timer button golden failed on macOS and Windows with a 30 px (0.54%) diff, over the 0.5% threshold. The diff was the button's right edge column and two shadow pixels. On Linux the engine measures the six Ahem glyphs of "Button" as 83.9996 px wide, and 84 px on macOS. The button is sized by its text, so on Linux it was 115.9996 px wide and its right edge was antialiased into the last pixel column. The same diff is present before 0d7b586, so the fork's change did not cause it. The "Future<Image>" in the failure message is not a bug: with obscured text, alchemist passes the blocked-text image as a Future<ui.Image>, which matchesGoldenFile accepts, and the message prints that value as "Actual". The button now has a fixed 120x48 size, so its geometry does not depend on the text width. The golden is regenerated on Linux x64 (Flutter 3.47.1), 120x48 instead of 116x48, and the macOS render is identical to it. The test runs on every OS again.
Fix the golden tests that hang and the timer golden mismatch
…rk of A daily check: alerts once the oldest commit of Betterment/alchemist not merged here is a week old. The pin is the last upstream commit merged; update it with each merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Story
Description
Each call to goldenTest() was calling goldenTestAdapter.setUp(_setUpGoldenTests), appending a new setUp callback to the current test group every time. With N golden tests, all N callbacks ran before each of the 2N variant runs, producing 2N² total setUp executions (e.g. 20,000 for 100 tests).
The fix guards the registration with a boolean so it happens exactly once.
Benchmarked against a real project (442 golden tests, 63 files, --concurrency=1):
The system CPU drop (46s → 18s) reflects kernel time spent on memory management (page faults, VM pressure) servicing the pathological allocation pattern.
A regression test is included that intercepts setUpFn/testWidgetsFn to prove the accumulation and its quadratic effect.
Type of Change