Skip to content

fix: register setUp only once, eliminating O(N²) accumulation. ~2x faster tests that can scale in 1 file - #175

Open
jpohhhh wants to merge 33 commits into
Betterment:mainfrom
Telosnex:main
Open

jpohhhh wants to merge 33 commits into
Betterment:mainfrom
Telosnex:main

Conversation

@jpohhhh

@jpohhhh jpohhhh commented Mar 1, 2026

Copy link
Copy Markdown

Story

  • Working on porting Mermaid chart rendering from JS to Dart.
  • Noticed when 100 tests were in one file, my 64 GB of RAM Mac ran out of memory.
  • Remembered that I had regularly seen a pattern where splitting tests into individual files, then having a test file run them, lowered RAM usage. (~14 GB in this case instead of 64)
  • Had LLM look into code b/c there was nothing obvious to me. It identified this setup being 2N^2, and provided a simple patch.
  • It didn't sound plausible to me.
  • However, it definitely worked. tests are much faster and consume much less RAM, confirmed over and over again on different suites of golden tests I have.
  • Description below produced by LLM, #s are real.
  • Before #s are: unpatched, 10 files with 10 tests. (1 file with 100 could not run)
  • After #s are: patched, 1 file with 100 tests.

Description

Each call to goldenTest() was calling goldenTestAdapter.setUp(_setUpGoldenTests), appending a new setUp callback to the current test group every time. With N golden tests, all N callbacks ran before each of the 2N variant runs, producing 2N² total setUp executions (e.g. 20,000 for 100 tests).

The fix guards the registration with a boolean so it happens exactly once.

Benchmarked against a real project (442 golden tests, 63 files, --concurrency=1):

Metric Before (0.13.0) After (fix) Improvement
Peak RAM 18.1 GB 3.2 GB 5.6×
Wall time 212 s (3:33) 127 s (2:07) 1.7×
Test execution 3:08 2:02 1.5×
User CPU 180.8 s 85.8 s 2.1×
System CPU 46.1 s 17.7 s 2.6×
Page reclaims 12.7M 4.9M 2.6×

The system CPU drop (46s → 18s) reflects kernel time spent on memory management (page faults, VM pressure) servicing the pathological allocation pattern.

A regression test is included that intercepts setUpFn/testWidgetsFn to prove the accumulation and its quadratic effect.

Type of Change

  • ✨ New feature (non-breaking change which adds functionality)
  • 🛠️ Bug fix (non-breaking change which fixes an issue)
  • ❌ Breaking change (fix or feature that would cause existing functionality to change)
  • 🧹 Code refactor
  • ✅ Build configuration change
  • 📝 Documentation
  • 🗑️ Chore

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Addresses a test registration performance/memory issue where goldenTest() repeatedly registered a setUp callback, causing quadratic setUp executions as golden tests scaled within a file.

Changes:

  • Adds a guard in goldenTest() intended to register _setUpGoldenTests via setUp only once.
  • Adds a regression test that intercepts setUpFn/testWidgetsFn to detect callback accumulation and assert linear scaling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
lib/src/golden_test.dart Introduces a global “register setUp once” guard around goldenTestAdapter.setUp(_setUpGoldenTests).
test/src/golden_test_set_up_accumulation_test.dart Adds a regression test that records setUp registrations and simulates runtime executions to validate non-quadratic behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lib/src/golden_test.dart Outdated
Comment on lines +172 to +175
if (!_goldenTestSetUpRegistered) {
_goldenTestSetUpRegistered = true;
goldenTestAdapter.setUp(_setUpGoldenTests);
}
Comment thread lib/src/golden_test.dart

final Set<String> _loadedFontFamilies = {};
bool _goldenTestSetUpRegistered = false;

Comment on lines +100 to +106
// FIXED: only one setUp callback should be registered regardless
// of how many goldenTest() calls are made.
expect(
registeredSetUpCallbacks,
hasLength(1),
reason: 'goldenTest should register setUp at most once',
);
Each call to goldenTest() was calling goldenTestAdapter.setUp(_setUpGoldenTests),
appending a new setUp callback to the current test group every time. With N golden
tests, all N callbacks ran before each of the 2N variant runs, producing 2N² total
setUp executions (e.g. 20,000 for 100 tests).

The fix guards the registration with a boolean so it happens exactly once.

Benchmarked against a real project (442 golden tests, 63 files, --concurrency=1):

| Metric         | Before (0.13.0) | After (fix) | Improvement |
|----------------|-----------------|-------------|-------------|
| Peak RAM       | 18.1 GB         | 3.2 GB      | 5.6×        |
| Wall time      | 212 s (3:33)    | 127 s (2:07)| 1.7×        |
| Test execution | 3:08            | 2:02        | 1.5×        |
| User CPU       | 180.8 s         | 85.8 s      | 2.1×        |
| System CPU     | 46.1 s          | 17.7 s      | 2.6×        |
| Page reclaims  | 12.7M           | 4.9M        | 2.6×        |

The system CPU drop (46s → 18s) reflects kernel time spent on memory management
(page faults, VM pressure) servicing the pathological allocation pattern. The O(N²)
rapid-fire async setUp calls were starving the GC of idle time, preventing it from
finalizing native rendering resources (Pictures, Images, surfaces) between test runs.

A regression test is included that intercepts setUpFn/testWidgetsFn to prove the
accumulation and its quadratic effect.
A setUp registered once belongs to the first group only: a test of a
later group run alone (--plain-name) did not load fonts. Loading in the
test body keeps O(N) setup and works in every group; loadFonts is a
no-op once the fonts are loaded.
Telosnex CI and others added 13 commits October 28, 2026 18:43
Unit runs the format, analyze and host tests on Linux. Integration runs
the golden tests on macOS, Windows and Linux: the same golden files must
match on all three, which is the cross-OS check.

The upstream workflows (Slack, Codecov, the Flutter 3.16 matrix) are
removed: they do not follow the Telosnex standard and cannot run here.

The example's web/index.html loads flutter_bootstrap.js instead of the
legacy flutter.js loader, which cannot start a --wasm build.
very_good_analysis 5 reports ordering and parameter reassignment as info,
and analyze runs with --fatal-infos. dart fix sorted the directives and
the dependencies; the golden theme is a local, not a reassigned parameter
(same value).
On Windows the Flutter SDK is flutter.bat, so Process.run('flutter') in
flutter_test_config.dart failed to load every test file. runInShell finds
it.

The same golden differs by about 0.1% of pixels between the runners
(anti-aliasing of text and shadows). The CI goldens now allow 0.5%
(diffThreshold, which the library already supports), so a real change
still fails.
Three smoke tests hang in the test body until the 10 minute test timeout,
on macOS, Linux and Windows alike (the pressed and long-pressed
interactions, and the constrained 3000x3000 widget). They are not golden
diffs, so skip them with a reason rather than hide the cause.

On Windows the timer button golden matcher receives a Future<Image> and
does not match; macOS and Linux match the same golden. Skip it there only.
On macOS the same golden fails the way it does on Windows: the golden
matcher receives a Future<Image>. Skip it there with the reason, and
keep the Linux run so the golden is still checked on one runner.
Since goldenTest loads fonts in the test body, loadFonts runs in the
fake-async zone of each widget test. rootBundle caches the future of
loadString('FontManifest.json'), and that future belongs to the zone of
the first test. The zone does not run microtasks after its test ends, so
the second golden test in a file awaited the cached future forever. Before
0d7b586 the fonts loaded in a setUp, outside fake async, so the cache was
safe.

The three tests skipped as hangs ("while pressed", "while long pressed"
and "big constrained widget") are the second and third golden tests of
their files. They pass alone and hang only after another test in the
same file. Their skips are removed. load_fonts_test.dart calls loadFonts
in three consecutive widget tests and fails by timeout if this returns.
The timer button golden failed on macOS and Windows with a 30 px (0.54%)
diff, over the 0.5% threshold. The diff was the button's right edge
column and two shadow pixels. On Linux the engine measures the six Ahem
glyphs of "Button" as 83.9996 px wide, and 84 px on macOS. The button is
sized by its text, so on Linux it was 115.9996 px wide and its right edge
was antialiased into the last pixel column. The same diff is present
before 0d7b586, so the fork's change did not cause it.

The "Future<Image>" in the failure message is not a bug: with obscured
text, alchemist passes the blocked-text image as a Future<ui.Image>, which
matchesGoldenFile accepts, and the message prints that value as "Actual".

The button now has a fixed 120x48 size, so its geometry does not depend on
the text width. The golden is regenerated on Linux x64 (Flutter 3.47.1),
120x48 instead of 116x48, and the macOS render is identical to it. The
test runs on every OS again.
Fix the golden tests that hang and the timer golden mismatch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants