Skip to content

fix(ci): pin Playwright jobs to Ubuntu 24.04 and kill timed-out runner commands on uutils - #8337

Open
mmabrouk wants to merge 1 commit into
mainfrom
ci/pin-ubuntu-24-04-runner-test-flake
Open

mmabrouk wants to merge 1 commit into
mainfrom
ci/pin-ubuntu-24-04-runner-test-flake

Conversation

@mmabrouk

@mmabrouk mmabrouk commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

What broke

Two CI checks fail on every PR today, including PRs based on current main. Both come from the same event: GitHub moved ubuntu-latest to Ubuntu 26.04.

  1. Railway preview tests. The "Install Playwright browser" step fails with Playwright does not support chromium on ubuntu26.04-x64. This stops the wait-for-readiness job and every test job after it.
  2. Runner unit test remote-command.test.ts, "is killed in the sandbox at its lifetime, with everything it started, though nobody polls it". Example: https://github.com/Agenta-AI/agenta/actions/runs/44095938800/job/134083100822

Why the runner test failed

It is not a timing flake. It is a real behavior difference.

Ubuntu 26.04 replaces GNU coreutils with uutils (Rust). The runner starts each sandbox command under timeout -s KILL <lifetime>, and relies on timeout to kill the whole process group when the lifetime ends. GNU timeout does that. uutils timeout kills only its direct child (the supervisor shell), exits 124, and leaves the command and everything it started running. In the test, the command's sleep 2.5; echo old > late.txt ran to the end and wrote the file after the 1 second lifetime.

I reproduced this in an ubuntu:26.04 container: the test fails on main and passes with this change. It also passes on the host with GNU timeout.

This matters outside CI too. Any sandbox image that ships uutils would let a command whose runner died run past its lifetime.

The fix

  • .github/workflows/44-railway-tests.yml: the two jobs that install Playwright (wait-for-readiness and run-web-tests) now run on ubuntu-24.04. No other workflow installs Playwright (main already removed it from 12-check-unit-tests.yml). Playwright 1.60.0 in the lockfile does not support 26.04 yet, so a pin is the smallest change.
  • services/runner/src/engines/inprocess/sandbox/remote-command.ts: a shell now wraps timeout. When timeout ends with 124 (uutils) or 137 (GNU killed by its own KILL), the shell reads the process group the supervisor recorded in pgid and kills that group. Both timeout versions put the command in their own group, so the recorded group is the right one. A normal exit does not kill anything, as before.

Tests

  • pnpm vitest run tests/unit/inprocess/ on the host (GNU coreutils): 161 passed.
  • The same remote-command.test.ts in ubuntu:26.04 (uutils 0.10.0): fails on main, passes with this change.
  • pnpm run typecheck: clean.

Devin Review

…up with any timeout

ubuntu-latest is now Ubuntu 26.04. Playwright cannot install chromium there,
so the Railway preview jobs that install it run on ubuntu-24.04.

Ubuntu 26.04 also ships uutils timeout, which at the lifetime kills only the
supervisor and exits 124, where GNU timeout kills the whole process group.
The command and its children then lived past their lifetime. A shell around
timeout now kills the recorded group on 124 or 137.

Entire-Checkpoint: 01M7N3J58NV25JBDF21FQXQYN0
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of commands that exceed their time limits, so related processes are stopped more reliably. This helps prevent timed-out work from continuing in the background and supports more consistent task execution.

The Railway readiness and web-test jobs now use Ubuntu 24.04. The remote command launcher runs the supervisor under a detached leader shell and kills the recorded timeout process group when timeout exits with status 124 or 137.

Changes: Railway test runner

Layer / File(s) Summary
Pin Railway test runners
.github/workflows/44-railway-tests.yml
The wait-for-readiness and run-web-tests jobs now use ubuntu-24.04 instead of ubuntu-latest.

Remote command timeout cleanup

Layer / File(s) Summary
Clean up the timeout process group
services/runner/src/engines/inprocess/sandbox/remote-command.ts
A detached leader shell runs timeout. When it exits with status 124 or 137, the leader kills the recorded timeout process group.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • Agenta-AI/agenta#8333: Changes timeout process-group cleanup for the same uutils behavior and references the same test failure.
Merge Risk: Low · up to a7271

The runner pin and timeout cleanup should fix the reported CI failures. In the rare case of a duplicate launch, the recorded process group can be removed, and a command that hits its lifetime limit may then keep running on uutils systems. This is a small risk that the owner should be aware of.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names both changes in the PR: pinning the Playwright CI jobs to Ubuntu 24.04 in the Railway workflow, and killing timed-out runner command process groups on uutils timeout. Both match the ra…
Description check Passed The description explains the failures on Ubuntu 26.04, the Playwright install error, the uutils timeout behavior that leaves children running, and the fix in both files. It also lists test and typeche…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names both changes in the PR: pinning the Playwright CI jobs to Ubuntu 24.04 in the Railway workflow, and killing timed-out runner command process groups on uutils timeout. Both match the raw summary and the objectives. At 89 characters it is somewhat long but still specific and descriptive.

Full details: Description check

Explanation

The description explains the failures on Ubuntu 26.04, the Playwright install error, the uutils timeout behavior that leaves children running, and the fix in both files. It also lists test and typecheck results. It matches the changeset closely.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 8d6f0df and a7271d6.

Files selected for processing: 2

  • .github/workflows/44-railway-tests.yml
  • services/runner/src/engines/inprocess/sandbox/remote-command.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

// The group is `timeout`'s (the session leader), not this shell's own pid.
// The group is `timeout`'s (it makes its own), not this shell's own pid.
`read -r _ _ _ _ pg _ < /proc/$$/stat && echo $pg > "$d/pgid.tmp" && mv -f "$d/pgid.tmp" "$d/pgid" || exit 1`,
`mkdir "$d/claim" 2>/dev/null || { rm -f "$d/pgid"; exit 0; }`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Stability & Availability · Heavy lift

A losing duplicate supervisor can delete the winner's pgid.

The header says a late duplicate launch is possible. A duplicate runs Line 179 first and overwrites pgid with its own group. If mkdir at Line 180 fails, it runs rm -f "$d/pgid". The winner is then running with no pgid.

Before this change, this made Stop and polls read "no pgid" for a live command. This change adds a second effect. At the lifetime, the leader at Line 198 reads an empty pg. On uutils, where timeout kills only the supervisor, the cleanup is skipped and the command keeps running. The new cleanup therefore depends on pgid surviving the loser branch. Write pgid only after the claim succeeds, or have the loser skip the rm. This conflicts with the header's rule that pgid is written before the claim, so a different ordering rule is needed. For example, the loser could write to a unique temp name and never touch pgid.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Railway Preview Environment

Item Value
Status stopped
Commit a7271d69523cb7f57459f75dcfecfad06ff66bc0
Manual expiry (UTC) Not granted
CI deadline (UTC) Not granted
Workflow https://github.com/Agenta-AI/agenta/actions/runs/44120083686

Dependent tests finished. Post /preview for manual review. Environment absence verified; retained storage requires separate provider accounting.

Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant