Skip to content

fix: log worker startup scheduler misses as warnings - #3210

Merged
MarcusSorealheis merged 2 commits into
TraceMachina:mainfrom
mvanhorn:fix/615-worker-startup-connect-logs
Nov 17, 2026
Merged

MarcusSorealheis merged 2 commits into
TraceMachina:mainfrom
mvanhorn:fix/615-worker-startup-connect-logs

Conversation

@mvanhorn

Copy link
Copy Markdown
Contributor

What and why

A worker that comes up before its scheduler is reachable records each failed attempt as an error. That startup order is normal, and the ERROR lines pile up in the worker log for as long as the scheduler is still down. Every reconnect failure in LocalWorker::run went through one handler that logged at error level with the message "Error". Connect failures, registration failures, and later disconnects all shared that path, and the level did not depend on ever_registered, so attempts made before the first successful registration were errors too.

The handler now takes a failure label, connect, register, or disconnect, and includes it on the log line. Before ever_registered is set the attempt is logged with warn!; after the worker has registered, the same attempt is logged with error!. The message is "Scheduler connection attempt failed, retrying". The worker tests require a startup connect failure to be a WARN line with failure="connect", and a disconnect after registration to be an ERROR.

Closes #615

How was this verified?

Ran cargo test locally for the nativelink change; it passed.
The tests passed with the nativelink change applied; a run with it reverted gave no clear result, so no test is shown to catch this bug.
The changed test nativelink-worker/tests/local_worker_test.rs did not run here: no test runner for it was available.

Risk

Scoped to the files listed above; no other caller or consumer changes.

AI assistance

AI was used for assistance.

The handler now takes a failure label, connect, register, or disconnect,
and includes it on the log line. Before ever_registered is set the
attempt is logged with warn!; after the worker has registered, the same
attempt is logged with error!. The message is "Scheduler connection
attempt failed, retrying". The worker tests require a startup connect
failure to be a WARN line with failure="connect", and a disconnect after
registration to be an ERROR.

Fixes TraceMachina#615
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Nov 17, 2026 10:25pm UTC
nativelink-aidm Ready Ready Preview Nov 17, 2026 10:25pm UTC

Request Review

@CLAassistant

CLAassistant commented Nov 17, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@MarcusSorealheis

Copy link
Copy Markdown
Member

@mvanhorn This is a straightforward improvement to the user/agent experience. We appreciate it. I'll take it for a spin now.

@MarcusSorealheis

Copy link
Copy Markdown
Member

/build-image nativelink

@MarcusSorealheis

Copy link
Copy Markdown
Member

/build-image nativelink-worker-init

@MarcusSorealheis

Copy link
Copy Markdown
Member

We have a flavor of this log decoration in our proprietary distribution, which relies heavily on readiness probes at startup but this exact feature should be in open source. We overlooked it.

@github-actions

Copy link
Copy Markdown
Contributor

Build results for this fork PR at c3c4365ec093:

Job Result Links
pull-request Passed Open the build · check run

Fork PRs cannot stream build events, so a trusted workflow publishes them after CI finishes and updates this comment as each job is published. Opening a build needs a Trace Machina sign-in; the check run lists the failing tests and their logs.

@github-actions

Copy link
Copy Markdown
Contributor

Image built and pushed!

ghcr.io/TraceMachina/nativelink-worker-init:2026-11-17-7836e5b22b3a7508207f8fff2a47346838b3b3f3

@github-actions

Copy link
Copy Markdown
Contributor

Image built and pushed!

ghcr.io/TraceMachina/nativelink:2026-11-17-7836e5b22b3a7508207f8fff2a47346838b3b3f3

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow! First-time contributor and passed the stress test. Thanks for the PR. Keep 'em comin!

@mvanhorn

Copy link
Copy Markdown
Contributor Author

Thanks for merging this, @MarcusSorealheis. Workers that start before the scheduler is up should now log warnings instead of a wall of errors.

@MarcusSorealheis

Copy link
Copy Markdown
Member

@mvanhorn what's your use case? Also, you can feel free to email me at marcus at tracemachina dot com if easier/unannounced.

This branch was successfully deployed

2 active and 1 inactive deployments
Preview – nativelink — dce16ea5 Deployed Nov 17, 2026 by vercel[bot]
Preview – nativelink-aidm — dce16ea5 Deployed Nov 17, 2026 by vercel[bot]
public-ci — dce16ea5 Deployed Nov 17, 2026 by MarcusSorealheis via NativeLink / Linux (PR) #798
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker connection failures on startup shouldn't raise errors

3 participants