Repository navigation
fix: log worker startup scheduler misses as warnings - #3210
MarcusSorealheis merged 2 commits into
Conversation
The handler now takes a failure label, connect, register, or disconnect, and includes it on the log line. Before ever_registered is set the attempt is logged with warn!; after the worker has registered, the same attempt is logged with error!. The message is "Scheduler connection attempt failed, retrying". The worker tests require a startup connect failure to be a WARN line with failure="connect", and a disconnect after registration to be an ERROR. Fixes TraceMachina#615
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@mvanhorn This is a straightforward improvement to the user/agent experience. We appreciate it. I'll take it for a spin now. |
|
/build-image nativelink |
|
/build-image nativelink-worker-init |
|
We have a flavor of this log decoration in our proprietary distribution, which relies heavily on readiness probes at startup but this exact feature should be in open source. We overlooked it. |
|
Build results for this fork PR at
Fork PRs cannot stream build events, so a trusted workflow publishes them after CI finishes and updates this comment as each job is published. Opening a build needs a Trace Machina sign-in; the check run lists the failing tests and their logs. |
|
Image built and pushed! |
|
Image built and pushed! |
MarcusSorealheis
left a comment
There was a problem hiding this comment.
Wow! First-time contributor and passed the stress test. Thanks for the PR. Keep 'em comin!
|
Thanks for merging this, @MarcusSorealheis. Workers that start before the scheduler is up should now log warnings instead of a wall of errors. |
|
@mvanhorn what's your use case? Also, you can feel free to email me at marcus at tracemachina dot com if easier/unannounced. |
What and why
A worker that comes up before its scheduler is reachable records each failed attempt as an error. That startup order is normal, and the ERROR lines pile up in the worker log for as long as the scheduler is still down. Every reconnect failure in LocalWorker::run went through one handler that logged at error level with the message "Error". Connect failures, registration failures, and later disconnects all shared that path, and the level did not depend on ever_registered, so attempts made before the first successful registration were errors too.
The handler now takes a failure label, connect, register, or disconnect, and includes it on the log line. Before ever_registered is set the attempt is logged with warn!; after the worker has registered, the same attempt is logged with error!. The message is "Scheduler connection attempt failed, retrying". The worker tests require a startup connect failure to be a WARN line with failure="connect", and a disconnect after registration to be an ERROR.
Closes #615
How was this verified?
Ran
cargo testlocally for the nativelink change; it passed.The tests passed with the nativelink change applied; a run with it reverted gave no clear result, so no test is shown to catch this bug.
The changed test nativelink-worker/tests/local_worker_test.rs did not run here: no test runner for it was available.
Risk
Scoped to the files listed above; no other caller or consumer changes.
AI assistance
AI was used for assistance.