Skip to content

Report queued and executing actions by age - #2929

Merged
MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
corcillo:corcillo-queue-age-metrics
Oct 13, 2026
Merged

MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
corcillo:corcillo-queue-age-metrics

Conversation

@corcillo

Copy link
Copy Markdown
Collaborator

What and why

The scheduler reports how many actions are queued, but not how long they have waited, and nothing about how long executing actions have been running. During a backlog the question an operator asks is "how much of the queue has waited over 30 minutes", and today that can only be estimated from execution.queue.time, which only covers actions that have already left the queue.

The matching pass already reads every queued action, and the executing sweep (#2923) reads every executing one, so each now counts what it read into age buckets (under 1m, 1–5m, 5–15m, 15–30m, 30–60m, over 60m):

  • scheduler.queue.age{age}: queued actions by time since the client submitted them (insert_timestamp, so requeues still count from the original submission)
  • scheduler.queue.oldest_age: the oldest queued action, in seconds
  • scheduler.executing.age{age}: executing actions by time since they started executing
  • scheduler.executing.oldest_age: the longest-running executing action, in seconds

How was this verified?

Not yet compiled or run locally: the Nix dev shell could not download the toolchain on a flaky network, so CI will be the first build. The code is formatted with the repository's nightly rustfmt (2026-08-26).

Tests added:

  • metrics_test.rs: each bucket bound lands in the bucket above it, the last bucket is open, and the oldest age is kept; a start time after "now" (clock skew between replicas) counts as zero age instead of failing; recording with no meter provider installed is a no-op.
  • simple_scheduler_state_manager_test.rs: the existing executing-sweep test now asserts that an action executing for five minutes is counted in the 5–15m bucket, with that age as the oldest.

The metrics reference is regenerated (gen:metrics-reference). Its header moves from v1.7.2 to v1.7.6, so the diff also picks up instruments added since v1.7.2 that the page had not listed.

Risk

Low. No extra store reads: both counts come from listings the scheduler already makes. Four new gauges, the two bucketed ones with six values of one attribute.

Every bucket is written on every pass, empty ones as zero, so a gauge never keeps a count from a backlog that has drained. Each replica reads the same shared state and reports the same numbers, so dashboards should aggregate with max, as for scheduler.queue.depth. The executing gauges only update while the executing sweep runs (executing_sweep_interval_s > 0, default 30s).

AI assistance

I iterated on a prompt, prompted an agent, and did several iterations of review on resulting code.

The scheduler reports how many actions are queued, but not how long they
have waited, and nothing about how long executing actions have run. During
a backlog the question an operator asks is how much of the queue has
waited over 30 minutes, and today that can only be estimated from the
queue-time histogram of actions that already left the queue.

The matching pass already reads every queued action, and the executing
sweep every executing one, so each now counts what it read into age
buckets (under 1m, 1-5m, 5-15m, 15-30m, 30-60m, over 60m) and records:

- scheduler.queue.age{age}: queued actions by time since the client
  submitted them, requeues included
- scheduler.queue.oldest_age: the oldest of them, in seconds
- scheduler.executing.age{age}: executing actions by time since they
  started executing
- scheduler.executing.oldest_age: the longest-running of them

No extra store reads. Every bucket is written on every pass, empty ones
as zero, so a gauge never holds a count from a backlog that has drained.
Every replica reads the same shared state, so aggregate with max.
@vercel

vercel Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 13, 2026 1:46pm UTC
nativelink-aidm Ready Ready Preview Oct 13, 2026 1:46pm UTC

Request Review

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MarcusSorealheis
MarcusSorealheis merged commit d06d9c9 into TraceMachina:main Oct 13, 2026
41 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — acfb73d1 Deployed Oct 13, 2026 by vercel[bot]
Preview – nativelink-aidm — acfb73d1 Deployed Oct 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants