Repository navigation
Report queued and executing actions by age - #2929
Merged
MarcusSorealheis merged 3 commits intoOct 13, 2026
Merged
Conversation
The scheduler reports how many actions are queued, but not how long they
have waited, and nothing about how long executing actions have run. During
a backlog the question an operator asks is how much of the queue has
waited over 30 minutes, and today that can only be estimated from the
queue-time histogram of actions that already left the queue.
The matching pass already reads every queued action, and the executing
sweep every executing one, so each now counts what it read into age
buckets (under 1m, 1-5m, 5-15m, 15-30m, 30-60m, over 60m) and records:
- scheduler.queue.age{age}: queued actions by time since the client
submitted them, requeues included
- scheduler.queue.oldest_age: the oldest of them, in seconds
- scheduler.executing.age{age}: executing actions by time since they
started executing
- scheduler.executing.oldest_age: the longest-running of them
No extra store reads. Every bucket is written on every pass, empty ones
as zero, so a gauge never holds a count from a backlog that has drained.
Every replica reads the same shared state, so aggregate with max.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
amankrx
approved these changes
Oct 13, 2026
This was referenced Oct 13, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The scheduler reports how many actions are queued, but not how long they have waited, and nothing about how long executing actions have been running. During a backlog the question an operator asks is "how much of the queue has waited over 30 minutes", and today that can only be estimated from
execution.queue.time, which only covers actions that have already left the queue.The matching pass already reads every queued action, and the executing sweep (#2923) reads every executing one, so each now counts what it read into age buckets (under 1m, 1–5m, 5–15m, 15–30m, 30–60m, over 60m):
scheduler.queue.age{age}: queued actions by time since the client submitted them (insert_timestamp, so requeues still count from the original submission)scheduler.queue.oldest_age: the oldest queued action, in secondsscheduler.executing.age{age}: executing actions by time since they started executingscheduler.executing.oldest_age: the longest-running executing action, in secondsHow was this verified?
Not yet compiled or run locally: the Nix dev shell could not download the toolchain on a flaky network, so CI will be the first build. The code is formatted with the repository's nightly rustfmt (2026-08-26).
Tests added:
metrics_test.rs: each bucket bound lands in the bucket above it, the last bucket is open, and the oldest age is kept; a start time after "now" (clock skew between replicas) counts as zero age instead of failing; recording with no meter provider installed is a no-op.simple_scheduler_state_manager_test.rs: the existing executing-sweep test now asserts that an action executing for five minutes is counted in the 5–15m bucket, with that age as the oldest.The metrics reference is regenerated (
gen:metrics-reference). Its header moves from v1.7.2 to v1.7.6, so the diff also picks up instruments added since v1.7.2 that the page had not listed.Risk
Low. No extra store reads: both counts come from listings the scheduler already makes. Four new gauges, the two bucketed ones with six values of one attribute.
Every bucket is written on every pass, empty ones as zero, so a gauge never keeps a count from a backlog that has drained. Each replica reads the same shared state and reports the same numbers, so dashboards should aggregate with
max, as forscheduler.queue.depth. The executing gauges only update while the executing sweep runs (executing_sweep_interval_s> 0, default 30s).AI assistance
I iterated on a prompt, prompted an agent, and did several iterations of review on resulting code.