Repository navigation
ci: run Bazel Native nightly and on merge, not on every PR - #2880
Merged
MarcusSorealheis merged 4 commits intoOct 20, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
b7r6
force-pushed
the
ci/native-bazel-nightly
branch
from
October 3, 2026 21:33
b97038a to
9ef2426
Compare
b7r6
force-pushed
the
ci/native-bazel-nightly
branch
from
October 3, 2026 21:37
9ef2426 to
b97038a
Compare
b7r6
pushed a commit
to b7r6/nativelink
that referenced
this pull request
Oct 4, 2026
Combines TraceMachina#2877 + TraceMachina#2878 + TraceMachina#2880 on one branch so a single CI run shows the true end-state per-PR wall clock. Not for merge: the individual PRs are the review units; this exists to measure their net effect.
b7r6
pushed a commit
to b7r6/nativelink
that referenced
this pull request
Oct 4, 2026
Combines TraceMachina#2877 + TraceMachina#2878 + TraceMachina#2880 on one branch so a single CI run shows the true end-state per-PR wall clock. Not for merge: the individual PRs are the review units; this exists to measure their net effect.
This was referenced Oct 7, 2026
Closed
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
…achina#2880) on fork main Mirrors the intended upstream per-PR layout ahead of land: Bazel Native and Cargo Native (incl. Windows) move to nightly + merge-to-main; per-PR build/test signal is carried by the Nix Bazel Dev lane. Keeps the fork representative of where upstream CI is going, not where main happens to sit today. TraceMachina#2897 retry logic preserved in native-bazel.yaml.
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly set.
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 14, 2026
…achina#2880) on fork main Mirrors the intended upstream per-PR layout ahead of land: Bazel Native and Cargo Native (incl. Windows) move to nightly + merge-to-main; per-PR build/test signal is carried by the Nix Bazel Dev lane. Keeps the fork representative of where upstream CI is going, not where main happens to sit today. TraceMachina#2897 retry logic preserved in native-bazel.yaml.
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 14, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly set.
b7r6
force-pushed
the
ci/native-bazel-nightly
branch
from
October 14, 2026 17:40
b97038a to
fe3f097
Compare
The non-Nix Bazel Native lane (Bazel 9.1.1 + 8.7.0) duplicates the per-PR build/test signal from the Nix Bazel Dev lane. Drop its pull_request trigger (keep push:main, add schedule + workflow_dispatch) to take ~12 min off every PR; non-Nix build breakage is caught nightly and on merge.
b7r6
force-pushed
the
ci/native-bazel-nightly
branch
from
October 15, 2026 11:19
fe3f097 to
69a9699
Compare
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly set.
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
…y (--config=nl-toolchain) Realizes the warm-cache contract TraceMachina#2874 set up. That change keys the per-PR Bazel Dev (nix) lane's reads on the endpoint-free nl-toolchain config (the hermetic LLVM platform + downloaded Rust toolchains that nl-rbe already uses), because toolchains keyed on downloaded content produce identical action digests on a bare runner, inside nix develop, or on a remote worker. But nothing writes that digest space: the main-push Bazel 9 Native lane warms with bare --extra_toolchains, and the Bazel Dev lane's own main-push run uploads nothing, since the nix-generated nativelink.bazelrc sets `build --remote_upload_local_results=false`, which outranks the `common --config=nl-write` the setup action writes. Measured on upstream main 818a71d (run 39631138721): Bazel Dev gets 18 remote cache hits out of 3979 processes (2705 run locally, 638s), while Bazel 9 Native gets 2723 hits against the same endpoint. Switch the Bazel 9 lane from --extra_toolchains=@rust_toolchains//:all to --config=nl-toolchain (which contains that same line plus the @llvm platform pin). Bazel Native now runs on main pushes and nightly only (TraceMachina#2880), and on main pushes it holds the write key, so it becomes the warmer for the hermetic digest space the per-PR Bazel Dev lane reads. The Bazel 8 lane is untouched (separate, lockfile_mode=off digest space). Transition cost: the first runs after this merges are cold until the first main push warms the new key space, which is one build. Measured so far is key ALIGNMENT, not the warm win: the aquery proof run shows nix+nl-toolchain command lines byte-identical to bare-runner nl-toolchain with no /nix/store references (https://github.com/b7r6/nativelink/actions/runs/37059855343). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
Main requires no build or test check to merge: the main ruleset requires only pre-commit-checks, CodeQL and vale (checked 2026-10-21), so merges are green-by-convention for every build and test lane. The obvious fix, marking individual workflows required, has a known trap: a workflow skipped by paths-ignore never reports, and a required check that never reports leaves the PR pending forever. That trap is also what keeps lane-selection work (skipping expensive lanes on diffs that cannot affect them) from being adoptable: every selected-away lane would wedge the PR. Add a Verification workflow with NO path filters, so it always runs and always reports, intended as the single required check. It waits for every other check run on the PR head commit, fails on any failed / timed-out / cancelled check, accepts 'skipped' (a job-level skip inside a workflow that ran, i.e. deliberate selection, still reports), and fails if an expected lane never reported at all, which catches the silent never-started cases: a broken workflow file, a runner outage. Expected lanes follow the per-PR topology after TraceMachina#2877 and TraceMachina#2880: pre-commit-checks always, and Bazel Dev / ubuntu-24.04 (the per-PR build and test signal now that Bazel Native and Cargo Native run on merge and nightly only) unless every changed file is one nix.yaml path-filters away (web/**, .github/styles/**). The gate needs two consecutive all-complete polls before it judges, because a job behind `needs:` (the rbe matrix behind generate-rbe-commands) gets its check run only after its dependency completes, and a single poll in that gap would pass a commit whose rbe lanes had not run yet. The script was exercised locally against a fake `gh` for all-green, missing lane, web-only, red lane and the `needs:` gap; the gap case passes falsely with a single poll. Costs and limits: the gate occupies one ubuntu runner while it polls (worst case ~45 min, typically it finishes with the slowest lane); the lane lists couple to job names and to nix.yaml's paths-ignore, and a rename fails loud (missing lane), not silent; and making it actually required is a ruleset setting only maintainers can change. This change only provides the check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
Draft PRs run the per-PR heavy matrix on every push even though nobody gates a draft on green checks. After TraceMachina#2877, TraceMachina#2879 and TraceMachina#2880 moved Bazel Native, Cargo Native, coverage, the OCI image and the integration lanes off pull_request, the per-PR heavy set is: Bazel Dev, Installation and the rbe-toolchain matrix (nix.yaml), the LRE lanes (lre.yaml) and asan (sanitizers.yaml). Gate those jobs on the PR not being a draft and add ready_for_review to the pull_request trigger types so the full set fires the moment a draft is promoted. In nix.yaml the guard is ANDed with the existing nightly skip; the rbe matrix follows generate-rbe-commands via `needs:`. Fast advisory lanes (vale, pre-commit, docs-lint, pr-template-check) keep running on drafts on purpose. Push, schedule and workflow_dispatch behavior is unchanged: the guard passes for every non-PR event. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 22, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly set. (cherry picked from commit 46e058d)
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 22, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly set. (cherry picked from commit 46e058d)
MarcusSorealheis
pushed a commit
that referenced
this pull request
Oct 22, 2026
* ci: defer the main.yaml integration-tests OCI lane off per-PR Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and #2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching #2880's pattern. Fork-local until folded into the upstream OCI/nightly set. (cherry picked from commit 46e058d) * Export telemetry over TLS to an https OTLP collector An `https://` collector endpoint stopped NativeLink at startup: `opentelemetry-otlp` was built without any of its TLS features and refuses to build a tonic exporter for HTTPS without one. The only collector a process could export to was one reached in plaintext, so a worker outside the cluster could not send its telemetry to the cluster's collector. Enable `tls-ring` on `opentelemetry-otlp` in nativelink-util. NativeLink already uses ring as its rustls provider, so this adds no crypto backend. An https endpoint, in `OTEL_EXPORTER_OTLP_ENDPOINT` or `NL_OTEL_ENDPOINT`, now gets server-authenticated TLS against the system trust store (tonic's bare `ClientTlsConfig::new()` trusts nothing, so the roots are enabled explicitly; a plaintext endpoint gets no TLS configuration at all, so a host without a certificate bundle still exports). Read the OpenTelemetry specification's certificate variables, which the Rust SDK does not: `OTEL_EXPORTER_OTLP_CERTIFICATE` names a private CA and is then the only CA trusted; `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` and `OTEL_EXPORTER_OTLP_CLIENT_KEY` name the identity for mTLS and have to be set together. The per-signal forms win over the general one. A file that cannot be read, or half an identity, stops the process at startup with an InvalidArgument naming the variable, which is also what a malformed `NL_OTEL_ENDPOINT` now does instead of panicking. Fixes #2983 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Give a signal with its own certificate variable its own NL_OTEL_ENDPOINT channel The shared channel `NL_OTEL_ENDPOINT` builds was made with the general certificate variables only, and an exporter handed a channel ignores its own `with_tls_config`, so `OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE` and the other per-signal variables worked with `OTEL_EXPORTER_OTLP_ENDPOINT` but not with `NL_OTEL_ENDPOINT`: the handshake failed with an unknown CA. Keep the one shared channel as the common case, and when any per-signal certificate variable is set build one channel per signal with that signal's resolution of the variables. `maybe_load_balanced_channel` stays as the general-variables channel for the integration tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Build no TLS for a plaintext OTLP signal and accept an uppercase https scheme A plaintext signal got a TLS configuration whenever a certificate variable was set, and the exporter still hands that to tonic, which loads the trust roots as it builds the connector and fails on a host without a certificate bundle: a deployment sending metrics over TLS with an identity and logs in plaintext died building the log exporter. A plaintext endpoint now gets no configuration whatever the variables say; only an https endpoint gets one. The scheme check was a case-sensitive `starts_with("https://")`, while the SDK accepts `HTTPS://` too, so such an endpoint started with an empty trust store. Parse the value as a URL and compare the scheme; a value that does not parse counts as plaintext and is left for the exporter to reject. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Spell unparsable the way the typos hook wants Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Say a directive that cannot be parsed, which both spell checkers accept Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Skip the SDK endpoint's TLS configuration for an exporter given the NL_OTEL_ENDPOINT channel An exporter handed the load-balanced channel sends on it and ignores the SDK's endpoint, but the builder still read that endpoint's certificate variables, so a stale OTEL_EXPORTER_OTLP_ENDPOINT pointing at an https collector with a missing CA file stopped a process that was going to connect in plaintext elsewhere. The transport is now chosen once per signal: the channel, or else the signal's own TLS configuration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: deus-ex-b7r6-0x02 <benjamin.reesman@tracemachina.com>
MarcusSorealheis
added a commit
that referenced
this pull request
Oct 23, 2026
* ci: defer the main.yaml integration-tests OCI lane off per-PR Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and #2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching #2880's pattern. Fork-local until folded into the upstream OCI/nightly set. (cherry picked from commit 46e058d) * Give a namespaced action a /proc that matches its PID namespace With use_namespaces an action is PID 1 of its own PID namespace, but /proc stayed the host's, so an action looking up a child it forked under /proc/<pid> read whichever host process had that number. The worker now always takes a private mount namespace as well, and the PID 1 child mounts a fresh proc over /proc before it execs, with the host /proc's locked flags mirrored so a user namespace accepts it. Where the host refuses the mount, as an unprivileged container that masks paths under /proc does, a startup probe warns and the action keeps the host's /proc as before. The probe shares its child-side code with namespaces_supported instead of repeating it. Fixes #3003 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Wait for the forked child to exec and compare /proc mounts in the proc tests Right after `sleep 30 &` the child can still be the shell, so reading its command line raced the exec and failed about once in a few hundred runs. The script now waits for the child to report `sleep 30`. The host leak check compared /proc/1/comm with "sh", which is wrong wherever the enclosing PID 1 is a shell; it now compares the test process's /proc mounts before and after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Keep the mount namespace optional without use_mount_namespace Without use_mount_namespace the mount namespace only carries the fresh /proc, but entering it and making it private had become mandatory, so a syscall policy that denies mount failed the namespace probe and the worker refused to start where it ran before. Now a refused CLONE_NEWNS or a refused private remount there falls back to the namespaces the action had before the fresh /proc, which is then skipped; with use_mount_namespace both stay required for the root mask. A test installs a seccomp filter denying mount in the pre-exec hook to show the action still runs as PID 1, and that the masked variant still fails with EPERM. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: deus-ex-b7r6-0x02 <benjamin.reesman@tracemachina.com> Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Bazel Nativebuilds and tests with the non-Nix Bazel toolchain on every pull request: Bazel 9.1.1, plus an 8.7.0 leg that checks lockfile compatibility. Nix's Bazel Dev lane already runs the samebazel test //...on every PR. This moves Bazel Native to a nightly run plus every merge tomain, withworkflow_dispatchfor on-demand runs. Its runs onmaintoday took 7 to 11 minutes.How was this verified?
The workflow parses. The job matrix and every step are unchanged: the
pull_requesttrigger is removed,push: mainstays, andscheduleandworkflow_dispatchare added. It is not a required status check, so dropping the PR trigger doesn't block merges.Our fork's
mainhas run this schedule since 2026-10-14, and today's scheduled Bazel Native run there passed.Risk
This is the most debatable change in the CI-deferral set. A break in the non-Nix Bazel build, or in the 8.7.0 lockfile check, would be caught on merge or overnight instead of on the PR. People who build without Nix would see that breakage after merge.
Bazel Dev still builds and tests everything on each PR, and
push: mainkeeps a check at merge time. No source, config or wire-format changes. If you'd rather keep non-Nix Bazel on PRs, this is the one to drop.AI assistance
Claude drafted this change and description under my direction.