Skip to content

ci: run Bazel Native nightly and on merge, not on every PR - #2880

Merged
MarcusSorealheis merged 4 commits into
TraceMachina:mainfrom
b7r6:ci/native-bazel-nightly
Oct 20, 2026
Merged

MarcusSorealheis merged 4 commits into
TraceMachina:mainfrom
b7r6:ci/native-bazel-nightly

Conversation

@b7r6

@b7r6 b7r6 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Bazel Native builds and tests with the non-Nix Bazel toolchain on every pull request: Bazel 9.1.1, plus an 8.7.0 leg that checks lockfile compatibility. Nix's Bazel Dev lane already runs the same bazel test //... on every PR. This moves Bazel Native to a nightly run plus every merge to main, with workflow_dispatch for on-demand runs. Its runs on main today took 7 to 11 minutes.

How was this verified?

The workflow parses. The job matrix and every step are unchanged: the pull_request trigger is removed, push: main stays, and schedule and workflow_dispatch are added. It is not a required status check, so dropping the PR trigger doesn't block merges.

Our fork's main has run this schedule since 2026-10-14, and today's scheduled Bazel Native run there passed.

Risk

This is the most debatable change in the CI-deferral set. A break in the non-Nix Bazel build, or in the 8.7.0 lockfile check, would be caught on merge or overnight instead of on the PR. People who build without Nix would see that breakage after merge.

Bazel Dev still builds and tests everything on each PR, and push: main keeps a check at merge time. No source, config or wire-format changes. If you'd rather keep non-Nix Bazel on PRs, this is the one to drop.

AI assistance

Claude drafted this change and description under my direction.

@vercel

vercel Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 19, 2026 11:47pm UTC
nativelink-aidm Ready Ready Preview Oct 19, 2026 11:47pm UTC

Request Review

@b7r6
b7r6 force-pushed the ci/native-bazel-nightly branch from b97038a to 9ef2426 Compare October 3, 2026 21:33
@b7r6
b7r6 force-pushed the ci/native-bazel-nightly branch from 9ef2426 to b97038a Compare October 3, 2026 21:37
b7r6 pushed a commit to b7r6/nativelink that referenced this pull request Oct 4, 2026
Combines TraceMachina#2877 + TraceMachina#2878 + TraceMachina#2880 on one branch so a single CI run shows the
true end-state per-PR wall clock. Not for merge: the individual PRs are the
review units; this exists to measure their net effect.
b7r6 pushed a commit to b7r6/nativelink that referenced this pull request Oct 4, 2026
Combines TraceMachina#2877 + TraceMachina#2878 + TraceMachina#2880 on one branch so a single CI run shows the
true end-state per-PR wall clock. Not for merge: the individual PRs are the
review units; this exists to measure their net effect.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
…achina#2880) on fork main

Mirrors the intended upstream per-PR layout ahead of land: Bazel Native and
Cargo Native (incl. Windows) move to nightly + merge-to-main; per-PR build/test
signal is carried by the Nix Bazel Dev lane. Keeps the fork representative of
where upstream CI is going, not where main happens to sit today. TraceMachina#2897 retry
logic preserved in native-bazel.yaml.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 14, 2026
…achina#2880) on fork main

Mirrors the intended upstream per-PR layout ahead of land: Bazel Native and
Cargo Native (incl. Windows) move to nightly + merge-to-main; per-PR build/test
signal is carried by the Nix Bazel Dev lane. Keeps the fork representative of
where upstream CI is going, not where main happens to sit today. TraceMachina#2897 retry
logic preserved in native-bazel.yaml.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 14, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
@b7r6
b7r6 force-pushed the ci/native-bazel-nightly branch from b97038a to fe3f097 Compare October 14, 2026 17:40
The non-Nix Bazel Native lane (Bazel 9.1.1 + 8.7.0) duplicates the per-PR
build/test signal from the Nix Bazel Dev lane. Drop its pull_request
trigger (keep push:main, add schedule + workflow_dispatch) to take ~12 min
off every PR; non-Nix build breakage is caught nightly and on merge.

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MarcusSorealheis
MarcusSorealheis merged commit 616c315 into TraceMachina:main Oct 20, 2026
29 checks passed
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
…y (--config=nl-toolchain)

Realizes the warm-cache contract TraceMachina#2874 set up. That change keys the
per-PR Bazel Dev (nix) lane's reads on the endpoint-free nl-toolchain
config (the hermetic LLVM platform + downloaded Rust toolchains that
nl-rbe already uses), because toolchains keyed on downloaded content
produce identical action digests on a bare runner, inside nix develop,
or on a remote worker. But nothing writes that digest space: the
main-push Bazel 9 Native lane warms with bare --extra_toolchains, and
the Bazel Dev lane's own main-push run uploads nothing, since the
nix-generated nativelink.bazelrc sets
`build --remote_upload_local_results=false`, which outranks the
`common --config=nl-write` the setup action writes. Measured on
upstream main 818a71d (run 39631138721): Bazel Dev gets 18 remote
cache hits out of 3979 processes (2705 run locally, 638s), while Bazel 9
Native gets 2723 hits against the same endpoint.

Switch the Bazel 9 lane from --extra_toolchains=@rust_toolchains//:all
to --config=nl-toolchain (which contains that same line plus the @llvm
platform pin). Bazel Native now runs on main pushes and nightly only
(TraceMachina#2880), and on main pushes it holds the write key, so it becomes the
warmer for the hermetic digest space the per-PR Bazel Dev lane reads.
The Bazel 8 lane is untouched (separate, lockfile_mode=off digest
space).

Transition cost: the first runs after this merges are cold until the
first main push warms the new key space, which is one build. Measured so
far is key ALIGNMENT, not the warm win: the aquery proof run shows
nix+nl-toolchain command lines byte-identical to bare-runner
nl-toolchain with no /nix/store references
(https://github.com/b7r6/nativelink/actions/runs/37059855343).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
Main requires no build or test check to merge: the main ruleset
requires only pre-commit-checks, CodeQL and vale (checked 2026-10-21),
so merges are green-by-convention for every build and test lane. The
obvious fix, marking individual workflows required, has a known trap: a
workflow skipped by paths-ignore never reports, and a required check
that never reports leaves the PR pending forever. That trap is also what
keeps lane-selection work (skipping expensive lanes on diffs that cannot
affect them) from being adoptable: every selected-away lane would wedge
the PR.

Add a Verification workflow with NO path filters, so it always runs and
always reports, intended as the single required check. It waits for
every other check run on the PR head commit, fails on any failed /
timed-out / cancelled check, accepts 'skipped' (a job-level skip inside
a workflow that ran, i.e. deliberate selection, still reports), and
fails if an expected lane never reported at all, which catches the
silent never-started cases: a broken workflow file, a runner outage.

Expected lanes follow the per-PR topology after TraceMachina#2877 and TraceMachina#2880:
pre-commit-checks always, and Bazel Dev / ubuntu-24.04 (the per-PR
build and test signal now that Bazel Native and Cargo Native run on
merge and nightly only) unless every changed file is one nix.yaml
path-filters away (web/**, .github/styles/**). The gate needs two
consecutive all-complete polls before it judges, because a job behind
`needs:` (the rbe matrix behind generate-rbe-commands) gets its check
run only after its dependency completes, and a single poll in that gap
would pass a commit whose rbe lanes had not run yet. The script was
exercised locally against a fake `gh` for all-green, missing lane,
web-only, red lane and the `needs:` gap; the gap case passes falsely
with a single poll.

Costs and limits: the gate occupies one ubuntu runner while it polls
(worst case ~45 min, typically it finishes with the slowest lane); the
lane lists couple to job names and to nix.yaml's paths-ignore, and a
rename fails loud (missing lane), not silent; and making it actually
required is a ruleset setting only maintainers can change. This change
only provides the check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
Draft PRs run the per-PR heavy matrix on every push even though nobody
gates a draft on green checks. After TraceMachina#2877, TraceMachina#2879 and TraceMachina#2880 moved Bazel
Native, Cargo Native, coverage, the OCI image and the integration lanes
off pull_request, the per-PR heavy set is: Bazel Dev, Installation and
the rbe-toolchain matrix (nix.yaml), the LRE lanes (lre.yaml) and asan
(sanitizers.yaml). Gate those jobs on the PR not being a draft and add
ready_for_review to the pull_request trigger types so the full set fires
the moment a draft is promoted. In nix.yaml the guard is ANDed with the
existing nightly skip; the rbe matrix follows generate-rbe-commands via
`needs:`. Fast advisory lanes (vale, pre-commit, docs-lint,
pr-template-check) keep running on drafts on purpose.

Push, schedule and workflow_dispatch behavior is unchanged: the guard
passes for every non-PR event.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 22, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.

(cherry picked from commit 46e058d)
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 22, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.

(cherry picked from commit 46e058d)
MarcusSorealheis pushed a commit that referenced this pull request Oct 22, 2026
* ci: defer the main.yaml integration-tests OCI lane off per-PR

Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and #2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching #2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.

(cherry picked from commit 46e058d)

* Export telemetry over TLS to an https OTLP collector

An `https://` collector endpoint stopped NativeLink at startup:
`opentelemetry-otlp` was built without any of its TLS features and refuses
to build a tonic exporter for HTTPS without one. The only collector a
process could export to was one reached in plaintext, so a worker outside
the cluster could not send its telemetry to the cluster's collector.

Enable `tls-ring` on `opentelemetry-otlp` in nativelink-util. NativeLink
already uses ring as its rustls provider, so this adds no crypto backend.
An https endpoint, in `OTEL_EXPORTER_OTLP_ENDPOINT` or `NL_OTEL_ENDPOINT`,
now gets server-authenticated TLS against the system trust store (tonic's
bare `ClientTlsConfig::new()` trusts nothing, so the roots are enabled
explicitly; a plaintext endpoint gets no TLS configuration at all, so a
host without a certificate bundle still exports).

Read the OpenTelemetry specification's certificate variables, which the
Rust SDK does not: `OTEL_EXPORTER_OTLP_CERTIFICATE` names a private CA and
is then the only CA trusted; `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` and
`OTEL_EXPORTER_OTLP_CLIENT_KEY` name the identity for mTLS and have to be
set together. The per-signal forms win over the general one. A file that
cannot be read, or half an identity, stops the process at startup with an
InvalidArgument naming the variable, which is also what a malformed
`NL_OTEL_ENDPOINT` now does instead of panicking.

Fixes #2983

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Give a signal with its own certificate variable its own NL_OTEL_ENDPOINT channel

The shared channel `NL_OTEL_ENDPOINT` builds was made with the general
certificate variables only, and an exporter handed a channel ignores its
own `with_tls_config`, so `OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE` and the
other per-signal variables worked with `OTEL_EXPORTER_OTLP_ENDPOINT` but
not with `NL_OTEL_ENDPOINT`: the handshake failed with an unknown CA.

Keep the one shared channel as the common case, and when any per-signal
certificate variable is set build one channel per signal with that
signal's resolution of the variables. `maybe_load_balanced_channel` stays
as the general-variables channel for the integration tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Build no TLS for a plaintext OTLP signal and accept an uppercase https scheme

A plaintext signal got a TLS configuration whenever a certificate variable
was set, and the exporter still hands that to tonic, which loads the trust
roots as it builds the connector and fails on a host without a certificate
bundle: a deployment sending metrics over TLS with an identity and logs in
plaintext died building the log exporter. A plaintext endpoint now gets no
configuration whatever the variables say; only an https endpoint gets one.

The scheme check was a case-sensitive `starts_with("https://")`, while the
SDK accepts `HTTPS://` too, so such an endpoint started with an empty trust
store. Parse the value as a URL and compare the scheme; a value that does
not parse counts as plaintext and is left for the exporter to reject.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Spell unparsable the way the typos hook wants

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Say a directive that cannot be parsed, which both spell checkers accept

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Skip the SDK endpoint's TLS configuration for an exporter given the NL_OTEL_ENDPOINT channel

An exporter handed the load-balanced channel sends on it and ignores the
SDK's endpoint, but the builder still read that endpoint's certificate
variables, so a stale OTEL_EXPORTER_OTLP_ENDPOINT pointing at an https
collector with a missing CA file stopped a process that was going to
connect in plaintext elsewhere. The transport is now chosen once per
signal: the channel, or else the signal's own TLS configuration.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: deus-ex-b7r6-0x02 <benjamin.reesman@tracemachina.com>
MarcusSorealheis added a commit that referenced this pull request Oct 23, 2026
* ci: defer the main.yaml integration-tests OCI lane off per-PR

Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and #2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching #2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.

(cherry picked from commit 46e058d)

* Give a namespaced action a /proc that matches its PID namespace

With use_namespaces an action is PID 1 of its own PID namespace, but
/proc stayed the host's, so an action looking up a child it forked
under /proc/<pid> read whichever host process had that number. The
worker now always takes a private mount namespace as well, and the
PID 1 child mounts a fresh proc over /proc before it execs, with the
host /proc's locked flags mirrored so a user namespace accepts it.

Where the host refuses the mount, as an unprivileged container that
masks paths under /proc does, a startup probe warns and the action
keeps the host's /proc as before. The probe shares its child-side
code with namespaces_supported instead of repeating it.

Fixes #3003

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Wait for the forked child to exec and compare /proc mounts in the proc tests

Right after `sleep 30 &` the child can still be the shell, so reading
its command line raced the exec and failed about once in a few hundred
runs. The script now waits for the child to report `sleep 30`. The host
leak check compared /proc/1/comm with "sh", which is wrong wherever the
enclosing PID 1 is a shell; it now compares the test process's /proc
mounts before and after.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Keep the mount namespace optional without use_mount_namespace

Without use_mount_namespace the mount namespace only carries the fresh
/proc, but entering it and making it private had become mandatory, so a
syscall policy that denies mount failed the namespace probe and the
worker refused to start where it ran before. Now a refused CLONE_NEWNS
or a refused private remount there falls back to the namespaces the
action had before the fresh /proc, which is then skipped; with
use_mount_namespace both stay required for the root mask. A test
installs a seccomp filter denying mount in the pre-exec hook to show
the action still runs as PID 1, and that the masked variant still
fails with EPERM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: deus-ex-b7r6-0x02 <benjamin.reesman@tracemachina.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — 6a5bd89d Deployed Oct 19, 2026 by vercel[bot]
Preview – nativelink-aidm — 6a5bd89d Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants