Skip to content

Isolate persistent worker by Action instance name. A fix for issue #2049 - #3142

Open
louisgregg wants to merge 23 commits into
TraceMachina:mainfrom
louisgregg:2049-isolate-persistent-worker-by-instance-name
Open

louisgregg wants to merge 23 commits into
TraceMachina:mainfrom
louisgregg:2049-isolate-persistent-worker-by-instance-name

Conversation

@louisgregg

@louisgregg louisgregg commented Nov 7, 2026 •

Copy link
Copy Markdown

What and why

Fixes #2049
This PR adds the ability to prevent persistent workers from sharing jobs if the don't share the same ActionInfo.instance_name() by including the instance_name in the WorkerKey. It prevents an Action originating on different instance from re-using the same idle worker as a previous Action even if the Actions appear to be otherwise identical. In multi-tenant deployments which use instance_name to define the boundary between tenants, this PR should preserve that boundary between workers assigned to jobs.

However, if a configuration puts multiple tenants in under the same instance, then this additional filter in WorkerKey will be insufficient to prevent information leakage between tenant jobs. This PR provides instance isolation, not total tenant isolation.

Updates to PR & default behavior

Following some feedback on the PR, this feature will be disabled by default. This was due to the potential performance impact isolation will cause for multi-tenant instances. It can be enabled by setting PoolConfig.isolate_by_instance_name to true.

Current configuration and information leakage / build poisoning risks

Currently, two Actions having the same WorkerKey may inherit a Worker from one another via the pool. The WorkerKey is defined by:

Consider two actions, Action 1 and Action 2, which are executed in sequence.

  1. Action 1 is assigned a new persistent Worker A.
  2. The build job completes without errors and Worker A is returned to the pool.
  3. Action 2 is started and its WorkerKey matches that of Action 1. Therefore it is assigned Worker A.

Worker A now executes Action 2 with access to the following:

  • in-process memory artifacts left over from Action 1, such as compiler/JVM caches or source code fragments. This may also include any secrets read from CLI inputs.
  • the persistent working_dir and its contents. This consists of anything the worker wrote outside of the per-action sandbox_dir (e.g. scratch files, caches, downloaded dependencies) during Action 1.

Note: Content-keyed caches generated during Action 1 will be available during Action 2's execution too. This could lead to build poisoning if Action 2 uses Action 1's cached output as input.

How was this verified?

I did not run this against a real instance of NativeLink. However, unit and integration tests were added for the following cases.

  • Check that the updated PersistentWorkersConfig deserializes from json correctly.
    • isolate_by_instance_name_defaults_to_false()
    • isolate_by_instance_name_deserializes_correctly_when_set()
  • Ensure the updated WorkerKey works as intended.
    • worker_key_distinguishes_different_instance_names()
  • Ensure that idle workers in the pool are assigned to new Actions correctly based on the PoolConfig.isolation_by_instance_name flag.
    • pool_shares_workers_for_different_instance_names_when_isolation_disabled()
    • pool_uses_different_workers_for_different_instance_names_when_isolation_enabled()
  • An integration test for RunningActionsManagerImpl to check Action/Worker assignment respects the isolation_by_instance_name flag, when it is set.
    • persistent_worker_isolates_actions_by_instance_name_when_enabled()
    • persistent_worker_does_not_isolate_actions_by_instance_name_by_default()

Risk

For a customer under a single trust domain using many instance_names, the worker pool hit-rate will drop, increasing the build time of otherwise identical actions and reducing overall performance. If the performance impact is severe, the PoolConfig.isolate_by_instance_name flag should be set to false.

AI assistance

I wrote all code in this PR myself, accepting some good ideas from the LLM and rejecting others. I used Claude Code as follows:

  • to conduct initial survey of open issues on the NativeLink Github and suggest suitable issues to work on.
  • to guide initial work on the repo, i.e. finding which files to update and suggesting code changes which align with the existing patterns/code style in the repo.
  • to find suitable places to add different unit tests/integration tests. The LLM also suggested some existing tests I could quickly modify to test behavior I introduced. I developed the tests myself and made sure the new tests were passing before creating the PR.

@CLAassistant

CLAassistant commented Nov 7, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

…normie-github:louisgregg/nativelink into 2049-isolate-persistent-worker-by-instance-name
…normie-github:louisgregg/nativelink into 2049-isolate-persistent-worker-by-instance-name
@vercel

vercel Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Nov 20, 2026 2:28am UTC
nativelink-aidm Ready Ready Preview Nov 20, 2026 2:28am UTC

Request Review

@louisgregg

Copy link
Copy Markdown
Author

@MarcusSorealheis @modernmedici this PR is ready for review when you get a chance. Thanks.

@MarcusSorealheis

Copy link
Copy Markdown
Member

Hi @louisgregg interesting approach. I have many thoughts. My first, though is should this isolation be off by default:PoolConfig.isolate_by_instance_name. Also, the performance penalty feels quite serious.

@louisgregg

Copy link
Copy Markdown
Author

@MarcusSorealheis thanks for the feedback. isolate_by_instance_name now defaults to false.

I'm not sure how to quantify the performance impact. The overall risk should be the product of the number of multi-tenant-but-single-instance nativelink users times the average slowdown-per-action each of them will experience.

Who might know how to estimate this? Nativelink user statistics may be necessary here, which I probably don't have access to.

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Isolate persistent workers by Action instance name

I read the full diff and checked the default against PersistentWorkersSettings in nativelink-worker/src/running_actions_manager.rs. I did not check out the branch or run the tests, and no Rust, Bazel, or NativeLink test jobs have run on this PR yet.

Blocking

The fix is off by default. isolate_by_instance_name defaults to false in nativelink-config/src/cas_server.rs (default_false) and in PoolConfig::default(). Multi-tenant deployments that use instance_name as their boundary, the case issue #2049 describes, stay exposed unless they opt in. The PR body contradicts this: it names isolate_by_instance_name_defaults_to_true and says to set the flag to false if the performance cost is severe, which only makes sense if the default is true. Either the default should be true, with a migration note for the lower hit rate, or the body should say the flag is opt-in and explain why the leak stays open by default.

Other issues

  1. The PR description does not match the tests. The body lists isolate_by_instance_name_defaults_to_true, persistent_worker_isolates_actions_by_instance_name, and persistent_worker_does_not_isolate_actions_by_instance_name_when_disabled. The diff has ..._defaults_to_false, ..._when_enabled, and ..._by_default. The body also says PoolConfig.isolation_by_instance_name, but the field is isolate_by_instance_name.
  2. A misleading test comment. In pool_uses_different_workers_for_different_instance_names_when_isolation_enabled (nativelink-worker/src/persistent_worker/pool.rs), the comment before the second acquire says "isolate_by_instance_name is false / So the same worker should be reused", but the test enables isolation and asserts two workers. It looks copy-pasted from the test above it.
  3. Config reference not regenerated. The new field has a doc comment, but web/apps/docs/content/docs/reference/nativelink-config/* is not in the diff. AGENTS.md requires gen:config-reference, not hand edits.
  4. Formatting and unrelated churn. The trailing comma added in worker_key_rejects_empty_argv is unrelated to this change. running_actions_manager_test.rs has trailing whitespace after }, in the new isolated-test config block. nativelink-config/tests/deserialization_test.rs has the typo "don't isolation by instance name", and the PoolConfig doc says "will be not be included".
  5. A misplaced doc line. pool.rs inserts /// Name of instance group included because it's not a secret. into the doc block on impl fmt::Debug for WorkerKey, where it reads as part of the existing "environment can carry a token" note.
  6. Simplicity. default_false is redundant, since #[serde(default)] on a bool already defaults to false. The two new tests in running_actions_manager_test.rs duplicate about 200 lines of setup; a shared helper that takes the flag would be simpler.
  7. Verification gaps. The status rollup shows only the PR template check, the CLA, and Vercel. The body lists test names but no commands or results, and does not say which checks were skipped, which AGENTS.md requires. Before this is marked ready, please run cargo test -p nativelink-config -p nativelink-worker and report the results.

What looks correct

  • The WorkerKey change is right. instance_name is part of equality, and acquire clears it when isolation is off, so the pool reuses workers as before.
  • The spawn path uses the instance name only for logging and errors, so it does not reach the child process.
  • Both end-to-end tests run with PersistentWorkersSettings, which is enabled by default, so the "by default" test exercises the pool as intended.
  • The body is candid that this is instance isolation, not full tenant isolation.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Build results for this fork PR at 4b230ac9bd07:

Job Result Links
pull-request Bazel exited BUILD_FAILURE Open the build · check run

Fork PRs cannot stream build events, so a trusted workflow publishes them after CI finishes and updates this comment as each job is published. Opening a build needs a Trace Machina sign-in; the check run lists the failing tests and their logs.

@louisgregg

Copy link
Copy Markdown
Author

@MarcusSorealheis I've addressed the problems you found and I believe the PR is ready for another CI/CD run. I'm curious if the test suite will pass there. Summary of fixes below.

  1. Formatting issue - The cause is trailing whitespace after the closing brace of the PersistentWorkersSettings block in the new isolated-test config (}, ). Removing the trailing spaces should clear both jobs.

Fixed. I ran bazel run --config=rustfmt @rules_rust//:rustfmt and cargo fmt -- --check which passed without errors.

Failing Tests

  1. Please also run cargo test -p nativelink-config -p nativelink-worker after the fix and post the results, since the Rust tests have not run yet.

Done. On my machine, all new/relevant tests pass, but these tests fail:

  • pressure_kill_reaches_a_job_control_group
  • straggler_sweep_reaches_an_orphaned_job_control_group
    both of which are in nativelink-worker/tests/running_actions_manager_test.rs. These tests failed on both this branch and main so I suspect its unrelated to my code change.

Both tests failed because the job that was supposed to be killed, survived. The generated errors were similar:

thread 'tests::pressure_kill_reaches_a_job_control_group' (178801) panicked at nativelink-worker/tests/running_actions_manager_test.rs:11562:9:
the set -m job survived the kill

and

thread 'tests::straggler_sweep_reaches_an_orphaned_job_control_group' (178854) panicked at nativelink-worker/tests/running_actions_manager_test.rs:11586:9:
the orphaned set -m job survived the sweep

Some jobs aren't being killed as expected. Is this related to my change somehow? I'm not running nix - could this be an environment issue? I'm interested to see if the CI/CD pipeline passes or not on these tests in particular.

Other issues

  1. Blocking: The fix is off by default.

The fix will stay off by default but I've updated the test names, comments and text of the PR body itself to reflect this.

  1. The PR description does not match the tests.

Fixed.

  1. A misleading test comment

Fixed.

  1. Config reference not regenerated

Done.

  1. Formatting and unrelated churn.
  • Removed trailing comma and trailing whitespace.
  • Fixed typo.
  1. A misplaced doc line

Fixed/clarified the doc line.

  1. Simplicity
  • removed the redundant default_false function bc serde defaults bools to false anyway.
  • Code de-duplicated between the two tests in running_actions_manager_test.rs and placed in two helpfer functions used by the tests.
  1. Verification gaps

See Testing section above. All tests related to my code change are passing but older tests fail and I'm not sure of root cause yet.

…normie-github:louisgregg/nativelink into 2049-isolate-persistent-worker-by-instance-name
…closes negative numbers in backticks so the vale pre-commit hook passes.
@louisgregg

Copy link
Copy Markdown
Author

I've added fixes for both formatting issues that were causing the following CI/CD issues:

  • the nix flake check pre-commit hook failed because using a dash in a negative number is forbidden by vale style check on the generated documentation.
  • the Run project-pinned bazel tests step failed because a variable name (in CamelCase) wasn't correctly enclosed in backticks.

The pipeline should pass now, when re-run.

@MarcusSorealheis

Copy link
Copy Markdown
Member

Can you test this one out and show it to us in a demo? Your agent should be able to help you set that up easily.

@louisgregg

Copy link
Copy Markdown
Author

Live demo plan

Hi @MarcusSorealheis I'd be happy to set up a demo and present it. I'll try to keep the demo short - I'll let you know when it's ready and we can schedule a video call.

I'll try to include:

  • a POC of artifacts/secrets leaking between tenants on different instances via the persistent worker selected from the worker pool.
  • a demo of issue being prevented when isolate_by_instance_name is enabled.

CI/CD checks and blocked merging

cargo test -p nativelink-config -p nativelink-worker is passing, except for the two tests I mentioned in my previous comment - this happens on main on my machine too and I think these failures are unrelated to my change.

I fixed the formatting issues that were causing the Vale checks to fail, but these got fixed by other people on main anyway, so I just accepted their changes instead. The failing nix flake check and Run project-pinned bazel tests steps should pass when CI/CD is re-run and the PR should be merge without issues.

@MarcusSorealheis

Copy link
Copy Markdown
Member

Check out pull request #3230

@MarcusSorealheis

Copy link
Copy Markdown
Member

That's what I mean by demo

This branch had an error being deployed

1 failed (outdated) and 2 active deployments
Preview – nativelink — 87686638 Deployed Nov 20, 2026 by vercel[bot]
Preview – nativelink-aidm — 87686638 Deployed Nov 20, 2026 by vercel[bot]
public-ci — 9f4133ad Deployed Nov 16, 2026 by louisgregg via NativeLink / Linux (PR) #736
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote Persistent Workers Present Security Risk

3 participants