Repository navigation
Isolate persistent worker by Action instance name. A fix for issue #2049 - #3142
louisgregg wants to merge 23 commits into
Conversation
… using PoolConfig.isolate_by_instance_name flag
…n test to ensure RunningActionsManagerImpl isolates Actions by instance_name when enabled in PoolConfig
…normie-github:louisgregg/nativelink into 2049-isolate-persistent-worker-by-instance-name
…normie-github:louisgregg/nativelink into 2049-isolate-persistent-worker-by-instance-name
…normie-github:louisgregg/nativelink into 2049-isolate-persistent-worker-by-instance-name
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@MarcusSorealheis @modernmedici this PR is ready for review when you get a chance. Thanks. |
|
Hi @louisgregg interesting approach. I have many thoughts. My first, though is should this isolation be off by default: |
|
@MarcusSorealheis thanks for the feedback. I'm not sure how to quantify the performance impact. The overall risk should be the product of the number of multi-tenant-but-single-instance nativelink users times the average slowdown-per-action each of them will experience. Who might know how to estimate this? Nativelink user statistics may be necessary here, which I probably don't have access to. |
MarcusSorealheis
left a comment
There was a problem hiding this comment.
Review: Isolate persistent workers by Action instance name
I read the full diff and checked the default against PersistentWorkersSettings in nativelink-worker/src/running_actions_manager.rs. I did not check out the branch or run the tests, and no Rust, Bazel, or NativeLink test jobs have run on this PR yet.
Blocking
The fix is off by default. isolate_by_instance_name defaults to false in nativelink-config/src/cas_server.rs (default_false) and in PoolConfig::default(). Multi-tenant deployments that use instance_name as their boundary, the case issue #2049 describes, stay exposed unless they opt in. The PR body contradicts this: it names isolate_by_instance_name_defaults_to_true and says to set the flag to false if the performance cost is severe, which only makes sense if the default is true. Either the default should be true, with a migration note for the lower hit rate, or the body should say the flag is opt-in and explain why the leak stays open by default.
Other issues
- The PR description does not match the tests. The body lists
isolate_by_instance_name_defaults_to_true,persistent_worker_isolates_actions_by_instance_name, andpersistent_worker_does_not_isolate_actions_by_instance_name_when_disabled. The diff has..._defaults_to_false,..._when_enabled, and..._by_default. The body also saysPoolConfig.isolation_by_instance_name, but the field isisolate_by_instance_name. - A misleading test comment. In
pool_uses_different_workers_for_different_instance_names_when_isolation_enabled(nativelink-worker/src/persistent_worker/pool.rs), the comment before the second acquire says "isolate_by_instance_name is false / So the same worker should be reused", but the test enables isolation and asserts two workers. It looks copy-pasted from the test above it. - Config reference not regenerated. The new field has a doc comment, but
web/apps/docs/content/docs/reference/nativelink-config/*is not in the diff. AGENTS.md requiresgen:config-reference, not hand edits. - Formatting and unrelated churn. The trailing comma added in
worker_key_rejects_empty_argvis unrelated to this change.running_actions_manager_test.rshas trailing whitespace after},in the new isolated-test config block.nativelink-config/tests/deserialization_test.rshas the typo "don't isolation by instance name", and thePoolConfigdoc says "will be not be included". - A misplaced doc line.
pool.rsinserts/// Name of instance group included because it's not a secret.into the doc block onimpl fmt::Debug for WorkerKey, where it reads as part of the existing "environment can carry a token" note. - Simplicity.
default_falseis redundant, since#[serde(default)]on aboolalready defaults tofalse. The two new tests inrunning_actions_manager_test.rsduplicate about 200 lines of setup; a shared helper that takes the flag would be simpler. - Verification gaps. The status rollup shows only the PR template check, the CLA, and Vercel. The body lists test names but no commands or results, and does not say which checks were skipped, which AGENTS.md requires. Before this is marked ready, please run
cargo test -p nativelink-config -p nativelink-workerand report the results.
What looks correct
- The
WorkerKeychange is right.instance_nameis part of equality, andacquireclears it when isolation is off, so the pool reuses workers as before. - The spawn path uses the instance name only for logging and errors, so it does not reach the child process.
- Both end-to-end tests run with
PersistentWorkersSettings, which is enabled by default, so the "by default" test exercises the pool as intended. - The body is candid that this is instance isolation, not full tenant isolation.
|
Build results for this fork PR at
Fork PRs cannot stream build events, so a trusted workflow publishes them after CI finishes and updates this comment as each job is published. Opening a build needs a Trace Machina sign-in; the check run lists the failing tests and their logs. |
|
@MarcusSorealheis I've addressed the problems you found and I believe the PR is ready for another CI/CD run. I'm curious if the test suite will pass there. Summary of fixes below.
Fixed. I ran Failing Tests
Done. On my machine, all new/relevant tests pass, but these tests fail:
Both tests failed because the job that was supposed to be killed, survived. The generated errors were similar: and Some jobs aren't being killed as expected. Is this related to my change somehow? I'm not running nix - could this be an environment issue? I'm interested to see if the CI/CD pipeline passes or not on these tests in particular. Other issues
The fix will stay off by default but I've updated the test names, comments and text of the PR body itself to reflect this.
Fixed.
Fixed.
Done.
Fixed/clarified the doc line.
See Testing section above. All tests related to my code change are passing but older tests fail and I'm not sure of root cause yet. |
…normie-github:louisgregg/nativelink into 2049-isolate-persistent-worker-by-instance-name
…closes negative numbers in backticks so the vale pre-commit hook passes.
|
I've added fixes for both formatting issues that were causing the following CI/CD issues:
The pipeline should pass now, when re-run. |
|
Can you test this one out and show it to us in a demo? Your agent should be able to help you set that up easily. |
…rker-by-instance-name
Live demo planHi @MarcusSorealheis I'd be happy to set up a demo and present it. I'll try to keep the demo short - I'll let you know when it's ready and we can schedule a video call. I'll try to include:
CI/CD checks and blocked merging
I fixed the formatting issues that were causing the Vale checks to fail, but these got fixed by other people on |
|
Check out pull request #3230 |
|
That's what I mean by demo |
What and why
Fixes #2049
This PR adds the ability to prevent persistent workers from sharing jobs if the don't share the same
ActionInfo.instance_name()by including theinstance_namein theWorkerKey. It prevents anActionoriginating on different instance from re-using the same idle worker as a previous Action even if the Actions appear to be otherwise identical. In multi-tenant deployments which useinstance_nameto define the boundary between tenants, this PR should preserve that boundary between workers assigned to jobs.However, if a configuration puts multiple tenants in under the same instance, then this additional filter in WorkerKey will be insufficient to prevent information leakage between tenant jobs. This PR provides instance isolation, not total tenant isolation.
Updates to PR & default behavior
Following some feedback on the PR, this feature will be disabled by default. This was due to the potential performance impact isolation will cause for multi-tenant instances. It can be enabled by setting
PoolConfig.isolate_by_instance_nametotrue.Current configuration and information leakage / build poisoning risks
Currently, two
Actions having the sameWorkerKeymay inherit a Worker from one another via the pool. The WorkerKey is defined by:Consider two actions, Action 1 and Action 2, which are executed in sequence.
WorkerKeymatches that of Action 1. Therefore it is assigned Worker A.Worker A now executes Action 2 with access to the following:
working_dirand its contents. This consists of anything the worker wrote outside of the per-actionsandbox_dir(e.g. scratch files, caches, downloaded dependencies) during Action 1.Note: Content-keyed caches generated during Action 1 will be available during Action 2's execution too. This could lead to build poisoning if Action 2 uses Action 1's cached output as input.
How was this verified?
I did not run this against a real instance of NativeLink. However, unit and integration tests were added for the following cases.
PersistentWorkersConfigdeserializes from json correctly.isolate_by_instance_name_defaults_to_false()isolate_by_instance_name_deserializes_correctly_when_set()WorkerKeyworks as intended.worker_key_distinguishes_different_instance_names()PoolConfig.isolation_by_instance_nameflag.pool_shares_workers_for_different_instance_names_when_isolation_disabled()pool_uses_different_workers_for_different_instance_names_when_isolation_enabled()RunningActionsManagerImplto check Action/Worker assignment respects theisolation_by_instance_nameflag, when it is set.persistent_worker_isolates_actions_by_instance_name_when_enabled()persistent_worker_does_not_isolate_actions_by_instance_name_by_default()Risk
For a customer under a single trust domain using many
instance_names, the worker pool hit-rate will drop, increasing the build time of otherwise identical actions and reducing overall performance. If the performance impact is severe, thePoolConfig.isolate_by_instance_nameflag should be set to false.AI assistance
I wrote all code in this PR myself, accepting some good ideas from the LLM and rejecting others. I used Claude Code as follows: