You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Worker occasionally dies with "UpdateForWorker stream closed early" #2115
2026-01-21T20:58:46.749339Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 9c671a53-b7fc-4c2b-ba51-5639f1b064b5
at nativelink-worker/src/running_actions_manager.rs:1415
in nativelink_worker::local_worker::worker_start_action
in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "9c671a53-b7fc-4c2b-ba51-5639f1b064b5", digest_function: BLAKE3
in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
in nativelink::worker with name: "worker_0"
2026-01-21T20:58:46.749389Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 6cac1188-bff7-4038-996f-4612bb138fe2
at nativelink-worker/src/running_actions_manager.rs:1415
in nativelink_worker::local_worker::worker_start_action
in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "6cac1188-bff7-4038-996f-4612bb138fe2", digest_function: BLAKE3
in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
in nativelink::worker with name: "worker_0"
2026-01-21T20:58:59.487142Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
at nativelink-worker/src/local_worker.rs:701
in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
in nativelink::worker with name: "worker_0"
thread 'main' (24873374) panicked at src/bin/nativelink.rs:702:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }
We used to get this error in the previous release, I have noticed this rarely in the latest release, but with the right configuration, you can get away with this error. I will document these configurations this week. Please let us know if you're still this facing this error with the latest release and the new configurations.
The panic happened when LocalWorker::run returned Err after in-transit actions failed to drain within 10 s of the UpdateForWorker stream closing; main's try_join_all then panicked. #2658 (v1.6.4) made that non-fatal: the worker now logs the error, calls kill_all() and reconnects. #2643 (v1.6.6) added disconnect_with_action_in_transit_reconnects_test. v1.7.3 should also make the disconnects themselves less frequent (#2797 liveness, #2807 reconnect backoff with jitter, #2856 stalled-connection reconnect).
Every report here is on v1.5.2 or older. @rejuvenile@Ratnesh4378, could you check whether the worker still exits on v1.7.3 or later (latest is v1.8.3)? You may still see Actions in transit did not reach zero… in the logs, but the process should stay up and reconnect. If nobody can reproduce the crash, I think this can be closed.
Related: operate/troubleshooting, operate/runbooks, deploy-bare-metal and deploy-docker-compose still say worker reconnect is a flat 0.5 s retry with no backoff, which hasn't been true since #2807. I'd like to fix those in a small docs PR that references this issue. Since the source links are pinned to v1.6.5, should that PR wait for the SOURCE_REF bump, or go ahead now?
Activity
amankrx commentedon Feb 15, 2026
We used to get this error in the previous release, I have noticed this rarely in the latest release, but with the right configuration, you can get away with this error. I will document these configurations this week. Please let us know if you're still this facing this error with the latest release and the new configurations.
Ratnesh4378 commentedon Jun 30, 2026
This error is still occurring very often. I am using the latest version , v1.5.2
amankrx commentedon Jun 30, 2026
Can you please share your config?
Ratnesh4378 commentedon Jun 30, 2026
apiVersion: v1
kind: ConfigMap
metadata:
name: nativelink-configs
namespace: nativelink
data:
cp.json: |
{
"stores": [
{
"name": "CAS_STORE",
"filesystem": {
"content_path": "/cas/cas_content",
"temp_path": "/cas/cas_tmp",
"eviction_policy": { "max_bytes": 240000000000 }
}
},
{
"name": "AC_STORE",
"filesystem": {
"content_path": "/cas/ac_content",
"temp_path": "/cas/ac_tmp",
"eviction_policy": { "max_bytes": 10000000000 }
}
}
],
"schedulers": [
{
"name": "MAIN_SCHEDULER",
"simple": {
"supported_platform_properties": {
"OSFamily": "exact",
"cpu_count": "minimum"
}
}
}
],
"servers": [
{
"listener": { "http": { "socket_address": "0.0.0.0:50051" } },
"services": {
"cas": { "main": { "cas_store": "CAS_STORE" } },
"ac": { "main": { "ac_store": "AC_STORE" } },
"execution": { "main": { "scheduler": "MAIN_SCHEDULER" } },
"bytestream": { "main": { "cas_store": "CAS_STORE" } },
"capabilities": { "main": {} },
"worker_api": { "scheduler": "MAIN_SCHEDULER" }
}
}
]
}
worker.json: |
{
"servers": [],
"stores": [
{
"name": "REMOTE_CAS",
"grpc": {
"instance_name": "main",
"endpoints": [{ "uri": "grpc://nl-cp.nativelink.svc.cluster.local:50051" }],
"store_type": "cas"
}
},
{
"name": "REMOTE_AC",
"grpc": {
"instance_name": "main",
"endpoints": [{ "uri": "grpc://nl-cp.nativelink.svc.cluster.local:50051" }],
"store_type": "ac"
}
}
],
"workers": [
{
"local": {
"worker_api_endpoint": {
"uri": "grpc://nl-cp.nativelink.svc.cluster.local:50051"
},
"cas_fast_slow_store": {
"fast_slow": {
"fast": {
"filesystem": {
"content_path": "/nl-work-dir/cas_content",
"temp_path": "/nl-work-dir/cas_tmp",
"eviction_policy": { "max_bytes": 20000000000 }
}
},
"slow": { "ref_store": { "name": "REMOTE_CAS" } }
}
},
"upload_action_result": {
"ac_store": { "ref_store": { "name": "REMOTE_AC" } }
},
"work_directory": "/nl-work-dir/work",
"platform_properties": {
"OSFamily": "linux",
"cpu_count": "4"
}
}
}
]
}
Ratnesh4378 commentedon Jul 1, 2026
apiVersion: v1
kind: ConfigMap
metadata:
name: nativelink-configs
namespace: nativelink
data:
cp.json: |
{
"stores": [
{
"name": "CAS_STORE",
"filesystem": {
"content_path": "/cas/cas_content",
"temp_path": "/cas/cas_tmp",
"eviction_policy": { "max_bytes": 240000000000 }
}
},
{
"name": "AC_STORE",
"filesystem": {
"content_path": "/cas/ac_content",
"temp_path": "/cas/ac_tmp",
"eviction_policy": { "max_bytes": 40000000000 }
}
}
],
"schedulers": [
{
"name": "MAIN_SCHEDULER",
"simple": {
"supported_platform_properties": {
"OSFamily": "exact",
"cpu_count": "minimum",
"label:action_large": "exact",
"container-image": "exact"
}
}
}
],
"servers": [
{
"listener": { "http": { "socket_address": "0.0.0.0:50051" } },
"services": {
"cas": [{ "instance_name": "main", "cas_store": "CAS_STORE" }],
"ac": [{ "instance_name": "main", "ac_store": "AC_STORE" }],
"execution": [{ "instance_name": "main", "cas_store": "CAS_STORE","scheduler": "MAIN_SCHEDULER" }],
"bytestream": [{ "instance_name": "main", "cas_store": "CAS_STORE" }],
"worker_api":{
"scheduler": "MAIN_SCHEDULER"
},
"capabilities": [
{
"instance_name": "main",
"remote_execution": {
"scheduler": "MAIN_SCHEDULER",
},
},
]
}
}
]
}
worker.json: |
{
"servers":[],
"stores": [
{
"name": "REMOTE_CAS",
"grpc": {
"instance_name": "main",
"endpoints": [{ "address": "grpc://:50051" }],
"store_type": "cas"
}
},
{
"name": "REMOTE_AC",
"grpc": {
"instance_name": "main",
"endpoints": [{ "address": "grpc://:50051" }],
"store_type": "ac"
}
},
{
"name": "WORKER_LOCAL_CACHE",
"filesystem": {
"content_path": "/nl-work-dir/cas_content",
"temp_path": "/nl-work-dir/cas_tmp",
"eviction_policy": { "max_bytes": 20000000000 }
}
},
{
"name": "WORKER_FAST_SLOW",
"fast_slow": {
"fast": { "ref_store": {"name": "WORKER_LOCAL_CACHE" } },
"slow": { "ref_store": {"name": "REMOTE_CAS" } }
}
}
],
"workers": [
{
"local": {
"worker_api_endpoint": {
"uri": "grpc://:50051"
},
"cas_fast_slow_store": "WORKER_FAST_SLOW",
"upload_action_result": {
"ac_store": "REMOTE_AC"
},
"work_directory": "/nl-work-dir/work",
"platform_properties": {
"OSFamily": {
"values": ["Linux"]
},
"container-image": {
"values":[""]
},
"label:action_large": {
"values":["1"]
}
}
}
}
]
}
Ratnesh4378 commentedon Jul 1, 2026
This is the exact config I am using for control plane and worker nodes
rajputdivyanshu81 commentedon Nov 3, 2026
Hi @amankrx @rejuvenile @Ratnesh4378, I looked into this.
The panic happened when LocalWorker::run returned Err after in-transit actions failed to drain within 10 s of the UpdateForWorker stream closing; main's try_join_all then panicked. #2658 (v1.6.4) made that non-fatal: the worker now logs the error, calls kill_all() and reconnects. #2643 (v1.6.6) added disconnect_with_action_in_transit_reconnects_test. v1.7.3 should also make the disconnects themselves less frequent (#2797 liveness, #2807 reconnect backoff with jitter, #2856 stalled-connection reconnect).
Every report here is on v1.5.2 or older. @rejuvenile @Ratnesh4378, could you check whether the worker still exits on v1.7.3 or later (latest is v1.8.3)? You may still see Actions in transit did not reach zero… in the logs, but the process should stay up and reconnect. If nobody can reproduce the crash, I think this can be closed.
Related: operate/troubleshooting, operate/runbooks, deploy-bare-metal and deploy-docker-compose still say worker reconnect is a flat 0.5 s retry with no backoff, which hasn't been true since #2807. I'd like to fix those in a small docs PR that references this issue. Since the source links are pinned to v1.6.5, should that PR wait for the SOURCE_REF bump, or go ahead now?