Skip to content

Worker occasionally dies with "UpdateForWorker stream closed early" #2115

Open

Description

@rejuvenile
  2026-01-21T20:58:46.749339Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 9c671a53-b7fc-4c2b-ba51-5639f1b064b5
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "9c671a53-b7fc-4c2b-ba51-5639f1b064b5", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:46.749389Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 6cac1188-bff7-4038-996f-4612bb138fe2
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "6cac1188-bff7-4038-996f-4612bb138fe2", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:59.487142Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
    at nativelink-worker/src/local_worker.rs:701
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


thread 'main' (24873374) panicked at src/bin/nativelink.rs:702:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }

Activity

  1. amankrx commented on Feb 15, 2026

    @amankrx
    Member

    We used to get this error in the previous release, I have noticed this rarely in the latest release, but with the right configuration, you can get away with this error. I will document these configurations this week. Please let us know if you're still this facing this error with the latest release and the new configurations.

  2. Ratnesh4378 commented on Jun 30, 2026

    @Ratnesh4378

    This error is still occurring very often. I am using the latest version , v1.5.2

  3. amankrx commented on Jun 30, 2026

    @amankrx
    Member

    Can you please share your config?

  4. Ratnesh4378 commented on Jun 30, 2026

    @Ratnesh4378

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: nativelink-configs
    namespace: nativelink
    data:
    cp.json: |
    {
    "stores": [
    {
    "name": "CAS_STORE",
    "filesystem": {
    "content_path": "/cas/cas_content",
    "temp_path": "/cas/cas_tmp",
    "eviction_policy": { "max_bytes": 240000000000 }
    }
    },
    {
    "name": "AC_STORE",
    "filesystem": {
    "content_path": "/cas/ac_content",
    "temp_path": "/cas/ac_tmp",
    "eviction_policy": { "max_bytes": 10000000000 }
    }
    }
    ],
    "schedulers": [
    {
    "name": "MAIN_SCHEDULER",
    "simple": {
    "supported_platform_properties": {
    "OSFamily": "exact",
    "cpu_count": "minimum"
    }
    }
    }
    ],
    "servers": [
    {
    "listener": { "http": { "socket_address": "0.0.0.0:50051" } },
    "services": {
    "cas": { "main": { "cas_store": "CAS_STORE" } },
    "ac": { "main": { "ac_store": "AC_STORE" } },
    "execution": { "main": { "scheduler": "MAIN_SCHEDULER" } },
    "bytestream": { "main": { "cas_store": "CAS_STORE" } },
    "capabilities": { "main": {} },
    "worker_api": { "scheduler": "MAIN_SCHEDULER" }
    }
    }
    ]
    }
    worker.json: |
    {
    "servers": [],
    "stores": [
    {
    "name": "REMOTE_CAS",
    "grpc": {
    "instance_name": "main",
    "endpoints": [{ "uri": "grpc://nl-cp.nativelink.svc.cluster.local:50051" }],
    "store_type": "cas"
    }
    },
    {
    "name": "REMOTE_AC",
    "grpc": {
    "instance_name": "main",
    "endpoints": [{ "uri": "grpc://nl-cp.nativelink.svc.cluster.local:50051" }],
    "store_type": "ac"
    }
    }
    ],
    "workers": [
    {
    "local": {
    "worker_api_endpoint": {
    "uri": "grpc://nl-cp.nativelink.svc.cluster.local:50051"
    },
    "cas_fast_slow_store": {
    "fast_slow": {
    "fast": {
    "filesystem": {
    "content_path": "/nl-work-dir/cas_content",
    "temp_path": "/nl-work-dir/cas_tmp",
    "eviction_policy": { "max_bytes": 20000000000 }
    }
    },
    "slow": { "ref_store": { "name": "REMOTE_CAS" } }
    }
    },
    "upload_action_result": {
    "ac_store": { "ref_store": { "name": "REMOTE_AC" } }
    },
    "work_directory": "/nl-work-dir/work",
    "platform_properties": {
    "OSFamily": "linux",
    "cpu_count": "4"
    }
    }
    }
    ]
    }

  5. Ratnesh4378 commented on Jul 1, 2026

    @Ratnesh4378

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: nativelink-configs
    namespace: nativelink
    data:
    cp.json: |
    {
    "stores": [
    {
    "name": "CAS_STORE",
    "filesystem": {
    "content_path": "/cas/cas_content",
    "temp_path": "/cas/cas_tmp",
    "eviction_policy": { "max_bytes": 240000000000 }
    }
    },
    {
    "name": "AC_STORE",
    "filesystem": {
    "content_path": "/cas/ac_content",
    "temp_path": "/cas/ac_tmp",
    "eviction_policy": { "max_bytes": 40000000000 }
    }
    }
    ],
    "schedulers": [
    {
    "name": "MAIN_SCHEDULER",
    "simple": {
    "supported_platform_properties": {
    "OSFamily": "exact",
    "cpu_count": "minimum",
    "label:action_large": "exact",
    "container-image": "exact"
    }
    }
    }
    ],
    "servers": [
    {
    "listener": { "http": { "socket_address": "0.0.0.0:50051" } },
    "services": {
    "cas": [{ "instance_name": "main", "cas_store": "CAS_STORE" }],
    "ac": [{ "instance_name": "main", "ac_store": "AC_STORE" }],
    "execution": [{ "instance_name": "main", "cas_store": "CAS_STORE","scheduler": "MAIN_SCHEDULER" }],
    "bytestream": [{ "instance_name": "main", "cas_store": "CAS_STORE" }],
    "worker_api":{
    "scheduler": "MAIN_SCHEDULER"
    },
    "capabilities": [
    {
    "instance_name": "main",
    "remote_execution": {
    "scheduler": "MAIN_SCHEDULER",
    },
    },
    ]
    }
    }
    ]
    }
    worker.json: |
    {
    "servers":[],
    "stores": [
    {
    "name": "REMOTE_CAS",
    "grpc": {
    "instance_name": "main",
    "endpoints": [{ "address": "grpc://:50051" }],
    "store_type": "cas"
    }
    },
    {
    "name": "REMOTE_AC",
    "grpc": {
    "instance_name": "main",
    "endpoints": [{ "address": "grpc://:50051" }],
    "store_type": "ac"
    }
    },
    {
    "name": "WORKER_LOCAL_CACHE",
    "filesystem": {
    "content_path": "/nl-work-dir/cas_content",
    "temp_path": "/nl-work-dir/cas_tmp",
    "eviction_policy": { "max_bytes": 20000000000 }
    }
    },
    {
    "name": "WORKER_FAST_SLOW",
    "fast_slow": {
    "fast": { "ref_store": {"name": "WORKER_LOCAL_CACHE" } },
    "slow": { "ref_store": {"name": "REMOTE_CAS" } }
    }
    }
    ],
    "workers": [
    {
    "local": {
    "worker_api_endpoint": {
    "uri": "grpc://:50051"
    },
    "cas_fast_slow_store": "WORKER_FAST_SLOW",
    "upload_action_result": {
    "ac_store": "REMOTE_AC"
    },
    "work_directory": "/nl-work-dir/work",
    "platform_properties": {
    "OSFamily": {
    "values": ["Linux"]
    },
    "container-image": {
    "values":[""]
    },
    "label:action_large": {
    "values":["1"]
    }
    }
    }
    }
    ]
    }

  6. Ratnesh4378 commented on Jul 1, 2026

    @Ratnesh4378

    This is the exact config I am using for control plane and worker nodes

  7. rajputdivyanshu81 commented on Nov 3, 2026

    @rajputdivyanshu81
    Contributor

    Hi @amankrx @rejuvenile @Ratnesh4378, I looked into this.

    The panic happened when LocalWorker::run returned Err after in-transit actions failed to drain within 10 s of the UpdateForWorker stream closing; main's try_join_all then panicked. #2658 (v1.6.4) made that non-fatal: the worker now logs the error, calls kill_all() and reconnects. #2643 (v1.6.6) added disconnect_with_action_in_transit_reconnects_test. v1.7.3 should also make the disconnects themselves less frequent (#2797 liveness, #2807 reconnect backoff with jitter, #2856 stalled-connection reconnect).

    Every report here is on v1.5.2 or older. @rejuvenile @Ratnesh4378, could you check whether the worker still exits on v1.7.3 or later (latest is v1.8.3)? You may still see Actions in transit did not reach zero… in the logs, but the process should stay up and reconnect. If nobody can reproduce the crash, I think this can be closed.

    Related: operate/troubleshooting, operate/runbooks, deploy-bare-metal and deploy-docker-compose still say worker reconnect is a flat 0.5 s retry with no backoff, which hasn't been true since #2807. I'd like to fix those in a small docs PR that references this issue. Since the source links are pinned to v1.6.5, should that PR wait for the SOURCE_REF bump, or go ahead now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions