Skip to content

[Core] Add accelerator backend hooks for multi-platform support - #534

Open
Meihan-chen wants to merge 3 commits into
vllm-project:mainfrom
Meihan-chen:feat/accelerator-backend-hooks
Open

Meihan-chen wants to merge 3 commits into
vllm-project:mainfrom
Meihan-chen:feat/accelerator-backend-hooks

Conversation

@Meihan-chen

@Meihan-chen Meihan-chen commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Ray scheduling and weight synchronization still make hardware-specific choices outside the existing Accelerator abstraction: they request Ray's GPU resource, read CUDA device UUIDs, and construct vLLM's IPC/NCCL transfer configuration directly.

This PR moves those choices behind the existing accelerator interface and updates their callers in training and serving.

Changes

  • Ray scheduling: add helpers for placement-group bundles, actor resource requests, and assigned device IDs. These use the backend's resource name; the default remains Ray's native GPU APIs.
  • Weight synchronization: obtain the device UUID, transport name, and trainer initialization-info factory from the backend. IPC and NCCL remain the default implementations. Factories are resolved when a transfer is initialized.
  • Process environments: route training-actor, serving-actor, and serving-subprocess environment construction through device_env. A backend can supply its own module; the default preserves the caller's environment.
  • Megatron initialization: add before_import() and after_args(args) hooks for setup that must run before Megatron imports or after argument initialization. Both are no-ops by default. The conversion tool uses the same import hook as training.

Backend descriptions are class-level methods. A driver with an explicitly configured backend can obtain its resource and transport settings without constructing a device runtime on the driver.

Validation

569 tests passed in an isolated CPU environment, including default and custom backend dispatch, drivers without local devices, and existing weight-transfer tests. Applicable pre-commit checks passed. GPU execution has not been validated for this change.

Describe Ray resources, child environments, Megatron startup and weight-transfer factories through the existing accelerator registry. Explicit backend descriptions work on a driver without local devices; automatic selection retains its runtime probes.

Extract the generic interfaces and tests from vllm-project#489 at da5defa. Vendor implementations remain a separate change.

Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Use generic resource, environment, device UUID and weight-transfer lookups at the existing call sites. Invoke the no-op-by-default Megatron lifecycle before imports and after argument initialization.

Validation: 50 focused CPU tests; broader CPU regressions and vLLM configuration contracts; applicable pre-commit checks. GPU execution remains to be validated.
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
@read-the-docs-community

read-the-docs-community Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request abstracts accelerator-specific configurations for Ray resource scheduling, Megatron lifecycle hooks, child process environments, and weight transfer backends, removing hardcoded GPU and NCCL assumptions. The feedback suggests caching the resolved backend type in _describing_backend() to avoid repeated execution of factory callables on deviceless hosts.

Comment on lines +316 to +351
def _describing_backend() -> type[Accelerator]:
"""The backend class whose description applies, without requiring a device.

A process that only builds scheduling requests needs the description of
the accelerator the job runs on, not a working local device, and may
legitimately have none of its own. Resolution order is the already
selected backend, the explicitly requested one, whatever this host can
actually offer, and finally the neutral defaults on :class:`Accelerator`,
which are what the GPU path already assumes.

Side effects by branch, because they are not uniform:

* An already selected backend, and the neutral default: none.
* An explicit request: the registered class is returned as a class. A
registration whose factory is not itself a class is called once to
learn the type, so such a factory must stay cheap.
* Falling through to auto-detection: this runs the ordinary selection,
which probes for devices, may import a vendor runtime, and installs
the process-wide backend. A caller that must not reach a runtime names
the backend it wants through VIME_ACCELERATOR.
"""
if _ACCELERATOR is not None:
return type(_ACCELERATOR)
_register_builtin_backends()
requested = _requested_name()
if requested is not None:
entry = _REGISTRY.get(requested)
if entry is None:
registered = ", ".join(sorted(_REGISTRY))
raise ValueError(f"Unknown accelerator {requested!r}; registered backends: {registered}")
factory = entry.factory
return factory if isinstance(factory, type) else type(factory())
selected = initialize_accelerator()
if selected is not None:
return type(selected)
return Accelerator

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Issue: Repeated Execution of Accelerator Factory Callables on Deviceless Hosts

In _describing_backend(), when an explicit accelerator is requested (via VIME_ACCELERATOR) but _ACCELERATOR is not yet initialized, the code resolves the backend class. If the registered factory is not a class (e.g., a lambda or function wrapper), it calls factory() to determine the type:

return factory if isinstance(factory, type) else type(factory())

Because _describing_backend() is called repeatedly across various utility functions (such as ray_device_key(), device_env_qualname(), megatron_lifecycle_qualname(), etc.), this results in repeatedly executing the factory callable.

If the factory function performs any non-trivial initialization, imports heavy vendor libraries, or attempts to probe/initialize hardware, calling it repeatedly can introduce significant performance overhead or raise unexpected errors on deviceless hosts (e.g., CPU-only orchestrators).

Solution:

Cache the resolved backend type using a robust cache keyed by (requested, factory). Keying by both the requested name and the factory object ensures perfect test isolation and safety against registry modifications/clearing between tests.

_REQUESTED_TYPE_CACHE: dict[tuple[str, Any], type[Accelerator]] = {}


def _describing_backend() -> type[Accelerator]:
    """The backend class whose description applies, without requiring a device.

    A process that only builds scheduling requests needs the description of
    the accelerator the job runs on, not a working local device, and may
    legitimately have none of its own. Resolution order is the already
    selected backend, the explicitly requested one, whatever this host can
    actually offer, and finally the neutral defaults on :class:`Accelerator`,
    which are what the GPU path already assumes.

    Side effects by branch, because they are not uniform:

    * An already selected backend, and the neutral default: none.
    * An explicit request: the registered class is returned as a class. A
      registration whose factory is not itself a class is called once to
      learn the type, so such a factory must stay cheap.
    * Falling through to auto-detection: this runs the ordinary selection,
      which probes for devices, may import a vendor runtime, and installs
      the process-wide backend. A caller that must not reach a runtime names
      the backend it wants through VIME_ACCELERATOR.
    """
    if _ACCELERATOR is not None:
        return type(_ACCELERATOR)
    _register_builtin_backends()
    requested = _requested_name()
    if requested is not None:
        entry = _REGISTRY.get(requested)
        if entry is None:
            registered = ", ".join(sorted(_REGISTRY))
            raise ValueError(f"Unknown accelerator {requested!r}; registered backends: {registered}")
        factory = entry.factory
        cache_key = (requested, factory)
        if cache_key not in _REQUESTED_TYPE_CACHE:
            _REQUESTED_TYPE_CACHE[cache_key] = factory if isinstance(factory, type) else type(factory())
        return _REQUESTED_TYPE_CACHE[cache_key]
    selected = initialize_accelerator()
    if selected is not None:
        return type(selected)
    return Accelerator

Reuse the existing accelerator fixture and fake backend instead of three separate test modules. Keep deviceless Ray scheduling, inherited environments, lifecycle defaults and lazy backend overrides; remove duplicate resource and transport assertions covered by the existing weight-transfer tests.

Validation: 569 CPU checks and applicable pre-commit hooks passed. Production code is unchanged.
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant