Repository navigation
[Core] Add accelerator backend hooks for multi-platform support - #534
Meihan-chen wants to merge 3 commits into
Conversation
Describe Ray resources, child environments, Megatron startup and weight-transfer factories through the existing accelerator registry. Explicit backend descriptions work on a driver without local devices; automatic selection retains its runtime probes. Extract the generic interfaces and tests from vllm-project#489 at da5defa. Vendor implementations remain a separate change. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Use generic resource, environment, device UUID and weight-transfer lookups at the existing call sites. Invoke the no-op-by-default Megatron lifecycle before imports and after argument initialization. Validation: 50 focused CPU tests; broader CPU regressions and vLLM configuration contracts; applicable pre-commit checks. GPU execution remains to be validated. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
There was a problem hiding this comment.
Code Review
This pull request abstracts accelerator-specific configurations for Ray resource scheduling, Megatron lifecycle hooks, child process environments, and weight transfer backends, removing hardcoded GPU and NCCL assumptions. The feedback suggests caching the resolved backend type in _describing_backend() to avoid repeated execution of factory callables on deviceless hosts.
| def _describing_backend() -> type[Accelerator]: | ||
| """The backend class whose description applies, without requiring a device. | ||
|
|
||
| A process that only builds scheduling requests needs the description of | ||
| the accelerator the job runs on, not a working local device, and may | ||
| legitimately have none of its own. Resolution order is the already | ||
| selected backend, the explicitly requested one, whatever this host can | ||
| actually offer, and finally the neutral defaults on :class:`Accelerator`, | ||
| which are what the GPU path already assumes. | ||
|
|
||
| Side effects by branch, because they are not uniform: | ||
|
|
||
| * An already selected backend, and the neutral default: none. | ||
| * An explicit request: the registered class is returned as a class. A | ||
| registration whose factory is not itself a class is called once to | ||
| learn the type, so such a factory must stay cheap. | ||
| * Falling through to auto-detection: this runs the ordinary selection, | ||
| which probes for devices, may import a vendor runtime, and installs | ||
| the process-wide backend. A caller that must not reach a runtime names | ||
| the backend it wants through VIME_ACCELERATOR. | ||
| """ | ||
| if _ACCELERATOR is not None: | ||
| return type(_ACCELERATOR) | ||
| _register_builtin_backends() | ||
| requested = _requested_name() | ||
| if requested is not None: | ||
| entry = _REGISTRY.get(requested) | ||
| if entry is None: | ||
| registered = ", ".join(sorted(_REGISTRY)) | ||
| raise ValueError(f"Unknown accelerator {requested!r}; registered backends: {registered}") | ||
| factory = entry.factory | ||
| return factory if isinstance(factory, type) else type(factory()) | ||
| selected = initialize_accelerator() | ||
| if selected is not None: | ||
| return type(selected) | ||
| return Accelerator |
There was a problem hiding this comment.
Issue: Repeated Execution of Accelerator Factory Callables on Deviceless Hosts
In _describing_backend(), when an explicit accelerator is requested (via VIME_ACCELERATOR) but _ACCELERATOR is not yet initialized, the code resolves the backend class. If the registered factory is not a class (e.g., a lambda or function wrapper), it calls factory() to determine the type:
return factory if isinstance(factory, type) else type(factory())Because _describing_backend() is called repeatedly across various utility functions (such as ray_device_key(), device_env_qualname(), megatron_lifecycle_qualname(), etc.), this results in repeatedly executing the factory callable.
If the factory function performs any non-trivial initialization, imports heavy vendor libraries, or attempts to probe/initialize hardware, calling it repeatedly can introduce significant performance overhead or raise unexpected errors on deviceless hosts (e.g., CPU-only orchestrators).
Solution:
Cache the resolved backend type using a robust cache keyed by (requested, factory). Keying by both the requested name and the factory object ensures perfect test isolation and safety against registry modifications/clearing between tests.
_REQUESTED_TYPE_CACHE: dict[tuple[str, Any], type[Accelerator]] = {}
def _describing_backend() -> type[Accelerator]:
"""The backend class whose description applies, without requiring a device.
A process that only builds scheduling requests needs the description of
the accelerator the job runs on, not a working local device, and may
legitimately have none of its own. Resolution order is the already
selected backend, the explicitly requested one, whatever this host can
actually offer, and finally the neutral defaults on :class:`Accelerator`,
which are what the GPU path already assumes.
Side effects by branch, because they are not uniform:
* An already selected backend, and the neutral default: none.
* An explicit request: the registered class is returned as a class. A
registration whose factory is not itself a class is called once to
learn the type, so such a factory must stay cheap.
* Falling through to auto-detection: this runs the ordinary selection,
which probes for devices, may import a vendor runtime, and installs
the process-wide backend. A caller that must not reach a runtime names
the backend it wants through VIME_ACCELERATOR.
"""
if _ACCELERATOR is not None:
return type(_ACCELERATOR)
_register_builtin_backends()
requested = _requested_name()
if requested is not None:
entry = _REGISTRY.get(requested)
if entry is None:
registered = ", ".join(sorted(_REGISTRY))
raise ValueError(f"Unknown accelerator {requested!r}; registered backends: {registered}")
factory = entry.factory
cache_key = (requested, factory)
if cache_key not in _REQUESTED_TYPE_CACHE:
_REQUESTED_TYPE_CACHE[cache_key] = factory if isinstance(factory, type) else type(factory())
return _REQUESTED_TYPE_CACHE[cache_key]
selected = initialize_accelerator()
if selected is not None:
return type(selected)
return AcceleratorReuse the existing accelerator fixture and fake backend instead of three separate test modules. Keep deviceless Ray scheduling, inherited environments, lifecycle defaults and lazy backend overrides; remove duplicate resource and transport assertions covered by the existing weight-transfer tests. Validation: 569 CPU checks and applicable pre-commit hooks passed. Production code is unchanged. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Ray scheduling and weight synchronization still make hardware-specific choices outside the existing
Acceleratorabstraction: they request Ray'sGPUresource, read CUDA device UUIDs, and construct vLLM's IPC/NCCL transfer configuration directly.This PR moves those choices behind the existing accelerator interface and updates their callers in training and serving.
Changes
device_env. A backend can supply its own module; the default preserves the caller's environment.before_import()andafter_args(args)hooks for setup that must run before Megatron imports or after argument initialization. Both are no-ops by default. The conversion tool uses the same import hook as training.Backend descriptions are class-level methods. A driver with an explicitly configured backend can obtain its resource and transport settings without constructing a device runtime on the driver.
Validation
569 tests passed in an isolated CPU environment, including default and custom backend dispatch, drivers without local devices, and existing weight-transfer tests. Applicable pre-commit checks passed. GPU execution has not been validated for this change.