Skip to content

Add external store provider hooks - #3221

Draft
MarcusSorealheis wants to merge 1 commit into
mainfrom
feature/external-store-providers
Draft

MarcusSorealheis wants to merge 1 commit into
mainfrom
feature/external-store-providers

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Note

No version change. This PR deliberately bumps nothing: every crate stays at its current version. A release PR will be stacked on top of this branch when the next release is cut.

What and why

A distribution that links additional store implementations into the server has no way to register them today short of forking the store factory. This adds that seam in two logic-free pieces: an external store spec (provider name plus a passthrough config value), and a provider registry in nativelink-store that store_factory consults. The library split from #3150 already lets a distribution embed the server, so no changes under src/ are needed: an embedding binary registers its providers and then builds the stock server.

The stock binary registers no providers: an external spec there fails at startup naming the unknown provider and the (empty) registered set, so a config/binary mismatch is diagnosable from the first lines of output, and every successful external-store construction is logged with its provider name.

How was this verified?

New regression tests, both exercising new surface (they do not compile without the change):

  • nativelink-config/tests/external_spec_test.rs: the spec parses with and without config (defaulting to an empty object), rejects unknown fields, and requires provider.
  • nativelink-store/tests/external_store_test.rs: a registered test provider builds a working store through store_factory and round-trips data through the full Store API; an unregistered provider fails with InvalidArgument naming the provider and listing what is registered; duplicate registration is refused with AlreadyExists.

Beyond that, locally on this branch (rebased onto current main): cargo test across nativelink-config and nativelink-store passes (39 suites, 559 tests, 0 failures); a full cargo test --workspace passes; under Bazel, //nativelink-config/... and the new store test target pass with the rustfmt and clippy aspects; cargo clippy is clean for the touched crates. The main config reference page was regenerated (gen:config-reference main) and the docs lint:snippets and lint:anchors gates pass. Not verified locally: a real deployment; no cluster ran this build.

Risk

The external spec is new, opt-in config surface and inert unless a config names it: with no provider registered (the stock binary), naming it is a clean startup failure. Stores nested inside an external provider's config are opaque to config-wide processing such as grpc store adjustments; the spec documents this. There are no version, CLI, wire-protocol, or src/ changes.

AI assistance

Generated with Claude Code under human steering.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Nov 20, 2026 5:03pm UTC
nativelink-aidm Ready Ready Preview Nov 20, 2026 5:03pm UTC

Request Review

@MarcusSorealheis
MarcusSorealheis force-pushed the feature/external-store-providers branch from 0da41ec to eb82780 Compare November 18, 2026 17:27
@MarcusSorealheis MarcusSorealheis changed the title Add external store provider hooks and a server library entry point Add external store provider hooks Nov 18, 2026
@MarcusSorealheis
MarcusSorealheis force-pushed the feature/external-store-providers branch from eb82780 to 642e318 Compare November 19, 2026 17:07
@MarcusSorealheis
MarcusSorealheis force-pushed the feature/external-store-providers branch from 642e318 to a6fcd18 Compare November 19, 2026 21:49
Lets a distribution register additional store implementations without
carrying them in this tree. An `external` store spec selects a provider
by name with a passthrough config value, and the store factory builds it
through a registry the embedding binary populates before startup (the
library split in #3150 is what makes such a binary possible). The stock
binary registers no providers: an `external` spec there fails at startup
naming the unknown provider and the registered (empty) set. Regenerates
the main config reference.

This branch was successfully deployed

2 active and 2 inactive (1 outdated) deployments
Preview – nativelink — 58c0dc24 Deployed Nov 20, 2026 by vercel[bot]
Preview – nativelink-aidm — 58c0dc24 Deployed Nov 20, 2026 by vercel[bot]
cluster-ci-aws — 58c0dc24 Deployed Nov 20, 2026 by MarcusSorealheis via NativeLink / Linux #991
cluster-ci-benchmark — f053407b Deployed Nov 19, 2026 by MarcusSorealheis via NativeLink / Linux #927
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant