Skip to content

feat(py_info): support TreeArtifact venv entries via link_to_subpaths - #4396

Open
jvolkman wants to merge 6 commits into
bazel-contrib:mainfrom
jvolkman:refactor/action-install-prep
Open

jvolkman wants to merge 6 commits into
bazel-contrib:mainfrom
jvolkman:refactor/action-install-prep

Conversation

@jvolkman

@jvolkman jvolkman commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

Targets that install a Python package into a directory artifact
(TreeArtifact) during the build phase cannot populate individual File
objects in VenvSymlinkEntry.files when constructing virtual environment
symlinks. Without sub-path metadata, overlapping package directories
across multiple dependencies cannot be merged at file granularity.

Add an experimental VenvSymlinkEntry.link_to_subpaths field (and
builder support) so directory-backed entries can list their relative
file paths for conflict merging, and teach virtual environment link
generation to produce and merge TreeArtifact-backed entries.
Single-file entries whose files holds an ancestor directory of
link_to_path are now linked during conflict merging rather than
skipped.

Also prepare internal rule and tool helpers for build-action wheel
installation with no change in default behavior:

  • Support multi-file bootstrap interpreter programs by staging sibling
    Python modules into a private directory with bytecode writing
    disabled.
  • Extract pure in-memory helpers for wheel RECORD and console-script
    shebang rewriting so build-time installers can reuse them directly.

Targets that install a Python package into a directory artifact
(TreeArtifact) during the build phase cannot populate individual File
objects in VenvSymlinkEntry.files when constructing virtual environment
symlinks. Without sub-path metadata, overlapping package directories
across multiple dependencies cannot be merged at file granularity.

Add an experimental VenvSymlinkEntry.link_to_subpaths field (and
builder support) so directory-backed entries can list their relative
file paths for conflict merging, and teach virtual environment link
generation to produce and merge TreeArtifact-backed entries.
Single-file entries whose files holds an ancestor directory of
link_to_path are now linked during conflict merging rather than
skipped.

Also prepare internal rule and tool helpers for build-action wheel
installation with no change in default behavior:
* Support multi-file bootstrap interpreter programs by staging sibling
  Python modules into a private directory with bytecode writing
  disabled.
* Extract pure in-memory helpers for wheel RECORD and console-script
  shebang rewriting so build-time installers can reuse them directly.
@jvolkman

jvolkman commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

This PR was AI-assisted: the code was written by an AI coding agent under my direction and went through several rounds of agent-driven review, including checks that default-mode behavior is unchanged. I've reviewed the final change. I'll handle review comments myself; any agent-drafted replies or follow-up changes will be reviewed by me before they're posted or pushed.

@jvolkman

jvolkman commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

PTAL

@rickeylev

Copy link
Copy Markdown
Collaborator

@rsartor-cmd FYI some work Jeremy is doing related to sdist stuff

Comment thread python/private/py_info.bzl Resolved

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgive me if my questions seem dumb. Its bit awhile since I was in venv_runfiles. I think I was following along, but I'm not entirely sure. The main thing I wasn't sure about is it seemed like the lists of File | str-path-within-directory are now being mixed, so i'm having a hard time following the "relativeness" of objects.

Comment thread python/private/venv_runfiles.bzl Resolved
Comment thread python/private/py_info.bzl Resolved
Comment thread python/private/py_info.bzl Resolved
Comment thread python/private/venv_runfiles.bzl Resolved
Comment thread python/private/venv_runfiles.bzl Resolved
Comment thread python/private/venv_runfiles.bzl Resolved
Comment thread python/private/venv_runfiles.bzl Resolved
Drop the py_interpreter_program staging changes so it stays a thin
bootstrap shim, normalize get_venv_symlinks inputs into typed candidate
structs before grouping, clarify VenvSymlinkEntry and merge comments,
fix MyST directive nesting, and rename the changelog fragment to 4396.
@jvolkman

jvolkman commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

PTAL

@aignas aignas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a few questions.

Comment thread python/private/venv_runfiles.bzl Resolved
Comment thread python/private/venv_runfiles.bzl Resolved
Simplify the `files` expression on shared-library `VenvSymlinkEntry`
records in `get_venv_symlinks`, document `tree_venv_lib` and
`_assert_tree_artifact_equivalence`, and add
`tree_shared_lib_loading_test` to verify loading compiled C extensions
and transitive shared libraries from a `TreeArtifact`-backed venv at
runtime.
Switch `tree_venv_merge_test` and `tree_shared_lib_loading_test` from
`unittest.TestCase` + `py_reconfig_test` to `pytest_test` with plain
`pytest` test functions and assertions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants