Skip to content

Setup CI to publish a python package to PyPi #5

Closed

Description

@machshev

DVSim should be installable as any other Python package using pip install. While it is possible to reference a git repository directly, in general it's more convenient to publish packages to PyPi.

Activity

  1. added theissue type on Jul 23, 2025
  2. changed the title [-]Publish a python package to PyPi[/-] [+]Setup CI to publish a python package to PyPi[/+] on Jul 23, 2025
  3. pinned this issue on Jul 23, 2025
  4. self-assigned this
    on Aug 22, 2025
  5. machshev commented on Oct 2, 2025

    @machshev
    CollaboratorAuthor

    The issue we have at the moment is #16 doesn't work on master due to the branch protection rules. It seems adding a commit to master to update the version in pyproject.toml is not easy. The other option would be to move to dynamic version in the pyproject.toml and establish the package version from the release tags (which is the source of truth for python-semantic-release anyway).

    Proposal: drop the requirement for the commit to master and just use tags as the single source of truth for package version.

    [project]
    dynamic = ["version"]
    

    https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#static-vs-dynamic-metadata

  6. rswarbrick commented on Oct 3, 2025

    @rswarbrick
    Contributor

    My proposal:

    Stick something in the "merge my changes" process (I don't know how GitHub names this) so that:

    • It works through the commits since master (a bit like git rebase -i master..HEAD) and calls python-semantic-release version --no-commit and then git commit --amend each time.
    • Once that has run, we'll have fudged things so that the version in pyproject.toml always matches the logical version (not just on PR heads)
    • At this point, the repository (which I assume is the thing that gets merged to the "upstream repo") is a fast-forward from origin/master and has been updated correctly.

    Wouldn't that work?

  7. rswarbrick commented on Oct 3, 2025

    @rswarbrick
    Contributor

    About making releases on GitHub, you could then presumably get it to automatically create a GitHub release for the version that we end up with at the head of each PR

  8. rswarbrick commented on Oct 3, 2025

    @rswarbrick
    Contributor

    @KinzaQamar and I have just spent some more time (and she has taught me a bit about GitHub workflows!)

    New plan:

    • Add a pre-commit hook script somewhere in the repository that basically runs semantic-release --no-commit; git add pyproject.toml
    • Add some documentation that tells contributors to symlink .git/hooks/pre_commit to it.
    • Add a check in CI to make sure the contributor remembered to run it! This could use git rebase -i -exec ??? origin/master where ??? is something like semantic-release --no-commit; git status --porcelain | (check this is empty).

    I think this will work nicely! (Probably...)

  9. machshev commented on Oct 3, 2025

    @machshev
    CollaboratorAuthor

    @rswarbrick It's a nice idea, but not sure it will work? That only checks that the PR is correct at the time of submission. If there are several PRs going in parallel they will all have the same version (which is calculated relative to the version tags in the repo).

    At the point the first PR is merged then the version may be correct? (probably not if any other PR has been merged after it's creation). But all the following PRs will now be incorrect, even if the CI has passed.

    At best this means versions are missed, at worst it means versions could actually go backwards.
    In either case, the publish to pypi stage would fail as it will not accept pushing a package with the same version as an existing package.

    This is a fundamental problem with trying to calculate versions in a PR CI workflow, there will always be that race condition. As far as I can tell the only reliable way of doing this would be to do it on a merge to master, in a guaranteed sequential operation.

  10. rswarbrick commented on Oct 3, 2025

    @rswarbrick
    Contributor

    Gah, good point. I don't think I can see a way for the merging operation to futz with the thing that's being merged though. Maybe the right thing to do is to duplicate the third point (the "check in CI") so that it runs just as the serialization happens?

    I'm wondering about the merge_group action ?

  11. rswarbrick commented on Oct 3, 2025

    @rswarbrick
    Contributor

    If we made that change then you couldn't merge two concurrent PRs that both bumped the version. Somewhat irritating, but just a case of rebasing the later guy and re-pushing (and it's not like we're imaging a CI run to take hours).

  12. rswarbrick commented on Oct 3, 2025

    @rswarbrick
    Contributor

    Also, it's not too bad. If there are PRs A and B, the author can choose to do the serialization manually by making B include A and adding a note in the PR to merge A first.

  13. machshev commented on Oct 6, 2025

    @machshev
    CollaboratorAuthor

    After discussing this offline with @rswarbrick the conclusion we came to was that the extra commit is not necessary.

    On merge to master:

    • semantic-release calculates the version number based on delta change since the last release git tag.
    • Build the python package with the calculated version
    • Push to pypi
    • Git tag the new release version
    • Create a Github release for the version

    This means that for packages installed via pypi (actual released packages) the version will be embedded in the package. For development installs pip install -e <path> the version is ignored anyway, it depends on which commit is checked out + WIP as to the "version" you are running.

    Two potential implementation options:

    1. Run semantic-release with --no-commit, do the package build (uv build and uv publish).
    2. Calculate the version as per above, and explicitly pass it to the build tool. We can use dynamic versioning and pass the calculated version to the build tool (as per the above comment)

    The first option is probably the simplest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions