Skip to content

Run tests on wheels - #1977

Open
reinecke wants to merge 5 commits into
AcademySoftwareFoundation:mainfrom
reinecke:test-on-wheels
Open

reinecke wants to merge 5 commits into
AcademySoftwareFoundation:mainfrom
reinecke:test-on-wheels

Conversation

@reinecke

@reinecke reinecke commented Nov 9, 2025 •

Copy link
Copy Markdown
Collaborator

This change re-shuffles the CI so that:

  • A single py_coverage_build build runs and generates coverage using the same methodology our previous test builds used.
  • A separate "build and test only" set of platforms/python version run off the sdist (currently mingw)
  • Wheels are built in parallel with the smoke tests
  • The test suite is run for each OS/python combo against the wheels - the automation installs OTIO from the .whl before pulling the source code for running tests to ensure we don't accidentally rebuild from source.

tests/requirements.txt is a new file that CI uses to install test dependencies. It duplicates the dev extra, because the test jobs cannot use the extras otherwise they might install OTIO from source.

Note: ubuntu-24.04-arm and macos-15 cp39 wheels are built but not tested.

Ideally this would take the suggestion from @JeanChristopheMorinPerso and run against sdist - that change hasn't been made.
Update: after investigation, cibuildwheel isn't really meant to happen from sdist sources. I did a pass to make our build and test only phase run off it though - since those platforms would be running off sdist in the wild.

Additionally, if we could capture coverage from one of the cibuildwheel runs, we could probably eliminate the smoke test phase.

  • Consider dropping the smoke test and just harvest coverage from testing the wheels
    • Update: we decided against this because you need particular build options to generate needed coverage artifacts
  • Investigate having the wheel build to happen off the sdist
  • Currently the build and test only phase is hitting a test error - getting it to find the sdist tar.gz when the name is indeterminate (includes the version) is proving tricky. Resolved by using globbing
  • Re-shuffle the order of repo checkout and wheel install in the wheel test action to match the test only step
  • Reduce the matrix on the smoketest build so that it only does the one it needs for coverage - consider renaming explicitly as "generate_coverage" or similar

Fixes #1027
This supersedes #1961 - the changes have been added to this PR

Assisted-by: Claude Code / Claude Sonnet 5.5

@github-actions github-actions Bot added the ci label Nov 9, 2025
@reinecke
reinecke marked this pull request as ready for review November 9, 2025 21:52
@reinecke
reinecke marked this pull request as draft November 9, 2025 21:52
@reinecke
reinecke marked this pull request as ready for review November 9, 2025 21:52
@codecov-commenter

codecov-commenter commented Nov 9, 2025 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.62%. Comparing base (0bfd9d6) to head (7d38027).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1977   +/-   ##
=======================================
  Coverage   83.62%   83.62%           
=======================================
  Files         184      184           
  Lines       13627    13627           
  Branches     1270     1270           
=======================================
  Hits        11395    11395           
  Misses       2059     2059           
  Partials      173      173           
Flag Coverage Δ
py-unittests 83.62% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0bfd9d6...7d38027. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@reinecke
reinecke marked this pull request as draft November 10, 2025 00:21
@reinecke
reinecke force-pushed the test-on-wheels branch 2 times, most recently from ec030a9 to 742dc6c Compare November 12, 2025 22:04
@reinecke
reinecke marked this pull request as ready for review November 12, 2025 22:38
@reinecke

Copy link
Copy Markdown
Collaborator Author

I'm ready for feedback on this - just the one open questions about whether or not we really need the smoketests or if I should move the coverage reports to the tests we run on the equivalent wheel.

reinecke and others added 3 commits October 30, 2026 15:33
Re-tool the Python package workflow to test installed wheels (and the sdist
for platforms without a wheel, e.g. mingw) instead of the source checkout.
Replace the dev extra with tests/requirements.txt.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Eric Reinecke <ereinecke@netflix.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Eric Reinecke <ereinecke@netflix.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Eric Reinecke <ereinecke@netflix.com>
reinecke and others added 2 commits October 30, 2026 16:03
In test_wheels, install the wheel before checking out the source so the
tests run against the wheel and not the source tree, matching
py_testonly_build. Split the dependency install so the test requirements
are installed after the checkout.

Rename py_smoketest_build to py_coverage_build and drop its Python
matrix. It now builds on the coverage Python only (GH_COV_PY); all
Python versions are built and tested as wheels by package_wheels and
test_wheels.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Eric Reinecke <ereinecke@netflix.com>
Revert the removal of extras_require['dev'] and the related README and
Makefile hint changes. Removing the extra is a user-facing packaging
change that belongs in a separate PR. tests/requirements.txt stays,
because the wheel test jobs need to install the test dependencies
without building the package from source.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Eric Reinecke <ereinecke@netflix.com>

@jhodges10 jhodges10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get this merged!

I'm looking forward to the improved build times as well as just knowing that we're testing against the same build output that we're distributing.

Image

@JeanChristopheMorinPerso JeanChristopheMorinPerso left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a look and saw no major things that needs addressing. I only found minor things.

run: |
echo 'OTIO_CXX_DEBUG_BUILD=1' >> $GITHUB_ENV
sudo apt-get install lcov
- name: Install ZLIB (Linux)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Install ZLIB (Linux)
- name: Install zlib

The "Linux" bit is more confusing than anything.

run: sudo apt-get install -y zlib1g-dev
- name: Install python build dependencies
run: |
python -m pip install --upgrade pip setuptools wheel "flake8>=3.5" check-manifest

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need flake8 and check manifest if we are only interested in code coverage?

path: 'wheelhouse'
merge-multiple: true
- name: Install wheel for environment
run: pip install --no-cache-dir --no-index --find-links wheelhouse opentimelineio

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
run: pip install --no-cache-dir --no-index --find-links wheelhouse opentimelineio
run: python -m pip install --no-cache-dir --no-index --find-links wheelhouse opentimelineio

That's what we do everywhere else

include:
- { os: ubuntu-latest, shell: bash }
- { os: macos-15, shell: bash }
- { os: windows-2022, shell: pwsh }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious, why force push? I don't see any powershell code in the workflow.

Comment on lines +287 to +290

env:
OTIO_CXX_COVERAGE_BUILD: 'ON'
OTIO_CXX_BUILD_TMP_DIR: ${{ github.workspace }}/build

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
env:
OTIO_CXX_COVERAGE_BUILD: 'ON'
OTIO_CXX_BUILD_TMP_DIR: ${{ github.workspace }}/build

This is not needed since we aren't building.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Test pass for build artifacts

4 participants