Skip to content

feat: sample the on-screen picture with VideoSampleTap - #62

Merged
bilipp merged 1 commit into
mainfrom
feat/frame-sample-tap
Nov 3, 2026
Merged

bilipp merged 1 commit into
mainfrom
feat/frame-sample-tap

Conversation

@bilipp

@bilipp bilipp commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Summary

Adds VideoSampleTap, exposed as PlayerSession.videoSamples, so a host can drive something from what is on screen. Lume uses it for Philips Hue light sync on tvOS (bilipp/Lume#374, which needs this merged first).

Implementation

  • The video render pump offers each frame to the tap right after enqueueing it. The tap reads a sparse luma/chroma grid straight from the pixel buffer (read-only lock, never retained), converts it to linear light on BT.709 primaries (PQ/HLG scaled to SDR reference white, BT.2020/P3 converted), and keeps a short ring stamped with the frame PTS.
  • Hosts read sample(atOrBefore: currentTime), since frames are enqueued ahead of the clock.
  • Off by default; the render queue never waits on it, and an unreadable buffer is skipped. The ring empties on seek and on a newer serial.
  • New fixtures quadrant.mp4 (8-bit H.264) and quadrant10.mp4 (10-bit HEVC) in generate-fixtures.sh.

Note: the commit message says the tap hands over the last displayed frame; it actually samples a colour grid as described above. Worth fixing in the squash message.

Testing

  • FrameSampleTapTests (new) against both quadrant fixtures.
  • Verified end to end on an Apple TV driving a Hue bridge from Lume.

PlayerSession.videoSamples hands the host the frame the renderer last
displayed, so Lume can sample it for Philips Hue light sync. The tap is
lazy and costs nothing until a host attaches it. Adds quadrant fixtures
(8-bit H.264 and 10-bit HEVC) that pin the colour conversion.
@bilipp
bilipp merged commit e74f418 into main Nov 3, 2026
5 checks passed
@bilipp
bilipp deleted the feat/frame-sample-tap branch November 3, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant