Skip to content

CI: intermittent test failures under the strict cooperative pool #58

Open

Description

@bilipp

Since CI runs the suite with LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 only (#48), main fails intermittently. Strict mode puts all of Swift concurrency on one cooperative thread, on a 3-core VM. The failing tests change from run to run, and the same trees pass on their PR branches. Locally (strict mode, parallel), the suite passes every time.

Failures seen

  • "audio that stops reaching the renderer while video plays is reported" (AudioRouteTests.reportsLostAudio) fails on most CI runs with health is ok, and has never failed locally. AudioLossDetector only judges while the clock runs and video is ahead. If the synchronizer's clock stops once the audio renderer runs dry, as it may on the VM's virtual audio device, nothing is ever judged. fix: start playback after a seek's landing is checked; make the lost-audio test cut the feed #57 closed a separate gap: a pending pump retry kept feeding the renderer, and SystemRenderer.stopFeedingAudio() now stops it. The test still fails on CI after that. Quarantined on CI until this is understood. The detector's pure unit tests still run.
  • "open → play advances the clock in real time": the clock stayed at 0.05 s for 10 s. Once, on the CI run for feat: subtitle cue lifecycle and CEA-608 closed captions from the video #52's merge.
  • "a live channel that reconnects plays the replayed seconds once": dup=54, where the test expects over 100. Once, locally, with --no-parallel in strict mode. It passes 6 of 6 in isolation. The replayed byte range is fixed, so the count shouldn't depend on timing.
  • "resume-style seek over a range-less HTTP server keeps playing": a real ordering bug (playback started before the seek landing was checked). Fixed in fix: start playback after a seek's landing is checked; make the lost-audio test cut the feed #57.

Suspects

  • About 60 synchronous shutdown() calls (thread joins) in async tests: DemuxerTests, DecoderTests, AudioRouteTests.decode, HDRMetadataTests and others. TortureHTTPServer.init also waits on a semaphore. Each blocks the only cooperative thread. That is what the strict mode is meant to expose, so these should go through DedicatedThread, like PlayerSession does since chore(ci): strict cooperative pool for tests, offline docs check #48.
  • Behaviour of the virtual audio device on the CI VM (the clock stall and the lost-audio detection).

Numbers

  • Strict mode, parallel: about 70 s locally.
  • Strict mode, --no-parallel: 2.5 min locally.

Activity

  1. bilipp commented on Oct 13, 2026

    @bilipp
    OwnerAuthor

    Another one, on #53's merge run: "play prints startup, heartbeats and a clean summary" (LumeCtlTests.play) got 1 heartbeat where it expects at least 2. That's 2 s of playback with a 0.5 s interval, so the heartbeat timer was held up about 1.5 s. It's the same kind of cooperative-thread stall.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions