[deanq]

Why

Worker repos (worker-faster_whisper, llm-fine-tuning, vllm-loadbalancer-ep, worker-sdxl, worker-sdxl-turbo, cog-worker, worker-tensorrt-llm-triton) copied TEMPLATE-run_worker_tests.yml. It POSTs /run to a long-lived endpoint that boots this repo's runner image as a self-hosted runner. Once that endpoint idles it is paused with max_workers=0, and every run fails in about 4 seconds:

{"code":409,"message":{..."detail":"Endpoint is paused (max_workers=0). Set max_workers > 0 to accept work.","code":"ENDPOINT_PAUSED"}}

The template also sends the GitHub PAT in plaintext in the job input.

What

A composite action, test-image/, that tests a worker image on a temporary endpoint created for each run:

• rp flash dev --once (runpod-python 2.1.0) creates a live endpoint from the image with at most one worker. It runs every case in .runpod/tests.json (the Hub format) and deletes the endpoint when the tests pass, fail, or the run is cancelled.
• The live endpoint replaces the image's start command with the Flash runtime. The action forces versions through RUNPOD_PACKAGE_SPEC and RUNPOD_RUNTIME_PACKAGE_SPEC, which the launcher installs before starting. runpod-sdk-runtime isn't on PyPI, so it is pinned to the v0.2.0 tag's commit.
• On the worker, the job loads the image's handler file, captures the handler passed to runpod.serverless.start (at module level or under a __main__ guard), and calls it. Sync, async, generator and async generator handlers are supported.
• Flash execs the whole module on the worker, not just the function. The module's config travels in the endpoint env, so the worker never needs the tests file or cases.py.
• A test fails if the job fails or times out, returns no output, or returns a dict with an error key.

Caller usage:

- uses: runpod-workers/worker-github_runner/test-image@main
  with:
    image: runpod/worker-example:ci-pr-${{ github.event.number }}
    runpod-api-key: ${{ secrets.RUNPOD_API_KEY }}
    handler: /handler.py

Also removes TEMPLATE-run_worker_tests.yml, rewrites docs/usage.md, and adds a CI job for the action's unit tests.

Limitations (documented in docs/usage.md)

• The image's own start command does not run; the handler is called directly.
• The worker runs runpod==<runpod-version> (default 2.1.0), even if the image pins an older SDK.
• The Flash dev session puts RUNPOD_API_KEY in the endpoint env for the life of the run.
• Fork PRs get no secrets, so callers should skip the job for forks.

Test plan

• 24 unit tests: suite parsing, app naming, result checks, and handler loading. These include a test that execs the shipped module with only the worker env.
• Ran live with runpod/mock-worker:dev on an RTX A4000. The pass case passed in 2m02s with exit 0. A case returning {"error": ...} failed in 34s with exit 1. Both endpoints were deleted, and no endpoints were left behind.
• ruff and actionlint are clean.
• First real caller: worker-faster_whisper's CI (linked PR).

[Copilot]

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

5 open issues

• The inputs table uses || prefixes, which won’t render as a Markdown table in most renderers.… · New
• config.env parsing can raise KeyError (or TypeError if an entry isn’t a dict), which will… · New
• If the pid file exists but is empty/corrupted (e.g., partial write),… · New
• shell=True executes the command through a shell, which increases risk if server['command']… · New
• This opens the pid file without a context manager, which can leave the file descriptor open longer… · New

What changed in this PR

Adds a new test-image GitHub Action and supporting Flash app to run Hub-format worker tests on a temporary Runpod serverless endpoint (handler mode or server mode), then delete the endpoint.

Changes:

• Introduces test-image Flash app (flash_app.py) and suite parsing/checks (cases.py) to execute tests against a temporary endpoint.
• Adds unit tests covering suite parsing, handler execution, async/generator behavior, and server-mode behavior.
• Updates documentation and adds a CI workflow to run the new action’s unit tests; removes the older template workflow.

File	Description
test-image/​flash_app.py	Implements the Flash app entrypoint and worker-side execution for handler/server modes.
test-image/​cases.py	Parses Hub-format tests.json, normalizes server-mode requests, and validates outputs.
test-image/​action.yml	Defines the composite action that runs rp flash dev --once with forwarded env.
test-image/​tests/​test_run_case.py	Adds behavioral tests for handler mode, server mode, and worker-env shipping constraints.
test-image/​tests/​test_cases.py	Adds unit tests for suite parsing, request normalization, naming, and output checks.
test-image/​tests/​conftest.py	Ensures test-image package modules are importable in tests.
.github/​workflows/​CI-test_image_action.yml	Runs unit tests for test-image on PRs/pushes.
docs/​usage.md	Documents the new test-image action usage, modes, and failure cases.
README.md	Links to the new test-image action documentation.
.github/​workflows/​TEMPLATE-run_worker_tests.yml	Removes the old template workflow.

_{Review effort: Lite}