Skip to content

Fix git clone/fetch hanging indefinitely on unreachable repos - #875

Closed
zeevmoney wants to merge 1 commit into
masterfrom
zeev/per-13817-fix-git-clonefetch-operations-hanging-indefinitely-on
Closed

zeevmoney wants to merge 1 commit into
masterfrom
zeev/per-13817-fix-git-clonefetch-operations-hanging-indefinitely-on

Conversation

@zeevmoney

Copy link
Copy Markdown
Member

Summary

  • Add asyncio.wait_for timeout to pygit2 clone_repository and fetch operations in GitPolicyFetcher, preventing indefinite hangs when a git repo is unreachable
  • Wire existing POLICY_REPO_CLONE_TIMEOUT config (previously unused for scoped repos) into GitPolicyFetcher via ScopesService
  • Re-raise pygit2.GitError and asyncio.TimeoutError from _clone() instead of silently swallowing errors, so sync_scope can handle failures properly

Linear Issue

https://linear.app/permit/issue/PER-13817/fix-git-clonefetch-operations-hanging-indefinitely-on

Tests

  • Unit tests added for run_sync_with_timeout (5 tests)
  • Unit tests added for GitPolicyFetcher clone/fetch timeout and error handling (5 tests)
  • Mocks used for all external dependencies (pygit2, git operations)
  • Integration tests: not needed — changes are internal timeout/error handling
  • Test infrastructure: existing pytest + pytest-asyncio

Test plan

  • Verify POLICY_REPO_CLONE_TIMEOUT=0 (default) preserves existing behavior — no timeout applied
  • Verify POLICY_REPO_CLONE_TIMEOUT=30 causes clone/fetch to abort after 30s on unreachable repos
  • Verify error logs include clear timeout/failure messages
  • Verify scopes sharing the same repo lock are unblocked when a clone times out

🤖 Generated with Claude Code

Add timeout support to server-side pygit2 clone and fetch operations
via asyncio.wait_for. Wire existing POLICY_REPO_CLONE_TIMEOUT config
into GitPolicyFetcher. Re-raise errors from _clone() instead of
silently swallowing them so callers can handle failures properly.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@linear

linear Bot commented Feb 5, 2026

Copy link
Copy Markdown

@netlify

netlify Bot commented Feb 5, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for opal-docs canceled.

Name Link
🔨 Latest commit 1cc1c18
🔍 Latest deploy log https://app.netlify.com/projects/opal-docs/deploys/6984079dc2bfb70008479941

)
try:
repo: Repository = await run_sync(
repo: Repository = await run_sync_with_timeout(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to make sure that in case of a timeout, we actually terminate the Git subprocess

datetime.datetime.now()
)
try:
await run_sync_with_timeout(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timeout should be for the entire sync procedure, not individually fetch and clone. That would naturally double the actual timeout

return False
return last_fetched > t

async def fetch_and_notify_on_changes(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should instrument this function

async def run_sync_with_timeout(
func: Callable[P_args, T_result],
*args: P_args.args,
timeout: Optional[float] = None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the upstream function have "timeout" kwarg then you can't pass it to it, make it more unique

@zeevmoney

Copy link
Copy Markdown
Member Author

Closing: master already covers this.

#924 (git resilience: never stuck on an offline repo) added SCOPES_GIT_FETCH_TIMEOUT (default 120 s), which bounds every scope clone and fetch run through run_in_git_executor. A clone or fetch that fails or times out is:

  • logged and counted in opal_server.scopes.git_op_failures (reason: timeout or git_error);
  • recorded as a source failure, so the source backs off and the sync moves on instead of hanging;
  • retried on a later sync, after a partial clone directory has been removed.

#961 and #989 build on the same path. POLICY_REPO_CLONE_TIMEOUT is still used by the non-scope policy repo watcher, so it is not wired into the scope fetcher.

@zeevmoney zeevmoney closed this Nov 3, 2026
@zeevmoney
zeevmoney deleted the zeev/per-13817-fix-git-clonefetch-operations-hanging-indefinitely-on branch November 3, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants