Skip to content

feat(fetcher): support opt-in HTTP conditional requests (ETag / 304) (#990) - #998

Open
cocvn157 wants to merge 1 commit into
permitio:masterfrom
cocvn157:feat/http-fetcher-conditional-requests-990
Open

cocvn157 wants to merge 1 commit into
permitio:masterfrom
cocvn157:feat/http-fetcher-conditional-requests-990

Conversation

@cocvn157

@cocvn157 cocvn157 commented Nov 5, 2026 •

Copy link
Copy Markdown

Adds opt-in HTTP conditional request support (conditional_request: bool = False on HttpFetcherConfig) to HttpFetchProvider and DataUpdater so polling unchanged HTTP data sources uses ETag / If-None-Match and Last-Modified / If-Modified-Since instead of refetching and overwriting the policy store on every interval.

When conditional_request=True, HttpFetchProvider caches response validators per url|dst_path key and returns FETCH_NOT_MODIFIED on 304 Not Modified across both aiohttp and httpx backends, causing DataUpdater to skip _store_fetched_data() and report DataEntryReport(entry=entry, fetched=True, saved=False). Validator caches are cleared on get_base_policy_data() and reset() so initial loads and policy-store resets always perform a full fetch.

Fixes #990

@netlify

netlify Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for opal-docs ready!

Name Link
🔨 Latest commit e805c33
🔍 Latest deploy log https://app.netlify.com/projects/opal-docs/deploys/6aefdc07dacd683eb0c74d13
😎 Deploy Preview https://deploy-preview-998--opal-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@cocvn157
cocvn157 force-pushed the feat/http-fetcher-conditional-requests-990 branch from 22271fd to 8f9aceb Compare November 6, 2026 14:52
@cocvn157
cocvn157 force-pushed the feat/http-fetcher-conditional-requests-990 branch from 8f9aceb to e805c33 Compare November 8, 2026 02:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HTTP data fetcher: support conditional requests (ETag / If-None-Match → 304) to skip redundant full refetches

1 participant