HTTP data fetcher: support conditional requests (ETag / If-None-Match → 304) to skip redundant full refetches #990

Is your feature request related to a problem? Please describe.

The HTTP fetch provider always performs a full GET and the client always replaces the data in the policy store, even when the data source has not changed. With periodic_update_interval and update triggers, every client re-downloads and re-writes the full document on every cycle.

We run OPAL in production for an enterprise data platform:

• OPAL server and client 0.9.3 (planning to move to 0.9.9+). The fetcher/updater code paths below are unchanged between 0.9.3 and current master.
• 6–8 opal-client pods (HPA) serving ~1.5k authorization queries/s in total
• One HTTP data source, ~5 MB JSON, periodic_update_interval 600 s

For a document that rarely changes, this is roughly:

• ~144 full fetches per client per day → about 5.8 GB/day of transfer
• ~1,150 full policy-store data replacements per day across the fleet

A data source that supports standard HTTP conditional requests (a strong ETag, If-None-Match → 304 Not Modified) could avoid all of this, but the fetcher never sends the header. Our data source is a PEP-side service we built, and we are adding ETag / 304 support on that server side now — we would like the OPAL side to be able to use it as well.

Code references (current master):

• opal_common/fetcher/providers/http_fetch_provider.py — HttpFetcherConfig has headers / is_json / process_data / method / data only; no validator handling. A 304 would pass raise_for_status() and then fail in _response_to_data() on the empty body.
• opal_client/data/updater.py — calc_hash() is computed for the report only; identical data is still written via _set_policy_data().

Describe the solution you'd like

An opt-in conditional-request mode for the HTTP fetcher (default behaviour unchanged):

1. A new HttpFetcherConfig option (name open for discussion, e.g. conditional_request: bool = False).
2. When enabled, after a successful 200 the client remembers the response ETag (and Last-Modified if present) for that data source entry, and sends If-None-Match / If-Modified-Since on the next fetch.
3. On 304 Not Modified the fetcher returns a "not modified" result instead of trying to parse a body, and the updater skips the policy-store write (reported as fetched, not saved).
4. The validator is kept outside the provider instance (providers are created per fetch), keyed by entry (url + dst_path + config), in client memory only.

Points we would like your input on:

• Policy store resets: if the inline OPA restarts or the store is cleared while the client keeps its validators, a 304 would leave the store empty. We think validators should be cleared whenever the policy store is (re)initialised, and probably not used for the base fetch on (re)connect.
• Whether you prefer this in the HTTP provider/config as above, or as a generic "not modified" contract in the fetcher engine that other providers could also use.

Describe alternatives you've considered

• Client-side content hash check (skip the store write when calc_hash() matches the last saved hash). This avoids redundant writes but not the transfer. It could be a complementary change; we can propose it separately if useful.
• Longer periodic interval / disabling periodic updates. Reduces cost but also removes the safety net for missed update events.
• A custom fetch provider via FETCH_PROVIDER_MODULES. Works, but this seems generally useful for anyone fetching large, mostly static documents over HTTP.

Additional context

• We are happy to implement this and open a PR with tests if the direction looks good to you.
• We run the 0.9 line in production. Would a backport to release/0.9.x be acceptable, or should this target master (0.10.x) only?