Skip to content

feat(connectors): add connectors state storage using local files - #1948

Merged
spetz merged 3 commits into
masterfrom
connectors_state
Jul 4, 2025
Merged

spetz merged 3 commits into
masterfrom
connectors_state

Conversation

@spetz

@spetz spetz commented Jun 30, 2025

Copy link
Copy Markdown
Contributor

Allow the optional storage of source connectors state e.g. to keep track of the recently processed data, that can be persisted across the restart. For now, it only supports the local file storage.

@spetz
spetz force-pushed the connectors_state branch 3 times, most recently from 8892a70 to 1cfcf38 Compare June 30, 2025 19:44
@spetz spetz added the connectors Connectors runtime label Jun 30, 2025
@spetz
spetz force-pushed the connectors_state branch from 1cfcf38 to 70da535 Compare June 30, 2025 20:38
@spetz
spetz force-pushed the connectors_state branch from 70da535 to 6c46caa Compare July 3, 2025 18:57
@spetz

spetz commented Jul 3, 2025

Copy link
Copy Markdown
Contributor Author

Pushed additional extensions for the missing message metadata (id, timestamp etc.) and also fixed the tracing initializer for connectors.

Comment thread core/connectors/sdk/src/source.rs
Comment thread core/connectors/runtime/src/state.rs
@spetz
spetz force-pushed the connectors_state branch from d890e55 to 3a6208c Compare July 3, 2025 19:46
@spetz
spetz merged commit 7755084 into master Jul 4, 2025
@spetz
spetz deleted the connectors_state branch July 4, 2025 06:46
hageshiame pushed a commit to hageshiame/iggy that referenced this pull request Nov 7, 2025
…che#1948)

Allow the optional storage of source connectors state e.g. to keep track
of the recently processed data, that can be persisted across the restarts.
For now, it only supports the local file storage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Connectors runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants