Repository navigation
🐛 fix(docstores): validate add batches before changing stored documents - #888
Draft
adarshh347 wants to merge 2 commits into
Draft
adarshh347 wants to merge 2 commits into
adarshh347 wants to merge 2 commits into
Conversation
Stage documents until ID counts and duplicate checks pass. Preserve exist_ok last-document-wins behavior. AI-assisted fix; validation atomicity only, not filesystem transactions or thread safety.
Cover duplicate and mismatched IDs, no partial mutation, persisted store agreement, and exist_ok compatibility. AI-assisted tests; 17 scoped tests passed with one optional Haystack skip.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
InMemoryDocumentStore.addcurrently mutates its dictionary while validating IDs. If a later document conflicts, earlier documents in the rejected batch remain inserted. ForSimpleFileDocumentStore, the exception skipssave(), leaving the live corpus different from the reloaded corpus. Nonempty ID lists of the wrong length are also silently truncated byzip.This fix validates counts and stages the batch before updating the store. Existing-ID and within-batch duplicates are rejected without mutation.
exist_ok=Trueretains last-document-wins behavior, and optional/empty-ID fallback is unchanged.Reproduction
Start with ID
existing, then add two documents with IDs["new", "existing"]and defaultexist_ok=False. The operation raises, but upstream leavesnewin memory. The new regression verifies that only the original document remains, including agreement after file-store reload.Type of change
Compatibility note: malformed nonempty explicit ID lists now raise instead of silently dropping documents/IDs. This is intentional validation tightening.
Verification
Nine added regression cases fail on unmodified upstream; four cases pass (including the three original docstore tests and new
exist_okcompatibility case). After the fix:git diff --checkpassed.Command run from repository root:
Python 3.12; actual Document and store implementations, not stubs. The existing Elasticsearch test mocks transport as upstream does. Theflow default settings avoid loading the root UI configuration.
Limits
This provides validation atomicity only: it does not make filesystem writes transactional or provide thread safety. File-write failures and concurrent writers remain outside scope. Full project CI, full pre-commit, full editable development installation, and live integrations were not run. The local environment used a scoped dependency subset, with repository-lock Arrow/LanceDB versions to avoid unrelated incompatibility. NLTK attempted an optional resource download that the environment refused; no bypass was used and tests passed without it. No live user data, model calls, credentials, or paid services were used.
Checklist
No matching existing bug issue/PR was found in duplicate searches; this description and tests provide the reproduction. Prepared, reviewed, and tested with AI assistance. Submitted as a draft, not a claim of merge readiness.