Skip to content

🐛 fix(docstores): validate add batches before changing stored documents - #888

Draft
adarshh347 wants to merge 2 commits into
Cinnamon:mainfrom
adarshh347:fix/atomic-document-add
Draft

adarshh347 wants to merge 2 commits into
Cinnamon:mainfrom
adarshh347:fix/atomic-document-add

Conversation

@adarshh347

Copy link
Copy Markdown

Description

InMemoryDocumentStore.add currently mutates its dictionary while validating IDs. If a later document conflicts, earlier documents in the rejected batch remain inserted. For SimpleFileDocumentStore, the exception skips save(), leaving the live corpus different from the reloaded corpus. Nonempty ID lists of the wrong length are also silently truncated by zip.

This fix validates counts and stages the batch before updating the store. Existing-ID and within-batch duplicates are rejected without mutation. exist_ok=True retains last-document-wins behavior, and optional/empty-ID fallback is unchanged.

Reproduction

Start with ID existing, then add two documents with IDs ["new", "existing"] and default exist_ok=False. The operation raises, but upstream leaves new in memory. The new regression verifies that only the original document remains, including agreement after file-store reload.

Type of change

  • Bug fix: preserve store contents when input validation rejects an add batch.

Compatibility note: malformed nonempty explicit ID lists now raise instead of silently dropping documents/IDs. This is intentional validation tightening.

Verification

Nine added regression cases fail on unmodified upstream; four cases pass (including the three original docstore tests and new exist_ok compatibility case). After the fix:

  • Docstore suite: 13 passed.
  • Docstore plus document-schema suites: 17 passed, 1 skipped because optional Haystack is absent.
  • Scoped Black 22.3.0, isort 5.12.0, flake8 7.0.0, and git diff --check passed.
  • Independent review and scoped test rerun found no blocking issue.
  • Published branch fetched back and verified identical to the tested two local files (+62/-2).

Command run from repository root:

THEFLOW_SETTINGS_MODULE=theflow.settings.default PYTHONPATH=libs/kotaemon /tmp/kotaemon-review-venv/bin/python -m pytest libs/kotaemon/tests/test_docstores.py libs/kotaemon/tests/test_documents.py -q --tb=short

Python 3.12; actual Document and store implementations, not stubs. The existing Elasticsearch test mocks transport as upstream does. Theflow default settings avoid loading the root UI configuration.

Limits

This provides validation atomicity only: it does not make filesystem writes transactional or provide thread safety. File-write failures and concurrent writers remain outside scope. Full project CI, full pre-commit, full editable development installation, and live integrations were not run. The local environment used a scoped dependency subset, with repository-lock Arrow/LanceDB versions to avoid unrelated incompatibility. NLTK attempted an optional resource download that the environment refused; no bypass was used and tests passed without it. No live user data, model calls, credentials, or paid services were used.

Checklist

  • Focused code review completed
  • Regression and compatibility tests added
  • Behavior clarified in the method docstring
  • Full CI and maintainer review pending

No matching existing bug issue/PR was found in duplicate searches; this description and tests provide the reproduction. Prepared, reviewed, and tested with AI assistance. Submitted as a draft, not a claim of merge readiness.

Stage documents until ID counts and duplicate checks pass. Preserve exist_ok last-document-wins behavior. AI-assisted fix; validation atomicity only, not filesystem transactions or thread safety.
Cover duplicate and mismatched IDs, no partial mutation, persisted store agreement, and exist_ok compatibility. AI-assisted tests; 17 scoped tests passed with one optional Haystack skip.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant