Skip to content

Improve Wikidata enrichment, use MISP standard output and fix SPARQL injection - #1117

Merged
adulau merged 2 commits into
mainfrom
feat/wikidata-standard-expansion
Nov 12, 2026
Merged

adulau merged 2 commits into
mainfrom
feat/wikidata-standard-expansion

Conversation

@adulau

@adulau adulau commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

The Wikidata module interpolates untrusted text into a SPARQL query and returns only the first entity URI in the legacy response format. This change escapes query literals and returns bounded candidate matches as standard MISP annotation objects, with labels, descriptions and Wikidata links related to the source attribute.

Addresses the SPARQL injection described in #869, including quotes, backslashes and control characters. Language tags and numeric settings are validated before query construction.

  • Enable both expansion and hover, preserving the source attribute and its UUID references. Generated attributes have to_ids=false, with correlation disabled for descriptive text.
  • Match exact labels and aliases, remove duplicate entities, and add configurable language, result limits and request timeouts. Restrict the subquery to Q items before applying the limit, so properties and lexemes cannot consume the result budget. Defaults are English, 5 results and 15 seconds.
  • Use the current HTTPS endpoint and a MISP user agent. Return consistent error dictionaries for invalid requests, malformed responses and upstream failures; no matches returns an empty object list.
  • Add regression tests, update the live smoke test, and document configuration and response changes.

Compatibility: legacy top-level text requests remain accepted, but all responses now use misp_standard (results.Attribute and results.Object). Consumers of the previous results list must migrate.

Validation:

  • pytest -q tests/test_wiki.py tests/test_duckduckgo_search.py tests/test_expansion_smoke.py tests/test_dns.py: 224 passed.
  • Live HTTP module discovery and tests/test_expansions.py::TestExpansions::test_wikidata: passed against the running MISP modules service and Wikidata.
  • SPARQL parser regression checks for crafted input and strict validation against the standard MISP annotation template: passed.
  • Local RDF graph regression tests reproduce property/lexeme labels and aliases competing with Q items under LIMIT 1, including no-item cases: passed. A live Wikidata lookup with a one-result limit also passed.
  • Black, isort, flake8, documentation JSON/generated-section checks and git diff --check: passed.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T04:51:09.561666Z 7a2a7e3 PR opened
🔒 Security Review ✅ Completed 2026-11-12T04:52:11.496520Z 7a2a7e3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a2a7e3d9e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread misp_modules/modules/expansion/wiki.py Resolved
@adulau
adulau merged commit 09e32c8 into main Nov 12, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant