Repository navigation
fix(skill): coverage, chunk-parse, stale-extract and directed-explain bugs in skill templates (#5340) - #5351
Conversation
… bugs in skill templates (Graphify-Labs#5340) - B3.5 coverage check compares a PDF against its page count (pypdf, within the 50 MiB parse cap) instead of its newline count, and skips the check when the page count is unknown; the warning says pages for PDFs. - B3.5 skips an invalid or non-object chunk with a warning instead of aborting (as Step B3 already says), removes it so later steps never parse it, and no longer prints "Coverage OK" after skipping one. - --update deletes a leftover .graphify_extract.json up front, so a deletions-only run merges an empty extraction instead of an earlier run's stale one (which also blocked pruning of deleted files). - The inline explain fallback lists predecessors too on directed graphs; undirected output is unchanged. - query reference: the inline fallback is documented as unweighted (no IDF), matching the code. - hooks reference: `graphify claude install` writes to AGENTS.md when the project has one and no CLAUDE.md-style file, matching install.py. Fragments edited; generated skills regenerated with tools.skillgen --bless; gen.py sanctions the two changed monolith line classes. Regression tests added for each item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the pull request, @pawanshukla500. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.11 and 3.13, and that the change keeps extraction deterministic. |
There was a problem hiding this comment.
Graphify code review 
Review completed · 2a7c1a7334c2
Warning
Merge confidence: 3/5 · Review before merging · 🟠 High impact
1 P2, 1 P3 bugs · 9 possible issues not confirmed
Bugs (2)
P2 · Iterate over every edge-attribute mapping in G[neighbor][nid]: the inline directed fallback prints only the first incoming relationship when a predecessor has parallel edges. Inspect source
Detailed review · checked against the code
Directed explanations show only one incoming relationship per neighbor
When a directed graph has multiple relationships from the same node into the explained node, the fallback prints just one of them, hiding the others.
Expected: The explanation displays every incoming relationship from that predecessor.
Actually: The fallback selects and prints only the first edge’s attributes for that predecessor.
In the inline directed-graph fallback, incoming links are grouped by predecessor: G.predecessors(nid) visits a predecessor once, and each (neighbor, G[neighbor][nid], ...) stores that pair’s edge-attribute mapping. The loop then takes only next(iter(_raw.values()), {}) and prints one relationship. Thus, when one predecessor has parallel incoming edges, only one is displayed. The base fallback did not include incoming links; this incoming-edge path was added in the diff, so the behavior is introduced for the affected path.
- Independent source review: The main CLI’s
graphify explaincode iterates each edge returned byedge_datasfor successors and predecessors, so it handles parallel edges there. That is a separate route: the documented inline fallback used when the CLI is unavailable still selects one edge from the mapping. The inspected skill-generation test checks that the explain section exists, but does not exercise parallel incoming edges.
P3 · Skip the self-loop when adding directed predecessors in default.md: the inline fallback prints the same connection twice, with opposite arrow directions. Inspect source
Detailed review · checked against the code
Directed explanations print self-links twice
When a directed graph has a self-link on the explained node, the fallback prints that connection twice, with opposite arrow directions.
Expected: The self-loop is shown once as a connection of the explained node.
Actually: The successor and predecessor lists each include the self-loop, so the same edge is printed twice with opposite arrow directions.
In the inline explain fallback, the outgoing list uses G.neighbors(nid), and for directed graphs the code then appends G.predecessors(nid). A directed self-loop makes nid appear in both lists, so the same edge data is printed once with the outgoing arrow and once with the incoming arrow. The fragment is assigned as the query reference for generated platforms; the issue affects that inline fallback when used.
- Independent source review: Reviewed the related fallback tests: they cover directed incoming edges and undirected graphs, but no self-loop case. Neither the fallback code nor the inspected tests contain a check that deduplicates a self-loop.
Not confirmed (9 possible issues)
Leads worth a look; we didn't double-check these. They don't change the score.
Not confirmed · Hosts whose names contain a hyphen get no case in the runbook test, so their update.md is never run Inspect source
Detailed review · not confirmed
The case collector only accepts rendered paths whose host segment is word characters, so a host with a dash in its name is silently dropped even though the docstring says every host is covered.
Line 72 uses re.fullmatch with a word-character class between 'graphify/skills/' and '/references/update.md', and that class does not match a hyphen. Line 73 then continues on no match. The only guard, the assertion at line 81, fires only when no host matches at all, so one matching host is enough to pass.
-
The platform names were not visible in this change, so whether any current host name contains a hyphen is unchecked.
-
Found by the per-file sweep; not double-checked.
Not confirmed · The test drops every PATH directory that holds a graphify entry, which can also remove the directory holding the interpreter Inspect source
Detailed review · not confirmed
_env removes whole PATH directories that contain a graphify file or folder, not just the graphify launcher, so unrelated tools in those directories disappear for the runbook blocks.
Lines 89-90 keep or drop each PATH entry based on not (Path(p) / 'graphify').exists(), which drops the whole directory. An empty PATH entry resolves to the current directory, so a graphify entry there also removes it. The comment at lines 86-87 only asks to keep graphify off PATH, which this does more broadly.
-
Whether the blocks call a bare python3 or only the recorded interpreter was not checked for every host.
-
Found by the per-file sweep; not double-checked.
Not confirmed · If a runbook block spells the interpreter lookup differently, the test runs it with whatever python is first on PATH Inspect source
Detailed review · not confirmed
_bash only replaces the exact text $(cat graphify-out/.graphify_python). Any other spelling of the interpreter is left alone and resolved from PATH.
Lines 98-99 call str.replace with one literal and never check that a replacement happened. The comment at lines 96-97 assumes the blocks use the unquoted form, and nothing enforces that assumption.
-
The current rendered blocks were not inspected for alternate spellings of the interpreter lookup.
-
Found by the per-file sweep; not double-checked.
Not confirmed · The check for the CLAUDE.md-to-AGENTS.md note only inspects texts that already contain the claude sentence, so differently worded texts are Inspect source
Detailed review · not confirmed
The test says every rendered text describing graphify claude install must name the AGENTS.md it writes, but the loop only runs over artifacts that already contain the exact CLAUDE.md sentence. A rendered description of the same command with different wording passes the test without any check.
Line 986 builds the list with [a for a in gen.render_all(...) if sentence in a.content], so the filter itself decides which artifacts are checked. The loop at lines 989-991 only covers that filtered list. The docstring's promise that every rendered text describing the command says so is therefore not enforced for any text that drifts from the exact sentence.
-
Only the exact sentence 'This writes a
## graphifysection to the localCLAUDE.md' is recognised as describing the command. -
The rendered artifacts from gen.render_all were not inspected here, so it is unknown whether any other text describes the command.
-
Found by the per-file sweep; not double-checked.
Not confirmed · The test for an unknown PDF page count only blocks the pypdf import, so the unreadable-PDF case it names is never exercised Inspect source
Detailed review · not confirmed
The test is named for a PDF whose page count is unknown and its docstring also covers an unreadable PDF. It only sets pypdf to None, so the import fails and the file is never parsed. The case where pypdf is present but cannot read the file is not tested.
Line 1356 sets sys.modules['pypdf'] to None, so the import in the coverage body fails before any PDF is read. The fake PDF written at lines 1362-1363 is never parsed by pypdf, so the branch that handles a parse failure after a successful import is untested. The docstring at lines 1354-1355 says the unreadable case is covered, which the test does not show.
-
The coverage step body was not visible, so the exact exception handling for an unreadable PDF could not be confirmed.
-
Found by the per-file sweep; not double-checked.
Not confirmed · The AGENTS.md redirect sentence is required in every such text, but the test cannot tell whether the text states the condition it applies un Inspect source
Detailed review · not confirmed
The docstring says the redirect to AGENTS.md applies only when the project has an AGENTS.md and no CLAUDE.md-style file. The assertion only checks that a fixed sentence appears in each text. It does not check that the sentence is tied to that condition.
Lines 990-991 use substring checks only. The docstring makes a conditional claim about install._claude_instructions_file, but no assertion verifies the condition. The check passes on the fixed wording alone.
-
The surrounding wording of the rendered texts was not visible, so the condition may be stated nearby even though the test does not check it.
-
Found by the per-file sweep; not double-checked.
Not confirmed · Test helper unescapes every backslash-quote in the extracted fallback script, so the test may run code that differs from the shipped fragmen Inspect source
Detailed review · not confirmed
The _run_explain helper on line 222 removes every backslash followed by a double quote from the whole extracted script before executing it. It is meant to undo markdown escaping, but it cannot tell escaping apart from real code, so the test can exercise a different script than the one shipped in the fragment.
Line 222 applies script.replace(chr(92) + '"', '"') to the entire extracted script with no check for markdown escaping. The helper exists to run the inline fallback copies in the query reference, aider and devin unchanged, so any unescaping that alters real code undermines the three-way consistency these tests are meant to check.
-
The fragment text around the inline explain fallback was not inspected, so it is unknown whether backslash-quote sequences occur there.
-
FALLBACK_OPEN, FALLBACK_END and QUERY_FRAGMENT are defined outside the shown range and were not read.
-
Found by the per-file sweep; not double-checked.
Not confirmed · The explain direction allowlist accepts exact lines anywhere in the monolith, so unrelated edits can slip past the round-trip guard Inspect source
Detailed review · not confirmed
_is_explain_direction_fix_line matches whole stripped lines such as if G.is_directed(): and for neighbor in G.neighbors(nid): without checking where they appear. Any identical line added elsewhere in the monolith is treated as the sanctioned #5340 fix.
The predicate at lines 1483-1496 returns line.strip() in {...} with no context. _is_sanctioned_monolith_diff at line 1566 applies it to every added and removed line in the file, so the allowlist is global rather than limited to the explain fallback block.
-
I did not inspect the monolith body outside the diff, so I cannot tell whether duplicate matching lines exist today.
-
This assumes the round-trip check is the only guard for these monolith lines.
-
Found by the per-file sweep; not double-checked.
Not confirmed · The monolith_roundtrip docstring omits the new #5340 explain and AGENTS.md sanctioned changes Inspect source
Detailed review · not confirmed
The docstring lists the sanctioned change classes, but it does not mention the directed explain fix or the claude install AGENTS.md sentence that were added to _SANCTIONED_MONOLITH_DIFFS in this PR.
Lines 1575-1579 enumerate the predicates by name and PR number, and lines 1559-1560 add two predicates that the enumeration does not describe.
-
This is documentation drift only and does not change runtime behavior.
-
Reference checks do not establish safe deletion for dynamic or external consumers.
-
Found by the per-file sweep; not double-checked.
What this review covered
- We didn't review 116 files that aren't source code, such as docs, config, images, or generated or third-party code.
- 4 of 4 changed files inspected. This is scope, not a correctness guarantee.
Code graph and static checks · 1739 functions depend on this change
Impact & health
Impact — 1739 functions depend on the 1717 functions this change touches.
Health — grade A; 8 existing hotspot(s) in the area this change touches (pre-existing, not introduced here):
render()— 16 callers, 5 callees (high)audit_coverage()— 8 callers, 6 callees (high)main()— 3 callers, 11 callees (medium)_unreported_fixture()— 6 callers, 4 callees (medium)monolith_roundtrip()— 3 callers, 6 callees (medium)test_an_unreported_file_is_retried_instead_of_cached_as_complete()— 0 callers, 6 callees (medium)test_audit_catches_a_dropped_non_allowlisted_heading()— 0 callers, 6 callees (medium)test_update_preserves_undispatched_semantic_stamps()— 0 callers, 6 callees (medium)
Formal checks — 1739 functions this change can affect weren't formally checked in this review. Formal checks are informational only.
Gate & verification
✅ PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1739 function(s) in the blast radius were not formally verified this run
Test selection
594 of 594 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety (
noncode-change,uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/pi.test.ts— full-run-safetytests/test_4874_jvm_resolution.py— full-run-safetytests/test_4909_reference_binding_scale.py— full-run-safetytests/test_4978_reconcile_cost.py— full-run-safetytests/test_4992_ts_import_types.py— full-run-safetytests/test_5072_splittable_under_cap_retry.py— full-run-safetytests/test_affected_cli.py— full-run-safetytests/test_affected_defines.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_affected_overrides.py— full-run-safetytests/test_affected_path_file_node.py— full-run-safetytests/test_affected_seed_reference_forms.py— full-run-safetytests/test_agents_platform.py— full-run-safetytests/test_alias_folding.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— full-run-safetytests/test_astro_import_ids.py— full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_backend_opt_in.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_benchmark_real_query.py— full-run-safetytests/test_binary_media_words.py— full-run-safetytests/test_blade_extractor.py— full-run-safetytests/test_build.py— full-run-safetytests/test_build_located_semantic_identity.py— full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_ghost_fold_4955.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— full-run-safetytests/test_c_include_resolution.py— full-run-safetytests/test_cache.py— full-run-safetytests/test_cache_long_paths.py— full-run-safetytests/test_cache_stale_import_target.py— full-run-safetytests/test_cache_windows_case_4925.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_cargo_missing_manifest.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_only_decl_ids_4904.py— full-run-safetytests/test_case_sensitive_resolution.py— full-run-safety- … and 544 more
⚠️ non-code file(s) changed (graphify/skill-agents.md,graphify/skill-aider.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
⚠️ changed code file(s) with no mapped test (graphify/skill-agents.md,graphify/skill-aider.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
CHANGELOG.md§ 0.9.22 (2026-07-20) (lines 1388-1397): references changed symbols_env
What does this PR do?
Fixes the six problems reported in #5340, all in the skill templates under
tools/skillgen/fragments/. The generated skills are regenerated withtools.skillgen --bless.references/shared/extraction-spec.mdtells subagents to report a PDF's coverage in pages. The B3.5 check incore/core.mdcounted lines, so a fully read PDF was marked partial and re-extracted on every update. A PDF is now compared with its page count. The count comes frompypdf, and only for files within the same 50 MiB capdetectuses. If pypdf is missing, the file is too large or unreadable, or the count is unknown, the PDF is not checked at all. The warning says "pages" for PDFs. Non-PDF files behave as before.drop_unreported_chunk_sources()and the B3 merge also parse every chunk, so a plaincontinuewould have crashed them on the same file. Its files stay uncached and unstamped, so the next--updateretries them. B3.5 also no longer prints "Coverage OK" right after skipping a chunk.--updateno longer merges a stale.graphify_extract.json. The deletions-only branch writes an empty extraction onlyif [ ! -f graphify-out/.graphify_extract.json ]. Nothing in that run writes the file earlier, so one found there was left by a run that never reached Step 9's cleanup. Merging it brings back that run's nodes. And becausebuild_mergelets a re-extracted source win over a deleted one, a deleted file it still names is never pruned.--updatenow unlinks the file at the start, next to the existing.graphify_partial.txtcleanup.explainfallback lists callers on directed graphs.G.neighbors()returns only successors on aDiGraph, so graphs built with--directedshowed no callers or importers. WhenG.is_directed(), predecessors are listed too, with the arrow reversed (<--rel--). Undirected output is unchanged. This applies toreferences/query/default.mdand the aider/devin copies._score_query/_compute_idffromserve.pyfaithfully is not a small change, so the sentence is corrected instead.hooks.mdwording now matchesinstall.py. It saidgraphify claude installwrites to the localCLAUDE.md._claude_instructions_filewrites toAGENTS.mdwhen the project has one and noCLAUDE.md-style file, because a newCLAUDE.mdwould hideAGENTS.mdfrom Claude Code. The sentence now says so. This applies toreferences/shared/hooks.mdand the aider copy.tools/skillgen/gen.pyadds two sanctioned-diff predicates,_is_explain_direction_fix_lineand_is_claude_install_agents_md_line. They follow the existing pattern, so the aider/devin monolith round trip accepts exactly the changed lines.Type of change
Verification & Invariants
These invariants are protected:
--update.explainshows every edge of the node.No persisted state format changes. The only file-system effects are:
--updateremoves a leftover.graphify_extract.jsonbefore detecting changes.Both are scratch files that Step 9 already deletes.
New tests:
tests/test_skillgen.py:tests/test_query_fallback_edges.py:explainlists incoming edges forDiGraphandMultiDiGraphin all three fragments; undirected output is unchanged.tests/test_update_deletions_only_stale_extract.py(new): runs the rendered--updateblocks with bash on a built project that has one deleted doc and a leftover extraction. It uses the same skip gate astest_update_partial_manifest.py(bash and cat on a POSIX host).How was this tested?
Run on Windows 10, Python 3.12, in a venv with the project's dev extras. The branch is based on
340268a. It merges cleanly with currentv8(2a296fa, 0.9.113), and no file is touched by both sides.Graphify-specific checklist
uv run python -m tools.skillgen --bless) when changing their source fragments.Known follow-ups (not fixed here, to keep this PR to #5340)
explainfallback draws every edge as outgoing (--rel-->). It ignores the_srcdirection thatload_node_link_graph/ the CLI use. The new undirected test pins the current output, so it does not endorse it.hooks.mduninstall: thegraphify claude uninstall # remove the sectioncomment doesn't mention the AGENTS.md case..graphify_extract.jsonguard. Onlyreferences/shared/update.mdis changed here.Closes #5340.
🤖 Generated with Claude Code