Skip to content

fix(skill): coverage, chunk-parse, stale-extract and directed-explain bugs in skill templates (#5340) - #5351

Open
pawanshukla500 wants to merge 1 commit into
Graphify-Labs:v8from
pawanshukla500:fix/skill-template-logic-5340
Open

pawanshukla500 wants to merge 1 commit into
Graphify-Labs:v8from
pawanshukla500:fix/skill-template-logic-5340

Conversation

@pawanshukla500

Copy link
Copy Markdown

What does this PR do?

Fixes the six problems reported in #5340, all in the skill templates under tools/skillgen/fragments/. The generated skills are regenerated with tools.skillgen --bless.

  1. PDF coverage is now checked in pages, not newline bytes. references/shared/extraction-spec.md tells subagents to report a PDF's coverage in pages. The B3.5 check in core/core.md counted lines, so a fully read PDF was marked partial and re-extracted on every update. A PDF is now compared with its page count. The count comes from pypdf, and only for files within the same 50 MiB cap detect uses. If pypdf is missing, the file is too large or unreadable, or the count is unknown, the PDF is not checked at all. The warning says "pages" for PDFs. Non-PDF files behave as before.
  2. One invalid chunk no longer aborts B3.5. Step B3 already says an invalid chunk is skipped with a warning, but the coverage loop parsed every chunk with no guard. A chunk that is not valid JSON, or not a JSON object, now gets a warning and is removed. drop_unreported_chunk_sources() and the B3 merge also parse every chunk, so a plain continue would have crashed them on the same file. Its files stay uncached and unstamped, so the next --update retries them. B3.5 also no longer prints "Coverage OK" right after skipping a chunk.
  3. A deletions-only --update no longer merges a stale .graphify_extract.json. The deletions-only branch writes an empty extraction only if [ ! -f graphify-out/.graphify_extract.json ]. Nothing in that run writes the file earlier, so one found there was left by a run that never reached Step 9's cleanup. Merging it brings back that run's nodes. And because build_merge lets a re-extracted source win over a deleted one, a deleted file it still names is never pruned. --update now unlinks the file at the start, next to the existing .graphify_partial.txt cleanup.
  4. The inline explain fallback lists callers on directed graphs. G.neighbors() returns only successors on a DiGraph, so graphs built with --directed showed no callers or importers. When G.is_directed(), predecessors are listed too, with the arrow reversed (<--rel--). Undirected output is unchanged. This applies to references/query/default.md and the aider/devin copies.
  5. The query reference now describes the inline fallback correctly. It said the fallback "matches the same way" as the CLI (substring + IDF), but the fallback counts unweighted substring hits. Copying _score_query / _compute_idf from serve.py faithfully is not a small change, so the sentence is corrected instead.
  6. The hooks.md wording now matches install.py. It said graphify claude install writes to the local CLAUDE.md. _claude_instructions_file writes to AGENTS.md when the project has one and no CLAUDE.md-style file, because a new CLAUDE.md would hide AGENTS.md from Claude Code. The sentence now says so. This applies to references/shared/hooks.md and the aider copy.

tools/skillgen/gen.py adds two sanctioned-diff predicates, _is_explain_direction_fix_line and _is_claude_install_agents_md_line. They follow the existing pattern, so the aider/devin monolith round trip accepts exactly the changed lines.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Tests or CI
  • Refactor
  • Security fix

Verification & Invariants

These invariants are protected:

  • Step B3's rule that an invalid chunk is skipped and never fatal.
  • A file is only judged "partly read" against the unit its subagent reported.
  • A deleted file's nodes are pruned on --update.
  • explain shows every edge of the node.
  • Docs match code.

No persisted state format changes. The only file-system effects are:

  • B3.5 now removes an invalid chunk file.
  • --update removes a leftover .graphify_extract.json before detecting changes.

Both are scratch files that Step 9 already deletes.

  • Read the CONTRIBUTING.md guide.
  • Reproduced the issue and identified the invariant.
  • Made the smallest fix necessary.
  • Added a regression test (if bug fix) or isolated boundary test.
  • Kept the PR description synchronized with the final implementation.
  • Documented any limitations / unsupported cases explicitly.

New tests:

  • tests/test_skillgen.py:
    • PDF counted in pages
    • PDF skipped when the page count is unknown (pypdf import blocked)
    • invalid chunk skipped, for truncated JSON and for non-object JSON
    • no "Coverage OK" after a skipped chunk
    • the claude-install wording
    • the IDF sentence
  • tests/test_query_fallback_edges.py: explain lists incoming edges for DiGraph and MultiDiGraph in all three fragments; undirected output is unchanged.
  • tests/test_update_deletions_only_stale_extract.py (new): runs the rendered --update blocks with bash on a built project that has one deleted doc and a leftover extraction. It uses the same skip gate as test_update_partial_manifest.py (bash and cat on a POSIX host).

How was this tested?

Run on Windows 10, Python 3.12, in a venv with the project's dev extras. The branch is based on 340268a. It merges cleanly with current v8 (2a296fa, 0.9.113), and no file is touched by both sides.

python -m tools.skillgen --bless && python -m tools.skillgen
python -m tools.skillgen --check                 # check OK: 134 artifact(s) match committed output and expected/
python -m tools.skillgen --monolith-roundtrip    # monolith-roundtrip OK
wc -l graphify/skill.md                          # 839 (budget in test_install_references.py: < 840)

# New regression tests against the ORIGINAL fragments (base 340268a, tests copied in;
# the bash test's win32 skip lifted for this run, Git Bash on PATH):
pytest -k "pdf or invalid_chunk or skipping_a_chunk or claude_install_text or inline_fallback_weights or incoming_edges or unchanged_for_an_undirected or leftover_extraction"
#   -> 14 failed, 3 passed (the 3 passes are the "undirected output unchanged" guards)

# Targeted, on this branch:
pytest tests/test_skillgen.py tests/test_query_fallback_edges.py \
       tests/test_update_deletions_only_stale_extract.py \
       tests/test_install_references.py tests/test_update_partial_manifest.py
#   -> 120 passed, 10 skipped (POSIX-only bash tests skip on Windows)

# Full suite, branch vs base 340268a, same machine and venv:
pytest tests   # branch: 80 failed, 10955 passed, 525 skipped, 1 xfailed
pytest tests   # base:   80 failed, 10939 passed, 524 skipped, 1 xfailed
#   -> identical set of 80 failing test IDs on both trees (no new failures).
#      They are pre-existing on this Windows machine: test_terraform_modules (32),
#      test_terraform (23), test_pascal_singleton_resolution (9),
#      test_ext_tagged_id_clash (7), test_skillgen_input_path_injection (3),
#      test_csharp_verbatim_interpolation (3), and one each in
#      test_source_key_memoization, test_resolve_memoization, test_hook_bash_read.

ruff check tools/skillgen/gen.py tests/test_skillgen.py \
           tests/test_query_fallback_edges.py tests/test_update_deletions_only_stale_extract.py
#   -> All checks passed!

# Not run: pyright; the POSIX-only bash tests on Linux/macOS (left to CI).

Graphify-specific checklist

  • I updated generated skill artifacts (uv run python -m tools.skillgen --bless) when changing their source fragments.
  • I confirmed that AST/structural extraction remains deterministic (no ambient state dependencies like ENV variables).
  • I reviewed changes for security implications (no unsafe interpolation into shell/Python).
  • I confirmed no API keys or local-only graph data are included.
  • (If applicable) I disclosed AI authorship in my commit messages.

Known follow-ups (not fixed here, to keep this PR to #5340)

  • Images: image files still go through the newline-count coverage check, so a fully read image can be flagged as partial. This is the same class of bug as item 1, and it predates this change.
  • Default undirected graphs: on those graphs the explain fallback draws every edge as outgoing (--rel-->). It ignores the _src direction that load_node_link_graph / the CLI use. The new undirected test pins the current output, so it does not endorse it.
  • hooks.md uninstall: the graphify claude uninstall # remove the section comment doesn't mention the AGENTS.md case.
  • aider/devin: the aider/devin monoliths keep their own older copy of the deletions-only .graphify_extract.json guard. Only references/shared/update.md is changed here.

Closes #5340.

🤖 Generated with Claude Code

… bugs in skill templates (Graphify-Labs#5340)

- B3.5 coverage check compares a PDF against its page count (pypdf, within
  the 50 MiB parse cap) instead of its newline count, and skips the check
  when the page count is unknown; the warning says pages for PDFs.
- B3.5 skips an invalid or non-object chunk with a warning instead of
  aborting (as Step B3 already says), removes it so later steps never parse
  it, and no longer prints "Coverage OK" after skipping one.
- --update deletes a leftover .graphify_extract.json up front, so a
  deletions-only run merges an empty extraction instead of an earlier run's
  stale one (which also blocked pruning of deleted files).
- The inline explain fallback lists predecessors too on directed graphs;
  undirected output is unchanged.
- query reference: the inline fallback is documented as unweighted (no IDF),
  matching the code.
- hooks reference: `graphify claude install` writes to AGENTS.md when the
  project has one and no CLAUDE.md-style file, matching install.py.

Fragments edited; generated skills regenerated with tools.skillgen --bless;
gen.py sanctions the two changed monolith line classes. Regression tests
added for each item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Thanks for the pull request, @pawanshukla500. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.11 and 3.13, and that the change keeps extraction deterministic.

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify code review View in Graphify

Review completed · 2a7c1a7334c2

Warning

Merge confidence: 3/5 · Review before merging · 🟠 High impact
1 P2, 1 P3 bugs · 9 possible issues not confirmed

Bugs (2)

P2 · Iterate over every edge-attribute mapping in G[neighbor][nid]: the inline directed fallback prints only the first incoming relationship when a predecessor has parallel edges. Inspect source

Detailed review · checked against the code

Directed explanations show only one incoming relationship per neighbor

When a directed graph has multiple relationships from the same node into the explained node, the fallback prints just one of them, hiding the others.

Expected: The explanation displays every incoming relationship from that predecessor.
Actually: The fallback selects and prints only the first edge’s attributes for that predecessor.

In the inline directed-graph fallback, incoming links are grouped by predecessor: G.predecessors(nid) visits a predecessor once, and each (neighbor, G[neighbor][nid], ...) stores that pair’s edge-attribute mapping. The loop then takes only next(iter(_raw.values()), {}) and prints one relationship. Thus, when one predecessor has parallel incoming edges, only one is displayed. The base fallback did not include incoming links; this incoming-edge path was added in the diff, so the behavior is introduced for the affected path.

  • Independent source review: The main CLI’s graphify explain code iterates each edge returned by edge_datas for successors and predecessors, so it handles parallel edges there. That is a separate route: the documented inline fallback used when the CLI is unavailable still selects one edge from the mapping. The inspected skill-generation test checks that the explain section exists, but does not exercise parallel incoming edges.

P3 · Skip the self-loop when adding directed predecessors in default.md: the inline fallback prints the same connection twice, with opposite arrow directions. Inspect source

Detailed review · checked against the code

Directed explanations print self-links twice

When a directed graph has a self-link on the explained node, the fallback prints that connection twice, with opposite arrow directions.

Expected: The self-loop is shown once as a connection of the explained node.
Actually: The successor and predecessor lists each include the self-loop, so the same edge is printed twice with opposite arrow directions.

In the inline explain fallback, the outgoing list uses G.neighbors(nid), and for directed graphs the code then appends G.predecessors(nid). A directed self-loop makes nid appear in both lists, so the same edge data is printed once with the outgoing arrow and once with the incoming arrow. The fragment is assigned as the query reference for generated platforms; the issue affects that inline fallback when used.

  • Independent source review: Reviewed the related fallback tests: they cover directed incoming edges and undirected graphs, but no self-loop case. Neither the fallback code nor the inspected tests contain a check that deduplicates a self-loop.

Not confirmed (9 possible issues)

Leads worth a look; we didn't double-check these. They don't change the score.

Not confirmed · Hosts whose names contain a hyphen get no case in the runbook test, so their update.md is never run Inspect source

Detailed review · not confirmed

The case collector only accepts rendered paths whose host segment is word characters, so a host with a dash in its name is silently dropped even though the docstring says every host is covered.

Line 72 uses re.fullmatch with a word-character class between 'graphify/skills/' and '/references/update.md', and that class does not match a hyphen. Line 73 then continues on no match. The only guard, the assertion at line 81, fires only when no host matches at all, so one matching host is enough to pass.

  • The platform names were not visible in this change, so whether any current host name contains a hyphen is unchecked.

  • Found by the per-file sweep; not double-checked.

Not confirmed · The test drops every PATH directory that holds a graphify entry, which can also remove the directory holding the interpreter Inspect source

Detailed review · not confirmed

_env removes whole PATH directories that contain a graphify file or folder, not just the graphify launcher, so unrelated tools in those directories disappear for the runbook blocks.

Lines 89-90 keep or drop each PATH entry based on not (Path(p) / 'graphify').exists(), which drops the whole directory. An empty PATH entry resolves to the current directory, so a graphify entry there also removes it. The comment at lines 86-87 only asks to keep graphify off PATH, which this does more broadly.

  • Whether the blocks call a bare python3 or only the recorded interpreter was not checked for every host.

  • Found by the per-file sweep; not double-checked.

Not confirmed · If a runbook block spells the interpreter lookup differently, the test runs it with whatever python is first on PATH Inspect source

Detailed review · not confirmed

_bash only replaces the exact text $(cat graphify-out/.graphify_python). Any other spelling of the interpreter is left alone and resolved from PATH.

Lines 98-99 call str.replace with one literal and never check that a replacement happened. The comment at lines 96-97 assumes the blocks use the unquoted form, and nothing enforces that assumption.

  • The current rendered blocks were not inspected for alternate spellings of the interpreter lookup.

  • Found by the per-file sweep; not double-checked.

Not confirmed · The check for the CLAUDE.md-to-AGENTS.md note only inspects texts that already contain the claude sentence, so differently worded texts are Inspect source

Detailed review · not confirmed

The test says every rendered text describing graphify claude install must name the AGENTS.md it writes, but the loop only runs over artifacts that already contain the exact CLAUDE.md sentence. A rendered description of the same command with different wording passes the test without any check.

Line 986 builds the list with [a for a in gen.render_all(...) if sentence in a.content], so the filter itself decides which artifacts are checked. The loop at lines 989-991 only covers that filtered list. The docstring's promise that every rendered text describing the command says so is therefore not enforced for any text that drifts from the exact sentence.

  • Only the exact sentence 'This writes a ## graphify section to the local CLAUDE.md' is recognised as describing the command.

  • The rendered artifacts from gen.render_all were not inspected here, so it is unknown whether any other text describes the command.

  • Found by the per-file sweep; not double-checked.

Not confirmed · The test for an unknown PDF page count only blocks the pypdf import, so the unreadable-PDF case it names is never exercised Inspect source

Detailed review · not confirmed

The test is named for a PDF whose page count is unknown and its docstring also covers an unreadable PDF. It only sets pypdf to None, so the import fails and the file is never parsed. The case where pypdf is present but cannot read the file is not tested.

Line 1356 sets sys.modules['pypdf'] to None, so the import in the coverage body fails before any PDF is read. The fake PDF written at lines 1362-1363 is never parsed by pypdf, so the branch that handles a parse failure after a successful import is untested. The docstring at lines 1354-1355 says the unreadable case is covered, which the test does not show.

  • The coverage step body was not visible, so the exact exception handling for an unreadable PDF could not be confirmed.

  • Found by the per-file sweep; not double-checked.

Not confirmed · The AGENTS.md redirect sentence is required in every such text, but the test cannot tell whether the text states the condition it applies un Inspect source

Detailed review · not confirmed

The docstring says the redirect to AGENTS.md applies only when the project has an AGENTS.md and no CLAUDE.md-style file. The assertion only checks that a fixed sentence appears in each text. It does not check that the sentence is tied to that condition.

Lines 990-991 use substring checks only. The docstring makes a conditional claim about install._claude_instructions_file, but no assertion verifies the condition. The check passes on the fixed wording alone.

  • The surrounding wording of the rendered texts was not visible, so the condition may be stated nearby even though the test does not check it.

  • Found by the per-file sweep; not double-checked.

Not confirmed · Test helper unescapes every backslash-quote in the extracted fallback script, so the test may run code that differs from the shipped fragmen Inspect source

Detailed review · not confirmed

The _run_explain helper on line 222 removes every backslash followed by a double quote from the whole extracted script before executing it. It is meant to undo markdown escaping, but it cannot tell escaping apart from real code, so the test can exercise a different script than the one shipped in the fragment.

Line 222 applies script.replace(chr(92) + '"', '"') to the entire extracted script with no check for markdown escaping. The helper exists to run the inline fallback copies in the query reference, aider and devin unchanged, so any unescaping that alters real code undermines the three-way consistency these tests are meant to check.

  • The fragment text around the inline explain fallback was not inspected, so it is unknown whether backslash-quote sequences occur there.

  • FALLBACK_OPEN, FALLBACK_END and QUERY_FRAGMENT are defined outside the shown range and were not read.

  • Found by the per-file sweep; not double-checked.

Not confirmed · The explain direction allowlist accepts exact lines anywhere in the monolith, so unrelated edits can slip past the round-trip guard Inspect source

Detailed review · not confirmed

_is_explain_direction_fix_line matches whole stripped lines such as if G.is_directed(): and for neighbor in G.neighbors(nid): without checking where they appear. Any identical line added elsewhere in the monolith is treated as the sanctioned #​5340 fix.

The predicate at lines 1483-1496 returns line.strip() in {...} with no context. _is_sanctioned_monolith_diff at line 1566 applies it to every added and removed line in the file, so the allowlist is global rather than limited to the explain fallback block.

  • I did not inspect the monolith body outside the diff, so I cannot tell whether duplicate matching lines exist today.

  • This assumes the round-trip check is the only guard for these monolith lines.

  • Found by the per-file sweep; not double-checked.

Not confirmed · The monolith_roundtrip docstring omits the new #​5340 explain and AGENTS.md sanctioned changes Inspect source

Detailed review · not confirmed

The docstring lists the sanctioned change classes, but it does not mention the directed explain fix or the claude install AGENTS.md sentence that were added to _SANCTIONED_MONOLITH_DIFFS in this PR.

Lines 1575-1579 enumerate the predicates by name and PR number, and lines 1559-1560 add two predicates that the enumeration does not describe.

  • This is documentation drift only and does not change runtime behavior.

  • Reference checks do not establish safe deletion for dynamic or external consumers.

  • Found by the per-file sweep; not double-checked.

What this review covered
  • We didn't review 116 files that aren't source code, such as docs, config, images, or generated or third-party code.
  • 4 of 4 changed files inspected. This is scope, not a correctness guarantee.
Code graph and static checks · 1739 functions depend on this change

Impact & health

Impact — 1739 functions depend on the 1717 functions this change touches.

Health — grade A; 8 existing hotspot(s) in the area this change touches (pre-existing, not introduced here):

  • render() — 16 callers, 5 callees (high)
  • audit_coverage() — 8 callers, 6 callees (high)
  • main() — 3 callers, 11 callees (medium)
  • _unreported_fixture() — 6 callers, 4 callees (medium)
  • monolith_roundtrip() — 3 callers, 6 callees (medium)
  • test_an_unreported_file_is_retried_instead_of_cached_as_complete() — 0 callers, 6 callees (medium)
  • test_audit_catches_a_dropped_non_allowlisted_heading() — 0 callers, 6 callees (medium)
  • test_update_preserves_undispatched_semantic_stamps() — 0 callers, 6 callees (medium)

Formal checks — 1739 functions this change can affect weren't formally checked in this review. Formal checks are informational only.

Gate & verification

✅ PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 1739 function(s) in the blast radius were not formally verified this run

Test selection

594 of 594 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety (noncode-change, uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/pi.test.ts — full-run-safety
  • tests/test_4874_jvm_resolution.py — full-run-safety
  • tests/test_4909_reference_binding_scale.py — full-run-safety
  • tests/test_4978_reconcile_cost.py — full-run-safety
  • tests/test_4992_ts_import_types.py — full-run-safety
  • tests/test_5072_splittable_under_cap_retry.py — full-run-safety
  • tests/test_affected_cli.py — full-run-safety
  • tests/test_affected_defines.py — full-run-safety
  • tests/test_affected_member_seed.py — full-run-safety
  • tests/test_affected_overrides.py — full-run-safety
  • tests/test_affected_path_file_node.py — full-run-safety
  • tests/test_affected_seed_reference_forms.py — full-run-safety
  • tests/test_agents_platform.py — full-run-safety
  • tests/test_alias_folding.py — full-run-safety
  • tests/test_analyze.py — full-run-safety
  • tests/test_anthropic_custom_endpoint.py — full-run-safety
  • tests/test_antigravity_install.py — full-run-safety
  • tests/test_apm_fallback_version.py — full-run-safety
  • tests/test_architecture_doc.py — full-run-safety
  • tests/test_astro_extraction.py — full-run-safety
  • tests/test_astro_import_ids.py — full-run-safety
  • tests/test_atomic_canvas_export.py — full-run-safety
  • tests/test_atomic_version_stamp.py — full-run-safety
  • tests/test_atomic_writes.py — full-run-safety
  • tests/test_backend_env_isolation.py — full-run-safety
  • tests/test_backend_extras.py — full-run-safety
  • tests/test_backend_opt_in.py — full-run-safety
  • tests/test_benchmark.py — full-run-safety
  • tests/test_benchmark_raw_graph.py — full-run-safety
  • tests/test_benchmark_real_query.py — full-run-safety
  • tests/test_binary_media_words.py — full-run-safety
  • tests/test_blade_extractor.py — full-run-safety
  • tests/test_build.py — full-run-safety
  • tests/test_build_located_semantic_identity.py — full-run-safety
  • tests/test_build_merge_dedup_scope.py — full-run-safety
  • tests/test_build_merge_ghost_fold_4955.py — full-run-safety
  • tests/test_build_merge_hyperedges_and_prune.py — full-run-safety
  • tests/test_build_merge_shrink_guard.py — full-run-safety
  • tests/test_builtin_global_type_refs.py — full-run-safety
  • tests/test_c_include_resolution.py — full-run-safety
  • tests/test_cache.py — full-run-safety
  • tests/test_cache_long_paths.py — full-run-safety
  • tests/test_cache_stale_import_target.py — full-run-safety
  • tests/test_cache_windows_case_4925.py — full-run-safety
  • tests/test_callflow_html.py — full-run-safety
  • tests/test_cargo_introspect.py — full-run-safety
  • tests/test_cargo_missing_manifest.py — full-run-safety
  • tests/test_carried_hyperedge_remap.py — full-run-safety
  • tests/test_case_only_decl_ids_4904.py — full-run-safety
  • tests/test_case_sensitive_resolution.py — full-run-safety
  • … and 544 more

⚠️ non-code file(s) changed (graphify/skill-agents.md, graphify/skill-aider.md, graphify/skill-amp.md, graphify/skill-claw.md, graphify/skill-codex.md …) → running the full suite for safety (a code graph can't see config/fixture/data deps)

⚠️ changed code file(s) with no mapped test (graphify/skill-agents.md, graphify/skill-aider.md, graphify/skill-amp.md, graphify/skill-claw.md, graphify/skill-codex.md …) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Docs that may be stale (advisory)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Skill templates: five logic bugs in the inline Python snippets, plus an outdated note in hooks.md (0.9.112)

1 participant