Skip to content

perf(watch): rebase each preserved source path once per update - #5322

Open
Max17190 wants to merge 1 commit into
Graphify-Labs:v8from
Max17190:perf/cache-preserved-source-paths
Open

Max17190 wants to merge 1 commit into
Graphify-Labs:v8from
Max17190:perf/cache-preserved-source-paths

Conversation

@Max17190

Copy link
Copy Markdown
Contributor

Why

After a one-file change, graphify update still spends seconds on path bookkeeping for the whole graph. _reconcile_existing_graph calls rebase_preserved for every preserved node, edge and hyperedge, and the resolve context calls absolute_identity for every stored node and edge. Each call builds Path objects and walks parent directories, though the answer depends only on the stored source_file and roots fixed in __init__. Guava has 337k such items from 3.3k files, and its deep Java package paths make each call slower.

1d38e84 cached identity() and in_watch_root() for the same reason. These two calls were left uncached.

one-file graphify update reconcile on v8 this PR whole update on v8 this PR
Django, 188k items 3.12s 0.98s 31.1s 28.2s
Guava, 337k items 8.48s 1.96s 51.4s 43.8s

Median of 3 runs on an Apple M5 with Python 3.13. The whole-update saving is larger than the reconcile saving because the resolve context's absolute_identity calls sit outside _reconcile_existing_graph.

Summary

  • rebase_preserved works out the new source_file once per distinct stored value and reuses it. The rewrite logic moves unchanged into _rebased_uncached.
  • absolute_identity caches per (source_file, root), the same way identity() does.
  • Each _StoredSourcePaths lives for one update, and its roots are resolved before the update starts, so a cached answer cannot go stale.

Test plan

uv run pytest tests/test_watch_identity_cache.py -q   # 4 passed, 2 of them new
uv run pytest tests/ -q                               # 11581 passed, 134 skipped, 1 xfailed on Python 3.11 and 3.13
uv run ruff check .                                   # passes
uv run pyright graphify/watch.py tests/test_watch_identity_cache.py  # the same 11 errors as v8
uv run python -m tools.skillgen --check               # 134 artifacts match

The first new test counts normalizations and containment checks across 51 passes over three files. On v8 it fails with 210 and 153, and with this change both stop at 10 and 3 after the first pass. The second test pins the rewritten source_file for paths inside the root, ./ and .. segments, backslashes, a path outside the root, empty and missing values, and a watched root outside the project root. Its expected values pass on v8 too.

End to end on gin, ripgrep, axum, clap, excalidraw, Django and Guava, I ran a full build, three one-file edits with graphify update, and an update that deleted one file and added another, on v8 and on this branch at the same path. graph.json, graph.html and GRAPH_REPORT.md came out byte-identical after every step. manifest.json and the build state differ only in timestamps.

  • Read CONTRIBUTING.md
  • No generated skill files changed
  • Extraction unchanged and deterministic
  • No shell or Python interpolation, secrets, or local graph data
  • AI help disclosed in the commit trailer

An incremental update calls _StoredSourcePaths.rebase_preserved for every
preserved node, edge and hyperedge, and absolute_identity for every stored
node and edge of the resolve context. Each call builds Path objects and
walks parent directories, though the answer depends only on the stored
source_file and roots fixed in __init__. Guava has 337k such items from
3.3k files. Cache both per file, as 1d38e84 did for identity().

A one-file update on Guava goes from 51.4s to 43.8s, with the reconcile
step from 8.48s to 1.96s. graph.json, graph.html and GRAPH_REPORT.md are
byte-identical on seven repos after a full build, one-file edits, and a
delete plus add.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Max17190
Max17190 requested a review from safishamsi as a code owner November 18, 2026 18:36
@github-actions github-actions Bot added the contributor Pull request from a returning community contributor label Nov 18, 2026
@github-actions

Copy link
Copy Markdown

Thanks for the pull request, @Max17190. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.11 and 3.13, and that the change keeps extraction deterministic.

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify code review View in Graphify

Review completed · 5044b76025e1

Tip

Merge confidence: 5/5 · Looks safe to merge · 🟠 High impact
No issues reported

The change caches absolute-path identity and preserved-source rebasing results by source path.

Full summary

The change caches absolute-path identity and preserved-source rebasing results by source path. It adds tests for repeated-path cache reuse and unchanged rebasing results.

Code graph and static checks · 782 functions depend on this change

Impact & health

Impact — 782 functions depend on the 144 functions this change touches.

Health — grade A; 10 existing hotspot(s) in the area this change touches (pre-existing, not introduced here):

  • _rebuild_code() — 281 callers, 76 callees (high)
  • dispatch_command() — 8 callers, 156 callees (high)
  • main() — 128 callers, 3 callees (high)
  • watch() — 8 callers, 15 callees (high)
  • _reconcile_existing_graph() — 6 callers, 16 callees (high)
  • _check_shrink() — 18 callers, 5 callees (high)
  • _run_cli() — 11 callers, 8 callees (high)
  • _reconcile_graph_html() — 8 callers, 5 callees (high)
  • …and 2 more

Formal checks — 782 functions this change can affect weren't formally checked in this review. Formal checks are informational only.

Gate & verification

✅ PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 506 function(s) in the blast radius were not formally verified this run

Test selection

49 of 593 test file(s) selected (8%) via static blast radius.

  • tests/test_4978_reconcile_cost.py — impact
  • tests/test_build.py — impact
  • tests/test_cache_stale_import_target.py — impact
  • tests/test_code_only_scope_4923.py — impact
  • tests/test_compact_graph_json.py — impact
  • tests/test_cpp_forward_declarations.py — impact
  • tests/test_cpp_incremental_type_resolution.py — impact
  • tests/test_cpp_member_declaration_identity.py — impact
  • tests/test_csharp_method_owner_labels.py — impact
  • tests/test_csharp_object_creation.py — impact
  • tests/test_elixir_import_resolution.py — impact
  • tests/test_elixir_qualified_calls.py — impact
  • tests/test_elixir_unqualified_call_scope.py — impact
  • tests/test_export_direction.py — impact
  • tests/test_external_stub_endpoints.py — impact
  • tests/test_go_receiver_member_calls.py — impact
  • tests/test_incremental_unscanned_placeholders.py — impact
  • tests/test_java_member_calls.py — impact
  • tests/test_java_nested_type_identity.py — impact
  • tests/test_kotlin_grammar.py — impact
  • tests/test_labeling.py — impact
  • tests/test_language_override_lifecycle.py — impact
  • tests/test_linked_worktree_reconcile.py — impact
  • tests/test_lost_saved_semantic_nodes.py — impact
  • tests/test_markdown_code_spans.py — impact
  • tests/test_markdown_sections.py — impact
  • tests/test_no_cluster_undeclared_endpoints.py — impact
  • tests/test_pr10_watch_overrides.py — impact
  • tests/test_python_inherited_method_calls.py — impact
  • tests/test_python_injected_factory_calls.py — impact
  • tests/test_python_property_reads.py — impact
  • tests/test_python_pytest_fixture_receivers.py — impact
  • tests/test_qualified_type_competition.py — impact
  • tests/test_rust_use_trees.py — impact
  • tests/test_semantic_origin_stamp.py — impact
  • tests/test_skill_persisted_scan_options.py — impact
  • tests/test_solidity_inherited_calls.py — impact
  • tests/test_terraform_modules.py — impact
  • tests/test_undirected_two_file_import_cycle.py — impact
  • tests/test_update_incremental.py — impact
  • tests/test_update_js_resolution_inputs.py — impact
  • tests/test_update_lock_contention.py — impact
  • tests/test_update_missing_graph.py — impact
  • tests/test_update_no_change_gate.py — impact
  • tests/test_watch.py — impact
  • tests/test_watch_identity_cache.py — impact, changed-test
  • tests/test_watch_inotify_overflow.py — impact
  • tests/test_watch_manifest_location.py — impact
  • tests/test_watch_pruned.py — impact

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor Pull request from a returning community contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant