Repository navigation
Show scan scope, budget semantics, and expected output before paid work begins #25
Description
Summary
Operators need enough information before a scan to choose a useful scope and budget, and to understand what work will be retained if the scan stops early.
The initial report described reaching a $20 budget without useful security output. Later reports showed that completed scans can produce valuable findings, but file count alone does not predict cost and the first durable results can arrive late. The remaining request is practical pre-spend planning, rather than a guarantee that every scan will finish within a predicted price.
What has changed
- fix(scan): safely inventory explicitly scoped source files #88 and Run Standard scans with focused investigators #313 improved authoritative scope inventory before model work starts.
- fix(sdk): probe the Codex sandbox before starting a billed scan #1084 added a Linux/macOS sandbox-startup check before billed work.
- Current cost reporting explains API-price estimates and their limitations. Those estimates do not measure remaining ChatGPT subscription allowance.
These changes address specific failures reported in the discussion. They do not yet provide the full work/output plan requested here.
Remaining operator questions
- What will be scanned? Show the resolved target and authoritative inventory, including relevant first-party, generated, and dependency content.
- What work is planned? Explain the selected model/effort, scan mode, worker limits, and any known orchestration assumptions that materially affect cost.
- What does the budget mean? State the pricing basis and uncertainty, what the limiter measures, and why in-flight requests can finish above the limit. Any predicted range must be clearly labeled as an estimate.
- Can the environment run the workflow? Surface available runtime/authentication/sandbox capability results before paid work, without implying that an advisory access check can guarantee every later request will succeed.
- What will survive an early stop? Identify the first useful durable milestone, what findings/review evidence can be retained, and the supported recovery or resume behavior.
Acceptance criteria
- The pre-spend information is consistent with the scope and settings used by the actual scan.
- Available facts, estimates, and unknowns are distinguishable.
- Setup directories and inventory alone are not presented as completed security analysis.
- Interrupted execution retains supported completed work and accurately reports the terminal execution state and remaining scope notes.
A machine-readable ScanPlan was proposed in the discussion. Its exact format and any public CLI/API additions remain design proposals, not requirements to introduce a new command or flag.
Historical evidence and related work
The discussion contains budget-limited runs and later natural completions. Early phase: preflight or zero review counters should not be taken as proof that no worker review occurred; later investigation corrected that interpretation.
Related: #29 for allowance/usage visibility, #70 for truthful progress, #191 for CI artifact-production failures, and #56/#276 for refusal handling.
Activity
brahimhamichan commentedon Jul 29, 2026
Hitting this too, with data that may help isolate it: scan cost appears to be independent of scan scope.
CLI
0.1.1, modelgpt-5.6-sol,--mode standard, macOS, Node 26.5.0, Python 3.14.6, auth via stored ChatGPT sign-in.I ran the same repo three times, varying only scope and reasoning effort:
--path× 5)xhighxhighmedium--max-costinputTokenscachedInputTokenscacheWriteInputTokensoutputTokensprogress.phasepreflightpreflightpreflightfindingCountTwo independent levers, neither of which controls spend:
xhightomediumdid not reduce the burn rate; run 3 simply hit a lower ceiling sooner. Input tokens per dollar were ~1.23M in run 2 and ~1.10M in run 3.All three runs were killed by the cost limiter in
preflightwithworklistRows: 0andclosedRows: 0— file review never started, so no run could produce a finding. The limiter works exactly as documented; the problem is that nothing reaches the phase that generates findings before the budget is gone.Two things stand out:
cacheWriteInputTokens: 0alongside 31.6McachedInputTokens. If nothing is being written to cache, it's worth confirming the cached-input accounting is attributing reads correctly in the cost estimator.The artifacts match the counters: on run 2,
02_discovery/rank_input.jsonlwas written with exactly 419 records, but03_coverage/,04_reconciliation/and05_findings/were all created and left empty. Same on runs 1 and 3.On the original question in this issue — how to budget a scan up front — the practical problem is that
--pathandmodel_reasoning_effortare the two documented levers for controlling spend, and on this evidence neither one does. That leaves no way to buy a smaller scan: the only outcome available at any budget I tried was a full spend with zero findings. A cost estimate from--dry-run, or making the ranking phase's context scale with the selected scope, would each address that directly.Happy to supply the full scan records or rerun with different flags if it would help narrow this down.
createsomethingtoday commentedon Jul 29, 2026
I now have natural-terminal data from two report-only
standardscans usinggpt-5.6-sol/xhigh, with the scope/finalizer fixes in #71. The fixes allow scans to complete, but they confirm that cost is not predictable from source-file count.distfiles)The smaller tracked package was ~38% more expensive. Its 12-session tree spent ~95.5% of input tokens as cached input. Artifact timing was also highly back-loaded: deterministic inventory existed before model start, but the first candidate ledger did not persist until roughly 19 minutes; final findings appeared near 28 minutes and sealing completed at 32:44.
This changes the diagnosis from “the cost cap prevents all value” to “natural-terminal scans can produce substantial value, but neither scope size nor dry-run metadata gives an operator a usable cost estimate, and most spend is repeated cached context across delegated sessions.” Generated artifacts are another material lever: this 12-file package became a 36-row exhaustive inventory because shipped
distoutput is intentionally included.#71 fixes deterministic standard inventory and workbench target binding; it does not address cost estimation or orchestration duplication. The most useful next product controls from these runs would be: (1) a dry-run estimate based on planned session topology plus inventory classes, (2) configurable first-party/generated/dependency inventory policy, and (3) incremental durable candidate/validation checkpoints so a late stop retains more than setup artifacts.
thedaviddias commentedon Jul 29, 2026
Additional reproduction data from CLI
0.1.3(bundled plugin0.1.14) shows the same behavior on both the default model and the README's lower-cost model/effort example.Environment:
24.15.03.14.6I progressively narrowed the scope and then switched from
gpt-5.6-sol/xhightogpt-5.6-terra/high:All four scans recorded:
progress.phase: preflightfindingCount: 0worklistRows: 0closedRows: 0report.md,findings.json, orcoverage.jsonThe most focused run did create deterministic discovery artifacts and six
review_slice_*.txtfiles, but the scan stopped before any candidate or coverage row became durable. Across the four attempts the estimated cost was about $19.48 with no actionable output.The 294-file run is particularly useful evidence that reducing scope and using Terra/high does not currently provide a predictable small scan: it consumed more input and output tokens than the much larger sol/xhigh runs before stopping. The $5 one-app run also ended at $6.08 because in-flight delegated work completed after the configured estimate was crossed.
This reinforces the requests already raised here for:
Happy to provide the four scan IDs or sanitized
scans showoutput if useful.createsomethingtoday commentedon Jul 30, 2026
Extended exact-commit use found additional real project issues, then exposed a separate completion defect: immutable commit/range scans had no snapshotDigest and could not be saved after model work completed. PR #112 adds the deterministic diff binding and regression. This is separate from the scope-inventory fix in PR #71.
mldangelo-oai commentedon Jul 31, 2026
Maintainer reproduction on @openai/codex-security@0.1.4: a supposedly single-file scan using
--path .github/release.yml,gpt-5.6-luna, low effort, and one worker generated anin_scope_files.txtcontaining 186 repository files. The separate rank-input ledger correctly contained one requested file, but the agent followed the oversized inventory and had already consumed approximately $0.20 before I stopped it.This demonstrates that scan cost cannot currently be predicted from requested path scope when the model-generated inventory broadens to the entire repository. The candidate fix is #88, which creates and attests the exact scope inventory before starting Codex or spending model tokens. Nested desktop sandboxing can additionally prevent the agent from writing required artifacts at all; that should fail before model execution rather than surfacing only as a missing manifest afterward.
createsomethingtoday commentedon Aug 12, 2026
Follow-up after PR #88 shipped in
@openai/codex-security@0.1.10: I completed a zero-spend audit of the released/current public source and a synthetic two-file--dry-run. No model or scan request was started.The shipped improvement is important: the exact scoped inventory is now authoritative before Codex starts. The remaining pre-spend gap is that dry run reports configuration (
target, mode, output, auth selection, model, effort, and max cost), but not the exact inventory count/classes, planned worker/session topology, pricing basis or spend range, execution-sandbox capability, or Trusted Access status. In the real path, host runtime/auth/Python/output setup and scan registration happen beforethread.runStreamed(...), while the scan agent's shell/write behavior and Trusted Access signal are still exercised after model work can begin.I propose one host-owned, machine-readable
ScanPlanreceipt before spend:--max-costremains the runtime breaker;granted,not_granted, orunknown) plus fail/warn policy; andThe paired minimum durable checkpoint should bind scan ID, target identity, inventory digest, completed review scope, candidate rows (or an explicit zero-candidate result), and a typed incomplete reason with a safe retry/resume key. Setup directories, helper scripts, or inventory alone are diagnostics, not a security result.
This puts #191's shell/write failure and #276's Trusted Access/refusal failure on the same pre-spend contract instead of adding separate late warnings.
Would maintainers prefer this contract refined here first, or an invited focused PR? Because this repository is a one-way public mirror, I will not open a parallel implementation without that routing.
mldangelo-oai commentedon Aug 15, 2026
Earlier scope-related fixes landed in #88 and #313. Keeping this open for pre-spend planning, including the selected scope, model settings, budget behavior, and expected durable output.
[-]Expensive cost, nothing to show for it.[/-][+]Show scan scope, budget semantics, and expected output before paid work begins[/+]mldangelo-oai commentedon Oct 8, 2026
Related: merged #1084 checks Unix sandbox startup before billed work. That covers one pre-spend failure mode, but not the requested cost estimate or expected-output plan.