Skip to content

Log when API Security sampling is skipped for lack of a post-processing permit - #12954

Draft
jandro996 wants to merge 3 commits into
masterfrom
APPSEC-71153
Draft

jandro996 wants to merge 3 commits into
masterfrom
APPSEC-71153

Conversation

@jandro996

@jandro996 jandro996 commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

What Does This Do

  • Logs at debug level when ApiSecuritySamplerImpl.preSampleRequest declines a request because no post-processing permit is available (counter.tryAcquire() fails). The line includes the permit pool size (MAX_POST_PROCESSING_TASKS) and the endpoint hash.
  • No behavior change: the permit pool stays at 4 and the preSampleRequest contract is untouched.

Motivation

The system-test tests/appsec/api_security/test_schemas.py::Test_Schema_Request_Json_Body::test_request_method is flaky on Java + vertx4 (APPSEC_API_SECURITY): the failing span has no _dd.appsec.s.* tag at all. Timing analysis of a failing run points to the permit pool being exhausted: permits are taken on the request thread and only released by AppSecSpanPostProcessor on the TraceProcessingWorker thread, which also sends trace payloads synchronously. When more than 4 sampled requests end during a flush, the next one is silently not sampled.

Today that decision leaves no trace in the logs, so the hypothesis cannot be confirmed from a debug run. With this log, a run with DD_TRACE_DEBUG=true shows exactly which span lost its permit: the new line is emitted on the same thread right after GatewayBridge.maybeSampleForApiSecurity logs Checking API Security for end of request handler on span: {spanId}.

Additional Notes

  • We considered making the pool size configurable through DD_API_SECURITY_MAX_CONCURRENT_REQUESTS (the APPSEC_API_SECURITY scenario sets it to 50), but no tracer implements that key today: dotnet only supported it in v2.46 to v2.49 and removed it in v2.50, and it is not in any tracer's supported configurations. API Security sampling is best effort, so the flake will be handled on the system-tests side.
  • The log does not use AgentTracer.activeSpan() for trace/span ids: request end can run outside the server span scope (Lambda, async servers), so the active span may be null or unrelated. The preceding GatewayBridge line carries the correct span id from the request-ended callback.
  • The log is not rate limited on purpose: it is debug only, the same request path already emits a per-request debug line in GatewayBridge.maybeSampleForApiSecurity, and limiting it would hide the bursts this is meant to surface.

Contributor Checklist

Jira ticket: APPSEC-71153

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

- Match logSamplingDecision if/else style in logNoPermitAvailable
@jandro996 jandro996 added comp: asm waf Application Security Management (WAF) type: refactoring tag: no release notes Changes to exclude from release notes labels Oct 20, 2026
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T09:41:36.674686Z 6c875d1 Manual request
🔒 Security Review ✅ Completed 2026-10-20T09:42:11.448615Z 6c875d1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6c875d1b2f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6c875d1b2f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.96 s 13.86 s [+0.0%; +1.3%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.87 s 12.92 s [-1.1%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 16.95 s 16.24 s [+0.1%; +8.7%] (maybe worse)
startup:petclinic:iast:Agent 16.89 s 16.95 s [-1.2%; +0.6%] (no difference)
startup:petclinic:profiling:Agent 16.57 s 16.58 s [-1.1%; +0.9%] (no difference)
startup:petclinic:sca:Agent 16.96 s 16.72 s [+0.5%; +2.4%] (maybe worse)
startup:petclinic:tracing:Agent 16.07 s 15.72 s [-2.1%; +6.7%] (no difference)

Commit: 6d1a0a2b · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant