diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 05a80700f3..4f3fe0bc45 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -526,11 +526,25 @@ what changed in the server they use: judgment, so it is written by hand, never generated. 7. **`## Thanks for helping us improve`**: one list for the whole release, as the Inspector's release notes have it, crediting the community members - whose issues the listed PRs close, one line per person, most issues first. + whose issues the listed PRs close and whose advisories (`--advisory`) the + release fixes, one line per person, most references first; then the + prototype paragraph, one line per `--thank`. GitHub adds everyone `@`-mentioned in a release body to that release's **Contributors** strip, so the people credited appear there too. +**Collect both kinds of credit the helper cannot find on its own before the +preview**, since the generator writes the whole Thanks section and nothing is +added to the draft by hand: + +- **Every advisory this release fixes**, as `--advisory GHSA-xxxx-yyyy-zzzz`, + once each: the `[GHSA-…]` cards on the board for the milestone's work + (`security-advisory`). Its fix shipped from a private fork with no public + issue, so its reporters are no issue's author. A draft advisory works. +- **Every outside contributor credited on one of the milestone's issues**, as + `--thank "@login (#PR), , for #ISSUE"`, once per line + of the prototype paragraph. The preview lists each candidate (below). + Preview first. The default run prints the notes and creates nothing: ```sh @@ -545,20 +559,34 @@ git fetch origin main && git rev-parse "$MERGE_SHA^{tree}" # must be the tree NOTES=$(mktemp) npm run --silent release:notes -- --milestone "$MILESTONE" --merge-sha "$MERGE_SHA" \ --merge-branch "v2/chore/$N-release-$MILESTONE" --ledger-url "" \ + --advisory GHSA-xxxx-yyyy-zzzz --thank "@login (#PR), , for #ISSUE" \ > "$NOTES"; echo "EXIT=$?" ``` Add `--known-issue ""` once per known issue. The helper reads the manifests at `$MERGE_SHA` from GitHub, so it needs no checkout of that commit. It takes a few minutes: it looks up every listed PR's files and -closing issues, and every reporter's permission. +closing issues, every reporter's permission, and, in a preview, every +comment on the milestone's closed issues. + +**Read the preview before going on**, and its standard error with it: -**Read the preview before going on.** Then create the draft with the same -arguments plus `--draft`: +- **`credit notes on 's closed issues`** lists every credit a + maintainer left on one of them (an issue body or comment saying "credit" or + "thanks to" and naming a non-maintainer), each person marked + `(--thank given)` or `(NO --thank)`. Add a `--thank` for each one marked + `NO` whose prototype ships in this release, and re-run the preview. A note + that only credits a reporter the list already has needs none. +- **`PENDING advisory credit`** names a credit its recipient has not accepted + yet. It is listed, and may still be declined: settle it with the reporter + (`security-advisory`) before publishing, or re-run once they answer. + +Then create the draft with the same arguments plus `--draft`: ```sh npm run --silent release:notes -- --milestone "$MILESTONE" --merge-sha "$MERGE_SHA" \ - --merge-branch "v2/chore/$N-release-$MILESTONE" --ledger-url "" --draft + --merge-branch "v2/chore/$N-release-$MILESTONE" --ledger-url "" \ + --advisory GHSA-xxxx-yyyy-zzzz --thank "@login (#PR), , for #ISSUE" --draft ``` `--draft` creates a **draft** Release tagged `$MILESTONE` at `$MERGE_SHA`. It @@ -582,14 +610,21 @@ The rules the helper applies: - **Maintainers and bots are excluded by permission, not by name.** A maintainer is anyone with `admin`, `maintain` or `write` on the repo. Bots, deleted accounts, PR numbers and other repos' issues are dropped. -- **Outside PRs are credited through their issue, by hand.** This repo +- **An advisory's credits are read from the advisory**, accepted and pending + ones, any credit type. A declined credit, a bot and a maintainer are + dropped. An advisory in `triage` (not accepted), `closed` or `withdrawn` + aborts the run: it has no fix in this release to credit. +- **Outside PRs are credited through their issue, with `--thank`.** This repo closes an outside PR and files an issue for a fix worth keeping, crediting the PR's author in that issue (`issue-triage`, Harvest). The helper credits - the issue's *author*, who is then a maintainer and is excluded. Read the - milestone's issues for such credits and add those people to the draft's - Thanks section by hand before publishing. + the issue's *author*, who is then a maintainer and is excluded, so the + credit comes from a `--thank`, and the preview's discovery list is what + shows which ones are still missing. Discovery only reads notes a maintainer + wrote; a community member offering a fix in a comment is credited when a + maintainer says so on the issue. - **"Addresses", not "fixes."** The credited issues include feature requests. -- **The Thanks section is left out** when no community reporter remains. +- **The Thanks section is left out** when nobody is credited: no community + reporter, no advisory credit and no `--thank`. - **It fails fast.** A failed `gh` call, a GraphQL error, a missing PR or issue, an unknown permission value, a generated line it does not recognize, or a registry answer other than 200 or 404 aborts the run before anything is diff --git a/RELEASING.md b/RELEASING.md index c55aadbc4b..c44cf95773 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -83,7 +83,7 @@ A release ships one **milestone**. The step-by-step procedure, with its commands A milestone that touches both languages and needs an audit fix has all three. They are merged before the next step starts. 3. **The merge PR** takes `v2/main` into `main`. It is a pure merge: its branch is `origin/v2/main` with no commits of its own, and the merged tree equals `origin/v2/main`'s. Its body starts with `Part of #`, not `Closes`, because `main` is the default branch and a closing keyword there would close the issue before anything is published. It is merged with a merge commit, never squashed. 4. **The release ledger** is linked from the merge PR. It records, for the merge branch's tree: `npm run local:gate`; `npm run pack:verify` for each package; each server-facing issue in the milestone exercised through a client; and a targeted probe for each issue with no client surface. The maintainers approve the merge on it. A finding is fixed on `v2/main` and merged in again, never committed to the merge branch. -5. **A maintainer publishes a GitHub Release** targeting the merge commit on `main` (the commit the ledger verified, chosen by SHA rather than by the branch name, which may have moved). [`scripts/release-notes.mjs`](scripts/release-notes.mjs) (`npm run release:notes`) writes the notes, one section per published package with its PRs and a Thanks section for the community members whose issues the release addresses, and with `--draft` creates the draft Release under a tag named for the milestone (`vX.Y.Z`). The maintainer reads the draft and publishes it. The tag is a label, not a version. +5. **A maintainer publishes a GitHub Release** targeting the merge commit on `main` (the commit the ledger verified, chosen by SHA rather than by the branch name, which may have moved). [`scripts/release-notes.mjs`](scripts/release-notes.mjs) (`npm run release:notes`) writes the notes, one section per published package with its PRs and a Thanks section for the community members whose issues and advisories (`--advisory`) the release addresses and whose prototypes ship in it (`--thank`), and with `--draft` creates the draft Release under a tag named for the milestone (`vX.Y.Z`). The maintainer reads the draft and publishes it. The tag is a label, not a version. 6. **The build jobs run, then the maintainer approves the `release` environment deployments.** Each package publishes if its version is not on the registry yet; the rest skip cleanly. 7. **The release issue is closed by hand**, and its card moves to Done. diff --git a/scripts/release-notes.mjs b/scripts/release-notes.mjs index 73ba9d7cba..1a97c992bc 100644 --- a/scripts/release-notes.mjs +++ b/scripts/release-notes.mjs @@ -22,8 +22,10 @@ // ## New Contributors / Full Changelog GitHub's own trailer, verbatim // ## Known issues only when passed in: a maintainer's call // ## Thanks for helping us improve one release-wide list of the community -// members whose issues the PRs close, laid -// out as the Inspector's release notes are +// members whose issues the PRs close and +// whose advisories it fixes, laid out as +// the Inspector's release notes are, then +// the contributors whose prototypes ship // // The PR list is GitHub's generated one (`releases/generate-notes`, what the // UI's button uses) from the latest PUBLISHED Release to the merge commit. A @@ -33,6 +35,21 @@ // GitHub adds everyone `@`-mentioned in a release body to its Contributors // strip, so that section is what puts the reporters there. // +// Two kinds of credit never come from an issue's author, and were once pasted +// in by hand, which is how v2.0.0 published without its Thanks (#5250): +// +// * An advisory's fix ships from its private fork with no public issue, so +// each `--advisory GHSA-…` is read for its credits (a draft works: notes are +// drafted before advisories publish). Declined credits, bots and +// maintainers are dropped; a credit still pending acceptance is listed and +// flagged, since the reporter may yet decline it. +// * An outside PR is closed and its author credited on an issue (Harvest), so +// each `--thank "@login (#ref), "` is a line of the prototype +// paragraph. Which issues carry such a credit is a reading, not a lookup, +// so the preview lists every credit note a maintainer left on the +// milestone's closed issues that names a non-maintainer, marking those no +// `--thank` names yet. +// // FAIL FAST. Every `gh` call and registry request is checked and a failure // throws before anything is created: a permission lookup that errors must // never read as "community" (that could credit a maintainer), a rate-limited @@ -61,8 +78,22 @@ const COMMUNITY_PERMISSIONS = new Set(["triage", "read", "none"]); export const THANKS_LEAD_IN = "This release addresses issues reported by these community members. Thank you for taking the time to file them:"; +export const ADVISORY_LEAD_IN = + "This release addresses vulnerabilities reported by these community members. Thank you for taking the time to report them:"; +export const MIXED_LEAD_IN = + "This release addresses issues and vulnerabilities reported by these community members. Thank you for taking the time to report them:"; +const POLICY_LINK = `[issues, not pull requests](https://github.com/${REPO}/blob/v2/main/CONTRIBUTING.md)`; export const REPOSITORY_HEADING = "## Repository"; +// An advisory id as GitHub mints it, and the advisory states a fix can ship +// from: a triage advisory has not been accepted, and a closed or withdrawn one +// is not a vulnerability this release fixes. +const GHSA_ID = /^GHSA(-[23456789cfghjmpqrvwx]{4}){3}$/; +const SHIPPABLE_ADVISORY_STATES = new Set(["draft", "published"]); +// A credit note a maintainer leaves on an issue: Harvest's "credits its +// author", "Credit for the release notes: …", "Thanks to @x". +const CREDIT_NOTE = /\b(?:credit(?:s|ed|ing)?|thanks to|thank you to)\b/i; + /** * @typedef {{ dir: string, name: string, version: string, registry: "npm" | "PyPI" }} Package * @typedef {Package & { published: boolean }} CheckedPackage @@ -83,6 +114,8 @@ export function parseNotesArgs(argv) { "merge-branch": { type: "string" }, "ledger-url": { type: "string" }, "known-issue": { type: "string", multiple: true }, + advisory: { type: "string", multiple: true }, + thank: { type: "string", multiple: true }, "previous-tag": { type: "string" }, draft: { type: "boolean" }, }, @@ -114,12 +147,30 @@ export function parseNotesArgs(argv) { if (values.draft && values["previous-tag"] !== undefined) { throw new Error("--previous-tag is preview-only"); } + const advisories = [...new Set(values.advisory ?? [])]; + for (const id of advisories) { + if (!GHSA_ID.test(id)) { + throw new Error(`--advisory "${id}" is not a GHSA-xxxx-xxxx-xxxx id`); + } + } + const thanks = values.thank ?? []; + for (const line of thanks) { + // One bullet, naming whom it credits first, so the Contributors strip and + // the discovery check both find them. + if (!/^@[A-Za-z0-9-]+(?![\w/-])[^\n]*$/.test(line)) { + throw new Error( + `--thank "${line}" is not one line starting with @login, as "@login (#ref), "`, + ); + } + } return { milestone, mergeSha, mergeBranch, ledgerUrl, knownIssues: values["known-issue"] ?? [], + advisories, + thanks, previousTag: values["previous-tag"], mode: values.draft ? "draft" : "preview", }; @@ -492,29 +543,43 @@ export function isMaintainer(spawn, login) { throw new Error(`unexpected permission "${permission}" for @${login}`); } +/** + * `isMaintainer`, asked once per login per run, whichever lookup asks first. + * + * @param {Spawn} spawn + * @param {Map} cache lowercased login → maintainer + * @param {string} login + * @returns {boolean} + */ +function maintainerCached(spawn, cache, login) { + const key = login.toLowerCase(); + let maintainer = cache.get(key); + if (maintainer === undefined) { + maintainer = isMaintainer(spawn, login); + cache.set(key, maintainer); + } + return maintainer; +} + /** * Community reporter → the issues of theirs the listed PRs close. * * @param {Spawn} spawn * @param {number[]} pulls + * @param {Map} [maintainer] shared permission cache * @returns {Map} */ -export function collectReporters(spawn, pulls) { +export function collectReporters(spawn, pulls, maintainer = new Map()) { const issues = new Set(); for (const pr of pulls) { for (const n of issuesClosedBy(spawn, pr)) issues.add(n); } /** @type {Map} */ const reporters = new Map(); - /** @type {Map} */ - const maintainer = new Map(); for (const n of [...issues].sort((a, b) => a - b)) { const login = creditableAuthor(spawn, n); if (login === null) continue; - if (!maintainer.has(login)) { - maintainer.set(login, isMaintainer(spawn, login)); - } - if (maintainer.get(login)) continue; + if (maintainerCached(spawn, maintainer, login)) continue; const list = reporters.get(login) ?? []; list.push(n); reporters.set(login, list); @@ -523,22 +588,199 @@ export function collectReporters(spawn, pulls) { } /** - * One line per person, most issues first, then by name; "" when nobody. + * Community member → the advisories that credit them, in the order given, + * plus each credit still pending acceptance. The advisory endpoint answers + * for a draft too, for anyone who can read it, which is whoever drafts the + * notes. * - * @param {Map} reporters + * @param {Spawn} spawn + * @param {string[]} ids + * @param {Map} [maintainer] shared permission cache + * @returns {{ credits: Map, pending: string[] }} */ -export function formatThanks(reporters) { - if (reporters.size === 0) return ""; - const lines = [...reporters] - .sort( - ([a, ia], [b, ib]) => - ib.length - ia.length || - a.localeCompare(b, "en", { sensitivity: "base" }), - ) - .map( - ([login, nums]) => `* @${login} (${nums.map((n) => `#${n}`).join(", ")})`, +export function collectAdvisoryCredits(spawn, ids, maintainer = new Map()) { + /** @type {Map} */ + const credits = new Map(); + /** @type {string[]} */ + const pending = []; + for (const id of ids) { + const advisory = JSON.parse( + run(spawn, "gh", ["api", `repos/${REPO}/security-advisories/${id}`]), + ); + if (advisory.ghsa_id !== id) { + throw new Error(`asked for ${id}, GitHub answered ${advisory.ghsa_id}`); + } + if (!SHIPPABLE_ADVISORY_STATES.has(advisory.state)) { + throw new Error( + `${id} is ${advisory.state}: only a draft or published advisory has a fix to credit`, + ); + } + if (!Array.isArray(advisory.credits_detailed)) { + throw new Error(`${id} has no credits_detailed list`); + } + for (const credit of advisory.credits_detailed) { + if (credit.state === "declined") continue; + if (credit.state !== "accepted" && credit.state !== "pending") { + throw new Error(`unexpected credit state "${credit.state}" on ${id}`); + } + const login = credit.user?.login; + if (credit.user?.type !== "User" || typeof login !== "string") continue; + if (maintainerCached(spawn, maintainer, login)) continue; + if (credit.state === "pending") pending.push(`@${login} on ${id}`); + const list = credits.get(login) ?? []; + if (!list.includes(id)) list.push(id); + credits.set(login, list); + } + } + return { credits, pending }; +} + +/** + * The lead-in to the prototype paragraph, worded for one contributor or + * several, and as a continuation when a reporter list comes before it. + * + * @param {number} count + * @param {boolean} following + */ +export function prototypeLeadIn(count, following) { + return count === 1 + ? `${following ? "And to this contributor" : "Thank you to this contributor"}, whose pull request prototyped a change that ships in this release. Under the repository's ${POLICY_LINK} policy the PR was closed and the change rebuilt by the maintainers, crediting its author on the issue:` + : `${following ? "And to these contributors" : "Thank you to these contributors"}, whose pull requests prototyped changes that ship in this release. Under the repository's ${POLICY_LINK} policy their PRs were closed and the changes rebuilt by the maintainers, crediting them on the issue:`; +} + +/** + * The Thanks section: reporters of issues and advisories, one line per + * person with their issues then their advisories, most first, then by name; + * then the prototype paragraph, one line per `--thank`. "" when nobody. + * + * @param {{ reporters?: Map, advisories?: Map, thanks?: string[] }} credits + */ +export function formatThanks({ + reporters = new Map(), + advisories = new Map(), + thanks = [], +}) { + /** @type {Map} */ + const people = new Map(); + for (const [login, nums] of reporters) { + people.set( + login, + nums.map((n) => `#${n}`), + ); + } + for (const [login, ids] of advisories) { + people.set(login, [...(people.get(login) ?? []), ...ids]); + } + const parts = []; + if (people.size) { + const leadIn = + reporters.size && advisories.size + ? MIXED_LEAD_IN + : advisories.size + ? ADVISORY_LEAD_IN + : THANKS_LEAD_IN; + const lines = [...people] + .sort( + ([a, ra], [b, rb]) => + rb.length - ra.length || + a.localeCompare(b, "en", { sensitivity: "base" }), + ) + .map(([login, refs]) => `* @${login} (${refs.join(", ")})`); + parts.push(leadIn, lines.join("\n")); + } + if (thanks.length) { + parts.push( + prototypeLeadIn(thanks.length, people.size > 0), + thanks.map((line) => `* ${line}`).join("\n"), + ); + } + if (parts.length === 0) return ""; + return ["## Thanks for helping us improve", ...parts].join("\n\n"); +} + +/** + * The `@login`s a text names in prose: not in code, a quote or an HTML + * comment (see `proseOf`), not an email address, and not a scoped package + * name such as `@modelcontextprotocol/server-memory`. + * + * @param {string} text + * @returns {string[]} + */ +export function mentionsOf(text) { + const pattern = + /(? m[1]))]; +} + +/** @param {string} out one JSON value per line, as `gh --jq` prints them */ +function jsonLines(out) { + return out === "" ? [] : out.split("\n").map((line) => JSON.parse(line)); +} + +/** + * Every credit note a maintainer left on the milestone's closed issues (the + * issue body or a comment) and the non-maintainers it names: what the person + * drafting the notes reads to decide each `--thank`. A mention that is not an + * account (a typo, a team) names nobody, so its 404 is skipped; any other + * failure aborts, as everywhere else. + * + * @param {Spawn} spawn + * @param {string} milestone + * @param {Map} [maintainer] shared permission cache + * @returns {Array<{ issue: number, url: string, logins: string[] }>} + */ +export function discoverCredits(spawn, milestone, maintainer = new Map()) { + const number = run(spawn, "gh", [ + "api", + "--paginate", + `repos/${REPO}/milestones?state=all&per_page=100`, + "--jq", + `.[] | select(.title == ${JSON.stringify(milestone)}) | .number`, + ]); + if (!/^\d+$/.test(number)) { + throw new Error(`no milestone titled ${milestone}`); + } + /** @type {Array<{ number: number, url: string, body: string | null, author: string, type: string }>} */ + const issues = jsonLines( + run(spawn, "gh", [ + "api", + "--paginate", + `repos/${REPO}/issues?milestone=${number}&state=closed&per_page=100`, + "--jq", + ".[] | select(.pull_request == null) | {number, url: .html_url, body, author: .user.login, type: .user.type}", + ]), + ).sort((a, b) => a.number - b.number); + /** @param {string} login */ + const community = (login) => { + try { + return !maintainerCached(spawn, maintainer, login); + } catch (error) { + if (/HTTP 404/.test(/** @type {Error} */ (error).message)) return false; + throw error; + } + }; + const notes = []; + for (const issue of issues) { + /** @type {Array<{ url: string, body: string | null, author: string, type: string }>} */ + const comments = jsonLines( + run(spawn, "gh", [ + "api", + "--paginate", + `repos/${REPO}/issues/${issue.number}/comments?per_page=100`, + "--jq", + ".[] | {url: .html_url, body, author: .user.login, type: .user.type}", + ]), ); - return `## Thanks for helping us improve\n\n${THANKS_LEAD_IN}\n\n${lines.join("\n")}`; + for (const note of [issue, ...comments]) { + const body = note.body ?? ""; + if (note.type !== "User" || !CREDIT_NOTE.test(proseOf(body))) continue; + if (!maintainerCached(spawn, maintainer, note.author)) continue; + const logins = mentionsOf(body).filter(community); + if (logins.length) + notes.push({ issue: issue.number, url: note.url, logins }); + } + } + return notes; } /** @param {Section} section */ @@ -690,12 +932,50 @@ export async function main( console.error( `crediting reporters of issues closed by ${entries.length} PRs`, ); - const thanks = formatThanks( - collectReporters( - spawn, - entries.map((e) => e.pr), - ), + /** @type {Map} */ + const maintainer = new Map(); + const reporters = collectReporters( + spawn, + entries.map((e) => e.pr), + maintainer, + ); + const { credits: advisories, pending } = collectAdvisoryCredits( + spawn, + args.advisories, + maintainer, ); + for (const credit of pending) { + console.error( + `PENDING advisory credit: ${credit} is not accepted yet; it may still be declined`, + ); + } + const thanks = formatThanks({ + reporters, + advisories, + thanks: args.thanks, + }); + + if (args.mode === "preview") { + // The prototype paragraph is only as complete as the --thank flags, so + // the preview shows every credit on record next to what covers it. + const thanked = new Set( + args.thanks.flatMap(mentionsOf).map((l) => l.toLowerCase()), + ); + const credited = discoverCredits(spawn, args.milestone, maintainer); + console.error( + credited.length + ? `credit notes on ${args.milestone}'s closed issues (each needs a --thank, unless it credits a reporter already listed):` + : `no credit notes on ${args.milestone}'s closed issues`, + ); + for (const { issue, url, logins } of credited) { + const who = logins.map( + (l) => + `@${l} ${thanked.has(l.toLowerCase()) ? "(--thank given)" : "(NO --thank)"}`, + ); + console.error(` #${issue} ${url}: ${who.join(", ")}`); + } + } + const notes = assembleNotes({ packages: formatPackages(packages), sections, diff --git a/scripts/release-notes.test.mjs b/scripts/release-notes.test.mjs index 64f566eaec..89939f0507 100644 --- a/scripts/release-notes.test.mjs +++ b/scripts/release-notes.test.mjs @@ -3,12 +3,17 @@ // Repository catch-all dropping no PR, the release-wide Thanks section (laid // out as the Inspector's) with every exclusion (maintainers by permission, // bots, deleted accounts, PRs, other repos), keyword parsing, pagination, -// fail-fast at every lookup, and that nothing is created without --draft. Run via +// fail-fast at every lookup, and that nothing is created without --draft. +// #5250 added the credits that never come from an issue's author: advisory +// reporters (--advisory), the prototype paragraph (--thank), and the +// preview's discovery of credit notes on the milestone's closed issues. Run via // `npm run test:scripts`. import assert from "node:assert/strict"; import { test } from "node:test"; import { + ADVISORY_LEAD_IN, + MIXED_LEAD_IN, REPO, THANKS_LEAD_IN, assembleNotes, @@ -19,11 +24,16 @@ import { formatPackages, formatThanks, main, + mentionsOf, parseGenerated, parseNotesArgs, + prototypeLeadIn, registryUrl, } from "./release-notes.mjs"; +const GHSA_A = "GHSA-66qx-2266-pmj6"; +const GHSA_B = "GHSA-4gxg-m5jx-9j57"; + const SHA = "a".repeat(40); const BASE = [ "--milestone", @@ -45,6 +55,8 @@ test("parseNotesArgs defaults to a preview and validates its inputs", () => { mergeBranch: "v2/chore/1-release-v2.0.0", ledgerUrl: "https://l.example/x", knownIssues: [], + advisories: [], + thanks: [], previousTag: undefined, mode: "preview", }); @@ -84,6 +96,46 @@ test("parseNotesArgs defaults to a preview and validates its inputs", () => { assert.throws(() => parseNotesArgs([...BASE, "--publish"]), /publish/); }); +test("parseNotesArgs takes advisories and hand credits, and validates both", () => { + const args = parseNotesArgs([ + ...BASE, + "--advisory", + GHSA_A, + "--advisory", + GHSA_B, + "--advisory", + GHSA_A, + "--thank", + "@amy (#1), the fix for #2", + "--thank", + "@bob-c and @dee (#3), the docs", + ]); + // A repeated advisory is read and listed once. + assert.deepEqual(args.advisories, [GHSA_A, GHSA_B]); + assert.deepEqual(args.thanks, [ + "@amy (#1), the fix for #2", + "@bob-c and @dee (#3), the docs", + ]); + for (const id of ["GHSA-66qx-2266", "ghsa-66qx-2266-pmj6", "CVE-2026-1"]) { + assert.throws( + () => parseNotesArgs([...BASE, "--advisory", id]), + /is not a GHSA-xxxx-xxxx-xxxx id/, + ); + } + for (const line of [ + "amy (#1), the fix", + "thanks @amy", + "@amy (#1)\n* @eve, injected", + "@scope/pkg, a package", + "@my-scope/pkg, a hyphenated scope", + ]) { + assert.throws( + () => parseNotesArgs([...BASE, "--thank", line]), + /is not one line starting with @login/, + ); + } +}); + test("registryUrl asks for the exact version on each registry", () => { assert.equal( registryUrl({ @@ -266,7 +318,7 @@ test("formatThanks orders by issue count, then name case-insensitively", () => { ["bob", [4]], ]); assert.equal( - formatThanks(reporters), + formatThanks({ reporters }), [ "## Thanks for helping us improve", "", @@ -278,7 +330,114 @@ test("formatThanks orders by issue count, then name case-insensitively", () => { "* @zed (#5)", ].join("\n"), ); - assert.equal(formatThanks(new Map()), ""); + assert.equal(formatThanks({}), ""); + assert.equal( + formatThanks({ reporters: new Map(), advisories: new Map(), thanks: [] }), + "", + ); +}); + +test("formatThanks: advisories alone, mixed with issues, one line per person", () => { + assert.equal( + formatThanks({ + advisories: new Map([ + ["solo", [GHSA_B]], + ["two", [GHSA_A, GHSA_B]], + ]), + }), + [ + "## Thanks for helping us improve", + "", + ADVISORY_LEAD_IN, + "", + `* @two (${GHSA_A}, ${GHSA_B})`, + `* @solo (${GHSA_B})`, + ].join("\n"), + ); + // A person who filed an issue and reported an advisory is listed once, + // issues first, and counted by every reference. + assert.equal( + formatThanks({ + reporters: new Map([ + ["both", [7]], + ["filer", [3, 4]], + ]), + advisories: new Map([ + ["both", [GHSA_A, GHSA_B]], + ["finder", [GHSA_A]], + ]), + }), + [ + "## Thanks for helping us improve", + "", + MIXED_LEAD_IN, + "", + `* @both (#7, ${GHSA_A}, ${GHSA_B})`, + "* @filer (#3, #4)", + `* @finder (${GHSA_A})`, + ].join("\n"), + ); +}); + +test("formatThanks: the prototype paragraph, alone or after the reporters", () => { + assert.equal( + formatThanks({ thanks: ["@ace (#1), the assertion"] }), + [ + "## Thanks for helping us improve", + "", + prototypeLeadIn(1, false), + "", + "* @ace (#1), the assertion", + ].join("\n"), + ); + assert.match( + prototypeLeadIn(1, false), + /^Thank you to this contributor, whose pull request prototyped a change/, + ); + assert.match( + prototypeLeadIn(2, true), + /^And to these contributors, whose pull requests prototyped changes/, + ); + assert.match( + prototypeLeadIn(2, false), + /\[issues, not pull requests\]\(https:\/\/github\.com\/modelcontextprotocol\/servers\/blob\/v2\/main\/CONTRIBUTING\.md\)/, + ); + assert.equal( + formatThanks({ + reporters: new Map([["r", [1]]]), + thanks: ["@a (#2), x", "@b (#3), y"], + }), + [ + "## Thanks for helping us improve", + "", + THANKS_LEAD_IN, + "", + "* @r (#1)", + "", + prototypeLeadIn(2, true), + "", + "* @a (#2), x", + "* @b (#3), y", + ].join("\n"), + ); +}); + +test("mentionsOf finds logins in prose only", () => { + assert.deepEqual( + mentionsOf( + [ + "Credit to @amy and @Bob-C's prototype, (@dee), @amy again.", + "Mail me@example.com; install @modelcontextprotocol/server-memory.", + "Not `@code`, not ,", + "> nor @quoted", + "```", + "@fenced", + "```", + "nor a-@b, nor @-x, nor @trailing-.", + ].join("\n"), + ), + ["amy", "Bob-C", "dee"], + ); }); test("formatKnownIssues pluralizes and is omitted when empty", () => { @@ -340,6 +499,12 @@ function world({ releases = ["2026.8.31"], /** @type {string[]} */ tagRefs = [], + /** @type {Record} */ + advisories = {}, + /** @type {Record} */ + milestones = { "v2.0.0": 4 }, + /** @type {Array<{ number: number, body?: string | null, author: string, type?: string, comments?: Array<{ body: string | null, author: string, type?: string }> }>} */ + milestoneIssues = [], } = {}) { /** @type {Array<{ cmd: string, args: string[], input?: string }>} */ const calls = []; @@ -446,6 +611,58 @@ function world({ .join("\n"), ); } + const advisory = args[1].match(/security-advisories\/(.+)$/)?.[1]; + if (advisory) { + assert.equal(args.length, 2); + return advisories[advisory] + ? ok(JSON.stringify(advisories[advisory])) + : { status: 1, stdout: "", stderr: "gh: Not Found (HTTP 404)" }; + } + // The --jq each listing passes, applied to the model: the milestone's + // number by title, then its closed issues and each one's comments as JSON + // lines. + if (args[2]?.includes("/milestones?")) { + const title = JSON.parse( + /** @type {string} */ (args[4].match(/== (".*?")\)/)?.[1]), + ); + return ok(String(milestones[title] ?? "")); + } + if (args[2]?.includes("/issues?milestone=")) { + assert.match(args[2], /state=closed/); + const number = Number(args[2].match(/milestone=(\d+)/)?.[1]); + assert.ok(Object.values(milestones).includes(number)); + return ok( + milestoneIssues + .map((i) => + JSON.stringify({ + number: i.number, + url: `https://github.com/${REPO}/issues/${i.number}`, + body: i.body ?? null, + author: i.author, + type: i.type ?? "User", + }), + ) + .join("\n"), + ); + } + const commentsOf = args[2]?.match(/issues\/(\d+)\/comments/)?.[1]; + if (commentsOf) { + const issue = milestoneIssues.find( + (i) => i.number === Number(commentsOf), + ); + return ok( + (issue?.comments ?? []) + .map((c, k) => + JSON.stringify({ + url: `https://github.com/${REPO}/issues/${commentsOf}#issuecomment-${k}`, + body: c.body, + author: c.author, + type: c.type ?? "User", + }), + ) + .join("\n"), + ); + } if (args[1].includes("/git/matching-refs/tags/")) return ok(tagRefs.join("\n")); if (args[1].endsWith("/releases/generate-notes")) return ok(generated); @@ -453,6 +670,14 @@ function world({ if (args[2] === `repos/${REPO}/releases`) return ok(releases.join("\n")); const login = args[1].match(/collaborators\/([^/]+)\/permission/)?.[1]; assert.ok(login, `unexpected gh call: ${joined}`); + // GitHub's answer for a login that is no account. + if (!(login in perms)) { + return { + status: 1, + stdout: "", + stderr: `gh: ${login} is not a user (HTTP 404)`, + }; + } return ok(perms[login]); }; spawn.calls = calls; @@ -478,12 +703,16 @@ const user = (/** @type {string} */ login, extra = {}) => ({ function quiet(t) { /** @type {string[]} */ const out = []; - t.mock.method(console, "error", () => {}); + /** @type {string[]} */ + const err = []; + t.mock.method(console, "error", (/** @type {string} */ line) => { + err.push(line); + }); t.mock.method(process.stdout, "write", (/** @type {string} */ chunk) => { out.push(chunk); return true; }); - return out; + return { out, err }; } /** The text of one `## …` section of the notes. */ @@ -495,7 +724,7 @@ function sectionOf(/** @type {string} */ notes, /** @type {string} */ heading) { } test("notes: per-server sections, every PR placed, one release-wide Thanks", async (t) => { - const out = quiet(t); + const { out } = quiet(t); const deps = world({ pulls: { 10: { files: ["src/git/a.py"], body: "Closes #1" }, @@ -932,3 +1161,237 @@ test("a file renamed out of a server's directory still places the PR there", asy ); assert.doesNotMatch(notes, /## Repository/); }); + +/** An advisory as `security-advisories/{id}` answers, credits as given. */ +function advisory( + /** @type {string} */ id, + /** @type {Array<[string, string, string?]>} */ credits, + state = "draft", +) { + return { + ghsa_id: id, + state, + credits_detailed: credits.map(([login, creditState, type = "User"]) => ({ + user: { login, type }, + type: "reporter", + state: creditState, + })), + }; +} + +test("--advisory credits each advisory's reporters, flags pending, drops the rest", async (t) => { + const { err } = quiet(t); + const deps = world({ + pulls: { 10: { files: ["src/git/a.py"], body: "Closes #1" } }, + issues: { 1: user("both") }, + advisories: { + [GHSA_A]: advisory(GHSA_A, [ + ["both", "accepted"], + ["waiting", "pending"], + ["nope", "declined"], + ["dependabot", "accepted", "Bot"], + ["maint", "accepted"], + ]), + [GHSA_B]: advisory( + GHSA_B, + [ + ["both", "accepted"], + ["finder", "accepted"], + ], + "published", + ), + }, + perms: { both: "read", waiting: "read", finder: "none", maint: "admin" }, + }); + const notes = await main( + [...BASE, "--advisory", GHSA_A, "--advisory", GHSA_B], + deps, + ); + assert.ok( + notes.endsWith( + [ + "## Thanks for helping us improve", + "", + MIXED_LEAD_IN, + "", + `* @both (#1, ${GHSA_A}, ${GHSA_B})`, + `* @finder (${GHSA_B})`, + `* @waiting (${GHSA_A})`, + "", + ].join("\n"), + ), + notes, + ); + assert.doesNotMatch(notes, /@nope|@dependabot|@maint/); + assert.ok( + err.some((line) => + line.startsWith(`PENDING advisory credit: @waiting on ${GHSA_A}`), + ), + err.join("\n"), + ); + // A declined credit and a bot are never looked up; everyone else once, + // across issues and advisories. + const asked = deps.spawn.calls + .map((c) => c.args[1]?.match(/collaborators\/([^/]+)\/permission/)?.[1]) + .filter(Boolean); + assert.deepEqual(asked, ["both", "waiting", "maint", "finder"]); +}); + +test("an advisory that cannot carry a shipped fix, or answers oddly, aborts", async (t) => { + quiet(t); + const run = (/** @type {object} */ answer) => + main( + [...BASE, "--advisory", GHSA_A, "--draft"], + world({ advisories: { [GHSA_A]: answer }, perms: { x: "read" } }), + ); + for (const state of ["triage", "closed", "withdrawn"]) { + await assert.rejects( + run(advisory(GHSA_A, [], state)), + new RegExp(`${GHSA_A} is ${state}: only a draft or published advisory`), + ); + } + await assert.rejects( + run(advisory(GHSA_B, [])), + new RegExp(`asked for ${GHSA_A}, GitHub answered ${GHSA_B}`), + ); + await assert.rejects( + run({ ghsa_id: GHSA_A, state: "draft" }), + /has no credits_detailed list/, + ); + await assert.rejects( + run(advisory(GHSA_A, [["x", "maybe"]])), + /unexpected credit state "maybe"/, + ); + // An advisory GitHub does not show this caller (or that is not ours). + const deps = world(); + await assert.rejects( + main([...BASE, "--advisory", GHSA_A, "--draft"], deps), + /Not Found \(HTTP 404\)/, + ); + assert.equal( + deps.spawn.calls.some((c) => c.args[1] === "create"), + false, + ); +}); + +test("--thank lines become the prototype paragraph, after the reporters", async (t) => { + quiet(t); + const notes = await main( + [...BASE, "--thank", "@ace (#9), the fix for #5", "--draft"], + world({ + pulls: { 10: { files: ["src/git/a.py"], body: "Closes #1" } }, + issues: { 1: user("filer") }, + perms: { filer: "read" }, + }), + ); + assert.ok( + notes.endsWith( + [ + THANKS_LEAD_IN, + "", + "* @filer (#1)", + "", + prototypeLeadIn(1, true), + "", + "* @ace (#9), the fix for #5", + "", + ].join("\n"), + ), + notes, + ); +}); + +test("the preview lists maintainers' credit notes on the milestone's closed issues", async (t) => { + const { err } = quiet(t); + const deps = world({ + milestones: { "v1.9.0": 3, "v2.0.0": 4 }, + milestoneIssues: [ + { + number: 21, + author: "maint", + body: "Harvested from #900, crediting @harvested as its prototype.", + comments: [ + // Credited, and already given a --thank. + { author: "maint", body: "Thanks to @thanked for #901." }, + // Not a credit note at all. + { author: "maint", body: "cc @harvested, any update?" }, + // A community member's own words are not a maintainer's credit. + { author: "outsider", body: "Credit to @friend please" }, + { author: "github-actions", type: "Bot", body: "credit @botpal" }, + { author: "maint", body: null }, + ], + }, + { + number: 20, + author: "maint", + // Only maintainers, a quoted mention and a non-account are named. + body: "Credit: @maint2 and @ghost-typo.\n> credit @quoted", + }, + ], + perms: { + maint: "maintain", + maint2: "write", + harvested: "read", + thanked: "triage", + outsider: "read", + }, + }); + await main([...BASE, "--thank", "@Thanked (#901), the fix"], deps); + const lines = err.slice(err.findIndex((l) => l.startsWith("credit notes"))); + assert.deepEqual(lines.slice(0, 3), [ + "credit notes on v2.0.0's closed issues (each needs a --thank, unless it credits a reporter already listed):", + ` #21 https://github.com/${REPO}/issues/21: @harvested (NO --thank)`, + ` #21 https://github.com/${REPO}/issues/21#issuecomment-0: @thanked (--thank given)`, + ]); + assert.equal(lines.length, 4); + // The milestone's number came from its title. + assert.ok( + deps.spawn.calls.some((c) => c.args[2]?.includes("issues?milestone=4&")), + ); +}); + +test("discovery is preview-only, and says so when it finds nothing", async (t) => { + const { err } = quiet(t); + await main(BASE, world()); + assert.ok(err.includes("no credit notes on v2.0.0's closed issues")); + const draft = world(); + await main([...BASE, "--draft"], draft); + assert.equal( + draft.spawn.calls.some((c) => c.args.join(" ").includes("milestones")), + false, + ); +}); + +test("discovery aborts on a missing milestone or a failed lookup", async (t) => { + quiet(t); + await assert.rejects( + main(BASE, world({ milestones: {} })), + /no milestone titled v2\.0\.0/, + ); + const issue = { + number: 1, + author: "maint", + body: "credit @who", + }; + for (const failOn of ["/milestones?", "issues?milestone=", "/comments"]) { + await assert.rejects( + main( + BASE, + world({ milestoneIssues: [issue], perms: { maint: "admin" }, failOn }), + ), + /rate limit exceeded/, + ); + } + // Only a 404 means "not an account"; any other failed permission lookup + // aborts, as it does for a reporter. + await assert.rejects( + main( + BASE, + world({ + milestoneIssues: [issue], + perms: { maint: "admin", who: "superuser" }, + }), + ), + /unexpected permission "superuser" for @who/, + ); +});