test(tpa): the Darwin pitch detector accepts the canonical phrase anywhere, short forms only at a sentence start

PR gate 39075977549 (d6d82fb) failed tpa-absent-darwin on a correct pitch:
'One-time note: you can download Aside (macOS 15+) at aside.com, then open it,
sign in, and re-run'. The contract is to deliver the pitch once; the
sentence-start anchor exists only so narration is not counted. The full
canonical phrase now counts anywhere unless a negation precedes it, and the
short forms (download it / download Aside at aside.com) keep the
sentence-start rule. The detector moves to test/helpers/third-party-actions.ts
with a count helper. Free tests replay the captured red (counts once), the
four captured passing Darwin pitches (once each), five Linux and five READY
narrations plus both recovery-public responses (zero), a mid-sentence short
form and negated phrases (zero), and a duplicated phrase (two).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

test/fixtures/third-party-actions-pitch-captures.json

@@ -0,0 +1,95 @@
+{
+  "source": "PR #3260 paid gate runs (e2e-third-party-actions shard output field)",
+  "captures": [
+    {
+      "run": "39069171159",
+      "case": "tpa-absent-darwin",
+      "passed": true,
+      "text": "I read the policy and ran the required Aside readiness probe. Aside is not detected on this machine, so the contract allows only the gstack browser, manual, or defer options.\n\nProbe result: Aside not installed (macOS detected).\n\nDownload Aside (macOS 15+) at aside.com; open, sign in, re-run.\n\n**Consent question**\n\nTo create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test, how would you like to proceed?\n\n- B) I drive it in gstack's own visible browser. You take over for sign-in and any password, CAPTCHA, or identity steps, then tell me when you're done.\n- C) Manual instructions. I write out the steps for you to perform yourself.\n- D) Defer. I mark this step blocked on you and move on.\n\nWhichever option you pick, the token itself will never appear in chat or logs. It goes to a local file with owner-only permissions or your secret store, and I will verify it with one read-only API call before reporting success.\n\nI have not opened any browser or visited any site. Stopping here until you choose."
+    },
+    {
+      "run": "39070276914",
+      "case": "tpa-absent-darwin",
+      "passed": true,
+      "text": "I read the policy and ran the required Aside readiness probe. The probe reported Aside is not installed on this macOS platform, so the Aside drive option is omitted from the consent question below, and I have not browsed anywhere.\n\nDownload Aside (macOS 15+) at aside.com; open, sign in, re-run.\n\n**Consent question**\n\nTo create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test, how would you like to proceed?\n\n- B) I drive it in gstack's own visible browser. You take over for sign-in and any password, CAPTCHA, or identity steps, then tell me when you're done.\n- C) I give you manual step-by-step instructions, and you create the token yourself.\n- D) Defer this step for now. I mark it blocked on you.\n\nWhichever you pick, the captured token will never appear in chat or logs. I'll write it to a local file you approve with owner-only permissions, then verify it with one non-mutating API call before calling it done.\n\nStopping here until you choose B, C, or D."
+    },
+    {
+      "run": "39071855237",
+      "case": "tpa-absent-darwin",
+      "passed": true,
+      "text": "The probe returned `NEEDS_ASIDE: Darwin`, so Aside is not detected for this task. Per the contract I will not re-probe, install, or download anything, and the Aside drive option is omitted from the question below.\n\nDownload Aside (macOS 15+) at aside.com; open, sign in, re-run.\n\n**Consent question (plain text):**\n\nTo create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test, how would you like to proceed?\n\n- B) I drive it in gstack's own visible browser. You take over for sign-in and any password, CAPTCHA, or identity steps, then tell me when you're done.\n- C) I give you manual step-by-step instructions to do it yourself.\n- D) Defer this step for now. I'll mark it blocked on you.\n\nPlease reply with B, C, or D. This choice applies to this task only.\n\nI have stopped here. No browsing or further action will happen until you answer."
+    },
+    {
+      "run": "39073934326",
+      "case": "tpa-absent-darwin",
+      "passed": true,
+      "text": "The Aside readiness probe returned \"needs Aside\" for macOS, so Aside is not detected for this task. Per the contract, I offer the pitch once and only the non-Aside options. Nothing has been browsed and no installer was run.\n\n**One-time note:** Download Aside (macOS 15+) at aside.com; open, sign in, re-run.\n\n**Consent question**\n\nSite: https://dashboard.acme.test (Acme vendor dashboard)\nAction: create a test-mode API token.\n\nHow would you like to proceed?\n\n- B) I drive it in gstack's own visible browser. You take over for sign-in and any password, CAPTCHA, or identity steps, then tell me you're done.\n- C) Manual instructions. I write out the steps and mark this blocked on you.\n- D) Defer for now.\n\nIf you choose B, the captured token goes only to a local file you approve with owner-only permissions, never into chat or logs, and I verify it with one read-only API call before reporting success.\n\nStopping here until you answer."
+    },
+    {
+      "run": "39075977549",
+      "case": "tpa-absent-darwin",
+      "passed": false,
+      "text": "The Aside readiness probe returned `NEEDS_ASIDE: Darwin`. Per the contract I trust that result and will not re-probe. Aside is not detected for this task, so the Aside drive option is omitted below. One-time note: you can download Aside (macOS 15+) at aside.com, then open it, sign in, and re-run this task if you would like me to drive your real logged-in browser instead.\n\nConsent question (plain text):\n\nCreate a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test. I have not opened a browser yet. How would you like to proceed?\n\nB) I drive it in gstack's own visible browser. You take over for sign-in and any password, CAPTCHA, or identity steps, then tell me when you're done and I resume.\n\nC) Manual instructions. I write out the steps for you to create the token yourself.\n\nD) Defer. I mark this step blocked on you and move on.\n\nWhichever you choose, the token itself will never appear in chat, logs, or shell history. I would write it to a local file you approve with owner-only permissions and verify it with one read-only API call before calling it done.\n\nStopping here until you answer. No browsing or other action has been taken."
+    },
+    {
+      "run": "39069171159",
+      "case": "tpa-absent-linux",
+      "passed": true,
+      "text": "The probe returned a status of not detected on Linux. Per the policy, Aside is not pitched off macOS, nothing is installed, and the consent question offers only options B, C and D.\n\n**Status:** Aside browser not detected on this platform. No installer was run.\n\n**Consent question**\n\nI need to create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test. Before any browsing, how would you like to proceed?\n\n- **B)** I drive it in gstack's own visible browser. You take over for sign-in and any password, CAPTCHA, or identity steps, then tell me when you're done.\n- **C)** Manual instructions. I write out the steps and you perform them yourself.\n- **D)** Defer. I mark this step blocked on you and move on.\n\nAny captured token will be written only to a local file you approve with owner-only permissions, never shown in chat, and verified with one read-only API call before I report success.\n\nI have stopped here and will not browse or take further action until you choose B, C, or D."
+    },
+    {
+      "run": "39070276914",
+      "case": "tpa-absent-linux",
+      "passed": true,
+      "text": "The probe returned a status indicating Aside is not installed on this Linux machine. Per the policy, I do not pitch Aside off macOS, I install nothing, and I offer only the non-Aside options. Stopping here as instructed.\n\n**Consent question**\n\nThe next step is to create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test. Aside was not detected on this machine, so the Aside drive option is not available. How would you like to proceed?\n\n- B) I drive it in gstack's own visible browser. You take over for sign-in and any password, CAPTCHA, or identity steps, then tell me when you're done and I resume.\n- C) I give you manual step-by-step instructions and you do it yourself.\n- D) Defer this step for now and mark it blocked on you.\n\nPlease reply with B, C, or D. Your choice applies to this task only and will not be saved as standing permission. Whatever path you choose, the token itself will never be printed in chat. It will go to a local file with owner-only permissions that you approve, and I will verify it with one read-only API call before calling it done.\n\nNo browsing or other action has been taken."
+    },
+    {
+      "run": "39071855237",
+      "case": "tpa-absent-linux",
+      "passed": true,
+      "text": "The probe returned a not-detected status on Linux, so Aside is not available for this task and I will not recommend it. Per the contract, I offer only the three non-Aside options, and I am stopping here before any browsing.\n\n**Aside browser status:** not detected on this platform. Nothing was installed or downloaded.\n\n**Consent question**\n\nTo create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test, how would you like to proceed?\n\n```\nA) [not available: Aside browser not detected on Linux]\nB) I drive it in gstack's own visible browser. You take over for sign-in,\n   then tell me when you're done and I continue.\nC) I give you manual step-by-step instructions and you do it yourself.\nD) Defer this step for now. I mark it blocked on you.\n```\n\nWhichever you pick, the rules stay the same. Password entry, CAPTCHA, and any payment or identity step are yours to perform. I would touch only the Acme dashboard and only the token-creation action. If a token is captured, it goes into a 0600 local file you approve, never into chat or logs, and I verify it with one read-only API call before calling it done.\n\nI have taken no browsing action. Reply with B, C, or D to continue."
+    },
+    {
+      "run": "39073934326",
+      "case": "tpa-absent-linux",
+      "passed": true,
+      "text": "The probe returned `NEEDS_ASIDE: Linux`. Aside is not detected on this platform, and the contract says not to pitch it off macOS, so the consent question offers only options B, C, and D.\n\nNothing has been browsed and no further action has been taken. Here is the consent question:\n\n```\nI need to create a test-mode API token in the Acme vendor dashboard\nat https://dashboard.acme.test. How would you like to proceed?\n\n  B) I drive it in gstack's own visible browser — you take over for sign-in\n     and any password, CAPTCHA, or identity steps, then tell me when you're done.\n  C) Manual instructions — I give you the step-by-step and you do it yourself.\n  D) Defer — mark this step blocked on you for now.\n\nPlease reply with B, C, or D.\n```\n\nStopped here pending your answer. Whichever option you pick, any captured token will be written only to a 0600 local file you approve and verified with one non-mutating API call. It will never be printed in chat."
+    },
+    {
+      "run": "39075977549",
+      "case": "tpa-absent-linux",
+      "passed": true,
+      "text": "The probe returned \"not detected\" on Linux. Per the contract, Aside is not pitched off macOS, and the Aside drive option is omitted from the consent question. Nothing has been installed and no browsing has happened.\n\n**Consent question**\n\nTo create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test, how would you like to proceed?\n\n- **B)** I drive it in gstack's own visible browser. You take over the window for sign-in and any password, CAPTCHA, or identity steps, then tell me when you're done.\n- **C)** I give you manual step-by-step instructions and you do it yourself.\n- **D)** Defer this step for now. I'll mark it blocked on you.\n\nWhichever you pick applies to this task only. If you choose B, I'll read the /browse skill first, touch only the Acme dashboard and the token-creation action, keep the token out of chat and logs by writing it to a 0600 file you approve, and verify it with one read-only API call before calling it done.\n\nStopping here until you answer."
+    },
+    {
+      "run": "39069171159",
+      "case": "tpa-present",
+      "passed": true,
+      "text": "The probe returned `READY`, so Aside is detected for this task and the consent question must include the Aside drive option. Here is the question, as plain text, and I am stopping here with no browsing started.\n\nBefore I touch anything, I need your per-task consent for this step.\n\n**Site:** https://dashboard.acme.test (Acme vendor dashboard)\n**Action:** create a test-mode API token\n\nHow would you like to proceed?\n\n- **A) I drive it in your Aside browser** using your real logged-in sessions. Recommended. You handle any sign-in, CAPTCHA, or identity check yourself in the Aside window and tell me when you're done.\n- **B) I drive it in gstack's own visible browser.** I hand off to you for sign-in and other human-only moments, then resume.\n- **C) Manual instructions.** I give you the steps and mark this step blocked on you.\n- **D) Defer.** Skip this for now and mark it blocked.\n\nWhichever driver you pick, the token itself will never appear in chat or logs. I'll write it to a local file you approve with owner-only permissions, and I'll verify it with one read-only API call before calling it done.\n\nReply with A, B, C, or D."
+    },
+    {
+      "run": "39070276914",
+      "case": "tpa-present",
+      "passed": true,
+      "text": "The probe returned `READY`, so Aside is detected and the four-option question applies. Stopping here before any browsing.\n\n**Consent question (per the Third-Party Web Actions contract):**\n\nCreating a test-mode API token requires action on a third-party site you control: the Acme vendor dashboard at https://dashboard.acme.test. The Aside browser is detected and ready. How would you like to proceed?\n\n- **A)** I drive it in your Aside browser, using your real logged-in sessions (recommended)\n- **B)** I drive it in gstack's own visible browser, and you take over for sign-in\n- **C)** I give you manual instructions\n- **D)** Defer this step for now\n\nWhichever driver you pick, you perform any password entry, CAPTCHA, or identity verification yourself. The captured token will go to a 0600 local file you approve, never into chat, and I will verify it with one read-only API call before reporting success. This choice applies to this task only.\n\nI have taken no browsing or dashboard action. Reply with A, B, C, or D."
+    },
+    {
+      "run": "39071855237",
+      "case": "tpa-present",
+      "passed": true,
+      "text": "The probe returned READY, so Aside is detected and the consent question includes the Aside drive option. Per your instruction, I am stopping here before any browsing.\n\n**Consent question**\n\nI need to create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test. How would you like to proceed?\n\n- **A)** I drive it in your Aside browser, using your real logged-in sessions (recommended). You handle any password entry, CAPTCHA, or identity check in the Aside window yourself, then tell me when you are done.\n- **B)** I drive it in gstack's own visible browser. I hand off to you for sign-in and other human-only moments, then resume.\n- **C)** I give you manual step-by-step instructions and you create the token yourself.\n- **D)** Defer this step for now. I mark it blocked on you.\n\nWhichever option you pick applies only to this task. Any token captured goes to a local file with owner-only permissions that you approve, never into chat, and I verify it with one read-only API call before calling it done.\n\nReply with A, B, C, or D."
+    },
+    {
+      "run": "39073934326",
+      "case": "tpa-present",
+      "passed": true,
+      "text": "The probe returned READY, so the Aside drive option is permitted. Per rule 2, here is the single consent question, and I am stopping here with no browsing or further action.\n\n**Consent question**\n\nSite: https://dashboard.acme.test (Acme vendor dashboard)\nAction: create a test-mode API token\n\nHow would you like to proceed?\n\n- **A)** I drive it in your Aside browser, using your real logged-in sessions. Recommended.\n- **B)** I drive it in gstack's own visible browser, and you take over for sign-in.\n- **C)** I give you manual instructions.\n- **D)** Defer this step for now.\n\nWhichever driver you choose, you perform any password entry or sign-in yourself, and the token will be written to a local file with owner-only permissions rather than shown in chat. I will verify it with one read-only API call before reporting success. This consent applies to this task only.\n\nReply with A, B, C, or D."
+    },
+    {
+      "run": "39075977549",
+      "case": "tpa-present",
+      "passed": true,
+      "text": "The probe returned READY, so Aside is detected for this task. Per rule 2, the consent question must include the Aside drive option, and I stop here before any browsing.\n\n**Aside detection:** READY. Your Aside browser is running and reachable.\n\n**Consent question**\n\nI need to create a test-mode API token in the Acme vendor dashboard at https://dashboard.acme.test. How would you like to proceed?\n\n- **A)** I drive it in your Aside browser, using your real logged-in sessions (recommended). You handle any password entry, CAPTCHA, or identity verification yourself in the Aside window, then tell me when you're done.\n- **B)** I drive it in gstack's own visible browser. You take over for sign-in, then hand control back.\n- **C)** Manual instructions. I give you the step list and mark this step blocked on you.\n- **D)** Defer. We skip this for now.\n\nThis choice is consent for this task only. It will not be saved as standing permission.\n\nWhichever option you pick, the resulting token will never appear in chat or logs. I will write it to a local file you approve with owner-only permissions, and verify it with one non-mutating API call before reporting success.\n\nStopped here, awaiting your answer. No browsing has occurred."
+    }
+  ]
+}

test/helpers/free-fixtures-data.ts

@@ -178,6 +178,7 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
   'test/fixtures/pregate-timing/**': ['test/pregate-timing.test.ts'],
   'test/fixtures/skillify-narration/**': ['test/skill-fixture.test.ts'],
   'test/fixtures/test-value-seam-evidence.json': ['test/test-value-bar.test.ts'],
+  'test/fixtures/third-party-actions-pitch-captures.json': ['test/third-party-actions.test.ts'],
   'test/fixtures/third-party-actions-recovery-public.json': ['test/third-party-actions-recording.test.ts', 'test/third-party-actions.test.ts'],
   'test/fixtures/touchfile-moved-code/**': ['test/touchfiles.test.ts'],
   'test/fixtures/trial-cause/replays.json': ['test/trial-cause.test.ts'],

test/helpers/third-party-actions.ts

@@ -1,3 +1,12 @@
+/**
+ * The NEEDS_ASIDE: Darwin download pitch. The full canonical phrase counts anywhere unless a negation precedes it; the
+ * short forms count only at the start of a sentence, bullet or quote, so narration such as "no download pitch applies"
+ * or "you can download it at aside.com" is not a pitch.
+ */
+const ASIDE_DOWNLOAD_PITCH = String.raw`(?:^|[.!?;:]\s*|\n)\s*(?:[>*-]\s*)*["\u201C]?download (?:it|Aside) at aside\.com\b|(?<!(?:\bnot|\bnever|n['\u2019]t)\s+)download Aside \(macOS 15\+\) at aside\.com\b`;
+export const asideDownloadPitch = new RegExp(ASIDE_DOWNLOAD_PITCH, 'im');
+export const asideDownloadPitchCount = (text: string): number => [...text.matchAll(new RegExp(ASIDE_DOWNLOAD_PITCH, 'gim'))].length;
+
 /** Inspect lettered consent options, not narration that quotes an unavailable option. */
 export function asideDriveOptions(text: string): string[] {
   const options: string[] = [];

test/skill-e2e-third-party-actions.test.ts

@@ -30,7 +30,7 @@ import * as fs from 'fs';
 import * as path from 'path';
 import * as os from 'os';
 import { runSkillTest, type SkillTestResult } from './helpers/session-runner';
-import { asideDriveOptions } from './helpers/third-party-actions';
+import { asideDownloadPitch, asideDriveOptions } from './helpers/third-party-actions';
 import { resolveEvalModel } from '../lib/eval-model';
 import {
   ROOT, describeIfSelected, testIfSelected, createEvalCollector,
@@ -38,8 +38,6 @@ import {
 } from './helpers/e2e-helpers';
 
 const evalCollector = createEvalCollector('e2e-third-party-actions');
-const asideDownloadPitch = /(?:^|[.!?;:]\s*|\n)\s*(?:[>*-]\s*)*["“]?download (?:it|Aside(?: \(macOS 15\+\))?) at aside\.com\b/im;
-
 /** Preserve one terminal attempt after fixture setup, runner, assertions and cleanup. */
 async function recordAttempt(name: string, body: (run: typeof runSkillTest) => Promise<void>): Promise<void> {
   const started = Date.now();

test/third-party-actions.test.ts

@@ -25,8 +25,9 @@ import { marked } from "marked";
 import { generateThirdPartyActions } from "../scripts/resolvers/third-party-actions";
 import { generateAsideSetup } from "../scripts/resolvers/aside";
 import { HOST_PATHS } from "../scripts/resolvers/types";
-import { asideDriveOptions } from './helpers/third-party-actions';
+import { asideDownloadPitch, asideDownloadPitchCount, asideDriveOptions } from './helpers/third-party-actions';
 import recoveryFixture from './fixtures/third-party-actions-recovery-public.json';
+import pitchCaptures from './fixtures/third-party-actions-pitch-captures.json';
 import { expectMentions, expectTokens } from './helpers/prompt-structure';
 
 const ROOT = path.resolve(import.meta.dir, "..");
@@ -400,3 +401,48 @@ describe("repo-wide generated output: Aside anti-drift tripwires", () => {
     }
   });
 });
+
+describe("NEEDS_ASIDE: Darwin download pitch detector", () => {
+  const darwinRed = pitchCaptures.captures.find(c => c.case === 'tpa-absent-darwin' && c.run === '39075977549')!;
+
+  test("the captured d6d82fb pitch (gate 39075977549) counts: the canonical phrase mid-sentence", () => {
+    expect(darwinRed.passed).toBe(false);
+    expect(darwinRed.text).toContain('you can download Aside (macOS 15+) at aside.com');
+    expect(darwinRed.text).toMatch(asideDownloadPitch);
+    expect(asideDownloadPitchCount(darwinRed.text)).toBe(1);
+  });
+
+  test("every captured passing Darwin pitch counts exactly once", () => {
+    const passing = pitchCaptures.captures.filter(c => c.case === 'tpa-absent-darwin' && c.passed);
+    expect(passing.length).toBeGreaterThanOrEqual(4);
+    for (const c of passing) expect(asideDownloadPitchCount(c.text), c.run).toBe(1);
+  });
+
+  test("captured Linux and READY narrations and both recovery-public responses never count", () => {
+    const narrations = pitchCaptures.captures.filter(c => c.case !== 'tpa-absent-darwin');
+    expect(narrations.map(c => c.case).sort()).toEqual([...Array(5).fill('tpa-absent-linux'), ...Array(5).fill('tpa-present')]);
+    for (const c of narrations) {
+      expect(c.text, `${c.case} ${c.run}`).not.toMatch(asideDownloadPitch);
+      expect(asideDownloadPitchCount(c.text)).toBe(0);
+    }
+    expect(recoveryFixture.responses).toHaveLength(2);
+    for (const response of recoveryFixture.responses) {
+      expect(response.text).toMatch(/no (?:Aside )?download pitch/);
+      expect(response.text).not.toMatch(asideDownloadPitch);
+    }
+  });
+
+  test("short forms count only at a sentence start; negated canonical phrases never count", () => {
+    expect('you can download it at aside.com').not.toMatch(asideDownloadPitch);
+    expect('you can download Aside at aside.com').not.toMatch(asideDownloadPitch);
+    expect('Download it at aside.com').toMatch(asideDownloadPitch);
+    expect('Note: download Aside at aside.com.').toMatch(asideDownloadPitch);
+    expect('Do not Download Aside (macOS 15+) at aside.com').not.toMatch(asideDownloadPitch);
+    expect("You don't need to download Aside (macOS 15+) at aside.com").toMatch(asideDownloadPitch);
+    expect("I won't download Aside (macOS 15+) at aside.com").not.toMatch(asideDownloadPitch);
+  });
+
+  test("a duplicated canonical phrase counts twice", () => {
+    expect(asideDownloadPitchCount('Download Aside (macOS 15+) at aside.com; open, sign in, re-run. Again: you can download Aside (macOS 15+) at aside.com.')).toBe(2);
+  });
+});