ci: Trunk Dynamic CI observes free-suite and macOS regressions, learning only (GST-1400) (#4012)

* ci: Trunk Dynamic CI observes free-suite and macos-named-regressions, learning only (GST-1400)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog.d: Trunk Dynamic CI learning-only fragment (GST-1400)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: name the observer check red case and the required-check vote caveat (GST-1400 review)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: a missing Trunk token skips the step, not the job (GST-1400 fix-check)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: secret guards catch lowercase, bracket and toJSON references; docs cite the pinned source (GST-1400 voices)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog.d: name job-start failures as the one red case (prose lint)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

.github/workflows/free-tests.yml

@@ -8,6 +8,8 @@ name: Free Tests
 # provider keys at all — least privilege, and fork PRs get real test signal
 # here (the eval matrix skips fork PRs because repository secrets can't reach
 # them). test/free-tests-workflow-wiring.test.ts fails CI if a secret sneaks in.
+# The one exception is TRUNK_API_TOKEN in the trunk-dynamic-ci observer, which
+# runs no pull-request code and which nothing depends on (see that job).
 #
 # This is a REQUIRED check from day one (branch protection lists it). If it's
 # red, fix or quarantine-with-issue — don't make it advisory; an advisory lane
@@ -82,6 +84,41 @@ permissions:
   contents: read
 
 jobs:
+  # Trunk Dynamic CI, LEARNING ONLY (GST-1400). Trunk records, for each
+  # same-repository pull request, whether it would have run or skipped the two
+  # named jobs below; every job still runs. It is an observer by construction:
+  # no job needs it, no condition reads it, it re-exports no outputs, so a mode
+  # change in Trunk's dashboard cannot skip anything here. It is the one job in
+  # this workflow that sees a secret, so it checks out nothing and runs no
+  # pull-request code: one pinned action and nothing else, with no GITHUB_TOKEN
+  # scopes (Trunk reads the diff itself) and telemetry off. It never fails the
+  # run: the action never fails its step, the step continues on error and has
+  # its own timeout, and the job continues on error. Fork pull requests, pushes
+  # and dispatches skip the job; a missing token skips its step. Review:
+  # docs/TESTING_INTERNALS.md.
+  trunk-dynamic-ci:
+    name: Trunk Dynamic CI (learning only)
+    if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-24.04
+    timeout-minutes: 5
+    continue-on-error: true
+    permissions: {}
+    env:
+      TRUNK_API_TOKEN: ${{ secrets.TRUNK_API_TOKEN }}
+    steps:
+      - name: Record Trunk's verdicts for free-suite and macos-named-regressions (nothing reads them)
+        if: env.TRUNK_API_TOKEN != ''
+        timeout-minutes: 3
+        continue-on-error: true
+        uses: trunk-io/dynamic-ci@5bcddbf3615c2b032c21eed1b514c91164bda4a8 # v0.1.7
+        env:
+          TRUNK_DISABLE_TELEMETRY: 'true'
+        with:
+          token: ${{ env.TRUNK_API_TOKEN }}
+          job-keys: free-suite,macos-named-regressions
+          github-token: ''
+          enable-annotation: 'true'
+
   # Main pushes and pull requests only, and the one job with actions: read (it
   # reads this repository's runs and their tested-tree artifacts). Every suite
   # job below runs unless this job succeeded and reused a run; a skipped or

changelog.d/garrytan-gst-1400-trunk-dynamic-ci-learning.md

@@ -0,0 +1 @@
+- **Trunk Dynamic CI now watches pull request CI in learning mode, and it can't skip anything.** A new `trunk-dynamic-ci` job in `free-tests.yml` asks Trunk, on every pull request from a branch in this repository, whether it would run or skip `free-suite` and `macos-named-regressions`. Trunk records the answer. No job reads the answer, so every test still runs, and every required check keeps its name and its result. The job runs one pinned action. It checks out no code, holds no GitHub token scopes, skips fork pull requests, skips its step when the token is missing, and never fails the run or a required check. The one exception is when the job itself can't start or finish, for example when GitHub can't fetch the pinned action. Then the job's own check shows red, and if a rerun doesn't clear it, the fix is to delete the job. Merge evidence (the `free-tests` aggregate, CI carry, the pregate, the train gate, the backstop and the release gate) doesn't read Trunk. Review the results one week after this merges. The details are in `docs/TESTING_INTERNALS.md`.

docs/TESTING_INTERNALS.md

@@ -1688,6 +1688,58 @@ per file inside a multi-file child and leaves the console stream unchanged.
 up to bun's own test count. Counts are evidence for comparing k values; they
 never decide a verdict.
 
+### Trunk Dynamic CI (learning only)
+
+Since GST-1400, free-tests.yml has a `trunk-dynamic-ci` job ("Trunk Dynamic CI
+(learning only)") that runs Trunk's `trunk-io/dynamic-ci` action on every
+pull request from a branch in this repository. For each run, Trunk records
+whether it would have run or skipped `free-suite` and
+`macos-named-regressions`, and why. It is one verdict per job, so the ten
+free-suite shards share one verdict. The verdicts show up in the job summary,
+on the pull request's Dynamic CI tab and at app.trunk.io/dynamic-ci.
+
+It never skips anything. No job needs the observer, no condition reads its
+verdicts, and it exports no outputs. Switching Trunk to enforcement in its
+dashboard would still skip nothing, because nothing in this workflow reads
+the verdicts. Every test runs, and every required check keeps its name and
+its result. Merge evidence never reads Trunk: the `free-tests` aggregate,
+the CI reuse profile, `gstack-ci-carry`, the pregate, the train gate
+(evals.yml), the main backstop, the release gate and windows-smoke are
+unchanged.
+
+A Trunk problem never turns the run or any required check red. The action
+never fails its own step. The step also continues on error and has a 3-minute
+timeout, and the job continues on error, so the run's conclusion and the
+`free-tests` aggregate keep their own results. One case still shows red: if
+GitHub can't fetch the pinned action at "Set up job" (the action's repository
+or commit disappears), the observer's own check reports failure, and
+`gstack-ship-queue ready` counts every check at the head. Rerun it once;
+if it fails again, delete the job. Fork pull requests, pushes and dispatches
+skip the job. A missing token skips only its step, so the job still fetches
+the action. It is the only job in the workflow that sees a secret
+(`TRUNK_API_TOKEN`), so it checks out nothing and runs none of the pull
+request's code. It holds no GITHUB_TOKEN scopes (`permissions: {}`). At the
+pinned commit (`src/config.ts`, `src/changed-files.ts`), the action makes one
+network call: a request for verdicts to
+`api.trunk.io/v2/dynamic-ci/generate-plan`, at most 3 attempts with a
+30-second budget each. Telemetry is off. With no checkout and an empty
+`github-token`, the action sends no file list, and Trunk reads the diff itself
+through its GitHub App.
+`test/free-tests-workflow-wiring.test.ts` and
+`test/ci-reuse-workflow-wiring.test.ts` lock that shape in place.
+
+**Review date: one week after it merges (2026-11-12 if it merges on
+2026-11-05).** Compare Trunk's would-skip verdicts with what the jobs it
+would have skipped actually found, and look hardest at any would-skip on a run
+where that job failed. Leave out runs where `reuse` skipped the suite: Trunk
+still records a verdict there, but there is no result to compare it with.
+Trunk's `required-check` vote treats both jobs as not
+required, because branch protection names the `free-tests` aggregate and
+not the jobs under it. But `free-tests` needs both jobs and every shard
+receipt, so discount that vote when you judge the would-skip rate. Any skip
+added later also has to pass the aggregate's receipt check. Turning on
+skipping is a separate decision for Garry, made after that review.
+
 ### Windows free lane
 
 Windows has minimal support ([support tiers](support-tiers.md)). The required

test/ci-reuse-workflow-wiring.test.ts

@@ -19,8 +19,8 @@ const ROOT = path.resolve(import.meta.dir, '..');
 const read = (rel: string) => fs.readFileSync(path.join(ROOT, rel), 'utf8');
 const MAIN_PUSH = "github.event_name == 'push' && github.ref == 'refs/heads/main'";
 const FILES = {
-  'free-tests.yml': { aggregate: 'free-tests', profile: 'ci-reuse/free-tests', plan: 'free-plan', exempt: [] as string[], manual: false, reuseIf: `(${MAIN_PUSH}) || github.event_name == 'pull_request'`, pullRequest: true },
-  'windows-free-tests.yml': { aggregate: 'windows-free-tests', profile: 'ci-reuse/windows-free-tests', plan: 'windows-plan', exempt: ['windows-record-durations'], manual: true, reuseIf: MAIN_PUSH, pullRequest: false },
+  'free-tests.yml': { aggregate: 'free-tests', profile: 'ci-reuse/free-tests', plan: 'free-plan', exempt: ['trunk-dynamic-ci'], observer: 'trunk-dynamic-ci', manual: false, reuseIf: `(${MAIN_PUSH}) || github.event_name == 'pull_request'`, pullRequest: true },
+  'windows-free-tests.yml': { aggregate: 'windows-free-tests', profile: 'ci-reuse/windows-free-tests', plan: 'windows-plan', exempt: ['windows-record-durations'], observer: null, manual: true, reuseIf: MAIN_PUSH, pullRequest: false },
 };
 const SKIP = "steps.route.outputs.reused != 'true'";
 const PROFILES = JSON.parse(read('.gstack/pregate.json')).run_profiles as Record<string, { workflow: string; jobs: string[]; artifact: string; events: string[]; os: string }>;
@@ -47,8 +47,9 @@ for (const [file, spec] of Object.entries(FILES)) {
       expect(jobs.reuse.if).toBe(spec.reuseIf);
       expect(jobs.reuse.permissions).toEqual({ contents: 'read', actions: 'read' });
       expect(wf.permissions).toEqual({ contents: 'read' });
-      for (const name of Object.keys(jobs).filter(n => n !== 'reuse')) expect([name, jobs[name].permissions]).toEqual([name, undefined]);
-      expect(source).not.toContain('secrets.');
+      for (const name of Object.keys(jobs).filter(n => n !== 'reuse' && n !== spec.observer)) expect([name, jobs[name].permissions]).toEqual([name, undefined]);
+      if (spec.observer) expect([jobs[spec.observer].permissions, jobs[spec.observer].needs, jobs[spec.observer].steps.some((s: any) => s.uses?.startsWith('actions/checkout@'))]).toEqual([{}, undefined, false]);
+      expect(source.match(/secrets\s*(?:\.|\[)\s*['"]?[A-Za-z0-9_-]*|toJSON\(\s*secrets\b/gi) ?? []).toEqual(spec.observer ? ['secrets.TRUNK_API_TOKEN'] : []);
       expect(source).not.toContain('pull_request_target');
       const find = jobs.reuse.steps.find((s: any) => s.id === 'find');
       expect(find.env).toEqual({ GH_TOKEN: '${{ github.token }}' });

test/cso-distribution.test.ts

@@ -402,7 +402,8 @@ describe('CSO runtime staging gates', () => {
     expect(job).toBeTruthy();
     const gate = job.steps.find((step: any) => step.run?.trim() === 'bun test --max-concurrency 1 test/cso-docker-integration.test.ts');
     expect(gate.env.GSTACK_CSO_DOCKER_TESTS).toBe('1');
-    expect(raw).not.toContain('secrets.');
+    expect(JSON.stringify(job)).not.toContain('secrets.');
+    expect(raw.match(/secrets\s*(?:\.|\[)\s*['"]?[A-Za-z0-9_-]*|toJSON\(\s*secrets\b/gi)).toEqual(['secrets.TRUNK_API_TOKEN']);
     expect(job.steps.map((s: any) => s.run ?? '').join('\n')).toContain('docker --host unix:///var/run/docker.sock info');
     expect(job['continue-on-error']).not.toBe(true);
     expect(gate['continue-on-error']).not.toBe(true);

test/free-tests-workflow-wiring.test.ts

@@ -5,9 +5,11 @@
  *   1. It invokes the CANONICAL runner (bun run test:free), not a raw
  *      `bun test <dirs>` glob — the runner owns TEST_ROOTS and strict-output
  *      classification, so a truncated run can't report green.
- *   2. It is SECRETLESS: free tests make no API calls, and keeping keys out
- *      means fork PRs get real signal here. Any `secrets.` reference is a
- *      regression.
+ *   2. It is SECRETLESS where code runs: free tests make no API calls, and
+ *      keeping keys out means fork PRs get real signal here. The one allowed
+ *      `secrets.` reference is TRUNK_API_TOKEN in the learning-only
+ *      trunk-dynamic-ci observer, which checks out and runs nothing of the PR's
+ *      and which no job depends on; any other reference is a regression.
  *   3. It triggers on `pull_request` (never `pull_request_target`, which
  *      would hand a fork PR the base repo's context).
  *
@@ -35,11 +37,39 @@ describe('free-tests workflow wiring', () => {
   });
 
   test('secretless: no secrets reach the free lane', () => {
-    expect(source).not.toContain('secrets.');
+    const jobs = (Bun.YAML.parse(source) as any).jobs;
+    expect(source.match(/secrets\s*(?:\.|\[)\s*['"]?[A-Za-z0-9_-]*|toJSON\(\s*secrets\b/gi)).toEqual(['secrets.TRUNK_API_TOKEN']);
+    expect(jobs['trunk-dynamic-ci'].env).toEqual({ TRUNK_API_TOKEN: '${{ secrets.TRUNK_API_TOKEN }}' });
     expect(source).not.toContain('ANTHROPIC_API_KEY');
     expect(source).not.toContain('OPENAI_API_KEY');
   });
 
+  // Value: protects=GST-1400's learning-only contract: Trunk Dynamic CI observes free-suite and macos-named-regressions on same-repository pull requests and can never skip, gate or fail anything; fails_when=a job needs or reads the observer, it re-exports outputs, checks out or runs PR code beside the token, gains token scopes, loses its fail-open settings, runs on forks or non-PR events, or the action is unpinned; seam=.github/workflows/free-tests.yml; no_claim=what Trunk recommends
+  test('trunk-dynamic-ci is a learning-only observer: pinned, scope-less, fail-open, read by no job', () => {
+    const jobs = (Bun.YAML.parse(source) as any).jobs;
+    const trunk = jobs['trunk-dynamic-ci'];
+    expect(trunk.if).toBe("github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository");
+    expect(trunk.permissions).toEqual({});
+    expect(trunk['continue-on-error']).toBe(true);
+    expect(trunk['timeout-minutes']).toBe(5);
+    expect(trunk.outputs).toBeUndefined();
+    expect(trunk.needs).toBeUndefined();
+    expect(trunk.steps).toHaveLength(1);
+    const [step] = trunk.steps;
+    expect(step.uses).toBe('trunk-io/dynamic-ci@5bcddbf3615c2b032c21eed1b514c91164bda4a8');
+    expect(step.run).toBeUndefined();
+    expect(step.if).toBe("env.TRUNK_API_TOKEN != ''");
+    expect(step['continue-on-error']).toBe(true);
+    expect(step['timeout-minutes']).toBe(3);
+    expect(step.env).toEqual({ TRUNK_DISABLE_TELEMETRY: 'true' });
+    expect(step.with).toEqual({ token: '${{ env.TRUNK_API_TOKEN }}', 'job-keys': 'free-suite,macos-named-regressions', 'github-token': '', 'enable-annotation': 'true' });
+    for (const key of step.with['job-keys'].split(',')) expect(jobs[key], key).toBeDefined();
+    for (const name of Object.keys(jobs).filter(n => n !== 'trunk-dynamic-ci')) {
+      expect([name, [jobs[name].needs ?? []].flat()]).not.toEqual([name, expect.arrayContaining(['trunk-dynamic-ci'])]);
+      expect([name, JSON.stringify(jobs[name])]).not.toEqual([name, expect.stringMatching(/trunk|dynamic-ci/i)]);
+    }
+  });
+
   test('pull_request trigger, never pull_request_target', () => {
     expect(source).toContain('pull_request:');
     expect(source).not.toContain('pull_request_target');