From f9e37dacb4820b4822fdc9844a54766e3b5d539b Mon Sep 17 00:00:00 2001
From: Garry Tan <garrytan@gmail.com>
Date: Sun, 16 Aug 2026 19:42:04 -0700
Subject: [PATCH] ci(evals): pre-seed one-time preamble markers for PTY smokes
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Root cause of the documented intermittent scope-gate-question-NOT-observed
failure (test/skill-e2e-plan-mode-no-op.test.ts, also PR #2593 rounds 3/11):
on a fresh runner every one-time preamble marker is missing, so each PTY
child runs first-run feature discovery before the behavior under test, and
touching .feature-prompted-model-overlay under ~/.claude/skills/gstack/
trips Claude Code's sensitive-file permission prompt — the run stalls on
that dialog (classified outcome=asked) and the scope gate never renders.
Dev machines never reproduce it because the operator's markers exist.

Seed ~/.gstack one-time markers (.activated, .first-loop-tip-shown,
.telemetry-prompted, .proactive-prompted, .completeness-intro-seen,
.plan-tune-nudge-shown) and both .feature-prompted-* markers (via the
gstack root symlink into the checkout) in the PTY-smoke registration step,
so no first-run prompt can preempt the assertion under test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 .github/workflows/evals.yml | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/.github/workflows/evals.yml b/.github/workflows/evals.yml
index 5400b19d4..40c775135 100644
--- a/.github/workflows/evals.yml
+++ b/.github/workflows/evals.yml
@@ -284,6 +284,27 @@ jobs:
             cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md"
             cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections"
           done
+          # Pre-seed every ONE-TIME preamble marker so no PTY child ever takes a
+          # first-run branch mid-test. On a fresh runner these are all missing, so
+          # each smoke's preamble fires feature discovery / telemetry / lake-intro
+          # prompts before the behavior under test — and touching the
+          # feature-discovery marker under ~/.claude/skills/gstack/ trips Claude
+          # Code's sensitive-file permission prompt, stalling the run before the
+          # scope gate renders (the documented intermittent
+          # scope-gate-question-NOT-observed failure: outcome=asked was the
+          # permission dialog, not the gate). Dev machines never hit this because
+          # the operator's markers already exist; CI must seed them explicitly.
+          mkdir -p "$HOME/.gstack"
+          touch "$HOME/.gstack/.activated" \
+                "$HOME/.gstack/.first-loop-tip-shown" \
+                "$HOME/.gstack/.telemetry-prompted" \
+                "$HOME/.gstack/.proactive-prompted" \
+                "$HOME/.gstack/.completeness-intro-seen" \
+                "$HOME/.gstack/.plan-tune-nudge-shown"
+          # These two resolve through the gstack root symlink into $REPO —
+          # untracked scratch in the CI checkout, exactly where the preamble looks.
+          touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \
+                "$SKILLS_DIR/gstack/.feature-prompted-model-overlay"
           echo "--- registry under $SKILLS_DIR ---"
           ls -la "$SKILLS_DIR/gstack" "$SKILLS_DIR/office-hours" "$SKILLS_DIR/plan-ceo-review"
           # Fail fast if any committed target moved/renamed — a dangling symlink