feat(evals): seedSkills opt-in for PTY slash-command tests + tripwire

Wire ClaudePtyOptions.seedSkills through launchClaudePty: when set (and
hermetic, and no per-test CLAUDE_CONFIG_DIR override), the child gets
hermeticSkillsConfigDir() so typed /skill slash commands resolve instead
of dying as Unknown command before any model turn. Opted in at the three
runPlanSkill* helpers and the four direct-launch slash-command tests
(plan-design-with-ui, plan-ceo-mode-routing, autoplan-chain,
ship-idempotency).

New static tripwire (test/pty-skill-seeding-wiring.test.ts): any test
file that sends a slash command over the PTY must route through a
runPlanSkill* helper or pass seedSkills: true — an unseeded slash-command
test spends money and measures nothing. hermetic-wiring.test.ts now
blesses the repo-tree seeding path explicitly (config dir under runRoot,
symlinks into the repo checkout, never operator ~/.claude).

The CI "Register gstack skills for PTY smoke" step keeps a keep-me note:
container cross-mount symlinks defeat the TUI scanner and HOME is not
hermeticized, so the real-file copies there must survive this change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 63c52269daaffb833b3105ea9b4b99be6df8fec7)
This commit is contained in:
Garry Tan
2026-08-12 15:31:47 -07:00
parent 5819203905
commit 281227262d
8 changed files with 122 additions and 1 deletions
+12 -1
View File
@@ -24,7 +24,7 @@
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { hermeticChildEnv, isHermeticEnabled } from './hermetic-env';
import { hermeticChildEnv, hermeticSkillsConfigDir, isHermeticEnabled } from './hermetic-env';
/** Strip ANSI escapes for pattern-matching against visible text. */
export function stripAnsi(s: string): string {
@@ -61,6 +61,11 @@ export function resolveClaudeBinary(): string | null {
}
export interface ClaudePtyOptions {
/** Register the repo's shipped skills in the child's user scope via
* hermeticSkillsConfigDir(). Required by any test that types a /skill
* slash command; without it hermetic claude rejects the command as
* Unknown before any model turn. No effect when EVALS_HERMETIC=0. */
seedSkills?: boolean;
/**
* Permission mode for the session.
* - 'plan' (default) — launches with --permission-mode plan
@@ -1285,6 +1290,9 @@ export async function launchClaudePty(
// Hermetic by default (test/helpers/hermetic-env.ts): operator session
// context never reaches the child; per-test opts.env merges last.
const childEnv = hermeticChildEnv(opts.env);
if (opts.seedSkills && hermetic && !opts.env?.CLAUDE_CONFIG_DIR) {
childEnv.CLAUDE_CONFIG_DIR = hermeticSkillsConfigDir();
}
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const proc = (Bun as any).spawn([claudePath, ...args], {
@@ -1670,6 +1678,7 @@ export async function runPlanSkillObservation(opts: {
extraArgs: opts.extraArgs,
env: opts.env,
model: opts.model,
seedSkills: true,
});
try {
@@ -1966,6 +1975,7 @@ export async function runPlanSkillCounting(opts: {
timeoutMs: timeoutMs + 60_000,
env: opts.env,
model: opts.model,
seedSkills: true,
});
const fingerprints: AskUserQuestionFingerprint[] = [];
@@ -2199,6 +2209,7 @@ export async function runPlanSkillFloorCheck(opts: {
timeoutMs: timeoutMs + 60_000,
env: opts.env,
model: opts.model,
seedSkills: true,
});
try {