test(W7h): retire the paid SKILL.md-summary cases in skill-e2e-plan

W7h retirement (t14 F6-F8): office-hours-spec-review, plan-ceo-review-benefits
and codex-offered-{office-hours,ceo-review,design-review,eng-review}. Each
asked a model to read a generated SKILL.md, write a summary, and grepped the
summary for facts the generated file states directly.

Registration cleanup in the same commit: E2E_TOUCHFILES, E2E_TIERS (all six
gate) and E2E_KINDS (rule) in test/helpers/touchfiles-data.ts; six
scripts/paid-test-durations.json case-shard entries; no
test/helpers/periodic-exclude-data.ts entries existed. The fast PR profile
(scripts/test-pr-profile.ts) drops plan-ceo-review-benefits and
office-hours-spec-review from PR_PROFILE_CASE_IDS and PR_PROFILE_FILES. Tests
that named the retired ids as live examples move to existing ones:
paid-free-boundary and paid-pr-profile (a CEO template change now plans
auq-format-gate plus the CEO judge; the case-shard fixture uses
plan-review-report), test-pr-profile, touchfiles (CEO selection count
21 -> 19), the touchfile-moved-code w5 golden (retired ids removed), the
behavior-lane case-shard census (78 -> 72) in cookie-validation-phases, and
the skill-e2e-plan timeout list and full-gate PR floor (78,425 -> 72,755 s)
in paid-retry-supervision. e2e-tier-alignment, eval-kinds,
eval-detach-timeout-floor and paid-shards stay green.

Coverage consequence: per the Eng Retirements item, plan-ceo-review-benefits
was the only gate-tier CEO E2E in the fast PR profile, so CEO invariants are
treated as periodic-only under the W7c gate (every CEO conversion in this
branch keeps one free check per invariant); the gate judge
'plan-ceo-review/SKILL.md modes' still runs for CEO prompt changes. An
office-hours prompt change now has no fast PR E2E and is reported as
deferred to broad coverage rather than run.

Retirement cards:
- office-hours-spec-review: detects the Spec Review Loop's stated dimensions,
  dispatch tool, iteration cap and metrics. Replacement: gen-skill-docs.test.ts
  'SPEC_REVIEW_LOOP resolver' (round-3 cap, prepare/check/finalize commands,
  verdict and round-file wiring) and test/office-hours-review.test.ts (verdict
  parsing). Control: changing the round limit in
  scripts/resolvers/spec-review.ts turns gen-skill-docs red.
- plan-ceo-review-benefits: detects the CEO /office-hours prerequisite and
  mid-session offer. Replacement: gen-skill-docs.test.ts 'plan-ceo-review
  offers /office-hours when no design doc exists and detects a lost user
  mid-session' (previous commit). Control: 4/4 red mutations there.
- codex-offered-* (x4): detects the `command -v codex` probe, a fallback and
  optionality. Replacement: gen-skill-docs.test.ts 'Codex outside voice is
  offered behind an executed availability probe' (probe + not_installed
  fallback for all four skills) and skill-validation.test.ts 'plan reviews run
  the Codex outside voice default-on'. Control: 4/4 red there.
- non_test_callers: none; history: c118e24 v1.64.1.0 (#2572).
- validation: the registration tests named above.

Controls log: .context/mutation-skill-e2e-plan-retirements.md.
This commit is contained in:
garrytan committed 2026-10-03 01:47:21 +00:00
1 parent 51f1d947ee
commit fc9049a8b5
11 files changed
+16 -297

No files matched your search

-6
View File
@@ -40,12 +40,6 @@
"test/skill-e2e-plan-prosons.test.ts": 1000,
"test/skill-e2e-plan-tune.test.ts": 63925,
"test/skill-e2e-plan.test.ts": 251000,
"test/skill-e2e-plan.test.ts#codex-offered-ceo-review": 49880,
"test/skill-e2e-plan.test.ts#codex-offered-design-review": 54895,
"test/skill-e2e-plan.test.ts#codex-offered-eng-review": 59818,
"test/skill-e2e-plan.test.ts#codex-offered-office-hours": 50442,
"test/skill-e2e-plan.test.ts#office-hours-spec-review": 33866,
"test/skill-e2e-plan.test.ts#plan-ceo-review-benefits": 48811,
"test/skill-e2e-plan.test.ts#plan-review-report": 76406,
"test/skill-e2e-qa-bugs.test.ts": 1000,
"test/skill-e2e-qa-callers.test.ts": 737164,
+3 -3
View File
@@ -12,8 +12,8 @@ export const PR_PROFILE_CASE_IDS = [
'qa-functional-cli-fix', 'qa-functional-webhook-fix',
'review-exploratory-small-cli', 'ship-exploratory-small-cli', 'ship-exploratory-unavailable',
'ship-exploratory-plan-checks', 'ship-exploratory-late-input',
'plan-ceo-review-benefits', 'plan-eng-coverage-audit', 'plan-review-report',
'auq-format-gate', 'plan-design-review-no-ui-scope', 'office-hours-spec-review',
'plan-eng-coverage-audit', 'plan-review-report',
'auq-format-gate', 'plan-design-review-no-ui-scope',
'tpa-present', 'tpa-absent-linux',
'ship-local-workflow', 'ship-coverage-audit', 'docsync-spawned',
'ship-docsync-completion', 'ship-docsync-current', 'ship-docsync-failure', 'ship-docsync-store',
@@ -41,7 +41,7 @@ export const PR_PROFILE_FILES: Record<string, readonly string[]> = {
'test/skill-e2e-review.test.ts': ['review-sql-injection'],
'test/skill-e2e-coverage-audit.test.ts': ['review-coverage-audit', 'plan-eng-coverage-audit'],
'test/skill-e2e-test-value.test.ts': ['ship-coverage-value', 'review-test-value', 'test-audit-report-only'],
'test/skill-e2e-plan.test.ts': ['plan-ceo-review-benefits', 'plan-review-report', 'office-hours-spec-review'],
'test/skill-e2e-plan.test.ts': ['plan-review-report'],
'test/skill-e2e-ask-user-question-format-compliance.test.ts': ['auq-format-gate'],
'test/skill-e2e-design.test.ts': ['plan-design-review-no-ui-scope'],
'test/skill-e2e-third-party-actions.test.ts': ['tpa-present', 'tpa-absent-linux'],