From 92002dc2283ae9c109acb3cc8570cf4cff892c6d Mon Sep 17 00:00:00 2001 From: garrytan Date: Mon, 26 Oct 2026 21:10:17 +0000 Subject: [PATCH] docs(ethos, plan-tune, eval-plan): absorb four pstack principles, the observable-questions rule and candidate-side blinding (GSTA-124) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- AGENTS.md | 3 ++ CHANGELOG.md | 15 ++++++ ETHOS.md | 108 ++++++++++++++++++++++++++++++++++++++ NOTICE.md | 1 + docs/TESTING_INTERNALS.md | 36 +++++++++++++ docs/proactive.md | 14 +++++ eval-plan/SKILL.md | 27 ++++++++++ eval-plan/SKILL.md.tmpl | 27 ++++++++++ plan-tune/SKILL.md | 10 ++++ plan-tune/SKILL.md.tmpl | 10 ++++ 10 files changed, 251 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index b49cc15a4..8ebd65145 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -153,6 +153,9 @@ When fixing failures or preparing `/ship`, follow this order: only when that prose is the contract. Do not lower thresholds, increase model budgets, skip cases, or rejudge a failure to manufacture a pass. A pre-registered fixed panel is not rejudging. + For any measured speedup, regression or eval delta, name the limiter that + moved and the run-to-run variation; a gap smaller than that variation is no + difference. For policy or validation repairs, exercise the actual registered callback with representative native input and assert that it uses the helper’s result. When renderer or parser failures recur at the same boundary, verify the diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c0916118..994953df8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## [Unreleased] + +**Four more principles, and three places gstack stops asking what it could observe (pstack absorb, lane C3, GSTA-124).** Ideas from Lauren Tan's pstack (MIT), written in gstack's words; NOTICE.md and the ETHOS.md credit line say so. + +### Added +- **ETHOS.md principles 5 to 8**: Attack the Premise (two fixes that share a premise and fail the same gate indict the premise; write it down and falsify it before a third fix), Separate Before Serializing Shared State (a lock is a smell to check; remove the race before refereeing it), Make Operations Idempotent (design for the run that happens twice; ask what happens if the previous run died at every point), Explain the Number (a measurement carries what was measured, the limiter it moved and the run-to-run variation; a gap inside the variation is no difference). "How They Work Together" gains one line each and a Credits section names the source. +- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures and lists the fixture directory names on the word list (no fixture is renamed here). +- **Observable questions are not the user's to answer**: a rule in `/plan-tune`'s Important Rules and a section in `docs/proactive.md`. A question a command, render or measurement can settle is settled that way and reported; only product and preference calls reach the user, with the observation already made. The question-log audit treats an observable question as a defect in the asking skill rather than a preference to tune. + +### Changed +- **AGENTS.md validation discipline step 4**: one sentence on measured deltas: name the limiter that moved and the run-to-run variation; a gap smaller than that variation is no difference. `agents-digest/gstack-AGENTS.md` is generated from `scripts/gen-agents-digest.ts` and `VERSION`, not from AGENTS.md, so the regenerated digest is unchanged. + +### Gate integrity +- No threshold, trial count, case kind or judge prompt changed. Prose-only change: `plan-tune` and `eval-plan` are unpinned skills; sizes measured in the PR body. + ## [1.91.256.0] - 2026-10-26 **The /ship review loop converges, and a receipt survives a base merge.** Five Captains' /ship runs hit the review cap on the same PR shape: every pass re-reviewed the whole diff, reviewers answered an empty range with an invented advisory, and each advisory earned another full pass; four more re-ran /ship on product code they had not changed because a conflict-only main merge or a test fix moved the head the receipt named. The loop now fixes blocking findings in the diff on pass 1, reads only the range since the previous pass afterwards, caps at three passes, and counts `passes= blocking_found= deferred=` in the ship receipt; the receipt carries to the new head when the product patch-id is unchanged. The blocking floor is the owner's call (Garry, 2026-10-26: "Ok set it at that"), and the policy lives in gstack rather than in any one team's rules (Garry, 2026-10-26: "Gstack should be the actual holder of how to do things not one off Capy files"): `docs/review-policy.md` is the canonical statement, the skills carry it as their written policy, and a crew rule can be one line pointing at it. diff --git a/ETHOS.md b/ETHOS.md index 4a4eef033..22e286830 100644 --- a/ETHOS.md +++ b/ETHOS.md @@ -173,11 +173,110 @@ against, and gstack's countermeasure for each, lives in --- +## 5. Attack the Premise + +A fix is an argument: "the failure happens because X, so changing Y ends it." +When the gate rejects the fix, the usual move is to adjust Y and try again. +Do that once. When a second fix built on the same X also fails the same gate, +the suspect is no longer Y. It is X. Write the premise down as a sentence that +can be false ("the retry fires before the lock is released"), then design the +cheapest observation that would falsify it, and run that before any third fix. +Two failures that share a premise are evidence about the premise, and a third +attempt that keeps it is a guess wearing a fix's clothes. + +The same test applies to the request itself. "Make this faster" assumes the +slow part is where the asker thinks it is; "add a retry" assumes the failure is +transient. Before building on a premise someone handed you, state it back and +check it against something observable: a profile, a log line, a reproduced +failure. Most wasted work in this craft is excellent work on a false premise. + +**Anti-patterns:** +- "Third time's the charm." (Two shared-premise failures mean the premise is the bug.) +- Rephrasing the same fix as a different one. (Same X, same gate, same result.) +- Treating the task description as ground truth. (It is a hypothesis you were handed.) + +--- + +## 6. Separate Before Serializing Shared State + +When two writers collide, the reflex is a lock, a queue or a retry loop. Each +of those keeps the collision and pays for it forever: a lock serializes every +caller to protect the rare conflict, a queue adds a component that can fill, a +retry turns one race into many. Treat "we need a lock" as a design smell to +check before it is the answer. Ask first whether the writers need to share the +state at all. One file per writer with a merge at read time, a per-unit +directory, an append-only log keyed by author, a partition by owner: each of +these removes the race instead of refereeing it, and leaves nothing to deadlock. + +This is why a lane in a gstack crew owns its files rather than taking turns on +shared ones, and why the few shared files that remain (the changelog, the +notices) are append-only and merged by section. Serialize only when the state is +truly singular (a version counter, the merge onto main), and then say so and +name the single writer. + +**Anti-patterns:** +- A mutex around a cache that could have been per-caller. +- A retry loop on a write conflict that a per-writer file would never produce. +- "Only one of these runs at a time" as an unwritten assumption. + +--- + +## 7. Make Operations Idempotent + +Any operation that touches state outside the process will one day run twice: +a retry after a timeout, an agent resumed from a checkpoint, a user pressing +the key again. Design for that run. An idempotent operation reaches the same +end state whether it runs once or five times, and a run that crashed halfway +can simply be run again rather than repaired by hand. + +The working question is: what happens if the previous run died at every +possible point? Walk the operation line by line and answer it. A create that +checks for the existing row, a write that lands in a temp file and renames, a +"done" marker written last, a cleanup that tolerates an already-missing file: +each answer removes a class of manual recovery. Where true idempotence is +impossible (sending an email, charging a card), make the operation +distinguishable instead: a key the receiver can deduplicate on, recorded before +the send. + +**Anti-patterns:** +- A setup script that fails on its second run because the directory exists. +- A migration that appends on every run instead of converging. +- "Run it again" as advice without knowing what the first run left behind. + +--- + +## 8. Explain the Number + +A measurement reported without its mechanism is a rumor. "40% faster" is not a +result until you can say what got faster, why that change caused it, and what +the run-to-run noise was. If you cannot explain the number, you do not know +whether you measured your change, a warm cache, a quieter machine, or a +different input, and the next person cannot tell either. + +So every number gstack reports carries three things: the thing measured and how +(the command, the sample size, the clock), the limiter it moved (CPU, I/O, a +model's output length, a sample that was too small), and the variation +(repeated runs, or an honest "one run, so no claim about the delta"). A gap +smaller than the variation is not a difference; say "no measurable change" and +mean it. A surprising number is a lead to investigate, not a headline to post: +the eval that jumps twenty points usually found a shortcut, not a capability. + +**Anti-patterns:** +- A speedup from one run on each side. +- A pass rate without the trial count. +- "Faster" with no theory of what changed. + +--- + ## How They Work Together Boil the Ocean says: **do the complete thing.** Search Before Building says: **know what exists before you decide what to build.** Honest Work says: **the proof is the work, and a report is a claim until re-executed.** +Attack the Premise says: **when fixes keep failing, test the belief they share.** +Separate Before Serializing says: **remove the race before you referee it.** +Make Operations Idempotent says: **design for the run that happens twice.** +Explain the Number says: **a result you cannot explain is not yet a result.** Together: search first, then build the complete version of the right thing. The worst outcome is building a complete version of something that already @@ -194,3 +293,12 @@ wanted it. Every feature was built because it was needed, not because it was requested. If you're building something for yourself, trust that instinct. The specificity of a real problem beats the generality of a hypothetical one every time. + +--- + +## Credits + +Principles 5 through 8 (Attack the Premise, Separate Before Serializing Shared +State, Make Operations Idempotent, Explain the Number) are gstack's statement of +four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins, +`pstack/skills/principle-*`, MIT). The words here are gstack's; see NOTICE.md. diff --git a/NOTICE.md b/NOTICE.md index 0d26c6b8f..7c97e604e 100644 --- a/NOTICE.md +++ b/NOTICE.md @@ -95,3 +95,4 @@ design is not a defect, missing context; independent agreement as the strongest `NO FINDINGS` as a correct result) in `scripts/resolvers/review-army.ts` stage 4 and `docs/review-policy.md` are written in gstack's words. No sentence is copied, so no license text is carried; this entry is a courtesy. +Also ideas, not text, from `pstack/skills/principle-*`, `poteto-mode` and its eval playbook: ETHOS.md principles 5 to 8, the observable-questions rule in `plan-tune/SKILL.md.tmpl` and `docs/proactive.md`, the blinding rules in `eval-plan/SKILL.md.tmpl` and `docs/TESTING_INTERNALS.md`, and the limiter sentence in AGENTS.md step 4 (GSTA-124). diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index ed8b885c4..5470ead54 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1499,6 +1499,42 @@ the `recordGraceMs` option of `test/helpers/office-hours-attempt.ts` (default 5000) and the `timeoutMs` argument of `clearCookieTargetStorage(page, origin, timeoutMs)` (default 15000). +## Candidate-side blinding + +A paid case has two readers that must not know what the other knows: the +candidate (the agent under test) must not know it is being graded, and the +judge must not know which arm it is grading. `/eval-plan` Step 1c states the +three rules a preregistration carries; this is what they mean for gstack's own +fixtures. + +- **The candidate's view carries no eval word.** Nothing the candidate can read + (its working directory, file names, the task prompt, commit messages, the + environment) says `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, + `compare`, `benchmark`, `candidate` or `arena`. The project's own `test/` + directory is exempt when the task is about those tests. The hermetic home a + case builds (`test/helpers/hermetic-env.ts`) is the boundary: what is copied + in is visible, what stays in the runner is not. +- **Fixture roots are project-shaped.** A fixture the candidate works in is + named like the project it imitates, with the files that project would hold. + Runner-side fixtures (captured transcripts, judge calibration sets, census + reports) are never shown to a candidate and may keep descriptive names. +- **The judge grades labels and reads, not claims.** A judge prompt names arms + `A` and `B`, never the model or run, and a chain-following dimension is scored + from the files the transcript shows the candidate opened, not from what the + candidate says it read. + +An audit on 2026-10-26 (GSTA-124) found these fixture directory names on the +word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/`, +`cso-eval/`, `eval-history/`, `judge-calibration/` and its children, two shard +names under `paid-report-census/` and the `test/` subdirectories of +`state-matrix-6792/`. All of these are runner-side: no candidate's working +directory is one of them. The one candidate-visible gap is the other rule: +`test/helpers/arm-benchmark-harness.ts` copies each fixture into +`os.tmpdir()/arm---*`, so the candidate's working directory is named +after the arm rather than after a project. The e2e hermetic roots +(`skill-e2e--*`) carry no listed word. Renaming is a separate change with +its own touchfile updates; the list is here so that change can be scoped. + ## Running evals as an agent: detach Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in diff --git a/docs/proactive.md b/docs/proactive.md index b47dda6e4..124771e52 100644 --- a/docs/proactive.md +++ b/docs/proactive.md @@ -56,3 +56,17 @@ if you stored `off` before that, it takes the answer as given and does not ask. If collisions with other skill packs keep coming up under `core`, the fallback Garry approved on 2026-10-25 is `off` by default. + +## Questions a skill asks on its own + +The setting above governs which skills gstack offers unasked. A second rule +governs what a running skill may ask you unasked: a question whose answer the +skill could observe is not yours to answer. Whether a change is faster, whether +two renders differ, whether an endpoint accepts a payload, whether a test still +fails: each of those is settled by running the thing (time it, render both, +call it, run the test) and reporting the result, not by a prompt. The only +questions that come to you are product and preference calls (what the product +should do, what you want, what you are willing to spend or risk), and they +arrive with the observation already in hand so the choice is between measured +options. `/plan-tune` applies the same rule when it audits the question log: an +observable question is a defect in the asking skill, not a preference to tune. diff --git a/eval-plan/SKILL.md b/eval-plan/SKILL.md index c0dc58ba2..905e434a4 100644 --- a/eval-plan/SKILL.md +++ b/eval-plan/SKILL.md @@ -427,6 +427,33 @@ newest first; `docs/eval-models.md` in gstack), `arms`, `dry-run`, `pilot start` the default is `full`. A refused arm (`EVAL_MODEL_DENIED`, `EVAL_MODEL_OLD_GENERATION`, `EVAL_MODEL_GENERATION_UNRESOLVED`) is replaced, never argued past. +### Step 1c: Candidate-side blinding + +A candidate that can tell it is being graded behaves differently from one that +thinks it is at work, and a judge that knows which arm it is reading grades the +label. Three rules hold for every arm, written into the preregistration (a +`## Blinding` section naming the fixture root and the judge's labels) and checked +by reading the fixture tree before the freeze commit; `prereg --check` does not +read fixture trees, so this check is yours: + +1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`, + `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, + `candidate` or `arena` in any directory name, file name, prompt, commit + message or environment variable the candidate can read. A project's own test + directory is the one exception, and only when the task is about those tests. +2. **The directory looks like a project.** Name the fixture root the way the + project it imitates would (`billing-service/`, `docs-site/`), with the files + that project would plausibly hold, never `fixture-a/`, `case-7/` or the + model's name. A candidate working in `arm-b/` knows there is an `arm-a/`. +3. **The judge sees sanitized labels and the candidate's real reads.** The judge + gets `A` and `B`, never a model id, arm name, price or run number, and it + grades whether the candidate followed the chain (read the file the task + pointed at, then the file that one named) from the files the transcript shows + it actually opened, never from the candidate's own account of what it read. + +A rule broken after a priced call is a new eval with a new preregistration, the +same as a changed bar. + ## Step 2: zero-cost stub-model dry run Run the pipeline over the real development rows with a stub model that costs nothing: diff --git a/eval-plan/SKILL.md.tmpl b/eval-plan/SKILL.md.tmpl index c541639ec..600e95c89 100644 --- a/eval-plan/SKILL.md.tmpl +++ b/eval-plan/SKILL.md.tmpl @@ -99,6 +99,33 @@ newest first; `docs/eval-models.md` in gstack), `arms`, `dry-run`, `pilot start` the default is `full`. A refused arm (`EVAL_MODEL_DENIED`, `EVAL_MODEL_OLD_GENERATION`, `EVAL_MODEL_GENERATION_UNRESOLVED`) is replaced, never argued past. +### Step 1c: Candidate-side blinding + +A candidate that can tell it is being graded behaves differently from one that +thinks it is at work, and a judge that knows which arm it is reading grades the +label. Three rules hold for every arm, written into the preregistration (a +`## Blinding` section naming the fixture root and the judge's labels) and checked +by reading the fixture tree before the freeze commit; `prereg --check` does not +read fixture trees, so this check is yours: + +1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`, + `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, + `candidate` or `arena` in any directory name, file name, prompt, commit + message or environment variable the candidate can read. A project's own test + directory is the one exception, and only when the task is about those tests. +2. **The directory looks like a project.** Name the fixture root the way the + project it imitates would (`billing-service/`, `docs-site/`), with the files + that project would plausibly hold, never `fixture-a/`, `case-7/` or the + model's name. A candidate working in `arm-b/` knows there is an `arm-a/`. +3. **The judge sees sanitized labels and the candidate's real reads.** The judge + gets `A` and `B`, never a model id, arm name, price or run number, and it + grades whether the candidate followed the chain (read the file the task + pointed at, then the file that one named) from the files the transcript shows + it actually opened, never from the candidate's own account of what it read. + +A rule broken after a priced call is a new eval with a new preregistration, the +same as a changed bar. + ## Step 2: zero-cost stub-model dry run Run the pipeline over the real development rows with a stub model that costs nothing: diff --git a/plan-tune/SKILL.md b/plan-tune/SKILL.md index 78d6a862f..7e79cfde1 100644 --- a/plan-tune/SKILL.md +++ b/plan-tune/SKILL.md @@ -978,6 +978,16 @@ For background mode (e.g., the user wants to keep working): - **The profile never changes defaults.** This skill inspects and configures. Per-question preferences can auto-decide questions; the declared/inferred profile is advisory only. +- **Observable questions are not the user's to answer.** A question whose + answer a command, a render or a measurement can produce is a question for the + machine: "is this faster" is two timed runs, "which layout reads better" is two + renders side by side, "does the API accept this shape" is one call. The skill + that would have asked does the observation and reports what it saw. Only a + product or preference call reaches the user (what the product should do, what + they want, what they will spend or risk), and it arrives with the observation + already made. When the question log shows a prompt whose answer was observable, + the fix belongs in the asking skill, not in a never-ask preference: flag it in + the audit as an observable question and name the check that replaces it. - **Completion status:** - DONE — did what the user asked (enable/inspect/set/update/disable) - DONE_WITH_CONCERNS — action taken but flagging something (e.g., "your diff --git a/plan-tune/SKILL.md.tmpl b/plan-tune/SKILL.md.tmpl index 2aa16edd8..83797838d 100644 --- a/plan-tune/SKILL.md.tmpl +++ b/plan-tune/SKILL.md.tmpl @@ -662,6 +662,16 @@ For background mode (e.g., the user wants to keep working): - **The profile never changes defaults.** This skill inspects and configures. Per-question preferences can auto-decide questions; the declared/inferred profile is advisory only. +- **Observable questions are not the user's to answer.** A question whose + answer a command, a render or a measurement can produce is a question for the + machine: "is this faster" is two timed runs, "which layout reads better" is two + renders side by side, "does the API accept this shape" is one call. The skill + that would have asked does the observation and reports what it saw. Only a + product or preference call reaches the user (what the product should do, what + they want, what they will spend or risk), and it arrives with the observation + already made. When the question log shows a prompt whose answer was observable, + the fix belongs in the asking skill, not in a never-ask preference: flag it in + the audit as an observable question and name the check that replaces it. - **Completion status:** - DONE — did what the user asked (enable/inspect/set/update/disable) - DONE_WITH_CONCERNS — action taken but flagging something (e.g., "your