diff --git a/AGENTS.md b/AGENTS.md index 92f2331b4..d18d3f598 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -155,6 +155,9 @@ When fixing failures or preparing `/ship`, follow this order: only when that prose is the contract. Do not lower thresholds, increase model budgets, skip cases, or rejudge a failure to manufacture a pass. A pre-registered fixed panel is not rejudging. + For any measured speedup, regression or eval delta, name the limiter that + moved and the run-to-run variation; a gap smaller than that variation is no + difference. For policy or validation repairs, exercise the actual registered callback with representative native input and assert that it uses the helper’s result. When renderer or parser failures recur at the same boundary, verify the diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d5bb3af6..87263afda 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## [1.91.265.0] - 2026-10-27 + +**Four more principles, and three places gstack stops asking what it could observe (pstack absorb, lane C3, GSTA-124).** Ideas from Lauren Tan's pstack (MIT), written in gstack's words; NOTICE.md and the ETHOS.md credit line say so. + +### Added +- **ETHOS.md principles 5 to 8**: Attack the Premise (two fixes that share a premise and fail the same gate indict the premise; write it down and falsify it before a third fix), Separate Before Serializing Shared State (a lock is a smell to check; remove the race before refereeing it), Make Operations Idempotent (design for the run that happens twice; ask what happens if the previous run died at every point), Explain the Number (a measurement carries what was measured, the limiter it moved and the run-to-run variation; a gap inside the variation is no difference). "How They Work Together" gains one line each and a Credits section names the source. +- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures, lists the fixture directory names on the word list and the candidate-visible temporary roots it found (`arm---*` from `test/helpers/arm-benchmark-harness.ts`, `skill-e2e-benchmark-*` from `test/skill-e2e-deploy.test.ts`, the `skill-e2e--*` roots of `test/skill-e2e-test-value.test.ts` whose ids carry `test`, `gstack-health-eval-*` from `test/skill-e2e-health.test.ts`; the list is a scan of `mkdtemp` prefixes in e2e tests, not a proof of completeness); no fixture is renamed here. +- **Observable questions are not the user's to answer**: a rule in `/plan-tune`'s Important Rules and a section in `docs/proactive.md`. A question a local, read-only, free command, render or measurement can settle is settled that way and reported (a call that changes state, reaches a live or third-party system or spends money stays a question); only product and preference calls reach the user, with the observation already made. The question-log audit treats an observable question as a defect in the asking skill rather than a preference to tune. + +### Changed +- **AGENTS.md validation discipline step 4**: one sentence on measured deltas: name the limiter that moved and the run-to-run variation; a gap smaller than that variation is no difference. `agents-digest/gstack-AGENTS.md` is generated from `scripts/gen-agents-digest.ts` and `VERSION`, not from AGENTS.md, so the regenerated digest is unchanged. + +### Gate integrity +- No threshold, trial count, case kind or judge prompt changed. Prose-only change: `plan-tune` and `eval-plan` are unpinned skills; sizes measured in the PR body. + ## [1.91.264.0] - 2026-10-27 **/design-motion turns an approved design into a showcase loop for X and the website; adapted from Elaya's design-to-motion (GSTA-92 plan item 15, Garry 2026-10-25 "Do it").** gstack could design a screen and build it, but not show it moving: a screen recording drops frames, jumps at the loop seam, and nothing tells you a card drifted out of its container. /design-motion rebuilds the exact frame from a /design-shotgun PNG, a /design-html `finalized.html` or a URL (a Figma link when `FIGMA_TOKEN` is set), loads it from an empty state, gives each visual one story, records frames on a fake clock, and exports only after the loop is proven seamless. diff --git a/ETHOS.md b/ETHOS.md index 4a4eef033..a8b5e4cba 100644 --- a/ETHOS.md +++ b/ETHOS.md @@ -173,11 +173,111 @@ against, and gstack's countermeasure for each, lives in --- +## 5. Attack the Premise + +A fix is an argument: "the failure happens because X, so changing Y ends it." +When the gate rejects the fix, the usual move is to adjust Y and try again. +Do that once. When a second fix built on the same X also fails the same gate, +the suspect is no longer Y. It is X. Write the premise down as a sentence that +can be false ("the retry fires before the lock is released"), then design the +cheapest observation that would falsify it, and run that before any third fix. +Two failures that share a premise are evidence about the premise, and a third +attempt that keeps it is a guess wearing a fix's clothes. + +The same test applies to the request itself. "Make this faster" assumes the +slow part is where the asker thinks it is; "add a retry" assumes the failure is +transient. Before building on a premise someone handed you, state it back and +check it against something observable: a profile, a log line, a reproduced +failure. Most wasted work in this craft is excellent work on a false premise. + +**Anti-patterns:** +- "Third time's the charm." (Two shared-premise failures mean the premise is the bug.) +- Rephrasing the same fix as a different one. (Same X, same gate, same result.) +- Treating the task description as ground truth. (It is a hypothesis you were handed.) + +--- + +## 6. Separate Before Serializing Shared State + +When two writers collide, the reflex is a lock, a queue or a retry loop. Each +of those keeps the collision and pays for it forever: a lock serializes every +caller to protect the rare conflict, a queue adds a component that can fill, a +retry turns one race into many. So when the sentence "we need a lock" comes up, +hear it as a prompt to look at the design, not as the fix. Ask first whether the +writers need to share the state at all. One file per writer with a merge at read time, a per-unit +directory, an append-only log keyed by author, a partition by owner: each of +these removes the race instead of refereeing it, and leaves nothing to deadlock. + +This is why a lane in a gstack crew owns its files rather than taking turns on +shared ones, and why the few shared files that remain (the changelog, the +notices) are append-only and merged by section. Serialize only when the state is +truly singular (a version counter, the merge onto main), and then say so and +name the single writer. + +**Anti-patterns:** +- A mutex around a cache that could have been per-caller. +- A retry loop on a write conflict that a per-writer file would never produce. +- "Only one of these runs at a time" as an unwritten assumption. + +--- + +## 7. Make Operations Idempotent + +Any operation that touches state outside the process will one day run twice: +a retry after a timeout, an agent resumed from a checkpoint, a user pressing +the key again. Design for that run. An idempotent operation reaches the same +end state whether it runs once or five times, and a run that crashed halfway +can simply be run again rather than repaired by hand. + +The working question: for each line of the operation, suppose the previous run +died right after it. What does this run find, and does it still converge? Walk +the operation line by line and answer that for every line. A create that +checks for the existing row, a write that lands in a temp file and renames, a +"done" marker written last, a cleanup that tolerates an already-missing file: +each answer removes a class of manual recovery. Where true idempotence is +impossible (sending an email, charging a card), make the operation +distinguishable instead: a key the receiver can deduplicate on, recorded before +the send. + +**Anti-patterns:** +- A setup script that fails on its second run because the directory exists. +- A migration that appends on every run instead of converging. +- "Run it again" as advice without knowing what the first run left behind. + +--- + +## 8. Explain the Number + +A measurement reported without its mechanism is a rumor. "40% faster" is not a +result until you can say what got faster, why that change caused it, and what +the run-to-run noise was. If you cannot explain the number, you do not know +whether you measured your change, a warm cache, a quieter machine, or a +different input, and the next person cannot tell either. + +So every number gstack reports carries three things: the thing measured and how +(the command, the sample size, the clock), the limiter it moved (CPU, I/O, a +model's output length, a sample that was too small), and the variation +(repeated runs, or an honest "one run, so no claim about the delta"). A gap +smaller than the variation is not a difference; say "no measurable change" and +mean it. A surprising number is a lead to investigate, not a headline to post: +the eval that jumps twenty points usually found a shortcut, not a capability. + +**Anti-patterns:** +- A speedup from one run on each side. +- A pass rate without the trial count. +- "Faster" with no theory of what changed. + +--- + ## How They Work Together Boil the Ocean says: **do the complete thing.** Search Before Building says: **know what exists before you decide what to build.** Honest Work says: **the proof is the work, and a report is a claim until re-executed.** +Attack the Premise says: **when fixes keep failing, test the belief they share.** +Separate Before Serializing says: **remove the race before you referee it.** +Make Operations Idempotent says: **design for the run that happens twice.** +Explain the Number says: **a result you cannot explain is not yet a result.** Together: search first, then build the complete version of the right thing. The worst outcome is building a complete version of something that already @@ -194,3 +294,13 @@ wanted it. Every feature was built because it was needed, not because it was requested. If you're building something for yourself, trust that instinct. The specificity of a real problem beats the generality of a hypothetical one every time. + +--- + +## Credits + +Principles 5 through 8 (Attack the Premise, Separate Before Serializing Shared +State, Make Operations Idempotent, Explain the Number) are gstack's statement of +four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins, its +`pstack/skills/principle-*` leaves and its `poteto-mode` skill, MIT). The words +here are gstack's; see NOTICE.md. diff --git a/NOTICE.md b/NOTICE.md index c0012a392..274809e6f 100644 --- a/NOTICE.md +++ b/NOTICE.md @@ -114,3 +114,4 @@ design is not a defect, missing context; independent agreement as the strongest `NO FINDINGS` as a correct result) in `scripts/resolvers/review-army.ts` stage 4 and `docs/review-policy.md` are written in gstack's words. No sentence is copied, so no license text is carried; this entry is a courtesy. +Also ideas, not text, from `pstack/skills/principle-*`, `poteto-mode` and its eval playbook: ETHOS.md principles 5 to 8, the observable-questions rule in `plan-tune/SKILL.md.tmpl` and `docs/proactive.md`, the blinding rules in `eval-plan/SKILL.md.tmpl` and `docs/TESTING_INTERNALS.md`, and the limiter sentence in AGENTS.md step 4 (GSTA-124). diff --git a/VERSION b/VERSION index 08e34f164..faf3b5765 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.264.0 +1.91.265.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index fe65cd75b..e80a8f112 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.264.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.265.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index aac8bef44..cc35d83fb 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1527,6 +1527,53 @@ the `recordGraceMs` option of `test/helpers/office-hours-attempt.ts` (default 5000) and the `timeoutMs` argument of `clearCookieTargetStorage(page, origin, timeoutMs)` (default 15000). +## Candidate-side blinding + +A paid case has two readers that must not know what the other knows: the +candidate (the agent under test) must not know it is being graded, and the +judge must not know which arm it is grading. `/eval-plan` Step 1c states the +three rules a preregistration carries; this is what they mean for gstack's own +fixtures. + +- **The candidate's view carries no eval word.** Nothing the candidate can read + (its working directory, file names, the task prompt, commit messages, the + environment) says `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, + `compare`, `benchmark`, `candidate` or `arena`. The project's own `test/` + directory is exempt when the task is about those tests, and so is the skill + text an arm loads as its own instructions (a gstack SKILL.md and the ETHOS.md it + points at use these words freely); the rule governs what the case builds. The + hermetic home a case builds (`test/helpers/hermetic-env.ts`) is the boundary: + what is copied in is visible, what stays in the runner is not. +- **Fixture roots are project-shaped.** A fixture the candidate works in is + named like the project it imitates, with the files that project would hold. + Runner-side fixtures (captured transcripts, judge calibration sets, census + reports) are never shown to a candidate and may keep descriptive names. +- **The judge grades labels and reads, not claims.** A judge prompt names arms + `A` and `B`, never the model or run, and a chain-following dimension is scored + from the files the transcript shows the candidate opened, not from what the + candidate says it read. + +An audit on 2026-10-26 (GSTA-124) found these fixture directory names on the +word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/`, +`cso-eval/`, `eval-history/`, `judge-calibration/` and its children, two shard +names under `paid-report-census/` and the `test/` subdirectories of +`state-matrix-6792/`. All of these are runner-side: no candidate's working +directory is one of them. Candidate-visible gaps are in the temporary roots +the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture +into `os.tmpdir()/arm---*`, so the candidate's working directory is +named after the arm rather than after a project; `test/skill-e2e-deploy.test.ts` +runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed +word; `test/skill-e2e-test-value.test.ts` creates `skill-e2e--*` roots as the +candidate's working directory, and two of its ids (`review-test-value`, +`test-audit-report-only`) carry the word `test`; and `test/skill-e2e-health.test.ts` +creates `gstack-health-eval-*`. That list comes from a scan of the `mkdtemp` +prefixes in the e2e tests, not from a proof that no other root carries a listed +word. The audit matched whole +lower-case words in path components under `test/fixtures/`; a substring match +(`factory-scorecard/`, `ios-qa/FixtureApp/Tests/`) was not counted. Renaming is a +separate change with its own touchfile updates; the list is here so that change +can be scoped. + ## Running evals as an agent: detach Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in diff --git a/docs/proactive.md b/docs/proactive.md index b47dda6e4..c2e7dba96 100644 --- a/docs/proactive.md +++ b/docs/proactive.md @@ -56,3 +56,21 @@ if you stored `off` before that, it takes the answer as given and does not ask. If collisions with other skill packs keep coming up under `core`, the fallback Garry approved on 2026-10-25 is `off` by default. + +## Questions a skill asks on its own + +The setting above governs which skills gstack offers unasked. A second rule +governs what a running skill may ask you unasked: a question whose answer the +skill could observe is not yours to answer. Whether a change is faster, whether +two renders differ, whether an endpoint accepts a payload, whether a test still +fails: each of those is settled by running the thing (time it, render both, +call it, run the test) and reporting the result, not by a prompt. That holds +for observations that are local, read-only and free; a call that changes state, +reaches a live or third-party system or spends money is still yours to approve. +That extends the one-way-door rule in `/plan-tune`, which covers destructive, +architectural and security questions, to spend and third-party reach. The only +questions that come to you are product and preference calls (what the product +should do, what you want, what you are willing to spend or risk), and they +arrive with the observation already in hand so the choice is between measured +options. `/plan-tune` applies the same rule when it audits the question log: an +observable question is a defect in the asking skill, not a preference to tune. diff --git a/eval-plan/SKILL.md b/eval-plan/SKILL.md index c0dc58ba2..2f5fca02b 100644 --- a/eval-plan/SKILL.md +++ b/eval-plan/SKILL.md @@ -427,6 +427,36 @@ newest first; `docs/eval-models.md` in gstack), `arms`, `dry-run`, `pilot start` the default is `full`. A refused arm (`EVAL_MODEL_DENIED`, `EVAL_MODEL_OLD_GENERATION`, `EVAL_MODEL_GENERATION_UNRESOLVED`) is replaced, never argued past. +### Step 1c: Candidate-side blinding + +A candidate that can tell it is being graded behaves differently from one that +thinks it is at work, and a judge that knows which arm it is reading grades the +label. Three rules hold for every arm, written into the preregistration (a +`## Blinding` section naming the fixture root and the judge's labels) and checked +by reading the fixture tree before the freeze commit; `prereg --check` does not +read fixture trees, so this check is yours: + +1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`, + `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, + `candidate` or `arena` in any directory name, file name, prompt, commit + message or environment variable the preregistration controls and the candidate + can read. Two things are outside the rule: a project's own test directory, when + the task is about those tests, and the skill text an arm loads as its own + instructions (a gstack SKILL.md uses these words freely); the rule is about the + fixture and the task, which the eval builds. +2. **The directory looks like a project.** Name the fixture root the way the + project it imitates would (`billing-service/`, `docs-site/`), with the files + that project would plausibly hold, never `fixture-a/`, `case-7/` or the + model's name. A candidate working in `arm-b/` knows there is an `arm-a/`. +3. **The judge sees sanitized labels and the candidate's real reads.** The judge + gets `A` and `B`, never a model id, arm name, price or run number, and it + grades whether the candidate followed the chain (read the file the task + pointed at, then the file that one named) from the files the transcript shows + it actually opened, never from the candidate's own account of what it read. + +A rule broken after a priced call is a new eval with a new preregistration, the +same as a changed bar. + ## Step 2: zero-cost stub-model dry run Run the pipeline over the real development rows with a stub model that costs nothing: diff --git a/eval-plan/SKILL.md.tmpl b/eval-plan/SKILL.md.tmpl index c541639ec..39ad672ee 100644 --- a/eval-plan/SKILL.md.tmpl +++ b/eval-plan/SKILL.md.tmpl @@ -99,6 +99,36 @@ newest first; `docs/eval-models.md` in gstack), `arms`, `dry-run`, `pilot start` the default is `full`. A refused arm (`EVAL_MODEL_DENIED`, `EVAL_MODEL_OLD_GENERATION`, `EVAL_MODEL_GENERATION_UNRESOLVED`) is replaced, never argued past. +### Step 1c: Candidate-side blinding + +A candidate that can tell it is being graded behaves differently from one that +thinks it is at work, and a judge that knows which arm it is reading grades the +label. Three rules hold for every arm, written into the preregistration (a +`## Blinding` section naming the fixture root and the judge's labels) and checked +by reading the fixture tree before the freeze commit; `prereg --check` does not +read fixture trees, so this check is yours: + +1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`, + `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, + `candidate` or `arena` in any directory name, file name, prompt, commit + message or environment variable the preregistration controls and the candidate + can read. Two things are outside the rule: a project's own test directory, when + the task is about those tests, and the skill text an arm loads as its own + instructions (a gstack SKILL.md uses these words freely); the rule is about the + fixture and the task, which the eval builds. +2. **The directory looks like a project.** Name the fixture root the way the + project it imitates would (`billing-service/`, `docs-site/`), with the files + that project would plausibly hold, never `fixture-a/`, `case-7/` or the + model's name. A candidate working in `arm-b/` knows there is an `arm-a/`. +3. **The judge sees sanitized labels and the candidate's real reads.** The judge + gets `A` and `B`, never a model id, arm name, price or run number, and it + grades whether the candidate followed the chain (read the file the task + pointed at, then the file that one named) from the files the transcript shows + it actually opened, never from the candidate's own account of what it read. + +A rule broken after a priced call is a new eval with a new preregistration, the +same as a changed bar. + ## Step 2: zero-cost stub-model dry run Run the pipeline over the real development rows with a stub model that costs nothing: diff --git a/package.json b/package.json index 68a69de87..2c7bfc2f1 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.264", + "version": "1.91.265", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/plan-tune/SKILL.md b/plan-tune/SKILL.md index 78d6a862f..03215352e 100644 --- a/plan-tune/SKILL.md +++ b/plan-tune/SKILL.md @@ -978,6 +978,19 @@ For background mode (e.g., the user wants to keep working): - **The profile never changes defaults.** This skill inspects and configures. Per-question preferences can auto-decide questions; the declared/inferred profile is advisory only. +- **Observable questions are not the user's to answer.** A question whose + answer a command, a render or a measurement can produce is a question for the + machine: "is this faster" is two timed runs, "which layout reads better" is two + renders side by side, "does the API accept this shape" is one call against a + local or test endpoint. The skill that would have asked does the observation + and reports what it saw, as long as the observation is local, read-only and + free; one that changes state, reaches a live or third-party system or spends + money is a one-way door and stays a question (see the rule above). Only a + product or preference call reaches the user (what the product should do, what + they want, what they will spend or risk), and it arrives with the observation + already made. When the question log shows a prompt whose answer was observable, + the fix belongs in the asking skill, not in a never-ask preference: flag it in + the audit as an observable question and name the check that replaces it. - **Completion status:** - DONE — did what the user asked (enable/inspect/set/update/disable) - DONE_WITH_CONCERNS — action taken but flagging something (e.g., "your diff --git a/plan-tune/SKILL.md.tmpl b/plan-tune/SKILL.md.tmpl index 2aa16edd8..b88d59bed 100644 --- a/plan-tune/SKILL.md.tmpl +++ b/plan-tune/SKILL.md.tmpl @@ -662,6 +662,19 @@ For background mode (e.g., the user wants to keep working): - **The profile never changes defaults.** This skill inspects and configures. Per-question preferences can auto-decide questions; the declared/inferred profile is advisory only. +- **Observable questions are not the user's to answer.** A question whose + answer a command, a render or a measurement can produce is a question for the + machine: "is this faster" is two timed runs, "which layout reads better" is two + renders side by side, "does the API accept this shape" is one call against a + local or test endpoint. The skill that would have asked does the observation + and reports what it saw, as long as the observation is local, read-only and + free; one that changes state, reaches a live or third-party system or spends + money is a one-way door and stays a question (see the rule above). Only a + product or preference call reaches the user (what the product should do, what + they want, what they will spend or risk), and it arrives with the observation + already made. When the question log shows a prompt whose answer was observable, + the fix belongs in the asking skill, not in a never-ask preference: flag it in + the audit as an observable question and name the check that replaces it. - **Completion status:** - DONE — did what the user asked (enable/inspect/set/update/disable) - DONE_WITH_CONCERNS — action taken but flagging something (e.g., "your