From 13a40c41b8f2dc2181ec090862bb1005f2dad7c3 Mon Sep 17 00:00:00 2001 From: garrytan Date: Mon, 26 Oct 2026 21:34:46 +0000 Subject: [PATCH] Reword ETHOS sentences, scope the blinding rule, name the second candidate-visible root Review pass 1 (frozen scope, five INFORMATIONAL findings, all in-diff, none blocking): two ETHOS.md sentences in principles 6 and 7 were too close to the credited source and are reworded; the Credits line names both pstack sources (principle-* leaves and poteto-mode); eval-plan Step 1c rule 1 is scoped to what the preregistration controls and exempts the skill text an arm loads; the observable-questions rule in plan-tune and docs/proactive.md requires the observation to be local, read-only and free and points at the one-way-door rule; docs/TESTING_INTERNALS.md names skill-e2e-benchmark-* (test/skill-e2e- deploy.test.ts:293) as a second candidate-visible root and states the audit's matching method. SKILL.md regenerated for both unpinned skills. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 2 +- ETHOS.md | 16 +++++++++------- docs/TESTING_INTERNALS.md | 24 +++++++++++++++--------- docs/proactive.md | 12 +++++++----- eval-plan/SKILL.md | 7 +++++-- eval-plan/SKILL.md.tmpl | 7 +++++-- plan-tune/SKILL.md | 7 +++++-- plan-tune/SKILL.md.tmpl | 7 +++++-- 8 files changed, 52 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 994953df8..bb442604c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ ### Added - **ETHOS.md principles 5 to 8**: Attack the Premise (two fixes that share a premise and fail the same gate indict the premise; write it down and falsify it before a third fix), Separate Before Serializing Shared State (a lock is a smell to check; remove the race before refereeing it), Make Operations Idempotent (design for the run that happens twice; ask what happens if the previous run died at every point), Explain the Number (a measurement carries what was measured, the limiter it moved and the run-to-run variation; a gap inside the variation is no difference). "How They Work Together" gains one line each and a Credits section names the source. -- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures and lists the fixture directory names on the word list (no fixture is renamed here). +- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures, lists the fixture directory names on the word list and names the two candidate-visible roots (`arm---*` from `test/helpers/arm-benchmark-harness.ts`, `skill-e2e-benchmark-*` from `test/skill-e2e-deploy.test.ts`); no fixture is renamed here. - **Observable questions are not the user's to answer**: a rule in `/plan-tune`'s Important Rules and a section in `docs/proactive.md`. A question a command, render or measurement can settle is settled that way and reported; only product and preference calls reach the user, with the observation already made. The question-log audit treats an observable question as a defect in the asking skill rather than a preference to tune. ### Changed diff --git a/ETHOS.md b/ETHOS.md index 22e286830..a8b5e4cba 100644 --- a/ETHOS.md +++ b/ETHOS.md @@ -202,9 +202,9 @@ failure. Most wasted work in this craft is excellent work on a false premise. When two writers collide, the reflex is a lock, a queue or a retry loop. Each of those keeps the collision and pays for it forever: a lock serializes every caller to protect the rare conflict, a queue adds a component that can fill, a -retry turns one race into many. Treat "we need a lock" as a design smell to -check before it is the answer. Ask first whether the writers need to share the -state at all. One file per writer with a merge at read time, a per-unit +retry turns one race into many. So when the sentence "we need a lock" comes up, +hear it as a prompt to look at the design, not as the fix. Ask first whether the +writers need to share the state at all. One file per writer with a merge at read time, a per-unit directory, an append-only log keyed by author, a partition by owner: each of these removes the race instead of refereeing it, and leaves nothing to deadlock. @@ -229,8 +229,9 @@ the key again. Design for that run. An idempotent operation reaches the same end state whether it runs once or five times, and a run that crashed halfway can simply be run again rather than repaired by hand. -The working question is: what happens if the previous run died at every -possible point? Walk the operation line by line and answer it. A create that +The working question: for each line of the operation, suppose the previous run +died right after it. What does this run find, and does it still converge? Walk +the operation line by line and answer that for every line. A create that checks for the existing row, a write that lands in a temp file and renames, a "done" marker written last, a cleanup that tolerates an already-missing file: each answer removes a class of manual recovery. Where true idempotence is @@ -300,5 +301,6 @@ every time. Principles 5 through 8 (Attack the Premise, Separate Before Serializing Shared State, Make Operations Idempotent, Explain the Number) are gstack's statement of -four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins, -`pstack/skills/principle-*`, MIT). The words here are gstack's; see NOTICE.md. +four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins, its +`pstack/skills/principle-*` leaves and its `poteto-mode` skill, MIT). The words +here are gstack's; see NOTICE.md. diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 5470ead54..749f1bc2b 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1511,9 +1511,11 @@ fixtures. (its working directory, file names, the task prompt, commit messages, the environment) says `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, `candidate` or `arena`. The project's own `test/` - directory is exempt when the task is about those tests. The hermetic home a - case builds (`test/helpers/hermetic-env.ts`) is the boundary: what is copied - in is visible, what stays in the runner is not. + directory is exempt when the task is about those tests, and so is the skill + text an arm loads as its own instructions (a gstack SKILL.md and the ETHOS.md it + points at use these words freely); the rule governs what the case builds. The + hermetic home a case builds (`test/helpers/hermetic-env.ts`) is the boundary: + what is copied in is visible, what stays in the runner is not. - **Fixture roots are project-shaped.** A fixture the candidate works in is named like the project it imitates, with the files that project would hold. Runner-side fixtures (captured transcripts, judge calibration sets, census @@ -1528,12 +1530,16 @@ word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/` `cso-eval/`, `eval-history/`, `judge-calibration/` and its children, two shard names under `paid-report-census/` and the `test/` subdirectories of `state-matrix-6792/`. All of these are runner-side: no candidate's working -directory is one of them. The one candidate-visible gap is the other rule: -`test/helpers/arm-benchmark-harness.ts` copies each fixture into -`os.tmpdir()/arm---*`, so the candidate's working directory is named -after the arm rather than after a project. The e2e hermetic roots -(`skill-e2e--*`) carry no listed word. Renaming is a separate change with -its own touchfile updates; the list is here so that change can be scoped. +directory is one of them. Two candidate-visible gaps are in the temporary roots +the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture +into `os.tmpdir()/arm---*`, so the candidate's working directory is +named after the arm rather than after a project, and `test/skill-e2e-deploy.test.ts` +runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed +word; the other `skill-e2e--*` roots do not. The audit matched whole +lower-case words in path components under `test/fixtures/`; a substring match +(`factory-scorecard/`, `ios-qa/FixtureApp/Tests/`) was not counted. Renaming is a +separate change with its own touchfile updates; the list is here so that change +can be scoped. ## Running evals as an agent: detach diff --git a/docs/proactive.md b/docs/proactive.md index 124771e52..d99d96f55 100644 --- a/docs/proactive.md +++ b/docs/proactive.md @@ -64,9 +64,11 @@ governs what a running skill may ask you unasked: a question whose answer the skill could observe is not yours to answer. Whether a change is faster, whether two renders differ, whether an endpoint accepts a payload, whether a test still fails: each of those is settled by running the thing (time it, render both, -call it, run the test) and reporting the result, not by a prompt. The only -questions that come to you are product and preference calls (what the product -should do, what you want, what you are willing to spend or risk), and they -arrive with the observation already in hand so the choice is between measured -options. `/plan-tune` applies the same rule when it audits the question log: an +call it, run the test) and reporting the result, not by a prompt. That holds +for observations that are local, read-only and free; a call that changes state, +reaches a live or third-party system or spends money is still yours to approve, +as the one-way-door rule in `/plan-tune` already says. The only questions that +come to you are product and preference calls (what the product should do, what +you want, what you are willing to spend or risk), and they arrive with the +observation already in hand so the choice is between measured options. `/plan-tune` applies the same rule when it audits the question log: an observable question is a defect in the asking skill, not a preference to tune. diff --git a/eval-plan/SKILL.md b/eval-plan/SKILL.md index 905e434a4..2f5fca02b 100644 --- a/eval-plan/SKILL.md +++ b/eval-plan/SKILL.md @@ -439,8 +439,11 @@ read fixture trees, so this check is yours: 1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, `candidate` or `arena` in any directory name, file name, prompt, commit - message or environment variable the candidate can read. A project's own test - directory is the one exception, and only when the task is about those tests. + message or environment variable the preregistration controls and the candidate + can read. Two things are outside the rule: a project's own test directory, when + the task is about those tests, and the skill text an arm loads as its own + instructions (a gstack SKILL.md uses these words freely); the rule is about the + fixture and the task, which the eval builds. 2. **The directory looks like a project.** Name the fixture root the way the project it imitates would (`billing-service/`, `docs-site/`), with the files that project would plausibly hold, never `fixture-a/`, `case-7/` or the diff --git a/eval-plan/SKILL.md.tmpl b/eval-plan/SKILL.md.tmpl index 600e95c89..39ad672ee 100644 --- a/eval-plan/SKILL.md.tmpl +++ b/eval-plan/SKILL.md.tmpl @@ -111,8 +111,11 @@ read fixture trees, so this check is yours: 1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, `candidate` or `arena` in any directory name, file name, prompt, commit - message or environment variable the candidate can read. A project's own test - directory is the one exception, and only when the task is about those tests. + message or environment variable the preregistration controls and the candidate + can read. Two things are outside the rule: a project's own test directory, when + the task is about those tests, and the skill text an arm loads as its own + instructions (a gstack SKILL.md uses these words freely); the rule is about the + fixture and the task, which the eval builds. 2. **The directory looks like a project.** Name the fixture root the way the project it imitates would (`billing-service/`, `docs-site/`), with the files that project would plausibly hold, never `fixture-a/`, `case-7/` or the diff --git a/plan-tune/SKILL.md b/plan-tune/SKILL.md index 7e79cfde1..03215352e 100644 --- a/plan-tune/SKILL.md +++ b/plan-tune/SKILL.md @@ -981,8 +981,11 @@ For background mode (e.g., the user wants to keep working): - **Observable questions are not the user's to answer.** A question whose answer a command, a render or a measurement can produce is a question for the machine: "is this faster" is two timed runs, "which layout reads better" is two - renders side by side, "does the API accept this shape" is one call. The skill - that would have asked does the observation and reports what it saw. Only a + renders side by side, "does the API accept this shape" is one call against a + local or test endpoint. The skill that would have asked does the observation + and reports what it saw, as long as the observation is local, read-only and + free; one that changes state, reaches a live or third-party system or spends + money is a one-way door and stays a question (see the rule above). Only a product or preference call reaches the user (what the product should do, what they want, what they will spend or risk), and it arrives with the observation already made. When the question log shows a prompt whose answer was observable, diff --git a/plan-tune/SKILL.md.tmpl b/plan-tune/SKILL.md.tmpl index 83797838d..b88d59bed 100644 --- a/plan-tune/SKILL.md.tmpl +++ b/plan-tune/SKILL.md.tmpl @@ -665,8 +665,11 @@ For background mode (e.g., the user wants to keep working): - **Observable questions are not the user's to answer.** A question whose answer a command, a render or a measurement can produce is a question for the machine: "is this faster" is two timed runs, "which layout reads better" is two - renders side by side, "does the API accept this shape" is one call. The skill - that would have asked does the observation and reports what it saw. Only a + renders side by side, "does the API accept this shape" is one call against a + local or test endpoint. The skill that would have asked does the observation + and reports what it saw, as long as the observation is local, read-only and + free; one that changes state, reaches a live or third-party system or spends + money is a one-way door and stays a question (see the rule above). Only a product or preference call reaches the user (what the product should do, what they want, what they will spend or risk), and it arrives with the observation already made. When the question log shows a prompt whose answer was observable,