Reword ETHOS sentences, scope the blinding rule, name the second candidate-visible root

Review pass 1 (frozen scope, five INFORMATIONAL findings, all in-diff, none
blocking): two ETHOS.md sentences in principles 6 and 7 were too close to the
credited source and are reworded; the Credits line names both pstack sources
(principle-* leaves and poteto-mode); eval-plan Step 1c rule 1 is scoped to
what the preregistration controls and exempts the skill text an arm loads;
the observable-questions rule in plan-tune and docs/proactive.md requires the
observation to be local, read-only and free and points at the one-way-door
rule; docs/TESTING_INTERNALS.md names skill-e2e-benchmark-* (test/skill-e2e-
deploy.test.ts:293) as a second candidate-visible root and states the audit's
matching method. SKILL.md regenerated for both unpinned skills.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-26 21:34:46 +00:00
1 parent 92002dc228
commit 13a40c41b8
8 files changed
+52 -30

No files matched your search

+1 -1
View File
@@ -6,7 +6,7 @@
### Added
- **ETHOS.md principles 5 to 8**: Attack the Premise (two fixes that share a premise and fail the same gate indict the premise; write it down and falsify it before a third fix), Separate Before Serializing Shared State (a lock is a smell to check; remove the race before refereeing it), Make Operations Idempotent (design for the run that happens twice; ask what happens if the previous run died at every point), Explain the Number (a measurement carries what was measured, the limiter it moved and the run-to-run variation; a gap inside the variation is no difference). "How They Work Together" gains one line each and a Credits section names the source.
- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures and lists the fixture directory names on the word list (no fixture is renamed here).
- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures, lists the fixture directory names on the word list and names the two candidate-visible roots (`arm-<fixture>-<arm>-*` from `test/helpers/arm-benchmark-harness.ts`, `skill-e2e-benchmark-*` from `test/skill-e2e-deploy.test.ts`); no fixture is renamed here.
- **Observable questions are not the user's to answer**: a rule in `/plan-tune`'s Important Rules and a section in `docs/proactive.md`. A question a command, render or measurement can settle is settled that way and reported; only product and preference calls reach the user, with the observation already made. The question-log audit treats an observable question as a defect in the asking skill rather than a preference to tune.
### Changed
+9 -7
View File
@@ -202,9 +202,9 @@ failure. Most wasted work in this craft is excellent work on a false premise.
When two writers collide, the reflex is a lock, a queue or a retry loop. Each
of those keeps the collision and pays for it forever: a lock serializes every
caller to protect the rare conflict, a queue adds a component that can fill, a
retry turns one race into many. Treat "we need a lock" as a design smell to
check before it is the answer. Ask first whether the writers need to share the
state at all. One file per writer with a merge at read time, a per-unit
retry turns one race into many. So when the sentence "we need a lock" comes up,
hear it as a prompt to look at the design, not as the fix. Ask first whether the
writers need to share the state at all. One file per writer with a merge at read time, a per-unit
directory, an append-only log keyed by author, a partition by owner: each of
these removes the race instead of refereeing it, and leaves nothing to deadlock.
@@ -229,8 +229,9 @@ the key again. Design for that run. An idempotent operation reaches the same
end state whether it runs once or five times, and a run that crashed halfway
can simply be run again rather than repaired by hand.
The working question is: what happens if the previous run died at every
possible point? Walk the operation line by line and answer it. A create that
The working question: for each line of the operation, suppose the previous run
died right after it. What does this run find, and does it still converge? Walk
the operation line by line and answer that for every line. A create that
checks for the existing row, a write that lands in a temp file and renames, a
"done" marker written last, a cleanup that tolerates an already-missing file:
each answer removes a class of manual recovery. Where true idempotence is
@@ -300,5 +301,6 @@ every time.
Principles 5 through 8 (Attack the Premise, Separate Before Serializing Shared
State, Make Operations Idempotent, Explain the Number) are gstack's statement of
four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins,
`pstack/skills/principle-*`, MIT). The words here are gstack's; see NOTICE.md.
four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins, its
`pstack/skills/principle-*` leaves and its `poteto-mode` skill, MIT). The words
here are gstack's; see NOTICE.md.
+15 -9
View File
@@ -1511,9 +1511,11 @@ fixtures.
(its working directory, file names, the task prompt, commit messages, the
environment) says `eval`, `test`, `judge`, `experiment`, `rubric`, `score`,
`compare`, `benchmark`, `candidate` or `arena`. The project's own `test/`
directory is exempt when the task is about those tests. The hermetic home a
case builds (`test/helpers/hermetic-env.ts`) is the boundary: what is copied
in is visible, what stays in the runner is not.
directory is exempt when the task is about those tests, and so is the skill
text an arm loads as its own instructions (a gstack SKILL.md and the ETHOS.md it
points at use these words freely); the rule governs what the case builds. The
hermetic home a case builds (`test/helpers/hermetic-env.ts`) is the boundary:
what is copied in is visible, what stays in the runner is not.
- **Fixture roots are project-shaped.** A fixture the candidate works in is
named like the project it imitates, with the files that project would hold.
Runner-side fixtures (captured transcripts, judge calibration sets, census
@@ -1528,12 +1530,16 @@ word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/`
`cso-eval/`, `eval-history/`, `judge-calibration/` and its children, two shard
names under `paid-report-census/` and the `test/` subdirectories of
`state-matrix-6792/`. All of these are runner-side: no candidate's working
directory is one of them. The one candidate-visible gap is the other rule:
`test/helpers/arm-benchmark-harness.ts` copies each fixture into
`os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is named
after the arm rather than after a project. The e2e hermetic roots
(`skill-e2e-<case>-*`) carry no listed word. Renaming is a separate change with
its own touchfile updates; the list is here so that change can be scoped.
directory is one of them. Two candidate-visible gaps are in the temporary roots
the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture
into `os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is
named after the arm rather than after a project, and `test/skill-e2e-deploy.test.ts`
runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed
word; the other `skill-e2e-<case>-*` roots do not. The audit matched whole
lower-case words in path components under `test/fixtures/`; a substring match
(`factory-scorecard/`, `ios-qa/FixtureApp/Tests/`) was not counted. Renaming is a
separate change with its own touchfile updates; the list is here so that change
can be scoped.
## Running evals as an agent: detach
+7 -5
View File
@@ -64,9 +64,11 @@ governs what a running skill may ask you unasked: a question whose answer the
skill could observe is not yours to answer. Whether a change is faster, whether
two renders differ, whether an endpoint accepts a payload, whether a test still
fails: each of those is settled by running the thing (time it, render both,
call it, run the test) and reporting the result, not by a prompt. The only
questions that come to you are product and preference calls (what the product
should do, what you want, what you are willing to spend or risk), and they
arrive with the observation already in hand so the choice is between measured
options. `/plan-tune` applies the same rule when it audits the question log: an
call it, run the test) and reporting the result, not by a prompt. That holds
for observations that are local, read-only and free; a call that changes state,
reaches a live or third-party system or spends money is still yours to approve,
as the one-way-door rule in `/plan-tune` already says. The only questions that
come to you are product and preference calls (what the product should do, what
you want, what you are willing to spend or risk), and they arrive with the
observation already in hand so the choice is between measured options. `/plan-tune` applies the same rule when it audits the question log: an
observable question is a defect in the asking skill, not a preference to tune.
+5 -2
View File
@@ -439,8 +439,11 @@ read fixture trees, so this check is yours:
1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`,
`judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`,
`candidate` or `arena` in any directory name, file name, prompt, commit
message or environment variable the candidate can read. A project's own test
directory is the one exception, and only when the task is about those tests.
message or environment variable the preregistration controls and the candidate
can read. Two things are outside the rule: a project's own test directory, when
the task is about those tests, and the skill text an arm loads as its own
instructions (a gstack SKILL.md uses these words freely); the rule is about the
fixture and the task, which the eval builds.
2. **The directory looks like a project.** Name the fixture root the way the
project it imitates would (`billing-service/`, `docs-site/`), with the files
that project would plausibly hold, never `fixture-a/`, `case-7/` or the
+5 -2
View File
@@ -111,8 +111,11 @@ read fixture trees, so this check is yours:
1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`,
`judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`,
`candidate` or `arena` in any directory name, file name, prompt, commit
message or environment variable the candidate can read. A project's own test
directory is the one exception, and only when the task is about those tests.
message or environment variable the preregistration controls and the candidate
can read. Two things are outside the rule: a project's own test directory, when
the task is about those tests, and the skill text an arm loads as its own
instructions (a gstack SKILL.md uses these words freely); the rule is about the
fixture and the task, which the eval builds.
2. **The directory looks like a project.** Name the fixture root the way the
project it imitates would (`billing-service/`, `docs-site/`), with the files
that project would plausibly hold, never `fixture-a/`, `case-7/` or the
+5 -2
View File
@@ -981,8 +981,11 @@ For background mode (e.g., the user wants to keep working):
- **Observable questions are not the user's to answer.** A question whose
answer a command, a render or a measurement can produce is a question for the
machine: "is this faster" is two timed runs, "which layout reads better" is two
renders side by side, "does the API accept this shape" is one call. The skill
that would have asked does the observation and reports what it saw. Only a
renders side by side, "does the API accept this shape" is one call against a
local or test endpoint. The skill that would have asked does the observation
and reports what it saw, as long as the observation is local, read-only and
free; one that changes state, reaches a live or third-party system or spends
money is a one-way door and stays a question (see the rule above). Only a
product or preference call reaches the user (what the product should do, what
they want, what they will spend or risk), and it arrives with the observation
already made. When the question log shows a prompt whose answer was observable,
+5 -2
View File
@@ -665,8 +665,11 @@ For background mode (e.g., the user wants to keep working):
- **Observable questions are not the user's to answer.** A question whose
answer a command, a render or a measurement can produce is a question for the
machine: "is this faster" is two timed runs, "which layout reads better" is two
renders side by side, "does the API accept this shape" is one call. The skill
that would have asked does the observation and reports what it saw. Only a
renders side by side, "does the API accept this shape" is one call against a
local or test endpoint. The skill that would have asked does the observation
and reports what it saw, as long as the observation is local, read-only and
free; one that changes state, reaches a live or third-party system or spends
money is a one-way door and stays a question (see the rule above). Only a
product or preference call reaches the user (what the product should do, what
they want, what they will spend or risk), and it arrives with the observation
already made. When the question log shows a prompt whose answer was observable,