From 27305c29f80a41c74ddbcd81f9fefe0f5aab7c67 Mon Sep 17 00:00:00 2001 From: garrytan Date: Mon, 26 Oct 2026 21:49:52 +0000 Subject: [PATCH] docs: correct the candidate-visible root list and the one-way-door attribution The blinding audit paragraph in docs/TESTING_INTERNALS.md and the CHANGELOG entry claimed two candidate-visible temporary roots and that the other skill-e2e--* roots carry no listed word; test/skill-e2e-test-value.test.ts:131 (skill-e2e-- with ids carrying 'test') and test/skill-e2e-health.test.ts:23 (gstack-health-eval-) show otherwise, so both now list the roots the mkdtemp scan found and say the list is not a proof of completeness. docs/proactive.md said the one-way-door rule already covers spend and third-party reach; it now says the new sentence extends that rule. The CHANGELOG bullet for the observable-questions rule names its local, read-only and free qualifier. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 4 ++-- docs/TESTING_INTERNALS.md | 11 ++++++++--- docs/proactive.md | 12 +++++++----- 3 files changed, 17 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bb442604c..8d7f7ddce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,8 +6,8 @@ ### Added - **ETHOS.md principles 5 to 8**: Attack the Premise (two fixes that share a premise and fail the same gate indict the premise; write it down and falsify it before a third fix), Separate Before Serializing Shared State (a lock is a smell to check; remove the race before refereeing it), Make Operations Idempotent (design for the run that happens twice; ask what happens if the previous run died at every point), Explain the Number (a measurement carries what was measured, the limiter it moved and the run-to-run variation; a gap inside the variation is no difference). "How They Work Together" gains one line each and a Credits section names the source. -- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures, lists the fixture directory names on the word list and names the two candidate-visible roots (`arm---*` from `test/helpers/arm-benchmark-harness.ts`, `skill-e2e-benchmark-*` from `test/skill-e2e-deploy.test.ts`); no fixture is renamed here. -- **Observable questions are not the user's to answer**: a rule in `/plan-tune`'s Important Rules and a section in `docs/proactive.md`. A question a command, render or measurement can settle is settled that way and reported; only product and preference calls reach the user, with the observation already made. The question-log audit treats an observable question as a defect in the asking skill rather than a preference to tune. +- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures, lists the fixture directory names on the word list and the candidate-visible temporary roots it found (`arm---*` from `test/helpers/arm-benchmark-harness.ts`, `skill-e2e-benchmark-*` from `test/skill-e2e-deploy.test.ts`, the `skill-e2e--*` roots of `test/skill-e2e-test-value.test.ts` whose ids carry `test`, `gstack-health-eval-*` from `test/skill-e2e-health.test.ts`; the list is a scan of `mkdtemp` prefixes in e2e tests, not a proof of completeness); no fixture is renamed here. +- **Observable questions are not the user's to answer**: a rule in `/plan-tune`'s Important Rules and a section in `docs/proactive.md`. A question a local, read-only, free command, render or measurement can settle is settled that way and reported (a call that changes state, reaches a live or third-party system or spends money stays a question); only product and preference calls reach the user, with the observation already made. The question-log audit treats an observable question as a defect in the asking skill rather than a preference to tune. ### Changed - **AGENTS.md validation discipline step 4**: one sentence on measured deltas: name the limiter that moved and the run-to-run variation; a gap smaller than that variation is no difference. `agents-digest/gstack-AGENTS.md` is generated from `scripts/gen-agents-digest.ts` and `VERSION`, not from AGENTS.md, so the regenerated digest is unchanged. diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 749f1bc2b..dce47c686 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1530,12 +1530,17 @@ word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/` `cso-eval/`, `eval-history/`, `judge-calibration/` and its children, two shard names under `paid-report-census/` and the `test/` subdirectories of `state-matrix-6792/`. All of these are runner-side: no candidate's working -directory is one of them. Two candidate-visible gaps are in the temporary roots +directory is one of them. Candidate-visible gaps are in the temporary roots the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture into `os.tmpdir()/arm---*`, so the candidate's working directory is -named after the arm rather than after a project, and `test/skill-e2e-deploy.test.ts` +named after the arm rather than after a project; `test/skill-e2e-deploy.test.ts` runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed -word; the other `skill-e2e--*` roots do not. The audit matched whole +word; `test/skill-e2e-test-value.test.ts` creates `skill-e2e--*` roots as the +candidate's working directory, and two of its ids (`review-test-value`, +`test-audit-report-only`) carry the word `test`; and `test/skill-e2e-health.test.ts` +creates `gstack-health-eval-*`. That list comes from a scan of the `mkdtemp` +prefixes in the e2e tests, not from a proof that no other root carries a listed +word. The audit matched whole lower-case words in path components under `test/fixtures/`; a substring match (`factory-scorecard/`, `ios-qa/FixtureApp/Tests/`) was not counted. Renaming is a separate change with its own touchfile updates; the list is here so that change diff --git a/docs/proactive.md b/docs/proactive.md index d99d96f55..c2e7dba96 100644 --- a/docs/proactive.md +++ b/docs/proactive.md @@ -66,9 +66,11 @@ two renders differ, whether an endpoint accepts a payload, whether a test still fails: each of those is settled by running the thing (time it, render both, call it, run the test) and reporting the result, not by a prompt. That holds for observations that are local, read-only and free; a call that changes state, -reaches a live or third-party system or spends money is still yours to approve, -as the one-way-door rule in `/plan-tune` already says. The only questions that -come to you are product and preference calls (what the product should do, what -you want, what you are willing to spend or risk), and they arrive with the -observation already in hand so the choice is between measured options. `/plan-tune` applies the same rule when it audits the question log: an +reaches a live or third-party system or spends money is still yours to approve. +That extends the one-way-door rule in `/plan-tune`, which covers destructive, +architectural and security questions, to spend and third-party reach. The only +questions that come to you are product and preference calls (what the product +should do, what you want, what you are willing to spend or risk), and they +arrive with the observation already in hand so the choice is between measured +options. `/plan-tune` applies the same rule when it audits the question log: an observable question is a defect in the asking skill, not a preference to tune.