mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-06 00:44:07 +01:00
Reword ETHOS sentences, scope the blinding rule, name the second candidate-visible root
Review pass 1 (frozen scope, five INFORMATIONAL findings, all in-diff, none blocking): two ETHOS.md sentences in principles 6 and 7 were too close to the credited source and are reworded; the Credits line names both pstack sources (principle-* leaves and poteto-mode); eval-plan Step 1c rule 1 is scoped to what the preregistration controls and exempts the skill text an arm loads; the observable-questions rule in plan-tune and docs/proactive.md requires the observation to be local, read-only and free and points at the one-way-door rule; docs/TESTING_INTERNALS.md names skill-e2e-benchmark-* (test/skill-e2e- deploy.test.ts:293) as a second candidate-visible root and states the audit's matching method. SKILL.md regenerated for both unpinned skills. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
92002dc228
commit
13a40c41b8
8 files changed
+52
-30
No files matched your search
+1
-1
@@ -6,7 +6,7 @@
|
||||
|
||||
### Added
|
||||
- **ETHOS.md principles 5 to 8**: Attack the Premise (two fixes that share a premise and fail the same gate indict the premise; write it down and falsify it before a third fix), Separate Before Serializing Shared State (a lock is a smell to check; remove the race before refereeing it), Make Operations Idempotent (design for the run that happens twice; ask what happens if the previous run died at every point), Explain the Number (a measurement carries what was measured, the limiter it moved and the run-to-run variation; a gap inside the variation is no difference). "How They Work Together" gains one line each and a Credits section names the source.
|
||||
- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures and lists the fixture directory names on the word list (no fixture is renamed here).
|
||||
- **`/eval-plan` Step 1c, candidate-side blinding**: three preregistration rules (no eval word anywhere the candidate can read; project-shaped fixture roots; the judge sees `A`/`B` labels and grades chain-following from the files the transcript shows the candidate opened). `prereg --check` does not read fixture trees, and the step says so. `docs/TESTING_INTERNALS.md#candidate-side-blinding` states the same rules for gstack's own fixtures, lists the fixture directory names on the word list and names the two candidate-visible roots (`arm-<fixture>-<arm>-*` from `test/helpers/arm-benchmark-harness.ts`, `skill-e2e-benchmark-*` from `test/skill-e2e-deploy.test.ts`); no fixture is renamed here.
|
||||
- **Observable questions are not the user's to answer**: a rule in `/plan-tune`'s Important Rules and a section in `docs/proactive.md`. A question a command, render or measurement can settle is settled that way and reported; only product and preference calls reach the user, with the observation already made. The question-log audit treats an observable question as a defect in the asking skill rather than a preference to tune.
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -202,9 +202,9 @@ failure. Most wasted work in this craft is excellent work on a false premise.
|
||||
When two writers collide, the reflex is a lock, a queue or a retry loop. Each
|
||||
of those keeps the collision and pays for it forever: a lock serializes every
|
||||
caller to protect the rare conflict, a queue adds a component that can fill, a
|
||||
retry turns one race into many. Treat "we need a lock" as a design smell to
|
||||
check before it is the answer. Ask first whether the writers need to share the
|
||||
state at all. One file per writer with a merge at read time, a per-unit
|
||||
retry turns one race into many. So when the sentence "we need a lock" comes up,
|
||||
hear it as a prompt to look at the design, not as the fix. Ask first whether the
|
||||
writers need to share the state at all. One file per writer with a merge at read time, a per-unit
|
||||
directory, an append-only log keyed by author, a partition by owner: each of
|
||||
these removes the race instead of refereeing it, and leaves nothing to deadlock.
|
||||
|
||||
@@ -229,8 +229,9 @@ the key again. Design for that run. An idempotent operation reaches the same
|
||||
end state whether it runs once or five times, and a run that crashed halfway
|
||||
can simply be run again rather than repaired by hand.
|
||||
|
||||
The working question is: what happens if the previous run died at every
|
||||
possible point? Walk the operation line by line and answer it. A create that
|
||||
The working question: for each line of the operation, suppose the previous run
|
||||
died right after it. What does this run find, and does it still converge? Walk
|
||||
the operation line by line and answer that for every line. A create that
|
||||
checks for the existing row, a write that lands in a temp file and renames, a
|
||||
"done" marker written last, a cleanup that tolerates an already-missing file:
|
||||
each answer removes a class of manual recovery. Where true idempotence is
|
||||
@@ -300,5 +301,6 @@ every time.
|
||||
|
||||
Principles 5 through 8 (Attack the Premise, Separate Before Serializing Shared
|
||||
State, Make Operations Idempotent, Explain the Number) are gstack's statement of
|
||||
four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins,
|
||||
`pstack/skills/principle-*`, MIT). The words here are gstack's; see NOTICE.md.
|
||||
four ideas from Lauren Tan's pstack (https://github.com/cursor/plugins, its
|
||||
`pstack/skills/principle-*` leaves and its `poteto-mode` skill, MIT). The words
|
||||
here are gstack's; see NOTICE.md.
|
||||
@@ -1511,9 +1511,11 @@ fixtures.
|
||||
(its working directory, file names, the task prompt, commit messages, the
|
||||
environment) says `eval`, `test`, `judge`, `experiment`, `rubric`, `score`,
|
||||
`compare`, `benchmark`, `candidate` or `arena`. The project's own `test/`
|
||||
directory is exempt when the task is about those tests. The hermetic home a
|
||||
case builds (`test/helpers/hermetic-env.ts`) is the boundary: what is copied
|
||||
in is visible, what stays in the runner is not.
|
||||
directory is exempt when the task is about those tests, and so is the skill
|
||||
text an arm loads as its own instructions (a gstack SKILL.md and the ETHOS.md it
|
||||
points at use these words freely); the rule governs what the case builds. The
|
||||
hermetic home a case builds (`test/helpers/hermetic-env.ts`) is the boundary:
|
||||
what is copied in is visible, what stays in the runner is not.
|
||||
- **Fixture roots are project-shaped.** A fixture the candidate works in is
|
||||
named like the project it imitates, with the files that project would hold.
|
||||
Runner-side fixtures (captured transcripts, judge calibration sets, census
|
||||
@@ -1528,12 +1530,16 @@ word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/`
|
||||
`cso-eval/`, `eval-history/`, `judge-calibration/` and its children, two shard
|
||||
names under `paid-report-census/` and the `test/` subdirectories of
|
||||
`state-matrix-6792/`. All of these are runner-side: no candidate's working
|
||||
directory is one of them. The one candidate-visible gap is the other rule:
|
||||
`test/helpers/arm-benchmark-harness.ts` copies each fixture into
|
||||
`os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is named
|
||||
after the arm rather than after a project. The e2e hermetic roots
|
||||
(`skill-e2e-<case>-*`) carry no listed word. Renaming is a separate change with
|
||||
its own touchfile updates; the list is here so that change can be scoped.
|
||||
directory is one of them. Two candidate-visible gaps are in the temporary roots
|
||||
the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture
|
||||
into `os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is
|
||||
named after the arm rather than after a project, and `test/skill-e2e-deploy.test.ts`
|
||||
runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed
|
||||
word; the other `skill-e2e-<case>-*` roots do not. The audit matched whole
|
||||
lower-case words in path components under `test/fixtures/`; a substring match
|
||||
(`factory-scorecard/`, `ios-qa/FixtureApp/Tests/`) was not counted. Renaming is a
|
||||
separate change with its own touchfile updates; the list is here so that change
|
||||
can be scoped.
|
||||
|
||||
## Running evals as an agent: detach
|
||||
|
||||
|
||||
+7
-5
@@ -64,9 +64,11 @@ governs what a running skill may ask you unasked: a question whose answer the
|
||||
skill could observe is not yours to answer. Whether a change is faster, whether
|
||||
two renders differ, whether an endpoint accepts a payload, whether a test still
|
||||
fails: each of those is settled by running the thing (time it, render both,
|
||||
call it, run the test) and reporting the result, not by a prompt. The only
|
||||
questions that come to you are product and preference calls (what the product
|
||||
should do, what you want, what you are willing to spend or risk), and they
|
||||
arrive with the observation already in hand so the choice is between measured
|
||||
options. `/plan-tune` applies the same rule when it audits the question log: an
|
||||
call it, run the test) and reporting the result, not by a prompt. That holds
|
||||
for observations that are local, read-only and free; a call that changes state,
|
||||
reaches a live or third-party system or spends money is still yours to approve,
|
||||
as the one-way-door rule in `/plan-tune` already says. The only questions that
|
||||
come to you are product and preference calls (what the product should do, what
|
||||
you want, what you are willing to spend or risk), and they arrive with the
|
||||
observation already in hand so the choice is between measured options. `/plan-tune` applies the same rule when it audits the question log: an
|
||||
observable question is a defect in the asking skill, not a preference to tune.
|
||||
+5
-2
@@ -439,8 +439,11 @@ read fixture trees, so this check is yours:
|
||||
1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`,
|
||||
`judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`,
|
||||
`candidate` or `arena` in any directory name, file name, prompt, commit
|
||||
message or environment variable the candidate can read. A project's own test
|
||||
directory is the one exception, and only when the task is about those tests.
|
||||
message or environment variable the preregistration controls and the candidate
|
||||
can read. Two things are outside the rule: a project's own test directory, when
|
||||
the task is about those tests, and the skill text an arm loads as its own
|
||||
instructions (a gstack SKILL.md uses these words freely); the rule is about the
|
||||
fixture and the task, which the eval builds.
|
||||
2. **The directory looks like a project.** Name the fixture root the way the
|
||||
project it imitates would (`billing-service/`, `docs-site/`), with the files
|
||||
that project would plausibly hold, never `fixture-a/`, `case-7/` or the
|
||||
|
||||
@@ -111,8 +111,11 @@ read fixture trees, so this check is yours:
|
||||
1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`,
|
||||
`judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`,
|
||||
`candidate` or `arena` in any directory name, file name, prompt, commit
|
||||
message or environment variable the candidate can read. A project's own test
|
||||
directory is the one exception, and only when the task is about those tests.
|
||||
message or environment variable the preregistration controls and the candidate
|
||||
can read. Two things are outside the rule: a project's own test directory, when
|
||||
the task is about those tests, and the skill text an arm loads as its own
|
||||
instructions (a gstack SKILL.md uses these words freely); the rule is about the
|
||||
fixture and the task, which the eval builds.
|
||||
2. **The directory looks like a project.** Name the fixture root the way the
|
||||
project it imitates would (`billing-service/`, `docs-site/`), with the files
|
||||
that project would plausibly hold, never `fixture-a/`, `case-7/` or the
|
||||
|
||||
+5
-2
@@ -981,8 +981,11 @@ For background mode (e.g., the user wants to keep working):
|
||||
- **Observable questions are not the user's to answer.** A question whose
|
||||
answer a command, a render or a measurement can produce is a question for the
|
||||
machine: "is this faster" is two timed runs, "which layout reads better" is two
|
||||
renders side by side, "does the API accept this shape" is one call. The skill
|
||||
that would have asked does the observation and reports what it saw. Only a
|
||||
renders side by side, "does the API accept this shape" is one call against a
|
||||
local or test endpoint. The skill that would have asked does the observation
|
||||
and reports what it saw, as long as the observation is local, read-only and
|
||||
free; one that changes state, reaches a live or third-party system or spends
|
||||
money is a one-way door and stays a question (see the rule above). Only a
|
||||
product or preference call reaches the user (what the product should do, what
|
||||
they want, what they will spend or risk), and it arrives with the observation
|
||||
already made. When the question log shows a prompt whose answer was observable,
|
||||
|
||||
@@ -665,8 +665,11 @@ For background mode (e.g., the user wants to keep working):
|
||||
- **Observable questions are not the user's to answer.** A question whose
|
||||
answer a command, a render or a measurement can produce is a question for the
|
||||
machine: "is this faster" is two timed runs, "which layout reads better" is two
|
||||
renders side by side, "does the API accept this shape" is one call. The skill
|
||||
that would have asked does the observation and reports what it saw. Only a
|
||||
renders side by side, "does the API accept this shape" is one call against a
|
||||
local or test endpoint. The skill that would have asked does the observation
|
||||
and reports what it saw, as long as the observation is local, read-only and
|
||||
free; one that changes state, reaches a live or third-party system or spends
|
||||
money is a one-way door and stays a question (see the rule above). Only a
|
||||
product or preference call reaches the user (what the product should do, what
|
||||
they want, what they will spend or risk), and it arrives with the observation
|
||||
already made. When the question log shows a prompt whose answer was observable,
|
||||
|
||||
Reference in new issue
Block a user