mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-13 03:48:06 +01:00
Reword ETHOS sentences, scope the blinding rule, name the second candidate-visible root
Review pass 1 (frozen scope, five INFORMATIONAL findings, all in-diff, none blocking): two ETHOS.md sentences in principles 6 and 7 were too close to the credited source and are reworded; the Credits line names both pstack sources (principle-* leaves and poteto-mode); eval-plan Step 1c rule 1 is scoped to what the preregistration controls and exempts the skill text an arm loads; the observable-questions rule in plan-tune and docs/proactive.md requires the observation to be local, read-only and free and points at the one-way-door rule; docs/TESTING_INTERNALS.md names skill-e2e-benchmark-* (test/skill-e2e- deploy.test.ts:293) as a second candidate-visible root and states the audit's matching method. SKILL.md regenerated for both unpinned skills. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
92002dc228
commit
13a40c41b8
8 files changed
+52
-30
No files matched your search
@@ -1511,9 +1511,11 @@ fixtures.
|
||||
(its working directory, file names, the task prompt, commit messages, the
|
||||
environment) says `eval`, `test`, `judge`, `experiment`, `rubric`, `score`,
|
||||
`compare`, `benchmark`, `candidate` or `arena`. The project's own `test/`
|
||||
directory is exempt when the task is about those tests. The hermetic home a
|
||||
case builds (`test/helpers/hermetic-env.ts`) is the boundary: what is copied
|
||||
in is visible, what stays in the runner is not.
|
||||
directory is exempt when the task is about those tests, and so is the skill
|
||||
text an arm loads as its own instructions (a gstack SKILL.md and the ETHOS.md it
|
||||
points at use these words freely); the rule governs what the case builds. The
|
||||
hermetic home a case builds (`test/helpers/hermetic-env.ts`) is the boundary:
|
||||
what is copied in is visible, what stays in the runner is not.
|
||||
- **Fixture roots are project-shaped.** A fixture the candidate works in is
|
||||
named like the project it imitates, with the files that project would hold.
|
||||
Runner-side fixtures (captured transcripts, judge calibration sets, census
|
||||
@@ -1528,12 +1530,16 @@ word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/`
|
||||
`cso-eval/`, `eval-history/`, `judge-calibration/` and its children, two shard
|
||||
names under `paid-report-census/` and the `test/` subdirectories of
|
||||
`state-matrix-6792/`. All of these are runner-side: no candidate's working
|
||||
directory is one of them. The one candidate-visible gap is the other rule:
|
||||
`test/helpers/arm-benchmark-harness.ts` copies each fixture into
|
||||
`os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is named
|
||||
after the arm rather than after a project. The e2e hermetic roots
|
||||
(`skill-e2e-<case>-*`) carry no listed word. Renaming is a separate change with
|
||||
its own touchfile updates; the list is here so that change can be scoped.
|
||||
directory is one of them. Two candidate-visible gaps are in the temporary roots
|
||||
the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture
|
||||
into `os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is
|
||||
named after the arm rather than after a project, and `test/skill-e2e-deploy.test.ts`
|
||||
runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed
|
||||
word; the other `skill-e2e-<case>-*` roots do not. The audit matched whole
|
||||
lower-case words in path components under `test/fixtures/`; a substring match
|
||||
(`factory-scorecard/`, `ios-qa/FixtureApp/Tests/`) was not counted. Renaming is a
|
||||
separate change with its own touchfile updates; the list is here so that change
|
||||
can be scoped.
|
||||
|
||||
## Running evals as an agent: detach
|
||||
|
||||
|
||||
+7
-5
@@ -64,9 +64,11 @@ governs what a running skill may ask you unasked: a question whose answer the
|
||||
skill could observe is not yours to answer. Whether a change is faster, whether
|
||||
two renders differ, whether an endpoint accepts a payload, whether a test still
|
||||
fails: each of those is settled by running the thing (time it, render both,
|
||||
call it, run the test) and reporting the result, not by a prompt. The only
|
||||
questions that come to you are product and preference calls (what the product
|
||||
should do, what you want, what you are willing to spend or risk), and they
|
||||
arrive with the observation already in hand so the choice is between measured
|
||||
options. `/plan-tune` applies the same rule when it audits the question log: an
|
||||
call it, run the test) and reporting the result, not by a prompt. That holds
|
||||
for observations that are local, read-only and free; a call that changes state,
|
||||
reaches a live or third-party system or spends money is still yours to approve,
|
||||
as the one-way-door rule in `/plan-tune` already says. The only questions that
|
||||
come to you are product and preference calls (what the product should do, what
|
||||
you want, what you are willing to spend or risk), and they arrive with the
|
||||
observation already in hand so the choice is between measured options. `/plan-tune` applies the same rule when it audits the question log: an
|
||||
observable question is a defect in the asking skill, not a preference to tune.
|
||||
Reference in new issue
Block a user