From a87d92c5e20cc09c274f18c0d723de8de81603ce Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sun, 8 Nov 2026 11:16:41 -0800 Subject: [PATCH] Open findings and results write-ups with two or three plain takeaways (#4173) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- ETHOS.md | 17 +++++++++++++++++ changelog.d/findings-open-with-takeaways.md | 1 + docs/TESTING_INTERNALS.md | 8 ++++++++ eval-plan/SKILL.md | 6 +++++- eval-plan/SKILL.md.tmpl | 6 +++++- lib/eval-plan.ts | 2 +- test/fixtures/context-budget.json | 2 +- 7 files changed, 38 insertions(+), 4 deletions(-) create mode 100644 changelog.d/findings-open-with-takeaways.md diff --git a/ETHOS.md b/ETHOS.md index aaac79291..5ef7a9aaa 100644 --- a/ETHOS.md +++ b/ETHOS.md @@ -266,10 +266,27 @@ smaller than the variation is not a difference; say "no measurable change" and mean it. A surprising number is a lead to investigate, not a headline to post: the eval that jumps twenty points usually found a shortcut, not a capability. +A write-up of what we learned is a gift to everyone building agents (Garry, +2026-11-08: "this is a gift"), so it leads with what they can use. Every +findings file, investigation write-up and results README opens with two or +three plain bullets: the big takeaways, phrased as "We learned X. You should do +X." What works comes first, with why it matters to a builder. What didn't work +comes right after, stated plainly with its number, and a null becomes a useful +lesson where the data supports one ("you don't need X; the plain model already +does Y"). Our own mechanics (tracker codes, ledgers, receipts, gate details) go +last. Every experiment names the public lesson it is after before it runs. +Spend stays in the ledger and internal reports; a findings file shows cost +only when it is part of the lesson, such as cost per correct answer between two +setups. Private is the default: open a repo, dataset or tool only with a stated +reason, helping builders or letting others reproduce a published finding. None +of this bends the rules above: no number is rounded in our favour and no caveat +is dropped. + **Anti-patterns:** - A speedup from one run on each side. - A pass rate without the trial count. - "Faster" with no theory of what changed. +- A results file whose lesson, or whose null, sits under a page of run ids. --- diff --git a/changelog.d/findings-open-with-takeaways.md b/changelog.d/findings-open-with-takeaways.md new file mode 100644 index 000000000..a7beff3db --- /dev/null +++ b/changelog.d/findings-open-with-takeaways.md @@ -0,0 +1 @@ +- **Findings and results write-ups now open with two or three plain takeaways, what works first.** ETHOS.md's Explain the Number says every findings file, investigation write-up and results README opens with "We learned X. You should do X." bullets: what works first, then what didn't, stated plainly with its number, and a null turned into the lesson its data supports. Internal mechanics go last, every experiment names the public lesson it is after, cost appears only when it is part of the lesson, and a repo, dataset or tool is opened only with a stated reason. No number is rounded in our favour and no caveat is dropped. /eval-plan's preregistration asks for a `lesson:` line under Bars, its results step points to the rule, and docs/TESTING_INTERNALS.md says the same beside its results guidance. diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 44cc24a7b..388b39ef1 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1973,6 +1973,14 @@ drops the row has not reported its result. `gstack-eval-plan results --check filled row for every Arms identity (`EVAL_REPORT_NO_GSTACK_ROW` otherwise), and no number is published before it passes. +**Writing it up.** A results file, findings file or investigation write-up +opens with two or three plain takeaways ("We learned X. You should do X."), +what works first and what didn't right after, with its number; a null becomes +the lesson its data supports. Run ids, ledgers and gate details go last. Cost +appears only when it is part of the lesson, and datasets or tools are opened +only with a stated reason. The rules and their honesty bars are in +[ETHOS.md, Explain the Number](../ETHOS.md#8-explain-the-number). + **Safety-rule and router evals: a descriptive third arm.** The W1 safety-rule evals compare `GSTACK_SAFETY_ARM=rule` (the shipped rule) with `removed` (the rule paragraph deleted), and the router failure-routing evals compare diff --git a/eval-plan/SKILL.md b/eval-plan/SKILL.md index 7dc0e5be6..adca9139c 100644 --- a/eval-plan/SKILL.md +++ b/eval-plan/SKILL.md @@ -134,7 +134,7 @@ Write (or open) the preregistration file. Each section is required and must be f | Section | What it holds | |---|---| -| Bars | one line per metric: name, pass bar, direction, what a miss means for the decision | +| Bars | `lesson:` the public lesson the run is after, then one line per metric: name, pass bar, direction, what a miss means for the decision | | Arms | one line per arm: the arm identity first (`` or `#`), settings, per-item cost from Step 1b | | Comparison | `stack: loaded` when any arm loads gstack or another skill stack; otherwise `stack: none` and a `why:` line | | Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves | @@ -335,6 +335,10 @@ leaves the run (a report, a READY block, a PR body), check it: whose filled row is missing from the table that holds the others; add it and rerun. A preregistration with `stack: none` needs no row. +The results file opens with two or three plain takeaways ("We learned X. You should do +X."): what works first, then what didn't, with its number. Cost appears only as part of a +lesson, and nothing is opened without a stated reason (ETHOS.md, Explain the Number). + ## Tier 3, after P7 (not in this skill yet) Each of these waits for the adoption number in the multi-agent wave plan (P7); one line each so the shape is known: diff --git a/eval-plan/SKILL.md.tmpl b/eval-plan/SKILL.md.tmpl index 47b62613d..4472766b0 100644 --- a/eval-plan/SKILL.md.tmpl +++ b/eval-plan/SKILL.md.tmpl @@ -56,7 +56,7 @@ Write (or open) the preregistration file. Each section is required and must be f | Section | What it holds | |---|---| -| Bars | one line per metric: name, pass bar, direction, what a miss means for the decision | +| Bars | `lesson:` the public lesson the run is after, then one line per metric: name, pass bar, direction, what a miss means for the decision | | Arms | one line per arm: the arm identity first (`` or `#`), settings, per-item cost from Step 1b | | Comparison | `stack: loaded` when any arm loads gstack or another skill stack; otherwise `stack: none` and a `why:` line | | Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves | @@ -250,6 +250,10 @@ leaves the run (a report, a READY block, a PR body), check it: whose filled row is missing from the table that holds the others; add it and rerun. A preregistration with `stack: none` needs no row. +The results file opens with two or three plain takeaways ("We learned X. You should do +X."): what works first, then what didn't, with its number. Cost appears only as part of a +lesson, and nothing is opened without a stated reason (ETHOS.md, Explain the Number). + ## Tier 3, after P7 (not in this skill yet) Each of these waits for the adoption number in the multi-agent wave plan (P7); one line each so the shape is known: diff --git a/lib/eval-plan.ts b/lib/eval-plan.ts index 49b2dc9f0..258ee975a 100644 --- a/lib/eval-plan.ts +++ b/lib/eval-plan.ts @@ -37,7 +37,7 @@ export function preregTemplate(title = 'Eval preregistration'): string { return `# ${title} ## Bars - + ## Arms diff --git a/test/fixtures/context-budget.json b/test/fixtures/context-budget.json index ef668c2fe..5b2a29593 100644 --- a/test/fixtures/context-budget.json +++ b/test/fixtures/context-budget.json @@ -25,7 +25,7 @@ "diagram": 4286, "document-generate": 6013, "document-release": 4454, - "eval-plan": 5107, + "eval-plan": 5700, "freeze": 1115, "gstack": 5350, "gstack-upgrade": 6297,