v1.91.139.0 fix(evals): docsync observer accepts read-only git describe (#3276)

* fix(evals): docsync observer accepts read-only git describe

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.139.0 fix(evals): docsync observer accepts read-only git describe

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-10-18 20:00:10 +00:00
1 parent 67783c1885
commit 3a64ab8a1f
6 files changed
+118 -4

No files matched your search

+9
View File
@@ -1,5 +1,14 @@
# Changelog
## [1.91.139.0] - 2026-10-18
**The docs-sync fixture's declared Git interface now includes read-only `git describe`, so `ship-docsync-store` no longer goes red when the /document-release child checks for release tags.** PR #3272's gate run 39153452209 failed that case on one command, `git describe --tags --always`, which the observer did not list. A focused paid rerun of `ship-docsync-store` on this tree passed ($3.83); the child ran the same command twice and the observer accepted it.
### Changed
- **Accepted:** `git describe` with any of `--tags`, `--always`, `--abbrev=<n>` and `--exact-match`, plus at most one literal revision. The observer prose that parents pass to children lists the same forms, so the code allowlist and the prompt agree.
- **Still refused:** `--dirty` and `--broken`, which take `index.lock` and rewrite `.git/index` (seen under strace on git 2.55), plus `--contains`, `--all`, any other flag, a second revision, redirection and composition. Write-capable Git commands such as `git tag` and `git update-index` stay outside the interface.
- **Gate integrity:** this widens one observer allowlist (`test/helpers/docsync-observer.ts`) and tightens nothing. Free regressions cover all 36 Bash commands of the captured run and run each accepted form against a tagged fixture with an out-of-date file timestamp, checking that the repository and the raw index bytes don't change. Negative controls show that `git describe --dirty` really rewrites the index.
## [1.91.138.0] - 2026-10-18
**`scripts/gate-cost-replay.ts --history` replays PR gates' red runs and READY times, so the merge-train plan's two estimates are measured instead of assumed.** Over Oct 3 to 18 (505 E2E Evals runs, 92 merged PRs), the screen-provisional rule run on each failing head's diff from its tested base would have missed the PR-caused red on 2 of the 13 PRs that had one (#3036, #3121): 2 of 72 paid-relevant PRs (2.8%), against the plan's 6% estimate. 13 of 92 merged PRs had a PR-caused red, matching the plan's figure. Replaying READY arrivals through trains of 5 with a 3-hour fill limit, the realized train averages about 2.3 PRs, so trains make 22 to 29 gate attempts a week rather than the 12 to 20 the train model assumed, and READY-to-merge p90 runs 5 to 8 hours. This is lane T0 of the plan to cut the paid gate bill about 4x; nothing changes what any gate runs.
+1 -1
View File
@@ -1 +1 @@
1.91.138.0
1.91.139.0
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.91.138.0 — regenerate/re-copy after upgrading gstack
# gstack digest v1.91.139.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gstack",
"version": "1.91.138",
"version": "1.91.139",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
+100
View File
@@ -306,3 +306,103 @@ test('subcommand --help is admitted alone and runs without effect; with any othe
expect(fs.existsSync(record)).toBe(false);
expect(repoSnapshot(fixture.repo)).toEqual(before);
});
// PR gate run 39153452209 (ship-docsync-store, head a844821583d664e3dea279fff12b935865e92f52) went red only on the
// document-release child's `git describe --tags --always` (bash#24). Every Bash command of the parent and its child,
// in order; <HOME> is the fixture home.
const STORE_39153452209 = [
'git status',
'git rev-parse HEAD',
'git ls-files',
'<HOME>/.claude/skills/gstack/bin/gstack-docs-candidate --help',
'git diff --cached',
'git diff',
'git branch --show-current',
'git log --oneline',
'git check-ignore -v .qa-state',
'ls -la <HOME>',
'<HOME>/.claude/skills/gstack/bin/gstack-skill-start --help',
'git ls-files -z --cached --others --exclude-standard',
'git show HEAD:handbook/reference/commands/widget.md.tmpl',
'ls -la <HOME>/state',
'git hash-object README.md handbook/reference/commands/widget.md.tmpl options.ts app.ts personal-note.txt',
'git log --oneline --decorate --all',
'<HOME>/.claude/skills/gstack/bin/gstack-docs-candidate snapshot --out <HOME>/ship-docs-20261018-c0048c5-a1.json --audit-id ship-docs-20261018-c0048c5-a1 --mode read-only --base c0048c54c542cf0ba46b7587cf7fea1b519c75e4 --select app.ts --select README.md --select options.ts --docs handbook --docs README.md --docs AGENTS.md --docs CHANGELOG.md --docs TODOS.md',
'GSTACK_SESSION_KIND=spawned <HOME>/.claude/skills/gstack/bin/gstack-skill-start --skill document-release --model claude',
'git status',
'git log --oneline',
'git diff c0048c54c542cf0ba46b7587cf7fea1b519c75e4 HEAD --stat',
'git diff --cached',
'git diff',
'git ls-files --cached --others --exclude-standard',
'git hash-object README.md options.ts app.ts personal-note.txt',
'git describe --tags --always',
'<HOME>/.claude/skills/gstack/bin/gstack-skill-end --skill document-release --outcome success --session-id 878-1792348356-d8b277ad --tel-start 1792348356 --used-browse no',
'<HOME>/.claude/skills/gstack/bin/gstack-docs-candidate compare <HOME>/ship-docs-20261018-c0048c5-a1.json',
'git status',
'git rev-parse HEAD',
'git hash-object README.md handbook/reference/commands/widget.md.tmpl options.ts app.ts personal-note.txt AGENTS.md CHANGELOG.md TODOS.md VERSION package.json',
'ls -la <HOME>',
'ls -la <HOME>/repo',
'cat <HOME>/ship-docs-20261018-c0048c5-a1-documentation.md >> <HOME>/ship-report.md',
'git status',
'ls -la <HOME>',
];
test('every Bash command of the captured run 39153452209 ship-docsync-store is inside the interface', () => {
const commands = STORE_39153452209.map(command => command.replaceAll('<HOME>', fixture.home));
for (const file of ['ship-docs-20261018-c0048c5-a1-documentation.md', 'ship-report.md']) fs.writeFileSync(path.join(fixture.home, file), '');
expect(commands).toHaveLength(36);
expect(commands).toContain('git describe --tags --always');
expect(commands.filter(command => !docsCommandAllowed(command, fixture))).toEqual([]);
expect(docsToolFailures(nativeResult('git describe --tags --always', 'c0048c5', 'docs-dispatch'), fixture, [], true)).toEqual([]);
});
const DESCRIBE_READS = ['git describe', 'git describe --tags', 'git describe --always', 'git describe --abbrev=7',
'git describe --exact-match', 'git describe HEAD', "git describe --tags --always --abbrev=12 'HEAD^{}'",
'git describe --exact-match --tags main', 'git describe --always main', 'git describe --tags --always 2>/dev/null'];
const DESCRIBE_REFUSED = ['git describe --dirty', 'git describe --dirty=-modified', 'git describe --always --dirty',
'git describe --broken', 'git describe --broken=-broken', 'git describe --contains HEAD', 'git describe --all',
'git describe --abbrev', 'git describe --abbrev=x', 'git describe HEAD main', 'git describe -- HEAD',
'git describe --tags > tag.md', 'git describe --tags; git tag v9', 'git describe --tags && git tag v9'];
test('read-only git describe forms are declared and leave the index and repository unchanged', () => {
expect(docsNativeInterface(fixture)).toContain('describe with only --tags, --always, --abbrev=<n> or --exact-match and at most one revision');
const own = fixtureDocs('store');
try {
gitAt(own.repo, 'tag', '-a', 'v0.1.0', '-m', 'fixture release', 'HEAD');
const stale = new Date(Date.now() - 3600_000);
fs.utimesSync(path.join(own.repo, 'AGENTS.md'), stale, stale);
const index = () => fs.readFileSync(path.join(own.repo, '.git/index')).toString('base64');
const before = { repo: repoSnapshot(own.repo), index: index() };
for (const command of DESCRIBE_READS) {
expect(docsCommandAllowed(command, own), command).toBe(true);
expect(docsToolFailures(nativeResult(command), own, [], true), command).toEqual([]);
const run = spawnSync('bash', ['-c', command], { cwd: own.repo, encoding: 'utf8', timeout: 10000 });
expect(run.status, `${command}: ${run.stderr}`).toBe(0);
expect({ repo: repoSnapshot(own.repo), index: index() }, command).toEqual(before);
}
} finally {
own.clean();
}
});
test('describe forms that refresh the index or fall outside the declared set, and Git writes, stay outside the interface', () => {
for (const command of [...DESCRIBE_REFUSED, 'git tag v9 HEAD', 'git update-index --refresh', 'git notes add -m x HEAD']) {
expect(docsCommandAllowed(command, fixture), command).toBe(false);
expect(docsToolFailures(nativeResult(command, 'successful tool acknowledgment', 'docs-dispatch'), fixture, [], true), command)
.toEqual(['command outside declared docs observation interface']);
}
const own = fixtureDocs('store');
try {
const stale = new Date(Date.now() - 3600_000);
fs.utimesSync(path.join(own.repo, 'AGENTS.md'), stale, stale);
const index = () => fs.readFileSync(path.join(own.repo, '.git/index')).toString('base64');
const before = index();
const run = spawnSync('git', ['describe', '--always', '--dirty'], { cwd: own.repo, encoding: 'utf8', timeout: 10000 });
expect(run.status, run.stderr).toBe(0);
expect(index()).not.toBe(before);
} finally {
own.clean();
}
});
+6 -1
View File
@@ -197,6 +197,7 @@ Keep artifacts concise: update the invocation record in place with ids, counts,
const DOCS_READ_COMMANDS = ['pwd', 'ls', 'cat', 'sha256sum', 'stat'];
const DOCS_GIT_READS = ['status', 'diff', 'show', 'log', 'ls-files', 'ls-tree', 'check-ignore', 'rev-parse', 'merge-base', 'hash-object'];
const DOCS_DESCRIBE_FLAGS = /^(?:--tags|--always|--exact-match|--abbrev=\d+)$/;
const DOCS_SAFE_GIT_CONFIG = /^(?:core\.pager=cat|core\.quotepath=(?:true|false|on|off)|color\.[a-z.]+=(?:true|false|never|always|auto))$/i;
/**
@@ -261,6 +262,10 @@ function literalDocsCommandAllowed(text: string, fixture: ReturnType<typeof fixt
if (rest.some(arg => /^(?:--output|--ext-diff|--textconv|-w)(?:=|$)/.test(arg))) return false;
if (rest[0] === 'hash-object' && rest.some(arg => /^-[^-]*w/.test(arg))) return false;
if (rest[0] === 'branch') return rest.length === 2 && rest[1] === '--show-current';
if (rest[0] === 'describe') {
const operands = rest.slice(1).filter(arg => !DOCS_DESCRIBE_FLAGS.test(arg));
return operands.length <= 1 && operands.every(arg => !arg.startsWith('-'));
}
return DOCS_GIT_READS.includes(rest[0]);
}
if (commandName === 'bun' || commandName === process.execPath) {
@@ -288,7 +293,7 @@ function literalDocsCommandAllowed(text: string, fixture: ReturnType<typeof fixt
export function docsNativeInterface(fixture: Pick<ReturnType<typeof fixtureDocs>, 'home' | 'repo' | 'skills'>, scripts: string[] = [], transport = false, insert = false, file?: string): string {
const skills = fixture.skills.split(path.sep).join('/');
return `Fixture observation interface (applies to parent and every child; ${file ? `saved at ${file}: each child prompt tells the child to Read that file first instead of copying this text` : 'include this interface in child prompts'}): Bash may execute only separate literal pwd, ls, cat, stat, sha256sum, Git read commands (status, diff, show, log, ls-files, ls-tree, check-ignore, rev-parse, merge-base, hash-object without -w, branch --show-current), the exact generated Preamble block with its spawned prefix, literal installed gstack-skill-start/gstack-skill-end commands for document-release (start requires GSTACK_SESSION_KIND=spawned) or their own --help, or the literal installed ${skills}/bin/gstack-docs-candidate snapshot (its --out a private .json under ${fixture.home}), compare <that .json> and --help commands. No shell composition, custom interpreters, arbitrary scripts, inline eval or memory-mapped writes. The only additional scripts are ${scripts.length ? scripts.join(', ') : 'none'}.${insert ? ` To insert one saved private Markdown artifact into another (a saved section file into a report), Bash may also run the single literal command cat SOURCE.md >> TARGET.md with absolute paths under ${fixture.home}.` : ''} Read/Glob/Grep remain available; Read skill and section files with Read (offset/limit for ranges), because Bash output over 30KB becomes a preview that no permitted Bash command can page. Use Write/Edit for permitted docs and private JSON/Markdown artifacts under ${fixture.home}; do not rewrite installed skills, config, actor state or scripts. No effects outside the owned fixture. The owner preserves evidence and cleans up. Missing observer coverage blocks acceptance; the Linux kernel monitor covers syscall writes in the product tree, not hostile processes or arbitrary external destinations.
return `Fixture observation interface (applies to parent and every child; ${file ? `saved at ${file}: each child prompt tells the child to Read that file first instead of copying this text` : 'include this interface in child prompts'}): Bash may execute only separate literal pwd, ls, cat, stat, sha256sum, Git read commands (status, diff, show, log, ls-files, ls-tree, check-ignore, rev-parse, merge-base, hash-object without -w, branch --show-current, describe with only --tags, --always, --abbrev=<n> or --exact-match and at most one revision), the exact generated Preamble block with its spawned prefix, literal installed gstack-skill-start/gstack-skill-end commands for document-release (start requires GSTACK_SESSION_KIND=spawned) or their own --help, or the literal installed ${skills}/bin/gstack-docs-candidate snapshot (its --out a private .json under ${fixture.home}), compare <that .json> and --help commands. No shell composition, custom interpreters, arbitrary scripts, inline eval or memory-mapped writes. The only additional scripts are ${scripts.length ? scripts.join(', ') : 'none'}.${insert ? ` To insert one saved private Markdown artifact into another (a saved section file into a report), Bash may also run the single literal command cat SOURCE.md >> TARGET.md with absolute paths under ${fixture.home}.` : ''} Read/Glob/Grep remain available; Read skill and section files with Read (offset/limit for ranges), because Bash output over 30KB becomes a preview that no permitted Bash command can page. Use Write/Edit for permitted docs and private JSON/Markdown artifacts under ${fixture.home}; do not rewrite installed skills, config, actor state or scripts. No effects outside the owned fixture. The owner preserves evidence and cleans up. Missing observer coverage blocks acceptance; the Linux kernel monitor covers syscall writes in the product tree, not hostile processes or arbitrary external destinations.
The working directory for parent and child Bash calls is already ${fixture.repo}. Run Git reads directly, for example: git status, git diff --cached, git merge-base main HEAD, git rev-parse HEAD. Do not use Git global options such as -C, -c, --git-dir or --work-tree, and do not prepend cd or another shell wrapper. The literal git subcommand must immediately follow git; an absolute owned repository path does not make git -C an allowed command.