mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 14:34:42 +01:00
v1.91.139.0 fix(evals): docsync observer accepts read-only git describe (#3276)
* fix(evals): docsync observer accepts read-only git describe Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.139.0 fix(evals): docsync observer accepts read-only git describe Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
67783c1885
commit
3a64ab8a1f
6 files changed
+118
-4
No files matched your search
@@ -1,5 +1,14 @@
|
||||
# Changelog
|
||||
|
||||
## [1.91.139.0] - 2026-10-18
|
||||
|
||||
**The docs-sync fixture's declared Git interface now includes read-only `git describe`, so `ship-docsync-store` no longer goes red when the /document-release child checks for release tags.** PR #3272's gate run 39153452209 failed that case on one command, `git describe --tags --always`, which the observer did not list. A focused paid rerun of `ship-docsync-store` on this tree passed ($3.83); the child ran the same command twice and the observer accepted it.
|
||||
|
||||
### Changed
|
||||
- **Accepted:** `git describe` with any of `--tags`, `--always`, `--abbrev=<n>` and `--exact-match`, plus at most one literal revision. The observer prose that parents pass to children lists the same forms, so the code allowlist and the prompt agree.
|
||||
- **Still refused:** `--dirty` and `--broken`, which take `index.lock` and rewrite `.git/index` (seen under strace on git 2.55), plus `--contains`, `--all`, any other flag, a second revision, redirection and composition. Write-capable Git commands such as `git tag` and `git update-index` stay outside the interface.
|
||||
- **Gate integrity:** this widens one observer allowlist (`test/helpers/docsync-observer.ts`) and tightens nothing. Free regressions cover all 36 Bash commands of the captured run and run each accepted form against a tagged fixture with an out-of-date file timestamp, checking that the repository and the raw index bytes don't change. Negative controls show that `git describe --dirty` really rewrites the index.
|
||||
|
||||
## [1.91.138.0] - 2026-10-18
|
||||
|
||||
**`scripts/gate-cost-replay.ts --history` replays PR gates' red runs and READY times, so the merge-train plan's two estimates are measured instead of assumed.** Over Oct 3 to 18 (505 E2E Evals runs, 92 merged PRs), the screen-provisional rule run on each failing head's diff from its tested base would have missed the PR-caused red on 2 of the 13 PRs that had one (#3036, #3121): 2 of 72 paid-relevant PRs (2.8%), against the plan's 6% estimate. 13 of 92 merged PRs had a PR-caused red, matching the plan's figure. Replaying READY arrivals through trains of 5 with a 3-hour fill limit, the realized train averages about 2.3 PRs, so trains make 22 to 29 gate attempts a week rather than the 12 to 20 the train model assumed, and READY-to-merge p90 runs 5 to 8 hours. This is lane T0 of the plan to cut the paid gate bill about 4x; nothing changes what any gate runs.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.91.138.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.91.139.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.91.138",
|
||||
"version": "1.91.139",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -306,3 +306,103 @@ test('subcommand --help is admitted alone and runs without effect; with any othe
|
||||
expect(fs.existsSync(record)).toBe(false);
|
||||
expect(repoSnapshot(fixture.repo)).toEqual(before);
|
||||
});
|
||||
|
||||
// PR gate run 39153452209 (ship-docsync-store, head a844821583d664e3dea279fff12b935865e92f52) went red only on the
|
||||
// document-release child's `git describe --tags --always` (bash#24). Every Bash command of the parent and its child,
|
||||
// in order; <HOME> is the fixture home.
|
||||
const STORE_39153452209 = [
|
||||
'git status',
|
||||
'git rev-parse HEAD',
|
||||
'git ls-files',
|
||||
'<HOME>/.claude/skills/gstack/bin/gstack-docs-candidate --help',
|
||||
'git diff --cached',
|
||||
'git diff',
|
||||
'git branch --show-current',
|
||||
'git log --oneline',
|
||||
'git check-ignore -v .qa-state',
|
||||
'ls -la <HOME>',
|
||||
'<HOME>/.claude/skills/gstack/bin/gstack-skill-start --help',
|
||||
'git ls-files -z --cached --others --exclude-standard',
|
||||
'git show HEAD:handbook/reference/commands/widget.md.tmpl',
|
||||
'ls -la <HOME>/state',
|
||||
'git hash-object README.md handbook/reference/commands/widget.md.tmpl options.ts app.ts personal-note.txt',
|
||||
'git log --oneline --decorate --all',
|
||||
'<HOME>/.claude/skills/gstack/bin/gstack-docs-candidate snapshot --out <HOME>/ship-docs-20261018-c0048c5-a1.json --audit-id ship-docs-20261018-c0048c5-a1 --mode read-only --base c0048c54c542cf0ba46b7587cf7fea1b519c75e4 --select app.ts --select README.md --select options.ts --docs handbook --docs README.md --docs AGENTS.md --docs CHANGELOG.md --docs TODOS.md',
|
||||
'GSTACK_SESSION_KIND=spawned <HOME>/.claude/skills/gstack/bin/gstack-skill-start --skill document-release --model claude',
|
||||
'git status',
|
||||
'git log --oneline',
|
||||
'git diff c0048c54c542cf0ba46b7587cf7fea1b519c75e4 HEAD --stat',
|
||||
'git diff --cached',
|
||||
'git diff',
|
||||
'git ls-files --cached --others --exclude-standard',
|
||||
'git hash-object README.md options.ts app.ts personal-note.txt',
|
||||
'git describe --tags --always',
|
||||
'<HOME>/.claude/skills/gstack/bin/gstack-skill-end --skill document-release --outcome success --session-id 878-1792348356-d8b277ad --tel-start 1792348356 --used-browse no',
|
||||
'<HOME>/.claude/skills/gstack/bin/gstack-docs-candidate compare <HOME>/ship-docs-20261018-c0048c5-a1.json',
|
||||
'git status',
|
||||
'git rev-parse HEAD',
|
||||
'git hash-object README.md handbook/reference/commands/widget.md.tmpl options.ts app.ts personal-note.txt AGENTS.md CHANGELOG.md TODOS.md VERSION package.json',
|
||||
'ls -la <HOME>',
|
||||
'ls -la <HOME>/repo',
|
||||
'cat <HOME>/ship-docs-20261018-c0048c5-a1-documentation.md >> <HOME>/ship-report.md',
|
||||
'git status',
|
||||
'ls -la <HOME>',
|
||||
];
|
||||
|
||||
test('every Bash command of the captured run 39153452209 ship-docsync-store is inside the interface', () => {
|
||||
const commands = STORE_39153452209.map(command => command.replaceAll('<HOME>', fixture.home));
|
||||
for (const file of ['ship-docs-20261018-c0048c5-a1-documentation.md', 'ship-report.md']) fs.writeFileSync(path.join(fixture.home, file), '');
|
||||
expect(commands).toHaveLength(36);
|
||||
expect(commands).toContain('git describe --tags --always');
|
||||
expect(commands.filter(command => !docsCommandAllowed(command, fixture))).toEqual([]);
|
||||
expect(docsToolFailures(nativeResult('git describe --tags --always', 'c0048c5', 'docs-dispatch'), fixture, [], true)).toEqual([]);
|
||||
});
|
||||
|
||||
const DESCRIBE_READS = ['git describe', 'git describe --tags', 'git describe --always', 'git describe --abbrev=7',
|
||||
'git describe --exact-match', 'git describe HEAD', "git describe --tags --always --abbrev=12 'HEAD^{}'",
|
||||
'git describe --exact-match --tags main', 'git describe --always main', 'git describe --tags --always 2>/dev/null'];
|
||||
const DESCRIBE_REFUSED = ['git describe --dirty', 'git describe --dirty=-modified', 'git describe --always --dirty',
|
||||
'git describe --broken', 'git describe --broken=-broken', 'git describe --contains HEAD', 'git describe --all',
|
||||
'git describe --abbrev', 'git describe --abbrev=x', 'git describe HEAD main', 'git describe -- HEAD',
|
||||
'git describe --tags > tag.md', 'git describe --tags; git tag v9', 'git describe --tags && git tag v9'];
|
||||
|
||||
test('read-only git describe forms are declared and leave the index and repository unchanged', () => {
|
||||
expect(docsNativeInterface(fixture)).toContain('describe with only --tags, --always, --abbrev=<n> or --exact-match and at most one revision');
|
||||
const own = fixtureDocs('store');
|
||||
try {
|
||||
gitAt(own.repo, 'tag', '-a', 'v0.1.0', '-m', 'fixture release', 'HEAD');
|
||||
const stale = new Date(Date.now() - 3600_000);
|
||||
fs.utimesSync(path.join(own.repo, 'AGENTS.md'), stale, stale);
|
||||
const index = () => fs.readFileSync(path.join(own.repo, '.git/index')).toString('base64');
|
||||
const before = { repo: repoSnapshot(own.repo), index: index() };
|
||||
for (const command of DESCRIBE_READS) {
|
||||
expect(docsCommandAllowed(command, own), command).toBe(true);
|
||||
expect(docsToolFailures(nativeResult(command), own, [], true), command).toEqual([]);
|
||||
const run = spawnSync('bash', ['-c', command], { cwd: own.repo, encoding: 'utf8', timeout: 10000 });
|
||||
expect(run.status, `${command}: ${run.stderr}`).toBe(0);
|
||||
expect({ repo: repoSnapshot(own.repo), index: index() }, command).toEqual(before);
|
||||
}
|
||||
} finally {
|
||||
own.clean();
|
||||
}
|
||||
});
|
||||
|
||||
test('describe forms that refresh the index or fall outside the declared set, and Git writes, stay outside the interface', () => {
|
||||
for (const command of [...DESCRIBE_REFUSED, 'git tag v9 HEAD', 'git update-index --refresh', 'git notes add -m x HEAD']) {
|
||||
expect(docsCommandAllowed(command, fixture), command).toBe(false);
|
||||
expect(docsToolFailures(nativeResult(command, 'successful tool acknowledgment', 'docs-dispatch'), fixture, [], true), command)
|
||||
.toEqual(['command outside declared docs observation interface']);
|
||||
}
|
||||
const own = fixtureDocs('store');
|
||||
try {
|
||||
const stale = new Date(Date.now() - 3600_000);
|
||||
fs.utimesSync(path.join(own.repo, 'AGENTS.md'), stale, stale);
|
||||
const index = () => fs.readFileSync(path.join(own.repo, '.git/index')).toString('base64');
|
||||
const before = index();
|
||||
const run = spawnSync('git', ['describe', '--always', '--dirty'], { cwd: own.repo, encoding: 'utf8', timeout: 10000 });
|
||||
expect(run.status, run.stderr).toBe(0);
|
||||
expect(index()).not.toBe(before);
|
||||
} finally {
|
||||
own.clean();
|
||||
}
|
||||
});
|
||||
@@ -197,6 +197,7 @@ Keep artifacts concise: update the invocation record in place with ids, counts,
|
||||
|
||||
const DOCS_READ_COMMANDS = ['pwd', 'ls', 'cat', 'sha256sum', 'stat'];
|
||||
const DOCS_GIT_READS = ['status', 'diff', 'show', 'log', 'ls-files', 'ls-tree', 'check-ignore', 'rev-parse', 'merge-base', 'hash-object'];
|
||||
const DOCS_DESCRIBE_FLAGS = /^(?:--tags|--always|--exact-match|--abbrev=\d+)$/;
|
||||
const DOCS_SAFE_GIT_CONFIG = /^(?:core\.pager=cat|core\.quotepath=(?:true|false|on|off)|color\.[a-z.]+=(?:true|false|never|always|auto))$/i;
|
||||
|
||||
/**
|
||||
@@ -261,6 +262,10 @@ function literalDocsCommandAllowed(text: string, fixture: ReturnType<typeof fixt
|
||||
if (rest.some(arg => /^(?:--output|--ext-diff|--textconv|-w)(?:=|$)/.test(arg))) return false;
|
||||
if (rest[0] === 'hash-object' && rest.some(arg => /^-[^-]*w/.test(arg))) return false;
|
||||
if (rest[0] === 'branch') return rest.length === 2 && rest[1] === '--show-current';
|
||||
if (rest[0] === 'describe') {
|
||||
const operands = rest.slice(1).filter(arg => !DOCS_DESCRIBE_FLAGS.test(arg));
|
||||
return operands.length <= 1 && operands.every(arg => !arg.startsWith('-'));
|
||||
}
|
||||
return DOCS_GIT_READS.includes(rest[0]);
|
||||
}
|
||||
if (commandName === 'bun' || commandName === process.execPath) {
|
||||
@@ -288,7 +293,7 @@ function literalDocsCommandAllowed(text: string, fixture: ReturnType<typeof fixt
|
||||
|
||||
export function docsNativeInterface(fixture: Pick<ReturnType<typeof fixtureDocs>, 'home' | 'repo' | 'skills'>, scripts: string[] = [], transport = false, insert = false, file?: string): string {
|
||||
const skills = fixture.skills.split(path.sep).join('/');
|
||||
return `Fixture observation interface (applies to parent and every child; ${file ? `saved at ${file}: each child prompt tells the child to Read that file first instead of copying this text` : 'include this interface in child prompts'}): Bash may execute only separate literal pwd, ls, cat, stat, sha256sum, Git read commands (status, diff, show, log, ls-files, ls-tree, check-ignore, rev-parse, merge-base, hash-object without -w, branch --show-current), the exact generated Preamble block with its spawned prefix, literal installed gstack-skill-start/gstack-skill-end commands for document-release (start requires GSTACK_SESSION_KIND=spawned) or their own --help, or the literal installed ${skills}/bin/gstack-docs-candidate snapshot (its --out a private .json under ${fixture.home}), compare <that .json> and --help commands. No shell composition, custom interpreters, arbitrary scripts, inline eval or memory-mapped writes. The only additional scripts are ${scripts.length ? scripts.join(', ') : 'none'}.${insert ? ` To insert one saved private Markdown artifact into another (a saved section file into a report), Bash may also run the single literal command cat SOURCE.md >> TARGET.md with absolute paths under ${fixture.home}.` : ''} Read/Glob/Grep remain available; Read skill and section files with Read (offset/limit for ranges), because Bash output over 30KB becomes a preview that no permitted Bash command can page. Use Write/Edit for permitted docs and private JSON/Markdown artifacts under ${fixture.home}; do not rewrite installed skills, config, actor state or scripts. No effects outside the owned fixture. The owner preserves evidence and cleans up. Missing observer coverage blocks acceptance; the Linux kernel monitor covers syscall writes in the product tree, not hostile processes or arbitrary external destinations.
|
||||
return `Fixture observation interface (applies to parent and every child; ${file ? `saved at ${file}: each child prompt tells the child to Read that file first instead of copying this text` : 'include this interface in child prompts'}): Bash may execute only separate literal pwd, ls, cat, stat, sha256sum, Git read commands (status, diff, show, log, ls-files, ls-tree, check-ignore, rev-parse, merge-base, hash-object without -w, branch --show-current, describe with only --tags, --always, --abbrev=<n> or --exact-match and at most one revision), the exact generated Preamble block with its spawned prefix, literal installed gstack-skill-start/gstack-skill-end commands for document-release (start requires GSTACK_SESSION_KIND=spawned) or their own --help, or the literal installed ${skills}/bin/gstack-docs-candidate snapshot (its --out a private .json under ${fixture.home}), compare <that .json> and --help commands. No shell composition, custom interpreters, arbitrary scripts, inline eval or memory-mapped writes. The only additional scripts are ${scripts.length ? scripts.join(', ') : 'none'}.${insert ? ` To insert one saved private Markdown artifact into another (a saved section file into a report), Bash may also run the single literal command cat SOURCE.md >> TARGET.md with absolute paths under ${fixture.home}.` : ''} Read/Glob/Grep remain available; Read skill and section files with Read (offset/limit for ranges), because Bash output over 30KB becomes a preview that no permitted Bash command can page. Use Write/Edit for permitted docs and private JSON/Markdown artifacts under ${fixture.home}; do not rewrite installed skills, config, actor state or scripts. No effects outside the owned fixture. The owner preserves evidence and cleans up. Missing observer coverage blocks acceptance; the Linux kernel monitor covers syscall writes in the product tree, not hostile processes or arbitrary external destinations.
|
||||
|
||||
The working directory for parent and child Bash calls is already ${fixture.repo}. Run Git reads directly, for example: git status, git diff --cached, git merge-base main HEAD, git rev-parse HEAD. Do not use Git global options such as -C, -c, --git-dir or --work-tree, and do not prepend cd or another shell wrapper. The literal git subcommand must immediately follow git; an absolute owned repository path does not make git -C an allowed command.
|
||||
|
||||
|
||||
Reference in new issue
Block a user