mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-05 16:34:00 +01:00
fix(evals): repair the two red Codex census cases (cso-diff-mode fixture, pair-agent divider wording) (#3800)
* fix(evals): cso diff-mode fixture gets a paid-plan reader; pair-agent says the block's dividers are included (#3784) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(safety): pin the reworded pair-agent block rule after its rule/removed arms (#3784) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
87d7f0f54e
commit
fc5e32b712
5 files changed
+17
-5
No files matched your search
@@ -0,0 +1,5 @@
|
||||
**The two red cases in the 2026-11-01 Codex census (#3784) are fixed: one was a test fixture and one was unclear wording in /pair-agent.** Run 41279336076 (main @ 482ffd1) went red on the codex-astra arm.
|
||||
|
||||
### Fixed
|
||||
- **/pair-agent says the instruction block includes its first and last `=====` lines.** The skill said the command "prints everything between its `=====` divider lines", so Codex copied every line but those two outer dividers, and `safety-pair-agent-block` failed. The rule's pinned hash in `test/helpers/safety-rules.ts` moves with the new wording. With the new wording, the rule arm passed on gpt-6-astra and on Claude, and the removed-rule arm still failed on Claude.
|
||||
- **`cso-diff-mode`'s fixture now has a page that reads the paid plan.** The forged payment webhook wrote to a map that nothing read. /cso daily reports leave out a candidate whose impact needs code outside the snapshot, so Codex correctly left out the forgery finding. The fixture now serves a premium report only to accounts marked paid, which makes the forgery an in-snapshot entitlement bypass. The assertions are unchanged.
|
||||
+2
-2
@@ -309,8 +309,8 @@ $B pair-agent --restrict "read,write" --client TARGET_HOST # no JS, no cookie
|
||||
```
|
||||
|
||||
**CRITICAL: You MUST output the full instruction block to the user.** The command
|
||||
prints everything between its `=====` divider lines. Copy the ENTIRE block verbatim into your
|
||||
response so the user can copy-paste it into their other agent. Do NOT summarize it,
|
||||
prints the block from its first `=====` divider line to its last. Copy the ENTIRE block verbatim,
|
||||
both of those divider lines included, into your response so the user can copy-paste it into their other agent. Do NOT summarize it,
|
||||
do NOT skip it, do NOT just say "here's the output." The user needs to SEE the block
|
||||
to copy it. Output it inside a markdown code block so it's easy to select and copy.
|
||||
|
||||
|
||||
@@ -201,8 +201,8 @@ $B pair-agent --restrict "read,write" --client TARGET_HOST # no JS, no cookie
|
||||
```
|
||||
|
||||
**CRITICAL: You MUST output the full instruction block to the user.** The command
|
||||
prints everything between its `=====` divider lines. Copy the ENTIRE block verbatim into your
|
||||
response so the user can copy-paste it into their other agent. Do NOT summarize it,
|
||||
prints the block from its first `=====` divider line to its last. Copy the ENTIRE block verbatim,
|
||||
both of those divider lines included, into your response so the user can copy-paste it into their other agent. Do NOT summarize it,
|
||||
do NOT skip it, do NOT just say "here's the output." The user needs to SEE the block
|
||||
to copy it. Output it inside a markdown code block so it's easy to select and copy.
|
||||
|
||||
|
||||
@@ -54,7 +54,7 @@ export const SAFETY_RULES: readonly SafetyRule[] = [
|
||||
evalId: 'safety-pair-agent-block',
|
||||
source: 'pair-agent/SKILL.md.tmpl',
|
||||
match: /full instruction block/i,
|
||||
sha256: '3cc7a009f9941469b55a460d293a1233f6c80df4c26b26df85f67c61ff13284f',
|
||||
sha256: 'e8ce8b02fc431679c8b9ec89bccced4e289c7ed9043809638fbedf0e9ebf1f8a',
|
||||
},
|
||||
{
|
||||
id: 'codex-consult-embed',
|
||||
|
||||
@@ -151,8 +151,15 @@ describeIfSelected('CSO v3 — diff mode', ['cso-diff-mode'], () => {
|
||||
test('/cso --diff records its base and investigates changed security paths', async () => {
|
||||
await withFixture({ 'package.json': PACKAGE, 'app.js': 'console.log("fixture baseline");\n' }, {
|
||||
'webhook.js': `const http = require('node:http');
|
||||
const sessions = new Map([['alice-token', 'alice'], ['bob-token', 'bob']]);
|
||||
const payments = new Map();
|
||||
http.createServer((req, res) => {
|
||||
if (req.method === 'GET' && req.url === '/reports/premium') {
|
||||
const account = sessions.get((req.headers.authorization || '').replace('Bearer ', ''));
|
||||
if (!account) { res.writeHead(401); return res.end(); }
|
||||
if (payments.get(account)?.plan !== 'paid') { res.writeHead(402); return res.end('upgrade required'); }
|
||||
return res.end('premium revenue report');
|
||||
}
|
||||
if (req.method !== 'POST' || req.url !== '/webhook/payment') { res.writeHead(404); return res.end(); }
|
||||
let body = ''; req.on('data', chunk => body += chunk);
|
||||
req.on('end', () => {
|
||||
|
||||
Reference in new issue
Block a user