fix(evals): repair the two red Codex census cases (cso-diff-mode fixture, pair-agent divider wording) (#3800)

* fix(evals): cso diff-mode fixture gets a paid-plan reader; pair-agent says the block's dividers are included (#3784)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(safety): pin the reworded pair-agent block rule after its rule/removed arms (#3784)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-05 00:22:40 +00:00
1 parent 87d7f0f54e
commit fc5e32b712
5 files changed
+17 -5

No files matched your search

@@ -0,0 +1,5 @@
**The two red cases in the 2026-11-01 Codex census (#3784) are fixed: one was a test fixture and one was unclear wording in /pair-agent.** Run 41279336076 (main @ 482ffd1) went red on the codex-astra arm.
### Fixed
- **/pair-agent says the instruction block includes its first and last `=====` lines.** The skill said the command "prints everything between its `=====` divider lines", so Codex copied every line but those two outer dividers, and `safety-pair-agent-block` failed. The rule's pinned hash in `test/helpers/safety-rules.ts` moves with the new wording. With the new wording, the rule arm passed on gpt-6-astra and on Claude, and the removed-rule arm still failed on Claude.
- **`cso-diff-mode`'s fixture now has a page that reads the paid plan.** The forged payment webhook wrote to a map that nothing read. /cso daily reports leave out a candidate whose impact needs code outside the snapshot, so Codex correctly left out the forgery finding. The fixture now serves a premium report only to accounts marked paid, which makes the forgery an in-snapshot entitlement bypass. The assertions are unchanged.
+2 -2
View File
@@ -309,8 +309,8 @@ $B pair-agent --restrict "read,write" --client TARGET_HOST # no JS, no cookie
```
**CRITICAL: You MUST output the full instruction block to the user.** The command
prints everything between its `=====` divider lines. Copy the ENTIRE block verbatim into your
response so the user can copy-paste it into their other agent. Do NOT summarize it,
prints the block from its first `=====` divider line to its last. Copy the ENTIRE block verbatim,
both of those divider lines included, into your response so the user can copy-paste it into their other agent. Do NOT summarize it,
do NOT skip it, do NOT just say "here's the output." The user needs to SEE the block
to copy it. Output it inside a markdown code block so it's easy to select and copy.
+2 -2
View File
@@ -201,8 +201,8 @@ $B pair-agent --restrict "read,write" --client TARGET_HOST # no JS, no cookie
```
**CRITICAL: You MUST output the full instruction block to the user.** The command
prints everything between its `=====` divider lines. Copy the ENTIRE block verbatim into your
response so the user can copy-paste it into their other agent. Do NOT summarize it,
prints the block from its first `=====` divider line to its last. Copy the ENTIRE block verbatim,
both of those divider lines included, into your response so the user can copy-paste it into their other agent. Do NOT summarize it,
do NOT skip it, do NOT just say "here's the output." The user needs to SEE the block
to copy it. Output it inside a markdown code block so it's easy to select and copy.
+1 -1
View File
@@ -54,7 +54,7 @@ export const SAFETY_RULES: readonly SafetyRule[] = [
evalId: 'safety-pair-agent-block',
source: 'pair-agent/SKILL.md.tmpl',
match: /full instruction block/i,
sha256: '3cc7a009f9941469b55a460d293a1233f6c80df4c26b26df85f67c61ff13284f',
sha256: 'e8ce8b02fc431679c8b9ec89bccced4e289c7ed9043809638fbedf0e9ebf1f8a',
},
{
id: 'codex-consult-embed',
+7
View File
@@ -151,8 +151,15 @@ describeIfSelected('CSO v3 — diff mode', ['cso-diff-mode'], () => {
test('/cso --diff records its base and investigates changed security paths', async () => {
await withFixture({ 'package.json': PACKAGE, 'app.js': 'console.log("fixture baseline");\n' }, {
'webhook.js': `const http = require('node:http');
const sessions = new Map([['alice-token', 'alice'], ['bob-token', 'bob']]);
const payments = new Map();
http.createServer((req, res) => {
if (req.method === 'GET' && req.url === '/reports/premium') {
const account = sessions.get((req.headers.authorization || '').replace('Bearer ', ''));
if (!account) { res.writeHead(401); return res.end(); }
if (payments.get(account)?.plan !== 'paid') { res.writeHead(402); return res.end('upgrade required'); }
return res.end('premium revenue report');
}
if (req.method !== 'POST' || req.url !== '/webhook/payment') { res.writeHead(404); return res.end(); }
let body = ''; req.on('data', chunk => body += chunk);
req.on('end', () => {