diff --git a/docs/pregate.md b/docs/pregate.md index 4e223e0cf..c9c36c8ba 100644 --- a/docs/pregate.md +++ b/docs/pregate.md @@ -612,8 +612,16 @@ the VM to stop at whichever comes first, that point or the cap's wall (at most 45 minutes, minus 3 for teardown). A deadline that leaves the VM under 300 s refuses before any VM (`PREGATE_REMOTE_TIMEOUT_TOO_SHORT`). At create time it prints the VM's name and the exact -`bash scripts/ubicloud/ubi-runner.sh down ` line to destroy it if the -run is killed before its teardown line. The whole script is parsed before it +`bash scripts/ubicloud/ubi-runner.sh down ` line to destroy it. + +A run killed before its teardown line leaves its VM running. A machine restart +kills a background pregate this way. Don't destroy the VM: rerun the same +`gstack-pregate` command on the same tree. It reattaches to that VM +(`reattaching to VM `), finishes the run, and bills from the VM's +original create. Use the `down` line only when you are giving up on the run +or the tree has changed. A reattach whose billed minutes leave the cap no room +refuses (`PREGATE_REMOTE_CAP_SPENT`) and prints both choices: destroy the VM, +or rerun with a higher `--spend-cap`. The whole script is parsed before it runs, so editing it during a run changes nothing. `--timeout 2700` gives a whole-tier run room inside the 45-minute wall: diff --git a/scripts/ubicloud/pregate-remote.sh b/scripts/ubicloud/pregate-remote.sh index c7eb3f90f..06299ddee 100755 --- a/scripts/ubicloud/pregate-remote.sh +++ b/scripts/ubicloud/pregate-remote.sh @@ -248,7 +248,7 @@ while :; do rm -f "$log.pid" "$log.timedout" "$log.left" t0=$(date +%s) vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$(( (stop_after + 900 + 3599) / 3600 ))" - [ -n "$prior_vm" ] || say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name" + [ -n "$prior_vm" ] || say "VM $vm_name; if this run is killed before its teardown line, rerun the same command to reattach; to give up on it, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name" UBI_LEFT_FILE="$log.left" python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \ bash -c "$vm_session" pregate-remote-vm "$stop_after" "$log" "$stream" "$here/ubi-runner.sh" "${args[@]}" -n "$vm_name" & vmrun=$! diff --git a/ship/DRIVER.md b/ship/DRIVER.md index ca721a499..8c0806e6b 100644 --- a/ship/DRIVER.md +++ b/ship/DRIVER.md @@ -13,7 +13,7 @@ outputs; print them verbatim. The model keeps only lead judgment, fixes, CHANGEL gates and the report. `STOP:` (exit 3) means nothing after it ran; `NEXT:` names the next act. Read a section only when a phase or the list at the end sends you, never up front. A STOP or decision this file does not cover: read `ship/SKILL.md` at that step, finish it -there, come back. Unattended, take a question's recommended option (log it); gate +there, come back. Unattended, take a question's recommended option; gate dispositions and approvals stay pending and end the run. Keep a private invocation record (`--record`): write its six headings at Phase 1, fill each as its phase ends: **Checks** (Phases 2-4: lanes, pregate, evals, `Coverage:`, the plan checks, `Exploratory QA:`, `Scope Check:`), **Decisions** (any phase: each answered or auto-taken question, verbatim), **Reviews** (Phases 3-5: the plan line, @@ -38,7 +38,7 @@ A dispatching program takes `BEFORE` (R and C) as readers start and passes `--dispatched-at --dispatched-credits --readers ` with the reading after the last settles. **Waits.** No call waits past 200 s: a program returns what still runs 200 s after its first line; the next -call resumes the poll. Lanes, pregate and `finish` run in background. Never end the turn while +call resumes the poll. Lanes, pregate and `finish` run in background; after a restart, rerun the same command. Never end the turn while readers run (each wakes it). A small waiting child saves nothing. Past 100 minutes, stop. Returns: ≤32 KB per program; split longer reads at headings, each byte once, before first use: phase programs end with `gstack-ship-parts next --run --phase --room <32000 - its output>`; on @@ -112,8 +112,8 @@ One re-audit at most. Commit in bisectable chunks; `gstack-ship-release finish --base --reviewed-tree --reviewed-base --docs-post --lane ... [--build ] --trailer ''`: `ROUTE:`, `DOCS:`, `EVIDENCE:` (FRESH; STALE or MISSING reruns once), `PREGATE:`; print them, the last `Gate edits:` and each `tests_ran`. Then -`gstack-ship-release push --branch [--gate-open ]`: `REDACT_PREPUSH:`, `SEALED:`, `PUSH:` -(pushed or ALREADY_PUSHED). Unattended, push defers the pre-push offer; else its STOP runs Step 17's guard block (`HOOK_STATE:`, `HOOKS_IN_GIT_DIR:`, `PREPUSH_PROMPTED:`). Never force-push. +`gstack-ship-release push --branch [--gate-open ]`: `REDACT_PREPUSH:`, `SEALED:`, `PUSH:`. +Unattended, push defers the pre-push offer; else its STOP runs Step 17's guard block (`HOOK_STATE:`, `HOOKS_IN_GIT_DIR:`, `PREPUSH_PROMPTED:`). Never force-push. ## 8. Publish: Steps 18, 19, 19-C, 20, 21, 22 diff --git a/ship/DRIVER.md.tmpl b/ship/DRIVER.md.tmpl index a0f0d8ac1..c32c2f88e 100644 --- a/ship/DRIVER.md.tmpl +++ b/ship/DRIVER.md.tmpl @@ -11,7 +11,7 @@ outputs; print them verbatim. The model keeps only lead judgment, fixes, CHANGEL gates and the report. `STOP:` (exit 3) means nothing after it ran; `NEXT:` names the next act. Read a section only when a phase or the list at the end sends you, never up front. A STOP or decision this file does not cover: read `ship/SKILL.md` at that step, finish it -there, come back. Unattended, take a question's recommended option (log it); gate +there, come back. Unattended, take a question's recommended option; gate dispositions and approvals stay pending and end the run. Keep a private invocation record (`--record`): write its six headings at Phase 1, fill each as its phase ends: **Checks** (Phases 2-4: lanes, pregate, evals, `Coverage:`, the plan checks, `Exploratory QA:`, `Scope Check:`), **Decisions** (any phase: each answered or auto-taken question, verbatim), **Reviews** (Phases 3-5: the plan line, @@ -36,7 +36,7 @@ A dispatching program takes `BEFORE` (R and C) as readers start and passes `--dispatched-at --dispatched-credits --readers ` with the reading after the last settles. **Waits.** No call waits past 200 s: a program returns what still runs 200 s after its first line; the next -call resumes the poll. Lanes, pregate and `finish` run in background. Never end the turn while +call resumes the poll. Lanes, pregate and `finish` run in background; after a restart, rerun the same command. Never end the turn while readers run (each wakes it). A small waiting child saves nothing. Past 100 minutes, stop. Returns: ≤32 KB per program; split longer reads at headings, each byte once, before first use: phase programs end with `gstack-ship-parts next --run --phase --room <32000 - its output>`; on @@ -110,8 +110,8 @@ One re-audit at most. Commit in bisectable chunks; `gstack-ship-release finish --base --reviewed-tree --reviewed-base --docs-post --lane ... [--build ] --trailer ''`: `ROUTE:`, `DOCS:`, `EVIDENCE:` (FRESH; STALE or MISSING reruns once), `PREGATE:`; print them, the last `Gate edits:` and each `tests_ran`. Then -`gstack-ship-release push --branch [--gate-open ]`: `REDACT_PREPUSH:`, `SEALED:`, `PUSH:` -(pushed or ALREADY_PUSHED). Unattended, push defers the pre-push offer; else its STOP runs Step 17's guard block (`HOOK_STATE:`, `HOOKS_IN_GIT_DIR:`, `PREPUSH_PROMPTED:`). Never force-push. +`gstack-ship-release push --branch [--gate-open ]`: `REDACT_PREPUSH:`, `SEALED:`, `PUSH:`. +Unattended, push defers the pre-push offer; else its STOP runs Step 17's guard block (`HOOK_STATE:`, `HOOKS_IN_GIT_DIR:`, `PREPUSH_PROMPTED:`). Never force-push. ## 8. Publish: Steps 18, 19, 19-C, 20, 21, 22 diff --git a/test/ubicloud-pregate-remote.test.ts b/test/ubicloud-pregate-remote.test.ts index 79afd1296..af53836c4 100644 --- a/test/ubicloud-pregate-remote.test.ts +++ b/test/ubicloud-pregate-remote.test.ts @@ -272,7 +272,7 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => { expect(create.exit, create.out).toBe(4); expect(create.out).toContain('PREGATE_REMOTE_INFRA: infrastructure failure (ubi-runner exit 1): ubi-runner: create failed'); expect(create.out).toMatch(/VM: eu-central-h1\/(ubirun-stub-\d+-[0-9a-f]{8}-t1); ubi-runner: gone: eu-central-h1\/\1 \(never existed or already destroyed\)/); - expect(create.out).toMatch(/VM (ubirun-stub-\d+-[0-9a-f]{8}-t1); if this run is killed before its teardown line, destroy it with: bash scripts\/ubicloud\/ubi-runner\.sh down \1\n/); + expect(create.out).toMatch(/VM (ubirun-stub-\d+-[0-9a-f]{8}-t1); if this run is killed before its teardown line, rerun the same command to reattach; to give up on it, destroy it with: bash scripts\/ubicloud\/ubi-runner\.sh down \1\n/); expect(create.out).toMatch(/log: \S+\/\.context\/pregate-remote\/\S+\/run\.log/); expect(create.out).toContain('spent=0.00 (0 billed min on standard-16)'); const setup = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_SETUP_FAIL: '1' });