v1.91.280.0 fix(flaky): ubi-runner quota wait counts down; pregate TERM teardown and design-lint linearity measure what they mean (GSTA-128) (#3517)

* fix(flaky): ubi-runner quota wait counts its budget down; pregate TERM teardown and design-lint linearity checks measure what they mean (GSTA-128)

Daily flaky pass, free-tier timing entries from the 2026-10-27 ledger.

- test/ubicloud-runner.test.ts quota case (Free Tests 40474956830, 40480063084,
  40488875742): quota_preflight pinned a wall-clock deadline before its first usage
  call, so a slow fake-CLI spawn on a loaded runner shaved the printed budget to
  "waiting up to 2s more". The budget now counts seconds spent waiting for room;
  usage reads between sleeps no longer eat it. Loaded local loop: 8/12 and 6/12
  green before, 12/12 after.
- test/ubicloud-pregate-remote.test.ts TERM teardown (40487321907 shard 13): the
  stub's `>>` creates the .down file before echo writes the line, and the test
  polled the inode, then read "". Both teardown waits now wait for the written line.
- test/design-lint-fifth-cycle.test.ts tracked-caps linearity (40494745147 shard
  15): the 8000-label run was timed once by wall clock on a shared runner (515 ms
  against the 500 ms floor). It now measures the lint's own CPU time, fastest of
  three runs, against the same unchanged bound.
- The hung-setup watchdog assertion now carries the runner's stderr, so its next
  red names the cause (one unexplained exit 2 in 46 local runs).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(ubi-runner): prove the quota wait countdown; CHANGELOG entry for the free-tier timing repairs (GSTA-128)

The existing WAIT=3 quota case passes under both the old wall-clock deadline
and the new countdown because its fake usage read is instant. The new case
slows the usage read to 1.2 s and asserts the printed 2 s budget (origin/main
prints 1 s), then stubs sleep for a 70 s wait to see the 30 s steps and the
release when the peer VM disappears. /ship Step 13 adds the [Unreleased]
CHANGELOG entry; the train re-heads it at the slot.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(ubi-runner): assert the quota wait budget on its log line, not on elapsed wall time (GSTA-128)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-10-27 17:39:03 +00:00
1 parent b3e94a3f8d
commit 703b0049eb
8 files changed
+67 -14

No files matched your search

+7 -4
View File
@@ -393,7 +393,7 @@ vcpus() { [[ "$1" =~ -([0-9]+)$ ]] && echo "${BASH_REMATCH[1]}"; }
# maximum the last refusal printed; without either the API's own refusal rules.
QUOTA_CACHE_MAX_AGE=86400
quota_preflight() {
local location=$1 name=$2 size=$3 need max used wait_until remaining source="UBI_QUOTA_VCPUS" cache="$STATE_ROOT/quota-max"
local location=$1 name=$2 size=$3 need max used remaining step source="UBI_QUOTA_VCPUS" cache="$STATE_ROOT/quota-max"
need=$(vcpus "$size") || return 0
max=$QUOTA_VCPUS
if [ -z "$max" ] && [ -f "$cache" ]; then
@@ -406,19 +406,22 @@ quota_preflight() {
fi
fi
[[ "$max" =~ ^[0-9]+$ ]] || return 0
wait_until=$(( $(date +%s) + QUOTA_WAIT ))
# The budget counts seconds spent waiting for room, not the usage calls
# between them: a slow first usage read must not shave the printed budget.
remaining=$QUOTA_WAIT
while :; do
used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}')
[[ "$used" =~ ^[0-9]+$ ]] || return 0
[ $(( used + need )) -gt "$max" ] || return 0
remaining=$(( wait_until - $(date +%s) ))
if [ "$remaining" -le 0 ]; then
log "quota preflight refused $location/$name ($size): it needs $need vCPUs and the project's VMs already use $used of $max (ceiling from $source; other usage such as managed runners counts on top)"
cmd_usage >&2 || true
die "create refused before any request: vCPU quota exhausted (quota full); wait for running VMs to finish (UBI_QUOTA_WAIT=<seconds> waits here), run $0 reap, use a smaller size, or set UBI_QUOTA_VCPUS to the project's real ceiling"
fi
log "quota: $used of $max vCPUs in use and $need needed; waiting up to ${remaining}s more for room"
sleep $(( remaining < 30 ? remaining : 30 ))
step=$(( remaining < 30 ? remaining : 30 ))
sleep "$step"
remaining=$(( remaining - step ))
done
}