Nightly Windows tests fit their time limit again (GST-2827) (#4409)

* Nightly Windows tests fit their time limit again (GST-2827)

The Windows duration seed missed 95 of 902 files, so the planner packed
them blind and one shard got about 26 minutes of work. Refresh the seed
from the 2026-11-16 recording and run eight shards of about 12 minutes.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Docs say the nightly Windows run uses eight shards (GST-2827)

CONTRIBUTING.md and docs/support-tiers.md still said six shards after the workflow moved to eight.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-17 08:25:01 +00:00
1 parent 18652ccc15
commit be7393888f
7 files changed
+903 -814

No files matched your search

+4 -3
View File
@@ -110,8 +110,9 @@ jobs:
bun-version: 1.4.2
- id: plan
name: Curate and balance the Windows-safe free suite
# Six shards: ~85 s of Linux-measured work each, ~2.5 min on a
# 4-vCPU Windows runner, so the lane lands near 4 min with setup.
# Eight shards: the 2026-11-16 Windows seed sums to about 96 min of
# per-file work, about 12 min a shard, which ran at 0.6-0.9x inside a
# shard, so each job lands near 10 min, under its 15-minute limit.
env:
GSTACK_FREE_TEST_DURATIONS: scripts/free-test-durations-windows.json
JOBS_OVERRIDE: ${{ inputs.windows_jobs }}
@@ -125,7 +126,7 @@ jobs:
if [ "$jobs" != "$WINDOWS_FREE_JOBS" ]; then
echo "- Sub-shard experiment: $jobs sub-shards per Windows shard (default $WINDOWS_FREE_JOBS); not a PR Windows Free Tests result" >> "$GITHUB_STEP_SUMMARY"
fi
matrix=$(bun run scripts/test-free-shards.ts --windows-only --ci-plan "$RUNNER_TEMP/windows-plan.json" --shards 6 --shard-jobs "$jobs")
matrix=$(bun run scripts/test-free-shards.ts --windows-only --ci-plan "$RUNNER_TEMP/windows-plan.json" --shards 8 --shard-jobs "$jobs")
echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
+1 -1
View File
@@ -193,7 +193,7 @@ the new defaults.
| Run the full free suite from a small machine | `bun run test:ubicloud` | Free suite on a billed 16-vCPU VM, about 5 minutes | `UBICLOUD_API_KEY`, `UBI_OWNER` |
| Clear pregate's test tier on a machine under 8 cores from CI, the default on gstack ([CI carry](docs/pregate.md#ci-carry)) | push, `gstack-ci-carry wait --base main`, then `gstack-pregate --allow-repo-commands` (a branch behind main first runs `gh workflow run free-tests.yml --ref <branch>`) | Free; reuses the green Free Tests run at your pushed tree | Pushed branch, `gh` |
| Clear pregate's test tier on a machine under 8 cores when no CI run covers your tree ([remote gate](docs/pregate.md#remote-gate), opt-in) | `gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands` (same command keyed or keyless) | Keyed: one billed 16-vCPU VM, about 7 minutes and $0.06 for the whole free tier, worst case $0.38; keyless: free, local parallel shards, practical for small selections only (98 files took 103 s on 4 cores) | Keyed: `UBICLOUD_API_KEY`, `UBI_OWNER`; keyless: nothing |
| Run the curated Windows-safe subset | `bun run test:windows` (CI: six `windows-latest` jobs, nightly on main; PRs run the shorter `windows-smoke` check, and a PR that touches path, file-name or process code also runs its changed Windows-safe test files in `Windows Scoped Tests`) | Free | Windows, Git Bash |
| Run the curated Windows-safe subset | `bun run test:windows` (CI: eight `windows-latest` jobs, nightly on main; PRs run the shorter `windows-smoke` check, and a PR that touches path, file-name or process code also runs its changed Windows-safe test files in `Windows Scoped Tests`) | Free | Windows, Git Bash |
| Rerun chosen Windows-safe test files on Windows CI | `gh workflow run windows-scoped-tests.yml --ref <branch> -f files='<file> <file>'` (pick files from `bun run test:windows --list`) | Free CI runners | Pushed branch, `gh` with workflow rights |
| Refresh the Windows duration seed | `gh workflow run windows-free-tests.yml --ref <branch> -f record_durations=true`, then commit the `free-test-durations-windows` artifact | Free CI runners | Pushed branch, `gh` with workflow rights |
| Run a native Windows qualification campaign | `gh workflow run native-qualification.yml --ref <branch> -f mode=<mode>` ([modes](docs/TESTING_INTERNALS.md#windows-free-lane)) | Free CI runners | Pushed branch, `gh` with workflow rights |
@@ -0,0 +1 @@
- **The nightly Windows test run finishes again.** Since 2026-11-06 one of its six shards ran past the 15-minute job limit almost every night, because 95 newer test files had no recorded Windows duration and the planner packed them blind, so one shard got 26 minutes of work while another got 12. The Windows durations are refreshed to a recording that covers all 902 files, and the run now uses eight shards of about 12 minutes each, so the suite fits with room to grow. No test was dropped and the time limit is unchanged.
+1 -1
View File
@@ -1808,7 +1808,7 @@ install change is `package.json`'s version line. Both filters are pinned by
a branch, dispatch `windows-scoped-tests.yml` with `-f files='<file> <file>'`.
The full curated suite below runs nightly on main and on dispatch, never on a PR.
`windows-free-tests.yml` plans on Linux (`--windows-only --ci-plan --shards 6
`windows-free-tests.yml` plans on Linux (`--windows-only --ci-plan --shards 8
--shard-jobs $WINDOWS_FREE_JOBS`), runs one strict serial shard per
`windows-latest` job
(`bun run test:windows --ci-run`), and verifies every shard receipt on Linux
+1 -1
View File
@@ -24,7 +24,7 @@ That is the whole promise. Other skills, optional parts such as `/cso`'s native
## Windows CI
- `windows-smoke` is the required Windows check on pull requests.
- The six-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs nightly on main, or when someone starts it by hand, and never blocks a pull request. The stable release cut ignores its results too, so a red or unfinished nightly never holds a release.
- The eight-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs nightly on main, or when someone starts it by hand, and never blocks a pull request. The stable release cut ignores its results too, so a red or unfinished nightly never holds a release.
- A red nightly run gets a fix only when it shows an install, startup or core-workflow break.
- Windows checks run by path (ship-90-90 plan D, approved 2026-11-03): `Windows Scoped Tests` runs a pull request's changed Windows-safe test files on one Windows job only when it touches path, file-name or process code (a red run blocks that pull request, like any failed check), and `Windows Setup E2E` runs only when install files change (a version-only `package.json` change doesn't count).
File diff suppressed because it is too large. Load diff
+3 -3
View File
@@ -34,7 +34,7 @@ test('the Windows lane runs nightly and by hand, plans, runs one strict shard pe
const plan = windows.jobs['windows-plan'];
expect(plan['runs-on']).toBe('ubuntu-24.04');
const planStep = plan.steps.find((step: any) => step.id === 'plan');
expect(planStep.run).toContain('--windows-only --ci-plan "$RUNNER_TEMP/windows-plan.json" --shards 6');
expect(planStep.run).toContain('--windows-only --ci-plan "$RUNNER_TEMP/windows-plan.json" --shards 8');
expect(planStep.env.GSTACK_FREE_TEST_DURATIONS).toBe('scripts/free-test-durations-windows.json');
const shard = windows.jobs['windows-free-shard'];
expect(shard['runs-on']).toBe('windows-latest');
@@ -72,8 +72,8 @@ test.skipIf(!Bun.which('bash'))('windows-plan passes the one sub-shard constant,
return { status: result.status, args: existsSync(path.join(directory, 'bun-args')) ? readFileSync(path.join(directory, 'bun-args'), 'utf8') : '' };
};
try {
expect(resolve('')).toEqual({ status: 0, args: expect.stringContaining(`--shards 6 --shard-jobs ${windows.env.WINDOWS_FREE_JOBS}`) });
expect(resolve('2')).toEqual({ status: 0, args: expect.stringContaining('--shards 6 --shard-jobs 2') });
expect(resolve('')).toEqual({ status: 0, args: expect.stringContaining(`--shards 8 --shard-jobs ${windows.env.WINDOWS_FREE_JOBS}`) });
expect(resolve('2')).toEqual({ status: 0, args: expect.stringContaining('--shards 8 --shard-jobs 2') });
for (const bad of ['0', '5', '2; true', 'x']) {
const refused = resolve(bad);
expect(refused.status, bad).not.toBe(0);