From b0ea2296d165c87301f0a91adc08d1614203a23a Mon Sep 17 00:00:00 2001 From: Sinabina Date: Fri, 17 Jul 2026 14:56:54 -0700 Subject: [PATCH] record final cross-platform validation evidence --- docs/gstack-2/ARCHITECTURE.md | 6 +- docs/gstack-2/HOST-COMPATIBILITY.md | 54 ++++++++--- docs/gstack-2/IOS-PHYSICAL-DEVICE.md | 23 ++++- docs/gstack-2/STATUS.md | 113 ++++++++++++---------- docs/gstack-2/TEST-EVIDENCE.md | 31 ++++--- docs/gstack-2/UPGRADE-AND-ROLLBACK.md | 15 ++- evals/ci/native-2026-07-17.json | 129 ++++++++++++++++++++++++++ 7 files changed, 279 insertions(+), 92 deletions(-) create mode 100644 evals/ci/native-2026-07-17.json diff --git a/docs/gstack-2/ARCHITECTURE.md b/docs/gstack-2/ARCHITECTURE.md index 23999fca3..4e83c7c77 100644 --- a/docs/gstack-2/ARCHITECTURE.md +++ b/docs/gstack-2/ARCHITECTURE.md @@ -240,13 +240,13 @@ means the replacement still lacks its required release evidence. | 5 | Single-host generation failures only warn | **Contained:** no per-host generation in the canonical path; canonical generation throws on failure. | Generator/parity suite and final build rerun are green. | | 6 | Setup continues after failed generation | **Implemented/contained:** standard installer handles skills; runtime setup fails before activation and preserves last known good. | Failure tests plus the real two-version default lifecycle and rollback pass. | | 7 | Removed/renamed generated skills are not pruned | **Implemented for canonical references:** regeneration removes each reference/asset tree and parity corpus before writing the fixed six. | Parity inventory test; stale-public-directory fixture pending. | -| 8 | Freshness misses generator/design/PDF changes | **Implemented:** build invokes canonical generation; parity hashes source, sections, assets, and fixtures. | Design is green at 101/0/381, PDF at 189/0/398, and the uninterrupted broad singleton run is green across all 383 files. | +| 8 | Freshness misses generator/design/PDF changes | **Implemented:** build invokes canonical generation; parity hashes source, sections, assets, and fixtures. | Design is green at 101/0/381, PDF at 189/0/398, and the uninterrupted broad singleton run is green at 6,255 pass / 226 expected skips / 0 fail across all 384 files. | | 9 | State paths bypass canonical resolver | **Implemented:** `runtime/paths.js` is authoritative for 2.0 runtime state. | `gstack2-runtime-core.test.ts`. | | 10 | Shell-evaluated path assignments are unsafe | **Implemented:** runtime paths are JavaScript values; shell-looking input remains literal. | Core test covers spaces, `$()`, semicolon, and `$HOME` text. | | 11 | Relinking omits carved section links | **Contained:** no per-section host relinking; sections are inlined into pinned modules. | Parity checks all 16 sections. | | 12 | External preambles omit PDF paths | **Contained:** PDF is indexed as an internal design capability/assets route, not copied host preamble prose. | Asset parity plus 189/0/398 PDF strict tests and a visually checked four-page live render pass. | | 13 | Production model benchmark imports test helpers | **Implemented:** the runner, pricing, providers, and optional judge live under `lib/model-benchmark/`; the production CLI imports only production modules. | `benchmark-production-boundary.test.ts` rejects imports from `test/` across `bin/` and `lib/`; focused runner and CLI tests exercise the relocated implementation. | -| 14 | Default tests omit `design/test` | **Implemented:** package default and free-test roots include `design/test`. | Design is green at 101/0/381 and is included in the uninterrupted 383-file broad pass. | +| 14 | Default tests omit `design/test` | **Implemented:** package default and free-test roots include `design/test`. | Design is green at 101/0/381 and is included in the uninterrupted 384-file broad pass. | | 15 | Default tests omit `ios-qa/daemon/test` | **Implemented:** package default and free-test roots include the daemon tests. | Focused daemon run: 95 pass / 0 fail / 229 assertions; daemon tests are also included in the uninterrupted broad pass. | | 16 | Host setup contradicts config-driven claim | **Contained:** host setup is no longer a 2.0 responsibility. | Standard installer CLI 1.5.19 passed all six configured host targets. | | 17 | Host-generated judgment copies drift | **Implemented:** one canonical module corpus with source-blob/render hashes. | 4,681 parity checks and installed-copy hashes pass. | @@ -254,7 +254,7 @@ means the replacement still lacks its required release evidence. | 19 | State identity crosses worktrees | **Implemented:** repo plus stable worktree identity selects state. | Linked-worktree core test passes. | | 20 | Partial ship failures are not reliably idempotent | **Implemented at runtime primitive:** claimed effects become uncertain and are not automatically repeated. | Crash/resume and completed-effect tests pass. End-to-end ship resume remains pending. | | 21 | Parser failures become empty success | **Implemented in iOS device discovery:** parse/tool failures are typed errors. | `tunnel-bootstrap.test.ts` malformed-JSON regression passes in the focused daemon suite. | -| 22 | Setup failures become product failures | **Partly implemented:** iOS discovery/setup categories and runtime doctor return actionable setup state. | Automated categories exist; live signing/provisioning gate pending. | +| 22 | Setup failures become product failures | **Partly implemented:** iOS discovery/setup categories and runtime doctor return actionable setup state. | Automated categories exist; live signing/build/install/launch/tunnel passed, while session acquisition and the five-check loop remain open and user-waived. | | 23 | Runtime network activity is not obvious | **Implemented for Context runtime:** selection, mode, and consent are explicit; status/doctor report them; zero lookup/fetch before Context selection+consent. | Context contract: 22 pass / 0 fail / 139 assertions, including persisted non-export fallbacks; verified-key official-endpoint live smoke passed. | | 24 | Context restore selects another worktree | **Implemented for canonical state resume:** current repo+worktree project ID scopes inspection/resume. | Linked-worktree identity test passes; compatibility end-to-end restore test pending. | | 25 | Preambles repeat large sections in every skill | **Implemented structurally:** six thin lazy dispatchers share infrastructure and load one preserved module on demand. | Current generated six-name/description catalog is 982 characters, about 246 token-equivalents versus the correctly parsed baseline of about 1,100 (77.6% lower). Re-measure if frontmatter changes. | diff --git a/docs/gstack-2/HOST-COMPATIBILITY.md b/docs/gstack-2/HOST-COMPATIBILITY.md index 66cc60122..06b8e2ac2 100644 --- a/docs/gstack-2/HOST-COMPATIBILITY.md +++ b/docs/gstack-2/HOST-COMPATIBILITY.md @@ -28,8 +28,8 @@ Review its detected-host prompt; detection must never silently enroll a host. Examples supported by the installer interface: ```bash -# One public skill only -npx skills add time-attack/gstack --skill plan +# One public skill only (point selection at the canonical skill directory) +npx skills add time-attack/gstack/skills --skill qa # Installer-managed global scope npx skills add time-attack/gstack -g @@ -54,6 +54,13 @@ ship appear as additional skills. Installing a subset must not pull the other five public entries unless the user selected them. +With `skills` CLI 1.5.19, the repository-root `--list` result is exactly these +six names. For selected installation, use the canonical `skills/` source as in +the example above. That CLI version's pre-filter display can count hidden +compatibility aliases, but the committed actual-host artifact proves that only +the selected `qa` directory was installed; the display count is not an +installed-skill count. + ## Candidate installer matrix The isolated standard-installer matrix passed 470/470 checks with `skills` CLI @@ -69,7 +76,7 @@ does not install the optional runtime. The committed evidence artifact is | Host | Portable | Project all-six | Global all-six | Selected-skill coverage | Installer tier | Host UI/process | |---|---|---|---|---|---|---| | Claude Code | yes | pass | pass | no separate subset case | **Verified — installer** | pending | -| OpenAI Codex | yes | pass | pass | global `qa`, `review`, `ship` pass + removal pass; opt-in alias covered | **Verified — installer** | v1/v2 failed; live v3 pending | +| OpenAI Codex | yes | pass | pass | global `qa`, `review`, `ship` pass + removal pass; actual selected `qa` runtime-absent run; opt-in alias covered | **Verified — installer**; runtime-absent invocation passed | live v1/v2/v3 failed; v3 was 3/4 | | Cursor | yes | pass | pass | project `qa`, `review`, `ship` pass + removal pass | **Verified — installer** | pending | | Pi | yes | pass | pass | no separate subset case | **Verified — installer** | pending | | OpenClaw | yes | pass | pass | project `ship` single-skill pass | **Verified — installer** | pending | @@ -83,11 +90,16 @@ Codex; neither alias nor unselected canonical skill was silently enrolled. `--copy` was advertised and used. This is filesystem/installer verification, not a claim that six host UIs loaded -or executed the skills. The Codex adversarial lane has no passing live result: -v1 and immutable v2 failed, while v3 is green only in its offline 18-test / -111-assertion harness and has not run live. The installer matrix used the -current local canonical projection through the published `npx skills` CLI, not -the still-unpushed GitHub branch URL. +or executed the skills. A separate actual Codex invocation installed only `qa` +from `time-attack/gstack/skills`, with the optional runtime absent, and passed +the judgment/setup-gate behavior without changing its workspace or creating a +runtime or browser. The Codex adversarial lane still has no passing live result: +v1 and immutable v2 failed; the v3 offline harness is green at 18 tests / 111 +assertions, but paid live v3 was a one-shot **3/4 failure** because review +failed compound inspection. It was not retried or relabeled. The installer +matrix used the current local canonical projection through the published +`npx skills` CLI. The release branch remained unpushed; native CI used the +temporary `codex/gstack-2-ci-20260717-39bc307b` ref only. Legacy generators currently know ten host names. That is historical breadth, not proof that all ten install correctly. Kiro's old Codex-rewrite behavior and @@ -125,8 +137,11 @@ bun run scripts/gstack2/test-install-matrix.ts --full \ The current matrix passed 470/470 installer CLI checks across 16 installs and two removals; its JSON artifact is committed at [`evals/installation/install-matrix.json`](../../evals/installation/install-matrix.json). -Steps 8–9, a passing live v3 adversarial run, and actual host UI loading remain -separate behavioral gates. +Steps 8–9 passed for the recorded Codex runtime-absent invocation; actual UI +loading for the other representative hosts remains separate. The paid live v3 +adversarial run is retained as a failed 3/4 gate, not a pending run. Evidence: +[`standard-codex-runtime-absent-2026-07-17.json`](../../evals/installation/standard-codex-runtime-absent-2026-07-17.json) +and [`2026-07-17T19-48-45Z-v3-live-gpt-5-4.json`](../../evals/host-adversarial/runs/2026-07-17T19-48-45Z-v3-live-gpt-5-4.json). ## Optional runtime/platform matrix @@ -147,7 +162,7 @@ activates the version, and writes stable POSIX and Windows launchers under Windows and includes the CoreDevice/iOS bundle only on Darwin. Add the bin directory to `PATH` if the short `gstack` command is desired. -Twenty-four focused installer tests pass with 336 assertions. They cover +Twenty-five focused installer tests pass with 341 assertions. They cover manifests, paths with spaces, source-root symlinks, internal-link/path-escape rejection, failed build/ validation/smoke rollback including native-load rollback smoke, @@ -166,10 +181,19 @@ nor model weights and reports the L4 capability unavailable. | Platform | Source-level target | Candidate evidence | |---|---|---| -| macOS | Node runtime + local browser + physical iOS where applicable | Runtime installer 24/336 and deterministic clean macOS arm64 bundle audit pass. The uninterrupted broad singleton run is green at 6,240 pass / 226 expected skips / 0 fail across 383 files. The signed-device gate remains pending. | -| Linux | Node runtime + local browser | Declared Dev Container GStack 2 suite passed at 136/0 with 1,127 assertions across 15 files; log: `/tmp/gstack2-devcontainer-gate-candidate-final4.log`. The one-assertion difference from the focused macOS result is the platform-conditional read-only-mode check. A clean Linux arm64 container used production-only install with the development SDK and model runtime absent, passed a local-browser journey and Sharp full-page screenshot, and uninstalled while preserving state. The candidate workflow targets GitHub-hosted `ubuntu-24.04`, but this local-only HEAD has no remote ref for that runner. Native-host broad Linux remains pending. | -| Native Windows | Node runtime; curated free tests; browser fallback where retained | **Blocked:** native Windows CI pending. The local Windows-safe singleton lane is green at 2,815 pass / 57 expected skips / 0 fail across 213 files, but it is not native evidence. The candidate workflow targets GitHub-hosted `windows-latest`; it cannot access this local-only HEAD without an explicitly authorized remote test ref. | -| Dev Container | Pure skills and optional runtime; browser only when container supports it | Declared image builds; generated freshness and the 136-test GStack 2 suite pass, as does the clean runtime install/browser smoke. Full broad Linux coverage remains pending. | +| macOS | Node runtime + local browser + physical iOS where applicable | Runtime installer 25/341 and the deterministic clean macOS arm64 bundle audit pass. The final native job passed 150/0 with 1,189 assertions. The uninterrupted broad singleton run is green at 6,255 pass / 226 expected skips / 0 fail and 25,509 assertions across 384 files. The signed-device gate is open but user-waived; no further device access is authorized. | +| Linux | Node runtime + local browser | Final native Ubuntu passed 150/0 with 1,189 assertions across 16 files. A clean Linux arm64 container also passed the production-only runtime/browser lifecycle. | +| Native Windows | Node runtime; curated free tests; browser fallback where retained | Final native Windows passed 150/0 with 1,145 assertions across 16 files and standard-installer discovery found exactly six skills. The local Windows-safe singleton lane also passed 2,829 / 57 expected skips / 0 fail with 8,648 assertions across all 214 selected files. | +| Dev Container | Pure skills and optional runtime; browser only when container supports it | The declared image built; the GStack 2 suite passed 150/0 with 1,188 assertions across 16 files, followed by the clean runtime install/browser smoke and state-preserving uninstall. | + +Native CI run [`29615621805`](https://github.com/time-attack/gstack/actions/runs/29615621805) +passed every job at commit `a8a5fa1aa381f9b948dfc57af26016092fc33277`: +macOS, Ubuntu, native Windows, the 470-check installer matrix, and the Dev +Container. The sanitized committed record is +[`evals/ci/native-2026-07-17.json`](../../evals/ci/native-2026-07-17.json). +Runs `29608904265`, `29611504979`, `29611757175`, `29612056517` (cancelled), +`29613668419`, `29614448170`, and `29614899434` are retained as superseded +diagnostic evidence, not substituted for the passing run. The six portable skills remain useful when runtime installation fails. A runtime failure must not remove or corrupt their standard-installer placement. diff --git a/docs/gstack-2/IOS-PHYSICAL-DEVICE.md b/docs/gstack-2/IOS-PHYSICAL-DEVICE.md index da7288ab0..e6091a3b2 100644 --- a/docs/gstack-2/IOS-PHYSICAL-DEVICE.md +++ b/docs/gstack-2/IOS-PHYSICAL-DEVICE.md @@ -35,14 +35,27 @@ That is a setup gate, not a DebugBridge failure. No app was installed or launched, and no pass artifact was written. The deploy skip and typed smoke failure must not be represented as a physical-device pass. -The latest user-selected target was a legacy iPhone (`iPhone10,6`) on iOS +The next user-selected target was a legacy iPhone (`iPhone10,6`) on iOS 16.7.10. Its lockdown/USB pairing validates, but `devicectl` reports `pairingState=unsupported`, no wired CoreDevice transport, and pairing fails with CoreDevice error 1011. The harness therefore returns typed code -`device_not_wired` before build or deploy. The other connected phone was not -used. Because CoreDevice is the locked backend, substituting a legacy or -third-party device driver would not satisfy this gate; a CoreDevice-compatible -iPhone is required. +`device_not_wired` before build or deploy. Because CoreDevice is the locked +backend, substituting a legacy or third-party device driver would not satisfy +this gate; a CoreDevice-compatible iPhone is required. + +The user then explicitly authorized the other connected, CoreDevice-compatible +iPhone. That run passed pairing, signing, build, install, launch, and tunnel +setup. The full five-check loop did not complete: `POST /session/acquire` +closed the socket before returning a session. At harness stop, the fixture app +was left installed with its data intact and no console/device session remained +attached. This is a retained operator observation, not a pass artifact; no pass +artifact was written. + +The user then explicitly stopped and waived further iPhone testing. No more +device access is authorized for this checkpoint. The preflight, Release guard, +typed setup-gate failures, and partial signed deployment above remain valid +evidence, but the waiver does not convert them into a P0 pass: the five-check +loop did not complete and no pass artifact exists. ## Hardware UDID versus CoreDevice UUID diff --git a/docs/gstack-2/STATUS.md b/docs/gstack-2/STATUS.md index 4070b5c5e..4a120dffe 100644 --- a/docs/gstack-2/STATUS.md +++ b/docs/gstack-2/STATUS.md @@ -4,10 +4,11 @@ The candidate contains substantial implementation, but it is not a released or verified GStack 2. `DONE` is prohibited until every P0 gate is backed by the -required evidence layer. Current blockers include a CoreDevice-compatible -signed physical iPhone, native Windows CI, a passing live v3 installed-host -adversarial run, native Linux evidence, and the signed physical-iPhone loop. No -branch push, draft PR, or PR-ready claim is authorized by this status. +required evidence layer. Current blockers include the waived physical-iPhone P0 +gate and the failed live v3 installed-host adversarial gate, including +incomplete representative host UI/process coverage. Native CI is green on +macOS, Ubuntu, Windows, and the Dev Container. No release-branch push, draft +PR, or PR-ready claim is authorized by this status. ## Implemented candidate surface @@ -27,6 +28,15 @@ branch push, draft PR, or PR-ready claim is authorized by this status. compatibility-alias cases, paths with spaces, source symlink, physical copies, and canonical hashes passed. The committed artifact is [`evals/installation/install-matrix.json`](../../evals/installation/install-matrix.json). +- [x] A standards-based Codex runtime-absent run passed. The canonical root + `--list` returned exactly six skills; the selected install used + `time-attack/gstack/skills --skill qa` and installed only byte-identical + `qa`. `skills` 1.5.19 can count hidden aliases in its pre-filter display, so + that display is not the installed subset count. Codex then preserved pure + judgment, reported the capability setup gate and one approval prompt, made + no file changes, started no browser, created no runtime, and contacted no + GStack external service. Artifact: + [`evals/installation/standard-codex-runtime-absent-2026-07-17.json`](../../evals/installation/standard-codex-runtime-absent-2026-07-17.json). - [x] Current six names/descriptions measure 982 characters (about 246 four-character token-equivalents), roughly 77.6% below the correctly parsed baseline of about 1,100 in the regenerated tree. @@ -47,14 +57,12 @@ branch push, draft PR, or PR-ready claim is authorized by this status. three retained Claude Haiku live samples are classified `REGRESSION`; they are preserved as noisy supplemental evidence, never cherry-picked as a primary gate or represented as green. -- [x] The focused macOS GStack 2 suite is green: 136 pass / 0 fail, 1,128 - assertions across 15 files. Log: `/tmp/gstack2-test-command-candidate-final2.log`. - This focused surface does not substitute for the broad or native-platform - gates. +- [x] The current macOS GStack 2 suite is green at commit `a8a5fa1a`: 150 pass + / 0 fail and 1,189 assertions across 16 files. - [x] Optional host-neutral runtime implemented with canonical paths, repo/worktree state identity, locks, atomic writes, effect claims, doctor/config/state/cleanup, migrations, upgrade/rollback, and uninstall. -- [x] Managed runtime installer coverage is green at 24 pass / 0 fail and 336 +- [x] Managed runtime installer coverage is green at 25 pass / 0 fail and 341 assertions. The deterministic clean macOS arm64 managed-bundle audit records 110 components, 1,829 files, 450,044,315 bytes, and 50 capability launchers. This is a platform-specific bundle measurement, not a universal byte count; @@ -95,77 +103,84 @@ branch push, draft PR, or PR-ready claim is authorized by this status. `signing_unavailable`, category `setup_gate`; it installed no app and wrote no pass artifact. A later attempt against the explicitly selected iPhone10,6 on iOS 16.7.10 returned `device_not_wired`; legacy USB pairing validates, but - CoreDevice pairing is unsupported. The other connected phone was not used. + CoreDevice pairing is unsupported. After the user explicitly authorized the + other connected phone, pairing/signing/build/install/launch/tunnel setup + passed, but `POST /session/acquire` closed its socket before the five-check + loop completed. At harness stop, no device session remained attached and the + fixture app was left installed. These are retained operator observations; + no pass artifact exists. +- [x] The user explicitly stopped and waived further physical-iPhone testing. + No more device access is authorized. The earlier partial/setup-gate evidence + remains retained, but it is not a physical-device P0 pass. - [x] Installed-host adversarial evidence is retained without relabeling: v1 **failed**; immutable v2 **failed** even though QA passed because the classifier produced false negatives for debug, review, and ship; v3 offline - harness coverage is green at 18 pass / 0 fail and 111 assertions. Live v3 - has not run and has no artifact, so there is no passing live installed-host - adversarial result. + harness coverage is green at 18 pass / 0 fail and 111 assertions. Paid live + v3 was a one-shot run and **failed 3/4** because review failed compound + inspection. It was not retried or relabeled. Artifact SHA-256: + `fcffdf2b0ee7bb9ac1351e246546af2cd352779bda7b1f8dc4a08f51fc66ef2f`. - [x] A current managed-bundle audit confirms no model-weight download path in setup. The standard Agent Skills installation remains Markdown-only and independent of the optional runtime. - [x] An isolated local-browser journey completed navigation, snapshot, screenshot, status, and stop cleanup. The stop acknowledgement regression has a 2 pass / 0 fail focused test. +- [x] The live aggregate SIGINT cancellation gate passed against an isolated + local-file fixture: shutdown returned an in-memory authenticated `503`, all + five owned processes exited, and no listener, root credential state, + credential-shaped file, or device session remained; an unrelated browser + daemon survived. The focused server-factory regression passed 34/0 with 62 + assertions. Artifact: + [`evals/browser/cancellation-2026-07-17.json`](../../evals/browser/cancellation-2026-07-17.json). - [x] The uninterrupted macOS broad suite is green under singleton isolation: - 6,240 pass / 226 expected skips / 0 fail and 25,449 assertions across all - 383 files. This includes the complete local-browser suite. The local - Windows-safe singleton lane is also green at 2,815 pass / 57 expected skips - / 0 fail and 8,586 assertions across all 213 curated files; it is not native - Windows evidence. + 6,255 pass / 226 expected skips / 0 fail and 25,509 assertions across all + 384 files. This includes the complete local-browser suite. The local + Windows-safe singleton lane is also green at 2,829 pass / 57 expected skips + / 0 fail and 8,648 assertions across all 214 selected files. +- [x] Native CI run + [`29615621805`](https://github.com/time-attack/gstack/actions/runs/29615621805) + passed at commit `a8a5fa1a`: macOS 150/0/1,189, Ubuntu 150/0/1,189, + Windows 150/0/1,145, Dev Container 150/0/1,188, and the standard installer + 470/470. Each native installer discovery found exactly six skills. Sanitized + artifact: + [`evals/ci/native-2026-07-17.json`](../../evals/ci/native-2026-07-17.json). - [x] Architecture, privacy, Context.dev, host compatibility, upgrade/rollback, migration, and governance documentation added. ## Blocking or incomplete P0 evidence -- [ ] Record a passing live v3 installed-host adversarial run and execute the - installed skill in each representative host UI. The six hosts are **Verified - at the installer layer only**; v1 and v2 are retained failed runs, and v3 is - offline-harness evidence only. -- [ ] Prove runtime-absent judgment behavior through an actual host invocation. - Runtime install failure and the real default capability lifecycle are covered. -- [ ] Complete native Linux and Windows matrices. The uninterrupted macOS - broad singleton run is green at 6,240/226/0 across 383 files, and the local - Windows-safe singleton lane is green at 2,815/57/0 across 213 files. Neither - substitutes for native execution on its named platform. The candidate - workflow targets GitHub-hosted `ubuntu-24.04` and `windows-latest`, but remote - execution requires explicit authorization for a CI-only ref because this - candidate is intentionally local-only before P0 passes. -- [ ] Integrate the passing complete local-browser suite and live journey into - final cancellation/leak evidence. The production dependency and managed - bundle audit contains no cloud-browser provider/path. -- [ ] Complete the physical-iPhone five-check loop with a signed test app. The - unsigned Release guard and earlier preflight passed, but the latest authorized - target cannot expose the required CoreDevice service on iOS 16.7.10. Provide - a CoreDevice-compatible iPhone without broadening the locked backend. No app - was installed and no pass artifact exists. -- [ ] Verify in one real cancellation run that no subprocess, listener, - credential, browser, or device-session remains after interruption. Existing - component tests cover each cleanup contract separately but not this aggregate - live gate. +- [ ] The paid live v3 installed-host adversarial gate is failed, not pending: + the immutable one-shot result is **3/4**, with review failing compound + inspection. Do not retry or relabel it. The six hosts remain **Verified at + the installer layer only**; representative UI/process coverage is incomplete. +- [ ] The physical-iPhone five-check loop remains incomplete and is explicitly + waived by the user. No further device access is authorized. The unsigned + Release guard and earlier preflight/setup-gate observations remain useful + partial evidence. A signed fixture was later installed and launched, but its + session-acquire request failed; no pass artifact exists and P0 remains open. - [ ] Finish final evidence-linked disposition for every infrastructure item; see the 25-row table in [ARCHITECTURE.md](./ARCHITECTURE.md). Current focused - evidence does not replace the remaining live and native-platform gates. + evidence does not replace the remaining live gates. ## Evidence index | Evidence | Path | State | |---|---|---| | Measured baseline | [BASELINE.md](./BASELINE.md) | Recorded | -| Candidate and baseline command ledger | [TEST-EVIDENCE.md](./TEST-EVIDENCE.md) | Broad macOS and local Windows-safe singleton runs green; live/native-platform gates pending | +| Candidate and baseline command ledger | [TEST-EVIDENCE.md](./TEST-EVIDENCE.md) | Runtime-absent, SIGINT, and native matrix pass; live v3 failed | +| Native CI matrix | [native-2026-07-17.json](../../evals/ci/native-2026-07-17.json) | macOS, Ubuntu, Windows, installer, and Dev Container green | | Complete skill migration | [SKILL-MIGRATION.md](./SKILL-MIGRATION.md) | Generated; 55/55 assignments | | Judgment provenance | [JUDGMENT-PROVENANCE.json](./JUDGMENT-PROVENANCE.json) | Generated; 4,681-check parity rerun green | | Parity contract | [JUDGMENT-PARITY.md](./JUDGMENT-PARITY.md) | Green for source/render/contract/asset fixtures | | Semantic parity | [SEMANTIC-PARITY.md](./SEMANTIC-PARITY.md) | Deterministic 295-check corpus green; retained live samples are regressions | -| Installed-host adversarial | [eval overview](../../evals/host-adversarial/README.md), [immutable v2 artifact](../../evals/host-adversarial/runs/2026-07-17T04-09-01-809Z-3d23a270.json) | V1 failed; immutable V2 failed; V3 offline 18/111 green, live V3 not run; no passing live run | +| Installed-host adversarial | [eval overview](../../evals/host-adversarial/README.md), [immutable live v3 artifact](../../evals/host-adversarial/runs/2026-07-17T19-48-45Z-v3-live-gpt-5-4.json) | V1/V2 failed; V3 offline 18/111 green; paid live V3 one-shot failed 3/4 (review) | | Structured scenarios | [SCENARIOS.md](./SCENARIOS.md) | 25/25 structured routing fixtures green | | Backlog traceability | [BACKLOG-MAP.json](./BACKLOG-MAP.json) | 755 unique items mapped | | Context integration | [CONTEXT-DEV.md](./CONTEXT-DEV.md) | Automated contract 22/139 green; verified-key official-endpoint live smoke passed | -| Host matrix | [HOST-COMPATIBILITY.md](./HOST-COMPATIBILITY.md) | 470/470 checks; six hosts Verified at installer layer; live v3/UI launch pending | +| Host matrix | [HOST-COMPATIBILITY.md](./HOST-COMPATIBILITY.md) | 470/470 checks; Codex runtime-absent run passed; live v3 failed; other UI launches pending | | Privacy boundary | [PRIVACY.md](./PRIVACY.md) | Implemented contract; full retained-tool egress audit pending | -| Physical iOS | [IOS-PHYSICAL-DEVICE.md](./IOS-PHYSICAL-DEVICE.md) | Earlier preflight 9 pass / 1 deploy skip / 29 assertions; latest selected target lacks a supported CoreDevice service; no app/pass artifact | -| Upgrade/recovery | [UPGRADE-AND-ROLLBACK.md](./UPGRADE-AND-ROLLBACK.md) | Runtime installer 24 pass / 336 assertions; deterministic clean macOS arm64 bundle audit recorded | +| Physical iOS | [IOS-PHYSICAL-DEVICE.md](./IOS-PHYSICAL-DEVICE.md) | Partial setup evidence only; user stopped/waived further device testing; no P0 pass artifact | +| Upgrade/recovery | [UPGRADE-AND-ROLLBACK.md](./UPGRADE-AND-ROLLBACK.md) | Runtime installer 25 pass / 341 assertions; deterministic clean macOS arm64 bundle audit recorded | ## Interpretation rules diff --git a/docs/gstack-2/TEST-EVIDENCE.md b/docs/gstack-2/TEST-EVIDENCE.md index bb0c4d272..de27cf050 100644 --- a/docs/gstack-2/TEST-EVIDENCE.md +++ b/docs/gstack-2/TEST-EVIDENCE.md @@ -9,7 +9,8 @@ pass from deterministic, offline, or filesystem-only evidence. | Command / probe | Observed result | What it proves / does not prove | |---|---|---| -| Focused macOS `bun test test/gstack2-*.test.ts` candidate run | **Exit 0: 136 pass / 0 fail**, 1,128 assertions across 15 files. Log: `/tmp/gstack2-test-command-candidate-final2.log`. | The focused GStack 2 routing/runtime/privacy/installer/upgrade and deterministic parity surface is green. It is not the broad repository or native-platform gate. | +| Earlier focused macOS `bun test test/gstack2-*.test.ts` candidate run | **Exit 0: 136 pass / 0 fail**, 1,128 assertions across 15 files. Log: `/tmp/gstack2-test-command-candidate-final2.log`. | Historical focused checkpoint retained rather than overwritten. | +| Current macOS `bun run test:gstack2`, commit `a8a5fa1a` | **Exit 0: 150 pass / 0 fail**, 1,189 assertions across 16 files. | Current generated freshness, routing, runtime, privacy, installer, upgrade, parity, and adversarial-harness surface is green. Native and broad evidence are recorded separately. | | `bun test --timeout 30000 test/gstack2-skills.test.ts test/gstack2-skills-routing.test.ts` after regeneration | **Exit 0: 3 pass / 0 fail**, 81 assertions. | The pinned corpus/parity test and both 25-scenario structured-routing tests are green. This remains structural/fixture evidence, not specialist live execution. | | `bun run scripts/gstack2/run-parity.ts`, 2026-07-17 rerun | **Exit 0: 4,681 checks passed**; 55 modules, 16 sections, 25 scenarios, 16 regressions, 78 assets. | Current source/render/provenance/contract/asset/fixture parity is green. It is deterministic parity, not live-host behavior. | | Earlier regenerated structural parity checkpoint | **Exit 0: 2,403 checks passed** with the then-current 55/16/25/16/45 inventory. | Historical candidate checkpoint before later thin-prelude and asset coverage; superseded by the current 4,681-check rerun. | @@ -18,26 +19,32 @@ pass from deterministic, offline, or filesystem-only evidence. | Retained Claude Haiku live semantic transcripts | **Three retained samples, all classified `REGRESSION`.** Two samples used now-obsolete visible-wrapper prompts. The current office-hours sample also penalizes independently sampled omissions even though those details remain byte-preserved in the candidate source. | Live-model sampling is supplemental and currently not green. These results must not be retried until favorable, cherry-picked, or used to overrule deterministic source loss. See [SEMANTIC-PARITY.md](./SEMANTIC-PARITY.md). | | Installed-host adversarial v1 | **FAILED.** The immutable v1 slash-invocation artifact is retained. | Unfavorable activation/classifier evidence; not a behavioral pass. See [eval overview](../../evals/host-adversarial/README.md). | | Installed-host adversarial v2 | **FAILED.** QA passed, while debug, review, and ship were false negatives from the v2 read-only-Git warning classifier. | All four dispatchers activated, but the top-level run failed and must stay failed. Immutable artifact: [`2026-07-17T04-09-01-809Z-3d23a270.json`](../../evals/host-adversarial/runs/2026-07-17T04-09-01-809Z-3d23a270.json), SHA-256 `7ab15ea575cb9a634b7d00212dd9d74902b1188281ae6a503a32ccf382facbf5`. | -| Installed-host adversarial v3 offline harness | **18 pass / 0 fail**, 111 assertions. | The classifier fix and offline harness are covered. Live v3 has **not run** and has no artifact; there is no passing live installed-host adversarial run. | +| Installed-host adversarial v3 offline harness | **18 pass / 0 fail**, 111 assertions. | The classifier fix and offline harness are covered. This deterministic result does not override the failed paid live run. | +| Installed-host adversarial v3 paid live run | **FAILED 3/4.** Debug, QA, and ship passed; review failed compound inspection. This was the one-shot run, with zero retries. | Unfavorable evidence is immutable and must not be retried or relabeled. Artifact: [`2026-07-17T19-48-45Z-v3-live-gpt-5-4.json`](../../evals/host-adversarial/runs/2026-07-17T19-48-45Z-v3-live-gpt-5-4.json), SHA-256 `fcffdf2b0ee7bb9ac1351e246546af2cd352779bda7b1f8dc4a08f51fc66ef2f`. | | Six-skill catalog measurement after regeneration | Six names/descriptions total 982 characters, about 246 four-character token-equivalents; baseline correctly parsed catalog was about 1,100. | 77.6% reduction, above the 75% gate. Re-measure if frontmatter changes. The buggy 4,214 baseline estimate is not used. | | `bun test ios-qa/daemon/test` | **95 pass / 0 fail**, 229 assertions. | Covers daemon regressions including malformed device JSON, hardware-UDID/CoreDevice selection, bounded proxy timeout, and expected-bundle mutation header. It is not a signed-app live pass. | | Focused DebugBridge/template build tests | **33 pass / 0 fail** in the candidate run, including Swift debug compilation/XCTest and Release symbol absence. | Static/build evidence for debug-only bridge wiring; still not an installed physical-app journey. | | Physical-iOS E2E preflight | **9 pass / 0 fail / 1 deploy check skipped**, 29 assertions. | The earlier connected-device/setup and unsigned Release checks pass. The deploy skip leaves the signed-device P0 gate open. | -| Direct physical-device smoke | **Typed setup-gate failures, no product failure:** the earlier target returned `signing_unavailable`; the latest explicitly selected target returned `device_not_wired`, and CoreDevice pairing returned error 1011. | The latest target is an iPhone10,6 on iOS 16.7.10: legacy USB pairing validates, but Xcode reports CoreDevice pairing unsupported and exposes no wired CoreDevice service. The other connected phone was not used. No app was installed or launched and no pass artifact was written. | +| Direct physical-device smoke | **Incomplete, not a pass:** one target returned `signing_unavailable`; the legacy iPhone10,6 target returned `device_not_wired` / CoreDevice error 1011; the subsequently authorized CoreDevice-compatible phone passed pairing, signing, build, install, launch, and tunnel setup but `POST /session/acquire` closed its socket before the five-check loop completed. | The harness process was stopped at the user's request. At harness stop, the signed fixture was left installed with data intact and no device session remained attached. These are retained operator observations; no pass artifact was written. Further iPhone testing is waived and unauthorized for this checkpoint. | | Context.dev contract (`gstack2-runtime-context.test.ts`) | **22 pass / 0 fail**, 139 assertions. | Persists explicit host/local-browser/none choices without consent, rejects private/credential URLs and request material plus private DNS, proves zero lookup/fetch before mode+consent, validates documented endpoint paths and exact failure taxonomy, and makes search typed unsupported without network. | | `gstack context smoke --url https://www.context.dev --json` | **PASS:** the official `/web/scrape/markdown` endpoint returned `ok: true` and credit metadata. | The verified key entered through protected stdin/environment worked. The temporary secrets file was mode `0600`, the isolated home was removed, and the safe output did not contain the key. Committed redacted artifact: [`evals/context-dev/live-smoke-2026-07-17.json`](../../evals/context-dev/live-smoke-2026-07-17.json). | | Standard installer matrix | **PASS: 470/470 checks**, 16 install cases, two removal cases, `skills` CLI 1.5.19. | Project/global installs pass for six hosts, selected-skill and opt-in compatibility-alias cases, copies, and hashes. This remains installer/filesystem evidence. Committed artifact: [`evals/installation/install-matrix.json`](../../evals/installation/install-matrix.json). | -| `bun test test/gstack2-runtime-install.test.ts` | **Exit 0: 24 pass / 0 fail**, 336 assertions. | Managed allowlist, hashes, spaces, source/internal-link rejection, production-only frozen dependencies, deterministic exact Sharp/ngrok platform closure, native-load rollback smoke, rollback/recovery, stable launchers, wrapper neutrality, and state-preserving uninstall pass. | +| Standard Agent Skills install + actual runtime-absent Codex invocation | **PASS.** Root `--list` returned exactly six. The selected source was `time-attack/gstack/skills --skill qa`; despite `skills` 1.5.19's pre-filter display counting hidden aliases, exactly one skill (`qa`) installed byte-identically. Codex exited 0 with `gstack` absent, reported `NEEDS_SETUP` and one approval prompt, changed no files, and created no runtime/browser/external-service activity. | Closes the actual-host runtime-absent judgment gate for Codex without overstating six-host UI coverage. Artifact: [`standard-codex-runtime-absent-2026-07-17.json`](../../evals/installation/standard-codex-runtime-absent-2026-07-17.json). | +| `bun test test/gstack2-runtime-install.test.ts` | **Exit 0: 25 pass / 0 fail**, 341 assertions. | Managed allowlist, hashes, spaces, source/internal-link rejection, production-only frozen dependencies, deterministic exact Sharp/ngrok platform closure, native-load rollback smoke, rollback/recovery, stable launchers, wrapper neutrality, and state-preserving uninstall pass. | | Deterministic clean macOS arm64 managed runtime bundle audit | **110 components, 1,829 files, 450,044,315 bytes, 50 capability launchers.** | This is a platform-specific bundle measurement, not a universal byte count; platform-native package payloads differ. Setup includes the Sharp/ngrok closure and excludes the development-only Claude Agent SDK. The Hugging Face sidecar is outside the bundle and its package is development-only, so production setup installs neither its inference runtime nor model weights; the L4 capability reports unavailable. The standard skill installer remains Markdown-only. Committed artifact: [`evals/runtime-bundle/darwin-arm64.json`](../../evals/runtime-bundle/darwin-arm64.json); reproduce with `bun run scripts/gstack2/audit-runtime-bundle.ts --output evals/runtime-bundle/darwin-arm64.json`. | -| Declared Linux Dev Container plus `bun run test:gstack2` inside it | **Exit 0: 136 pass / 0 fail, 1,127 assertions across 15 files.** Log: `/tmp/gstack2-devcontainer-gate-candidate-final4.log`. | Generated freshness, canonical-skill routing, parity, state, privacy, installer, recovery, and runtime behavior are green in the declared Linux container. The one-assertion difference from macOS is the platform-conditional read-only-mode check. This is not a native-host broad Linux run. | +| Earlier declared Linux Dev Container plus `bun run test:gstack2` inside it | **Exit 0: 136 pass / 0 fail, 1,127 assertions across 15 files.** Log: `/tmp/gstack2-devcontainer-gate-candidate-final4.log`. | Historical container checkpoint retained rather than overwritten; the current 150-test container result is recorded in the native-CI row below. | | `scripts/gstack2/runtime-install-smoke.sh` in the clean Linux arm64 container | **Pass:** production-only frozen dependencies installed with the development Agent SDK and Hugging Face/ONNX runtime absent; the managed Anthropic SDK, Sharp, and ngrok imports passed; prebuilt capabilities rebuilt; setup/doctor/version/design/PDF passed; a local-browser journey and Sharp full-page screenshot passed; uninstall preserved state. | Proves a source copy with spaces can build and complete the managed runtime lifecycle without Git history, an executable local-model stack, or Darwin-only iOS artifacts. It is not native Windows evidence. | -| Runtime lifecycle and external-effect matrix | **Pass:** real filesystem/subprocess tests cover clean install/uninstall, paths with spaces, source symlinks and internal-link rejection, macOS read-only reporting, interrupted-pointer rollback, crash-journal repair, last-known-good launcher recovery, and an actual local Git push that executes at most once across resume. | Closes the named local filesystem/recovery/idempotency gates. The aggregate live cancellation/leak gate remains open. | +| Runtime lifecycle and external-effect matrix | **Pass:** real filesystem/subprocess tests cover clean install/uninstall, paths with spaces, source symlinks and internal-link rejection, macOS read-only reporting, interrupted-pointer rollback, crash-journal repair, last-known-good launcher recovery, and an actual local Git push that executes at most once across resume. | Closes the named local filesystem/recovery/idempotency gates. Aggregate cancellation evidence is recorded separately below. | | `bun test design/test` | **101 pass / 0 fail**, 381 assertions. | The retained design suite is green and is also included in the later uninterrupted broad pass. | | `bun test make-pdf/test` plus combined-fixture render | **189 pass / 0 fail, 398 assertions;** four PDF pages rendered to PNG and visually inspected with no detected layout defect. | Retained strict PDF tests and a live internal render are green on this macOS host. Cross-platform visual equivalence remains a separate platform claim. | | Diagram suites | **51 pass / 0 fail / 1 skip**, 120 assertions; the opt-in paid lane recorded two skips. | Offline Mermaid/SVG/PNG/Excalidraw coverage is green. Skipped paid-provider cases are not live evidence. | | Isolated local-browser journey plus stop regression | Navigation, snapshot, screenshot, and status passed; stop returned success and left no observed process leak. `browse/test/stop-ack-before-shutdown.test.ts` is **2 pass / 0 fail**. | Provides one real local Chromium journey and a focused regression for acknowledging stop/restart before delayed shutdown. The complete browser suite is included in the later broad pass. | -| `bun test` through the strict singleton runner | **Exit 0: 6,240 pass / 226 expected skips / 0 fail**, 25,449 assertions; 383/383 shard headers and terminal single-file summaries. Log: `/tmp/gstack2-full-singleton-final-committed.log`. | One uninterrupted macOS broad pass after the browser lifecycle and retry-harness fixes. Expected skips are provider, credential, Poppler-environment, paid, or model-sidecar gates declared by their tests; external/live evidence remains separate. | -| Local `bun run test:windows` through singleton shards | **Exit 0: 2,815 pass / 57 expected skips / 0 fail**, 8,586 assertions; 213/213 shard headers and terminal single-file summaries. Log: `/tmp/gstack2-windows-singleton-final-committed.log`. | The curated Windows-safe subset is locally green under singleton isolation. This is not native Windows execution; native Windows CI remains blocked. | +| Live local-browser SIGINT aggregate | **PASS.** An isolated local-file journey completed `goto`, `status`, and `snapshot`; SIGINT produced an in-memory authenticated `503`, reduced five owned processes to zero, and left no listener, root credential state, credential-shaped file, or device session. The unrelated browser daemon survived. Focused regression: 34 pass / 0 fail, 62 assertions. | Closes the requested aggregate process/listener/state/credential cleanup gate without claiming device interaction. Artifact: [`evals/browser/cancellation-2026-07-17.json`](../../evals/browser/cancellation-2026-07-17.json). | +| Native CI run `29615621805`, commit `a8a5fa1a` | **PASS:** installer 470/470; macOS 150/0/1,189; Ubuntu 150/0/1,189; native Windows 150/0/1,145; Dev Container 150/0/1,188. All native installer-discovery steps found exactly six skills. | Complete native matrix evidence. Sanitized artifact: [`evals/ci/native-2026-07-17.json`](../../evals/ci/native-2026-07-17.json). Runs `29608904265`, `29611504979`, `29611757175`, `29612056517` (cancelled), `29613668419`, `29614448170`, and `29614899434` remain superseded diagnostics. | +| Earlier `bun test` strict singleton checkpoint | **Exit 0: 6,240 pass / 226 expected skips / 0 fail**, 25,449 assertions across 383 files. Log: `/tmp/gstack2-full-singleton-final-committed.log`. | Historical broad checkpoint retained rather than overwritten. | +| Current `bun test` through the strict singleton runner | **Exit 0: 6,255 pass / 226 expected skips / 0 fail**, 25,509 assertions; 384/384 shard headers and terminal single-file summaries. | One uninterrupted current-head macOS broad pass. Expected skips are provider, credential, Poppler-environment, paid, or model-sidecar gates declared by their tests; external/live evidence remains separate. | +| Earlier local `bun run test:windows` singleton checkpoint | **Exit 0: 2,815 pass / 57 expected skips / 0 fail**, 8,586 assertions across 213 files. Log: `/tmp/gstack2-windows-singleton-final-committed.log`. | Historical curated-lane checkpoint retained rather than overwritten. | +| Current local `bun run test:windows` through singleton shards | **Exit 0: 2,829 pass / 57 expected skips / 0 fail**, 8,648 assertions; all 214 selected files completed. | The current curated Windows-safe subset is locally green. Native Windows is independently green in run `29615621805`. | | Forbidden production-scope audit | Production dependencies and the 110-component managed bundle contain no cloud-browser provider, Hugging Face/ONNX inference runtime, model weights, or alternative physical-iOS backend. Transformers remains development-only for retained tests; CoreDevice/`devicectl` is the sole physical-iOS path. | Deterministic dependency/bundle/backend evidence. It does not substitute for native-platform, signed-device, or live-provider behavior. | ### Required final command ledger @@ -57,13 +64,13 @@ bun run test:windows bun test design/test bun test ios-qa/daemon/test host UI/process launch for six installer-verified hosts -passing live v3 installed-host adversarial run -runtime-absent judgment through an actual host invocation +passing live installed-host adversarial gate (live v3 one-shot is retained failed evidence) +runtime-absent judgment through an actual host invocation (Codex passed; broader host UI coverage remains) macOS + Linux + native Windows + Dev Container -local browser live journey + cancellation/leak cleanup +local browser live journey + cancellation/leak cleanup (passed) PDF strict + diagram suites Context.dev verified-key public-page smoke -physical signed-iPhone five-check loop + Release symbol check +physical signed-iPhone five-check loop + Release symbol check (waived; partial evidence is not a pass) upgrade/fail/recover/rollback/uninstall end-to-end ``` diff --git a/docs/gstack-2/UPGRADE-AND-ROLLBACK.md b/docs/gstack-2/UPGRADE-AND-ROLLBACK.md index 63bd928e0..1076de59f 100644 --- a/docs/gstack-2/UPGRADE-AND-ROLLBACK.md +++ b/docs/gstack-2/UPGRADE-AND-ROLLBACK.md @@ -69,7 +69,7 @@ written under `$GSTACK_HOME/bin` (default `~/.gstack/bin`), alongside `runtime-install.json`, which distinguishes managed paths from preserved config, secrets, projects, and plans. -Twenty-four focused installer tests pass with 336 assertions. They cover paths +Twenty-five focused installer tests pass with 341 assertions. They cover paths with spaces, a symlinked source root, internal-link rejection, runtime-only builder selection, deterministic exact Sharp/ngrok platform closure, failed build/validation/smoke rollback including native-load rollback smoke, @@ -90,9 +90,9 @@ with spaces, installed only frozen production dependencies with the development SDK absent, rebuilt runtime capabilities without Git history or skill regeneration, passed setup/doctor/version/design/PDF checks, completed a local browser journey and Sharp full-page screenshot, and uninstalled while -preserving state. Non-Darwin bundles omit the physical-iOS capability. Native -Windows execution remains a separate gate; source-level Windows launchers are -not native evidence. +preserving state. Non-Darwin bundles omit the physical-iOS capability. The +native Windows GStack 2 gate later passed 150/0 with 1,145 assertions in run +`29615621805`; source-level Windows launchers alone were not used as evidence. ## Runtime upgrade transaction @@ -193,11 +193,10 @@ newer-schema refusal, and non-mutating cleanup preview. The standard installer matrix separately passed 470/470 checks across six hosts, 16 installs, scopes, selections, and two removals; its artifact is [`evals/installation/install-matrix.json`](../../evals/installation/install-matrix.json). -The runtime installer is green at 24 pass / 0 fail and 336 assertions, and the +The runtime installer is green at 25 pass / 0 fail and 341 assertions, and the deterministic clean macOS arm64 110-component bundle audit is recorded. Its byte count is platform-specific because platform-native package payloads differ. The clean Linux Dev Container install/uninstall smoke passed. -Interrupted network acquisition/stage at OS level, a passing live v3 host run, -actual host UI execution, native-host Linux, and native Windows runs remain -gates. See +Interrupted network acquisition/stage at OS level, a passing installed-host +adversarial gate, and representative host UI execution remain gates. See [TEST-EVIDENCE.md](./TEST-EVIDENCE.md). diff --git a/evals/ci/native-2026-07-17.json b/evals/ci/native-2026-07-17.json new file mode 100644 index 000000000..66cd1d012 --- /dev/null +++ b/evals/ci/native-2026-07-17.json @@ -0,0 +1,129 @@ +{ + "schemaVersion": 1, + "capturedAt": "2026-07-17T21:46:03Z", + "repository": "time-attack/gstack", + "workflow": ".github/workflows/gstack2-gate.yml", + "temporaryRef": "codex/gstack-2-ci-20260717-39bc307b", + "releaseBranchPushed": false, + "finalRun": { + "id": 29615621805, + "url": "https://github.com/time-attack/gstack/actions/runs/29615621805", + "event": "workflow_dispatch", + "headSha": "a8a5fa1aa381f9b948dfc57af26016092fc33277", + "status": "completed", + "conclusion": "success", + "createdAt": "2026-07-17T21:42:54Z", + "completedAt": "2026-07-17T21:46:03Z", + "jobs": [ + { + "name": "installer-matrix", + "conclusion": "success", + "startedAt": "2026-07-17T21:42:59Z", + "completedAt": "2026-07-17T21:43:42Z", + "skillsCliVersion": "1.5.19", + "checks": 470, + "checksPassed": 470, + "installCases": 16, + "removalCases": 2 + }, + { + "name": "Native ubuntu-24.04", + "conclusion": "success", + "startedAt": "2026-07-17T21:42:59Z", + "completedAt": "2026-07-17T21:43:44Z", + "tests": 150, + "passed": 150, + "failed": 0, + "assertions": 1189, + "files": 16, + "testDurationSeconds": 14.13, + "standardInstallerDiscoveredSkills": 6 + }, + { + "name": "Native macos-14", + "conclusion": "success", + "startedAt": "2026-07-17T21:43:00Z", + "completedAt": "2026-07-17T21:43:47Z", + "tests": 150, + "passed": 150, + "failed": 0, + "assertions": 1189, + "files": 16, + "testDurationSeconds": 13.98, + "standardInstallerDiscoveredSkills": 6 + }, + { + "name": "dev-container", + "conclusion": "success", + "startedAt": "2026-07-17T21:42:59Z", + "completedAt": "2026-07-17T21:45:09Z", + "tests": 150, + "passed": 150, + "failed": 0, + "assertions": 1188, + "files": 16, + "testDurationSeconds": 19.49, + "cleanRuntimeInstallBrowserSmokeAndUninstall": "success" + }, + { + "name": "Native windows-latest", + "conclusion": "success", + "startedAt": "2026-07-17T21:43:01Z", + "completedAt": "2026-07-17T21:46:02Z", + "tests": 150, + "passed": 150, + "failed": 0, + "assertions": 1145, + "files": 16, + "testDurationSeconds": 52.27, + "standardInstallerDiscoveredSkills": 6 + } + ] + }, + "historicalDiagnosticRuns": [ + { + "id": 29608904265, + "conclusion": "failure", + "disposition": "superseded diagnostic", + "finding": "Ignored runtime payloads, Windows path assumptions, and Git safe.directory setup were incomplete." + }, + { + "id": 29611504979, + "conclusion": "failure", + "disposition": "superseded diagnostic", + "finding": "Windows exposed CRLF drift in generated Swift assets." + }, + { + "id": 29611757175, + "conclusion": "failure", + "disposition": "superseded diagnostic", + "finding": "Ubuntu exposed a lock-stress timeout and Windows exposed platform-dependent manifests." + }, + { + "id": 29612056517, + "conclusion": "cancelled", + "disposition": "superseded diagnostic", + "finding": "The Windows job hung in Context.dev DNS timeout coverage; the other matrix jobs were green." + }, + { + "id": 29613668419, + "conclusion": "failure", + "disposition": "superseded diagnostic", + "finding": "Windows completed after timer repair and exposed 21 filesystem, mode-validation, and launcher-harness failures." + }, + { + "id": 29614448170, + "conclusion": "failure", + "disposition": "superseded diagnostic", + "finding": "Windows passed 144 tests and exposed six test-only extensionless-launcher assumptions." + }, + { + "id": 29614899434, + "url": "https://github.com/time-attack/gstack/actions/runs/29614899434", + "headSha": "6d3eb795e83e98c2a5c607298a5adaec20ec4ded", + "conclusion": "failure", + "disposition": "superseded diagnostic", + "finding": "Windows passed 141 tests; eight integration tests exceeded Bun's 5-second default and one fixture corrupted the batch launcher it then executed." + } + ] +}