mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 22:45:02 +01:00
fix(scorecard): --now bounds every windowed line; recommended-harness week is 83 PRs (GST-135)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
959c8a6a4f
commit
40fe5c008e
7 files changed
+119
-43
No files matched your search
@@ -1,5 +1,17 @@
|
||||
# Changelog
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
**`gstack-factory-scorecard --now` now ends the window, so a past week reports that week's numbers.** Every line counts only what happened in `[now - since-days, now)`. Before, `--now` moved only the start, so `--now 2026-10-15T00:00:00Z --since-days 14` counted 290 merged PRs (every merge from 2026-10-01 through the day of the run) where the window holds 65, and the gate-run and release lines also counted through the day of the run. Rerun with the fix, the two weeks before 2026-10-15 show 65 merged PRs and the two weeks before 2026-10-29 show 197, matching GitHub search for both windows.
|
||||
|
||||
### Fixed
|
||||
- **`lib/factory-scorecard.ts`**: one window (`scoreWindow`, `inWindow`) bounds merged PRs and their lead times, gate runs and their cost, the red-train rate, flaky events, free-suite retries, issues and releases. The workflow-runs query asks GitHub for the window's days (`created=<start>..<end>`), and the issue searches use a closed time range instead of `>=start`.
|
||||
- **`scripts/lib/ci-history.ts`**: `listWorkflowRuns` takes an optional `until` (exclusive) and `createdFilter` builds the runs API's `created` query for both callers.
|
||||
- **`docs/recommended-harness.md`**: "Where gstack's own numbers stand" is remeasured for the exact week of 2026-10-19 through 2026-10-25 UTC: 83 merged PRs (not 103), median lead time 6.5h, 54 gate runs (52 green, 2 red), $1,361.94 of known gate cost, 0 releases. The earlier figures counted through midday on 2026-10-26.
|
||||
|
||||
### Tests
|
||||
- `test/factory-scorecard.test.ts`: a PR merged exactly at `--now`, and a gate run, free run and release after it, are left out; the runs and issue queries carry the window's end.
|
||||
|
||||
## [1.91.327.0] - 2026-10-30
|
||||
|
||||
**A new page shows how gstack and Capy run as a software factory, with measured numbers.** `capy/README.md` explains how Capy Captains, their crews and gstack's skills fit together, from the plan-first approval loop and `/ship` before READY to merge trains, fix waves, the daily flaky pass and peer Captains. It compares the 14 days after 2026-10-15 with the 14 days before: 197 PRs merged to gstack's main against 65, and 194 versions shipped against 62. It also says what those numbers leave out. The median PR shrank from 1,292 changed lines to 422, and the median time from opening a PR to merging it grew from 1.1 hours to 5.6, because trains batch merges. A footnote gives the windows and the method behind every figure, and the page ends with tips and how Garry uses the setup day to day.
|
||||
|
||||
@@ -15,6 +15,8 @@
|
||||
* [--out-dir <dir>] [--csv <file>] [--json] [--cache-dir <dir>]
|
||||
* [--fixtures <recorded gh responses.json>] [--now <iso>]
|
||||
*
|
||||
* The window is [now - since-days, now): --now (default: the current time) is
|
||||
* its end, and every line counts only items inside it.
|
||||
* --repo defaults to the GitHub owner/name of the origin remote; without one,
|
||||
* pass it. Every run writes this window's scorecard.{csv,md} and appends its
|
||||
* rows to trend.csv in --out-dir, so the weeks line up.
|
||||
|
||||
+14
-13
@@ -284,22 +284,23 @@ the gap next to the number.
|
||||
|
||||
### Where gstack's own numbers stand
|
||||
|
||||
Measured on gstack's repository with this tool for 2026-10-19 to 2026-10-26
|
||||
(the first full week of trains and `/ship`-before-READY; 103 merged PRs, 77
|
||||
gate runs, artifacts read for the newest 60):
|
||||
Measured on gstack's repository with this tool for the week of 2026-10-19
|
||||
through 2026-10-25, UTC (`--now 2026-10-26T00:00:00Z --since-days 7`; the first
|
||||
full week of trains and `/ship`-before-READY; 83 merged PRs, 54 gate runs,
|
||||
artifacts read for all 54):
|
||||
|
||||
| Line | gstack, that week | Against the bar |
|
||||
|---|---|---|
|
||||
| Merged PRs | 103 (crew 100, outside 3) | volume is there; outside contributions are few and are ported by the crew rather than merged from forks |
|
||||
| Lead time, created → merged | median 7.9h, p90 23.7h | inside the bar on both |
|
||||
| Lead time, READY → merged | no number: 0 of 103 PRs had a READY comment | the crew reports READY in-thread; step 6 above (post it on the PR) is what makes this measurable |
|
||||
| Review passes per PR | no number: 0 of 103 PRs carried a pass count | the receipt's `passes` field lands with /ship's convergence rule; until then crews record the count in the READY block, and the marker reads it from a PR comment |
|
||||
| Gate runs | 73 green / 4 red / 0 cancelled; known cost $1,628 on 59 of 77 runs | green rate 95%; cost unknown on 18 runs (screens that planned no cases, and runs older than the artifact cap) |
|
||||
| Known gate cost per merged PR | $15.81 (a floor) | about $24 per train of up to 5 PRs, plus screens; one gate per PR would have been about $33 each |
|
||||
| Red-train rate | 4% (1 red of 28 trains; 0 red backstop, 3 red screens) | inside the bar; the three red screens were one judge reading a moved file |
|
||||
| Flaky events | 21 paid cases passed on their one rerun (rerun spend $10.72); 17 free-suite retries in 17 of 103 runs | above the bar on both: 0.27 flaky cases per gate against 0.1, and 17% of free runs retried against 1%. One family (`ship-docsync-*`) repeated three times in two days |
|
||||
| Issues opened / closed | 50 / 56 | inside the bar |
|
||||
| Releases cut | 1 (the first weekly release) | on schedule, with the starvation caveat above |
|
||||
| Merged PRs | 83 (crew 80, outside 3) | volume is there; the 3 outside PRs are bot PRs (2 from github-actions, 1 from dependabot), and outside contributions are ported by the crew rather than merged from forks |
|
||||
| Lead time, created to merged | median 6.5h, p90 20.0h | inside the bar on both |
|
||||
| Lead time, READY to merged | no number: 0 of 83 PRs had a READY comment | the crew reports READY in-thread; step 6 above (post it on the PR) is what makes this measurable |
|
||||
| Review passes per PR | no number: 0 of 83 PRs carried a pass count | the receipt's `passes` field lands with /ship's convergence rule; until then crews record the count in the READY block, and the marker reads it from a PR comment |
|
||||
| Gate runs | 52 green / 2 red / 0 cancelled; known cost $1,361.94 on 52 of 54 runs | green rate 96%; cost unknown only on the 2 red runs (the red train and the red screen) |
|
||||
| Known gate cost per merged PR | $16.41 (a floor) | trains averaged $32.48 each for up to 5 PRs (21 trains with a known cost), screens $21.18 each (29 with a known cost) |
|
||||
| Red-train rate | 5% (1 red of 22 trains; 0 red backstop, 1 red screen) | 1 of 22 is 4.5%, just under the bar |
|
||||
| Flaky events | 17 paid cases passed on their one rerun (rerun spend $7.20); 17 free-suite retries in 17 of 83 runs | above the bar on both: 0.31 flaky cases per gate against 0.1, and 20% of free runs retried against 1%. One case (`ship-docsync-current`) did it four times in three days |
|
||||
| Issues opened / closed | 28 / 51 | inside the bar |
|
||||
| Releases cut | 0 | none inside the week: the first weekly release, v1.91.242.0, was published at 12:01 UTC on 2026-10-26, just after the window closed (see the starvation caveat above) |
|
||||
|
||||
Read the gap honestly: lead time and throughput are where they should be, the
|
||||
gate is green and its cost is mostly known, and flakiness is the line that is
|
||||
|
||||
+41
-20
@@ -5,6 +5,10 @@
|
||||
* counting rule next to the number, and a line whose input cannot be read says
|
||||
* so ("unavailable: <reason>") instead of printing 0.
|
||||
*
|
||||
* The window is [now - since-days, now): every input below counts an item only
|
||||
* when its timestamp falls inside it (inWindow), so a past window reproduces
|
||||
* after the fact.
|
||||
*
|
||||
* Inputs, all REST reads (no write, no paid work):
|
||||
* merged PRs pulls?state=closed&base=<base>, merged_at inside the window;
|
||||
* crew = --crew logins, else author_association OWNER/MEMBER/COLLABORATOR
|
||||
@@ -29,7 +33,7 @@ import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import {
|
||||
GH_CLIENT, downloadRunArtifacts, filesUnder, historyFetcher, listWorkflowRuns, readFlakeLedgerDir, unavailableReason,
|
||||
GH_CLIENT, createdFilter, downloadRunArtifacts, filesUnder, historyFetcher, listWorkflowRuns, readFlakeLedgerDir, unavailableReason,
|
||||
type GhClient,
|
||||
} from '../scripts/lib/ci-history';
|
||||
|
||||
@@ -92,6 +96,24 @@ export interface Scorecard {
|
||||
}
|
||||
export interface ScorecardRow { id: string; metric: string; value: string; benchmark: string; rule: string }
|
||||
|
||||
/** The scorecard's one window, [start, end) as ISO instants; every windowed read filters through inWindow. */
|
||||
export interface ScoreWindow { start: string; end: string }
|
||||
|
||||
export function scoreWindow(o: Pick<ScorecardOptions, 'now' | 'sinceDays'>): ScoreWindow {
|
||||
return { start: new Date(o.now - o.sinceDays * 86_400_000).toISOString(), end: new Date(o.now).toISOString() };
|
||||
}
|
||||
|
||||
export function inWindow(at: unknown, w: ScoreWindow): boolean {
|
||||
const t = typeof at === 'string' ? Date.parse(at) : NaN;
|
||||
return t >= Date.parse(w.start) && t < Date.parse(w.end);
|
||||
}
|
||||
|
||||
/** GitHub search's inclusive, whole-second range for the window: start rounded up, end one second before the next whole second at or after it. */
|
||||
export function searchRange(w: ScoreWindow): string {
|
||||
const second = (ms: number) => new Date(Math.ceil(ms / 1000) * 1000).toISOString().replace(/\.\d{3}Z$/, 'Z');
|
||||
return `${second(Date.parse(w.start))}..${second(Math.ceil(Date.parse(w.end) / 1000) * 1000 - 1000)}`;
|
||||
}
|
||||
|
||||
// --- Sources ---
|
||||
|
||||
export function ghSource(repo: string, cacheDir: string, client: GhClient = GH_CLIENT): ScorecardSource {
|
||||
@@ -140,14 +162,14 @@ function pullsPage(source: ScorecardSource, o: ScorecardOptions, page: number):
|
||||
}
|
||||
|
||||
/** Every PR merged into base inside the window; pages stop once a page's oldest update predates the window. */
|
||||
export function readMergedPrs(source: ScorecardSource, o: ScorecardOptions, since: string): MergedPr[] {
|
||||
export function readMergedPrs(source: ScorecardSource, o: ScorecardOptions, w: ScoreWindow): MergedPr[] {
|
||||
const out: MergedPr[] = [];
|
||||
const crewLogins = o.crew ? new Set(o.crew.map(l => l.toLowerCase())) : null;
|
||||
for (let page = 1; page <= MAX_PAGES; page++) {
|
||||
const pulls = pullsPage(source, o, page);
|
||||
for (const pr of pulls) {
|
||||
const mergedAt = typeof pr.merged_at === 'string' ? pr.merged_at : null;
|
||||
if (!mergedAt || mergedAt < since) continue;
|
||||
if (!mergedAt || !inWindow(mergedAt, w)) continue;
|
||||
const author = String(pr.user?.login ?? '');
|
||||
const association = String(pr.author_association ?? 'NONE');
|
||||
const comments = readComments(source, o.repo, Number(pr.number), { readyMarker: o.readyMarker, passMarker: o.passMarker, mergedAt });
|
||||
@@ -158,7 +180,7 @@ export function readMergedPrs(source: ScorecardSource, o: ScorecardOptions, sinc
|
||||
passes: bodyPasses === null ? comments.passes : Math.max(bodyPasses, comments.passes ?? 0) });
|
||||
}
|
||||
const oldest = pulls[pulls.length - 1]?.updated_at;
|
||||
if (pulls.length < PAGE || (typeof oldest === 'string' && oldest < since)) break;
|
||||
if (pulls.length < PAGE || (typeof oldest === 'string' && oldest < w.start)) break;
|
||||
}
|
||||
return out.sort((a, b) => a.mergedAt.localeCompare(b.mergedAt));
|
||||
}
|
||||
@@ -240,15 +262,15 @@ export function flakeRecords(dirs: string[]): { flakes: number; rerunUsd: number
|
||||
}
|
||||
|
||||
/** Gate runs in the window with kind, outcome, cost and flake records; artifacts are read for the newest maxArtifactRuns only. */
|
||||
export function readGateRuns(source: ScorecardSource, o: ScorecardOptions, since: string, prCost: Map<number, number>): { gates: GateRun[]; artifactRunsRead: number; artifactErrors: number } {
|
||||
const query = [`status=completed`, `per_page=${PAGE}`, `created=${encodeURIComponent(`>=${since.slice(0, 10)}`)}`,
|
||||
export function readGateRuns(source: ScorecardSource, o: ScorecardOptions, w: ScoreWindow, prCost: Map<number, number>): { gates: GateRun[]; artifactRunsRead: number; artifactErrors: number } {
|
||||
const query = [`status=completed`, `per_page=${PAGE}`, createdFilter(w.start, w.end),
|
||||
...(o.gateEvent ? [`event=${encodeURIComponent(o.gateEvent)}`] : [])].join('&');
|
||||
const gates: GateRun[] = [];
|
||||
for (let page = 1; page <= MAX_PAGES; page++) {
|
||||
const body = source.getJson(`repos/${o.repo}/actions/workflows/${o.gateWorkflow}/runs?${query}&page=${page}`) as { workflow_runs?: any[] };
|
||||
const runs = Array.isArray(body?.workflow_runs) ? body.workflow_runs : [];
|
||||
for (const run of runs) {
|
||||
if (String(run.created_at) < since || (o.gateEvent && run.event !== o.gateEvent)) continue;
|
||||
if (!inWindow(run.created_at, w) || (o.gateEvent && run.event !== o.gateEvent)) continue;
|
||||
const title = String(run.display_title ?? run.name ?? '');
|
||||
const branch = String(run.head_branch ?? '');
|
||||
const pr = Number(/#(\d+)/.exec(title)?.[1] ?? run.pull_requests?.[0]?.number ?? NaN);
|
||||
@@ -281,9 +303,9 @@ export function readGateRuns(source: ScorecardSource, o: ScorecardOptions, since
|
||||
}
|
||||
|
||||
/** Free-suite runs on base in the window and the flaky-pass entries their flake-ledger-* artifacts hold. */
|
||||
export function readFreeRetries(source: ScorecardSource, o: ScorecardOptions, since: string): FreeRetryRun[] {
|
||||
export function readFreeRetries(source: ScorecardSource, o: ScorecardOptions, w: ScoreWindow): FreeRetryRun[] {
|
||||
const client: GhClient = { getJson: p => source.getJson(p), downloadZip: () => { throw new Error('not used'); } };
|
||||
const runs = listWorkflowRuns(client, o.repo, o.freeWorkflow, { since, branch: o.base });
|
||||
const runs = listWorkflowRuns(client, o.repo, o.freeWorkflow, { since: w.start, until: w.end, branch: o.base });
|
||||
const retries = (runId: number): number | null => {
|
||||
try { return [...source.artifactDirs(runId, name => name.startsWith('flake-ledger-')).values()].flatMap(readFlakeLedgerDir).length; } catch { return null; }
|
||||
};
|
||||
@@ -295,30 +317,29 @@ function searchCount(source: ScorecardSource, q: string): number | null {
|
||||
return typeof body?.total_count === 'number' ? body.total_count : null;
|
||||
}
|
||||
|
||||
export function readReleases(source: ScorecardSource, repo: string, since: string): Array<{ tag: string; publishedAt: string }> {
|
||||
export function readReleases(source: ScorecardSource, repo: string, w: ScoreWindow): Array<{ tag: string; publishedAt: string }> {
|
||||
const body = source.getJson(`repos/${repo}/releases?per_page=${PAGE}&page=1`);
|
||||
return (Array.isArray(body) ? body : []).filter(r => !r.draft && typeof r.published_at === 'string' && r.published_at >= since)
|
||||
return (Array.isArray(body) ? body : []).filter(r => !r.draft && inWindow(r.published_at, w))
|
||||
.map(r => ({ tag: String(r.tag_name ?? ''), publishedAt: String(r.published_at) }));
|
||||
}
|
||||
|
||||
/** Every read, each guarded: one unreadable input marks its lines unavailable and the rest still print. */
|
||||
export function collectScorecard(source: ScorecardSource, o: ScorecardOptions): Scorecard {
|
||||
const since = new Date(o.now - o.sinceDays * 86_400_000).toISOString();
|
||||
const card: Scorecard = { repo: o.repo, base: o.base, windowStart: since, windowEnd: new Date(o.now).toISOString(),
|
||||
const w = scoreWindow(o);
|
||||
const card: Scorecard = { repo: o.repo, base: o.base, windowStart: w.start, windowEnd: w.end,
|
||||
prs: [], gates: [], freeRuns: [], artifactRunsRead: 0, passCap: o.passCap, artifactErrors: 0, issuesOpened: null, issuesClosed: null, releases: [], unavailable: [] };
|
||||
const guard = (what: string, fn: () => void) => { try { fn(); } catch (e) { card.unavailable.push(`${what}: ${unavailableReason(e)}`); } };
|
||||
guard('merged PRs', () => { card.prs = readMergedPrs(source, o, since); });
|
||||
guard('merged PRs', () => { card.prs = readMergedPrs(source, o, w); });
|
||||
guard('gate runs', () => {
|
||||
const prCost = new Map(card.prs.filter(p => p.commentCostUsd !== null).map(p => [p.number, p.commentCostUsd!] as const));
|
||||
Object.assign(card, readGateRuns(source, o, since, prCost));
|
||||
Object.assign(card, readGateRuns(source, o, w, prCost));
|
||||
});
|
||||
guard('free retries', () => { card.freeRuns = readFreeRetries(source, o, since); });
|
||||
const instant = since.replace(/\.\d{3}Z$/, 'Z');
|
||||
guard('free retries', () => { card.freeRuns = readFreeRetries(source, o, w); });
|
||||
guard('issues', () => {
|
||||
card.issuesOpened = searchCount(source, `repo:${o.repo} is:issue created:>=${instant}`);
|
||||
card.issuesClosed = searchCount(source, `repo:${o.repo} is:issue closed:>=${instant}`);
|
||||
card.issuesOpened = searchCount(source, `repo:${o.repo} is:issue created:${searchRange(w)}`);
|
||||
card.issuesClosed = searchCount(source, `repo:${o.repo} is:issue closed:${searchRange(w)}`);
|
||||
});
|
||||
guard('releases', () => { card.releases = readReleases(source, o.repo, since); });
|
||||
guard('releases', () => { card.releases = readReleases(source, o.repo, w); });
|
||||
return card;
|
||||
}
|
||||
|
||||
|
||||
@@ -95,21 +95,27 @@ export interface CiJob { id: number; name: string; conclusion: string | null; st
|
||||
|
||||
const PAGE = 100;
|
||||
|
||||
/** The runs API's `created` filter, by UTC day (both ends inclusive): `>=since`, or `since..until` when the window has an end. */
|
||||
export function createdFilter(since: string, until?: string): string {
|
||||
return `created=${encodeURIComponent(until ? `${since.slice(0, 10)}..${until.slice(0, 10)}` : `>=${since.slice(0, 10)}`)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every completed run of `workflow` created at or after `since`, newest first,
|
||||
* paging until a page is short or older than `since`. A failure on any page
|
||||
* throws: a partial history is never reported as a complete one.
|
||||
* Every completed run of `workflow` created at or after `since` (and before
|
||||
* `until` when given), newest first, paging until a page is short or older
|
||||
* than `since`. A failure on any page throws: a partial history is never
|
||||
* reported as a complete one.
|
||||
*/
|
||||
export function listWorkflowRuns(client: GhClient, repo: string, workflow: string,
|
||||
opts: { since: string; branch?: string; event?: string; maxPages?: number }): CiRun[] {
|
||||
opts: { since: string; until?: string; branch?: string; event?: string; maxPages?: number }): CiRun[] {
|
||||
const out: CiRun[] = [];
|
||||
const query = [`status=completed`, `per_page=${PAGE}`, `created=${encodeURIComponent(`>=${opts.since.slice(0, 10)}`)}`,
|
||||
const query = [`status=completed`, `per_page=${PAGE}`, createdFilter(opts.since, opts.until),
|
||||
...(opts.branch ? [`branch=${encodeURIComponent(opts.branch)}`] : []), ...(opts.event ? [`event=${encodeURIComponent(opts.event)}`] : [])];
|
||||
for (let page = 1; page <= (opts.maxPages ?? 20); page++) {
|
||||
const body = client.getJson(`repos/${repo}/actions/workflows/${workflow}/runs?${query.join('&')}&page=${page}`) as { workflow_runs?: any[] };
|
||||
const runs = Array.isArray(body?.workflow_runs) ? body.workflow_runs : [];
|
||||
for (const run of runs) {
|
||||
if (String(run.created_at) < opts.since) continue;
|
||||
if (String(run.created_at) < opts.since || (opts.until && Date.parse(String(run.created_at)) >= Date.parse(opts.until))) continue;
|
||||
if ((opts.branch && run.head_branch !== opts.branch) || (opts.event && run.event !== opts.event)) continue;
|
||||
out.push({ id: Number(run.id), attempt: Number(run.run_attempt ?? 1), sha: String(run.head_sha ?? ''), branch: String(run.head_branch ?? ''),
|
||||
event: String(run.event ?? ''), status: String(run.status ?? ''), conclusion: run.conclusion ?? null,
|
||||
|
||||
@@ -124,6 +124,40 @@ describe('factory-scorecard: recorded window', () => {
|
||||
expect(values.merged).toBe('3 (crew 2, outside 1)');
|
||||
});
|
||||
|
||||
test('--now closes the window: a PR, gate run, free run or release at or after now is left out, and the runs and issue queries carry the end', () => {
|
||||
// Value: protects=every windowed line counts [now - since-days, now) only, so a past window reproduces after the fact; fails_when=a read filters on the window start alone or a query drops the end bound; why_new=GST-135: --now 2026-10-15 counted 284 merged PRs (every merge through the run date) against 65 in the window, and the gate and release lines ignored --now; seam=none; no_claim=GitHub's server-side date filters, which the reads re-check per item
|
||||
const fx = fixture();
|
||||
const key = (part: string) => Object.keys(fx.responses).find(k => k.includes(part))!;
|
||||
(fx.responses[key('/pulls?')] as any[]).unshift({ number: 106, title: 'merged at the window end', user: { login: 'garry' }, author_association: 'OWNER',
|
||||
created_at: '2026-10-26T10:00:00Z', merged_at: '2026-10-26T12:00:00Z', updated_at: '2026-10-26T13:00:00Z', base: { ref: 'main' } });
|
||||
fx.responses['repos/acme/widgets/issues/106/comments?per_page=100&page=1'] = [];
|
||||
const gateRuns = fx.responses[key('/evals.yml/runs?')] as { workflow_runs: any[] };
|
||||
gateRuns.workflow_runs.unshift({ ...gateRuns.workflow_runs.find(r => r.id === 9002), id: 9007, conclusion: 'success', created_at: '2026-10-26T12:30:00Z' });
|
||||
const freeRuns = fx.responses[key('/free-tests.yml/runs?')] as { workflow_runs: any[] };
|
||||
freeRuns.workflow_runs.unshift({ ...freeRuns.workflow_runs[0], id: 8004, created_at: '2026-10-26T13:00:00Z' });
|
||||
(fx.responses[key('/releases?')] as any[]).unshift({ tag_name: 'v1.2.0', draft: false, prerelease: false, published_at: '2026-10-26T12:00:00Z' });
|
||||
const source = fixtureSource(fx, scratch());
|
||||
const card = collectScorecard(source, options());
|
||||
expect(card.unavailable).toEqual([]);
|
||||
expect(card.prs.map(p => p.number)).toEqual([101, 103, 102]);
|
||||
expect(card.gates.map(g => g.id)).toEqual([9004, 9003, 9002, 9001]);
|
||||
expect(card.freeRuns.map(r => r.id)).toEqual([8003, 8002, 8001]);
|
||||
expect(card.releases.map(r => r.tag)).toEqual(['v1.0.0']);
|
||||
const values = byId(scorecardRows(card));
|
||||
expect(values.merged).toBe('3 (crew 2, outside 1)');
|
||||
expect(values.lead_time).toBe('median 8.0h, p90 5.0d');
|
||||
expect(values.gates).toBe('2 / 1 / 1; known cost $43.00 on 2 of 4 runs');
|
||||
expect(values.releases).toBe('1 (v1.0.0)');
|
||||
for (const workflow of ['/evals.yml/runs?', '/free-tests.yml/runs?']) {
|
||||
expect(decodeURIComponent(source.calls.find(c => c.includes(workflow))!)).toContain('created=2026-10-19..2026-10-26');
|
||||
}
|
||||
const searches = source.calls.filter(c => c.startsWith('search/issues')).map(decodeURIComponent);
|
||||
expect(searches).toEqual([
|
||||
'search/issues?q=repo:acme/widgets is:issue created:2026-10-19T12:00:00Z..2026-10-26T11:59:59Z&per_page=1',
|
||||
'search/issues?q=repo:acme/widgets is:issue closed:2026-10-19T12:00:00Z..2026-10-26T11:59:59Z&per_page=1',
|
||||
]);
|
||||
});
|
||||
|
||||
test('--max-artifact-runs caps artifact reads: later runs count without cost or flake data', () => {
|
||||
const card = collectScorecard(fixtureSource(fixture(), scratch()), options({ maxArtifactRuns: 2 }));
|
||||
expect(card.artifactRunsRead).toBe(2);
|
||||
|
||||
+4
-4
@@ -133,7 +133,7 @@
|
||||
"body": "ship receipt follows\nSHIP_RECEIPT: pr=103 head=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb passes=3"
|
||||
}
|
||||
],
|
||||
"repos/acme/widgets/actions/workflows/evals.yml/runs?status=completed&per_page=100&created=%3E%3D2026-10-19&event=workflow_dispatch&page=1": {
|
||||
"repos/acme/widgets/actions/workflows/evals.yml/runs?status=completed&per_page=100&created=2026-10-19..2026-10-26&event=workflow_dispatch&page=1": {
|
||||
"total_count": 6,
|
||||
"workflow_runs": [
|
||||
{
|
||||
@@ -222,7 +222,7 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
"repos/acme/widgets/actions/workflows/free-tests.yml/runs?status=completed&per_page=100&created=%3E%3D2026-10-19&branch=main&page=1": {
|
||||
"repos/acme/widgets/actions/workflows/free-tests.yml/runs?status=completed&per_page=100&created=2026-10-19..2026-10-26&branch=main&page=1": {
|
||||
"total_count": 3,
|
||||
"workflow_runs": [
|
||||
{
|
||||
@@ -289,11 +289,11 @@
|
||||
"published_at": "2026-10-01T07:00:00Z"
|
||||
}
|
||||
],
|
||||
"search/issues?q=repo%3Aacme%2Fwidgets%20is%3Aissue%20created%3A%3E%3D2026-10-19T12%3A00%3A00Z&per_page=1": {
|
||||
"search/issues?q=repo%3Aacme%2Fwidgets%20is%3Aissue%20created%3A2026-10-19T12%3A00%3A00Z..2026-10-26T11%3A59%3A59Z&per_page=1": {
|
||||
"total_count": 5,
|
||||
"items": []
|
||||
},
|
||||
"search/issues?q=repo%3Aacme%2Fwidgets%20is%3Aissue%20closed%3A%3E%3D2026-10-19T12%3A00%3A00Z&per_page=1": {
|
||||
"search/issues?q=repo%3Aacme%2Fwidgets%20is%3Aissue%20closed%3A2026-10-19T12%3A00%3A00Z..2026-10-26T11%3A59%3A59Z&per_page=1": {
|
||||
"total_count": 7,
|
||||
"items": []
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user