ci: adopt tiered contributor validation

This commit is contained in:
Joseph Magly
2026-08-20 23:11:24 -04:00
parent 0d4d439f22
commit 38efd3dc0a
37 changed files with 916 additions and 121 deletions
+55 -30
View File
@@ -36,8 +36,9 @@ must be deliberate and follow [`docs/SUPPLY_CHAIN_POLICY.md`](docs/SUPPLY_CHAIN_
1. Start a focused branch from the current `main`.
2. Make the smallest coherent change that solves the linked problem.
3. Add tests with the implementation. New behavior without relevant tests is not
merge-ready.
3. Add focused tests with the implementation when practical. New behavior with
zero relevant coverage cannot pass the PR gate; maintainers may add remaining
test depth as an internal pre-merge task.
4. Update user, operator, policy, risk-map, and research documentation affected by
the change.
5. Run the applicable local checks below.
@@ -74,25 +75,32 @@ Tests must not download models or datasets, contact services, require an acceler
launch a UI, or use remote credentials unless they are explicitly assigned to a
conditional marker and gate.
CI currently requires:
CI keeps the contributor gate intentionally small enough to return useful review
feedback quickly. It currently requires:
- the full test matrix on Python 3.10, 3.11, and 3.12 with warnings treated as
errors;
- at least 75% repository line coverage and 60% repository branch coverage;
- at least 95% coverage of changed executable lines;
- no independent line or branch regression in a touched production module versus
the exact base commit;
- at least 80% line and 75% branch coverage for a new production module;
- at least 70% line coverage for each policy-designated critical boundary module;
- at least 94% line and 84% branch coverage for the mature CPU-testable scope;
- at least an 85% score in the bounded selective mutation campaign;
- zero unexpected warnings and no unowned, expired, or policy-invalid quarantine;
- Ruff F, actionlint, package, installed-wheel, installed-sdist, supply-chain, and
policy checks.
- one offline Python 3.12 core lane with warnings treated as errors;
- the versioned smoke suite, every changed CPU test, and contract tests selected
from the exact diff through `ci/test-risk-map.json`;
- at least 50% coverage of changed executable lines;
- Ruff F, actionlint, lock, policy, import, and CLI contract checks;
- a source and wheel build when package inputs change.
These values are floors, not targets. A change should strengthen behavioral
confidence rather than consume existing margin.
A green pull-request gate means the submission is ready for maintainer review. It
does not certify a release and does not guarantee immediate merge. Maintainers own
the additional risk-driven tests, cross-version checks, packaging depth, and
hardening needed for the merge decision.
### Tagged-release certification
The `v*` tag workflow, or an explicit manual release-validation run, executes the
full Python 3.10-3.12 matrix and retains the stricter repository, branch,
critical-module, touched-module, mature-scope, repeat, mutation, packaging,
Windows, and supply-chain gates. The full suite remains mandatory for a release;
the faster pull-request lane is not a release exemption.
### Test design standard
Each behavior-changing pull request must include:
@@ -121,20 +129,33 @@ uv run --extra dev python scripts/check_conditional_policy.py
## Local validation
Run focused tests while developing, then the complete baseline before requesting
review:
Run the focused pull-request baseline while developing. Replace `origin/main` with
the exact base commit when needed:
```bash
uv lock --check
uv run --extra dev python -m ruff check --select F app.py obliteratus tests scripts
uv run --extra dev python -m pytest
uv run --extra dev python -m build --sdist --wheel
mkdir -p test-results
uv run --extra dev python scripts/select_pr_tests.py --base-ref origin/main \
> test-results/selected-tests.txt
mapfile -t selected_tests < test-results/selected-tests.txt
uv run --extra dev python -m pytest "${selected_tests[@]}" \
--cov=app --cov-branch --cov-fail-under=0 \
--cov-report=json:test-results/coverage-pr-core.json
uv run --extra dev python scripts/check_coverage_thresholds.py \
test-results/coverage-pr-core.json --min-line 0 --min-branch 0 \
--min-changed 50 --base-ref origin/main
uv run --extra dev python -c 'import obliteratus; print(obliteratus.__version__)'
uv run --extra dev python -m obliteratus --help
uv run --extra dev python scripts/check_conditional_policy.py
uv run --extra dev python scripts/check_test_risk_map.py
```
If package inputs changed, also run
`uv run --extra dev python -m build --sdist --wheel`. Maintainers run the complete
`python -m pytest`, package-install,
cross-version, and quality-depth suite before a tagged release.
Ruff E501 remains a non-blocking legacy-debt report; new code should still respect
the configured 100-character line length. The exact reporting command lives in CI.
@@ -156,8 +177,9 @@ uv run --extra dev --group quality python scripts/check_mutation_score.py \
mutants/mutmut-cicd-stats.json --minimum 85
```
CI remains authoritative for exact-base coverage comparison, actionlint, platform
matrix results, installed-distribution checks, and retained evidence.
CI remains authoritative for exact-base changed-line coverage, actionlint, selected
test evidence, release platform-matrix results, installed-distribution checks, and
retained evidence.
## Risk-specific checks
@@ -234,23 +256,26 @@ import is not a performance result.
## Pull-request requirements
A merge-ready pull request:
A review-ready pull request:
- solves one coherent problem and links its canonical issue where one exists;
- explains user-visible behavior, risk surfaces, trust-boundary changes, and
compatibility impact;
- lists exact commands and outcomes, including checks not run and why;
- includes relevant code, tests, documentation, policy/risk-map updates, and lock or
digest changes in the same reviewable unit;
- includes relevant code and whatever focused tests the contributor can supply,
plus documentation, policy/risk-map updates, and lock or digest changes in the
same reviewable unit;
- contains no generated provider files, caches, unrelated cleanup, or drive-by
refactors;
- has signed commits, a current exact head, no unresolved review threads, no merge
conflicts, and every required CI check green.
- has signed commits, a current exact head, no merge conflicts, and every
contributor-facing CI check green.
Review is performed at an immutable head SHA. Pushing new commits invalidates prior
test and review conclusions until the new head is checked. Maintainers may split or
re-derive a stale legacy pull request and preserve its authorship, but new pull
requests are expected to arrive with their complete relevant test suite.
test and review conclusions until the new head is checked. Maintainers may add
tests or hardening in a separate attributable commit, resolve review threads, and
run deeper gates before merge. Behavior changes with no relevant test coverage
remain blocked; contributors are not expected to satisfy release-depth coverage
and mutation gates merely to request review.
## Contributing experiment results