CalvinBackup/OBLITERATUS
1
0
Fork 0
mirror of https://github.com/elder-plinius/OBLITERATUS.git synced 2026-08-17 16:37:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: align contributor standards with BT6 gates

Browse Source
This commit is contained in:
Joseph Magly
2026-08-16 17:36:34 -04:00
parent b25b3f3b66
commit 0f6be47948
5 changed files with 313 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.aiwg/aiwg.config
+10
View File
@@ -117,6 +117,16 @@
"delivery": {
"mode": "pr-required",
"default_branch": "main",
"committer": {
"name": "Joseph Magly",
"email": "1159087+jmagly@users.noreply.github.com"
},
"signing": {
"format": "openpgp",
"key": "62297562B1C7053088F405DB0117DAAA677A5BF2",
"enforce": "commits"
},
"require_signed_commits": true,
"require_ci_green": true,
"auto_close_issues": true,
"issue_comment_on_cycle": true,
.github/pull_request_template.md
+55 -14
View File
@@ -1,22 +1,63 @@
## Summary
- TBD
- Problem:
- Change:
- User-visible effect:
- Linked issue: Closes #
## Tests
## Risk and trust assessment
- [ ] `python -m ruff check --select F app.py obliteratus tests scripts`
- [ ] `python -m ruff check --select E501 --statistics obliteratus tests scripts/gemma4_12b_recursive_loop.py` reviewed as known non-blocking line-length debt
- [ ] `python -c 'import obliteratus; print(getattr(obliteratus, "__version__", "import ok"))'`
- [ ] `python -m obliteratus --help`
- [ ] `python -m pytest`
- [ ] `python -m build --sdist --wheel`
- Risk surfaces touched:
- Untrusted inputs or external dependencies:
- Remote code, deserialization, credentials, subprocess, network, or filesystem impact:
- Compatibility or migration impact:
## Test evidence
Exact head SHA: `TBD`
| Check | Result | Evidence or notes |
|---|---|---|
| Focused regression/contract tests | TBD | |
| Negative and boundary tests | TBD | |
| `python -m ruff check --select F app.py obliteratus tests scripts` | TBD | |
| `uv lock --check` | TBD | |
| `python -m pytest` | TBD | |
| `python -m build --sdist --wheel` | TBD | |
| Import and CLI smoke checks | TBD | |
| Applicable risk-surface checks | TBD | |
| Conditional hardware/service gates | not applicable / TBD | |
Coverage or mutation impact:
- Repository line/branch:
- Changed-line:
- Touched-module regression:
- Mature CPU scope:
- Mutation score, when applicable:
## Research or performance evidence
- Exact model/dataset revisions, configuration, seed, environment, and hardware:
- Raw evidence/artifacts and hashes:
- Baseline, uncertainty, limitations, and independent reproduction status:
Use `Not applicable` when this pull request makes no research or performance claim.
## Checklist
- [ ] Relevant tests were added or updated for new behavior.
- [ ] The full test suite passes locally or the CI result is linked.
- [ ] Documentation or examples were updated when user-facing behavior changed.
- [ ] No secrets, credentials, generated provider files, or unrelated changes are included.
- [ ] Any CI action or standalone-tool pin change also updates `ci/digests.txt`.
- [ ] I added or updated tests for every changed behavior.
- [ ] I covered relevant failure, boundary, and malformed-input paths.
- [ ] The default test path remains deterministic, offline, credential-free, and CPU-safe.
- [ ] I updated `ci/test-risk-map.json` or conditional policy when ownership changed.
- [ ] Documentation and examples match user-visible behavior.
- [ ] No secrets, private keys, credentials, caches, generated provider files, or unrelated changes are included.
- [ ] Remote code and executable deserialization remain disabled by default, or the explicit exception is justified and tested.
- [ ] Dependency, action, or tool changes include the reviewed lock/digest and supply-chain impact.
- [ ] Research and performance claims are traceable to exact-head evidence and scoped to what was tested.
- [ ] Every commit has a verifiable signature from its actual author or approved integration identity.
- [ ] The branch has not been force-pushed and the exact head is ready for review.
Legacy cleanup PRs may receive missing tests as a one-time maintainer courtesy when the change is already otherwise clean. New changes are expected to include relevant tests and keep the full suite passing.
New changes are expected to include their complete relevant test suite. The one-time
maintainer courtesy for already-reviewed legacy pull requests does not apply to new
submissions.
CONTRIBUTING.md
+235 -145
View File
@@ -1,193 +1,283 @@
# Contributing to OBLITERATUS
Thanks for your interest in contributing. This document covers everything you need to get started.
Thank you for contributing. OBLITERATUS is a research tool that mutates model
weights, evaluates behavior, persists results, and can cross local, accelerator,
network, and remote-execution trust boundaries. Changes therefore need both
ordinary software tests and evidence appropriate to the risk they affect.
## Development Setup
The authoritative requirements are the current CI workflow, the machine-readable
quality policies, and the BT6 maintainer profile:
- [`.github/workflows/ci.yml`](.github/workflows/ci.yml)
- [`ci/test-quality-policy.json`](ci/test-quality-policy.json)
- [`ci/test-risk-map.json`](ci/test-risk-map.json)
- [`ci/conditional-test-policy.json`](ci/conditional-test-policy.json)
- [`.aiwg/bt6-maintainer.yaml`](.aiwg/bt6-maintainer.yaml)
If this guide and an enforced policy disagree, the enforced policy wins. Update
the guide in the same pull request when intentionally changing a standard.
## Development setup
The supported Python versions are 3.10 through 3.12. Use the committed lock so
local resolution matches CI and Linux/Windows use the official CPU-only PyTorch
source by default:
```bash
git clone https://github.com/elder-plinius/OBLITERATUS.git
cd OBLITERATUS
python -m pip install --index-url https://download.pytorch.org/whl/cpu "torch>=2.0"
python - <<'PY' > /tmp/torch-cpu-constraint.txt
import torch
print(f"torch=={torch.__version__}")
PY
python -m pip install -e ".[dev]" -c /tmp/torch-cpu-constraint.txt
python -m pip install "uv==0.12.4"
uv sync --locked --extra dev
```
This installs CPU PyTorch first, then installs the package in editable mode with pinned development tools while constraining PyTorch to the already-installed CPU build.
Run project commands in the environment with `uv run --extra dev ...`, or activate
`.venv`. Never update `uv.lock` merely to make setup succeed. Dependency updates
must be deliberate and follow [`docs/SUPPLY_CHAIN_POLICY.md`](docs/SUPPLY_CHAIN_POLICY.md).
## Running Tests
## Change workflow
1. Start a focused branch from the current `main`.
2. Make the smallest coherent change that solves the linked problem.
3. Add tests with the implementation. New behavior without relevant tests is not
merge-ready.
4. Update user, operator, policy, risk-map, and research documentation affected by
the change.
5. Run the applicable local checks below.
6. Sign every commit and push the branch without rewriting published history.
7. Open a pull request using the repository template and keep its exact head green.
The repository does not permit force-pushes. Add corrective commits, or coordinate
with a maintainer when a history change is genuinely necessary.
### Signed commits and authorship
All commits must have a cryptographically verifiable signature. Contributors sign
with a key associated with their own forge identity; GitHub's verified web-flow
signature is also acceptable. Do not request, copy, or use a maintainer or project
private key.
Maintainer-authored and integration commits use the configured project publish key,
currently fingerprint
`62297562B1C7053088F405DB0117DAAA677A5BF2`, through the approved vault-backed
signing path. Maintainers preserve the original contributor as author when carrying
their work and keep maintainer hardening or test commits separately attributable.
Verify your local commit before pushing:
```bash
python -m pytest # full suite with coverage
python -m pytest tests/test_abliterate.py # single file
python -m pytest -x # stop on first failure
python -m pytest -k "test_name" # run specific test
python - <<'PY'
import obliteratus
print(getattr(obliteratus, "__version__", "import ok"))
PY
python -m obliteratus --help
git verify-commit HEAD
git log -1 --show-signature
```
All tests must pass before submitting a PR. Tests are designed to run on CPU without downloading models.
The mandatory gate currently requires at least 70% repository statement coverage,
55% branch coverage, 90% coverage of changed executable lines, and 70% statement
coverage in the device, loader, architecture-profile, CLI, simulated MLX,
evaluation-metric, reporting, community-contribution, and telemetry boundary
modules. New changes should raise these floors rather than consume the existing
margin. A separately measured mature CPU-only scope must remain at or above 90%
statement and 78% branch coverage; its environment-bound exclusions and their
conditional-test ownership are versioned in `ci/test-quality-policy.json`.
Every changed production module is also compared with coverage generated from
the exact base commit. Line and branch coverage may not regress independently,
and new modules start at 80% line / 75% branch coverage in addition to the 90%
changed-line requirement.
## Required pull-request baseline
The quality-depth job repeats the highest-consequence pure tests three times in
different file orders and with different deterministic hash seeds. It also runs
selective mutation testing over configuration, quality-policy, and pure
evaluation contracts. The mutation floor is 75%. Run these checks locally with:
The mandatory baseline is deterministic, offline, credential-free, and CPU-safe.
Tests must not download models or datasets, contact services, require an accelerator,
launch a UI, or use remote credentials unless they are explicitly assigned to a
conditional marker and gate.
CI currently requires:
- the full test matrix on Python 3.10, 3.11, and 3.12 with warnings treated as
errors;
- at least 75% repository line coverage and 60% repository branch coverage;
- at least 95% coverage of changed executable lines;
- no independent line or branch regression in a touched production module versus
the exact base commit;
- at least 80% line and 75% branch coverage for a new production module;
- at least 70% line coverage for each policy-designated critical boundary module;
- at least 94% line and 84% branch coverage for the mature CPU-testable scope;
- at least an 85% score in the bounded selective mutation campaign;
- zero unexpected warnings and no unowned, expired, or policy-invalid quarantine;
- Ruff F, actionlint, package, installed-wheel, installed-sdist, supply-chain, and
policy checks.
These values are floors, not targets. A change should strengthen behavioral
confidence rather than consume existing margin.
### Test design standard
Each behavior-changing pull request must include:
- a focused regression or contract test for the intended behavior;
- negative, boundary, malformed-input, and failure-path cases where applicable;
- assertions about externally observable outcomes, not only implementation calls;
- deterministic seeds or invariant/property assertions for numerical behavior;
- propagation and runtime evidence for new public options;
- cleanup, rollback, and atomicity checks for persistence or partial-failure paths;
- explicit tests at trust boundaries, including remote code, deserialization,
credentials, subprocesses, paths, network responses, and checkpoint provenance.
Do not weaken assertions, delete tests, add unconditional skips, suppress warnings,
or lower a threshold to make a change pass. A temporary exception must be narrowly
scoped, issue-linked, owner-assigned, reviewed, and time-bounded by the relevant
machine-readable policy.
Update [`ci/test-risk-map.json`](ci/test-risk-map.json) whenever source ownership,
contract coverage, or the conditional boundary changes. Validate it with:
```bash
uv run --extra dev python scripts/check_test_risk_map.py
uv run --extra dev python scripts/check_conditional_policy.py
```
## Local validation
Run focused tests while developing, then the complete baseline before requesting
review:
```bash
uv lock --check
uv run --extra dev python -m ruff check --select F app.py obliteratus tests scripts
uv run --extra dev python -m pytest
uv run --extra dev python -m build --sdist --wheel
uv run --extra dev python -c 'import obliteratus; print(obliteratus.__version__)'
uv run --extra dev python -m obliteratus --help
uv run --extra dev python scripts/check_conditional_policy.py
uv run --extra dev python scripts/check_test_risk_map.py
```
Ruff E501 remains a non-blocking legacy-debt report; new code should still respect
the configured 100-character line length. The exact reporting command lives in CI.
For changes to the quality system or mutation-owned code, also run:
```bash
uv sync --locked --extra dev --group quality
python scripts/run_repeat_gate.py --output test-results/repeat-gate.json
mutmut run --max-children 4
mutmut export-cicd-stats
python scripts/check_mutation_score.py mutants/mutmut-cicd-stats.json --minimum 75
uv run --extra dev --group quality python scripts/run_repeat_gate.py \
--output test-results/repeat-gate.json
uv run --extra dev --group quality python scripts/check_mutation_targets.py prepare
uv run --extra dev --group quality python scripts/prepare_mutation_coverage.py \
prepare-coverage --max-children 4
uv run --extra dev --group quality python scripts/prepare_mutation_coverage.py \
prepare-stats --max-children 4
uv run --extra dev --group quality python scripts/run_prepared_mutmut.py \
run --max-children 4
uv run --extra dev --group quality mutmut export-cicd-stats
uv run --extra dev --group quality python scripts/check_mutation_score.py \
mutants/mutmut-cicd-stats.json --minimum 85
```
The repeat and mutation selections are intentionally bounded; on the Wave A
calibration runner each completed in under one minute after environment setup.
CI retains normalized coverage/JUnit trends and repeat/mutation evidence for 90
days. Flake observations and time-bounded quarantines are governed in
`ci/test-quality-policy.json`; two observations within 30 days require an
owner, reason, repository issue, and expiry. Thresholds may move downward only
through a time-bounded exception linked to a reviewed repository issue.
CI remains authoritative for exact-base coverage comparison, actionlint, platform
matrix results, installed-distribution checks, and retained evidence.
The same policy owns mandatory-suite, registered-marker, individual-test, and
repeat-pass duration budgets. The normalized trend artifact records every
testcase duration and aggregates durations by test-layer marker. A testcase
that exceeds the default budget must have a named owner, repository issue,
reason, explicit ceiling, and expiring review window; otherwise CI fails.
## Risk-specific checks
Hardware, model-download, network-service, operator-UI, and remote-provider tests
run separately so the pull-request baseline stays offline and credential-free. See
[`docs/conditional-testing.md`](docs/conditional-testing.md) for manual commands,
runner labels, pinned resources, credential handling, costs, cadence, and evidence
freshness. Validate the mapping between those gates and CPU coverage exclusions with:
Run the focused checks for every risk surface your change touches, in addition to
the full baseline:
```bash
python scripts/check_conditional_policy.py
python scripts/check_test_risk_map.py
```
| Risk surface | Typical paths | Required focused checks |
|---|---|---|
| Model loading | `obliteratus/models/**`, `obliteratus/device.py`, `scripts/**` | `python -m pytest tests/test_strategies.py tests/test_module_imports.py` |
| Abliteration core | `obliteratus/abliterate.py`, `obliteratus/strategies/**` | `python -m pytest tests/test_abliterate.py` |
| Research metrics | `obliteratus/evaluation/**`, `obliteratus/analysis/**`, `paper/**`, `community_results/**` | `python -m pytest tests/test_advanced_metrics.py tests/test_breakthrough_modules.py tests/test_community.py` |
| User contracts | `obliteratus/cli.py`, `obliteratus/local_ui.py`, `app.py`, `notebooks/**` | `python -m pytest tests/test_cli.py tests/test_module_imports.py` and CLI help |
| CI and supply chain | `.github/workflows/**`, `ci/**`, dependency and policy files | `uv lock --check`, policy tests, and package build |
The machine-readable source-to-test ownership graph lives in
`ci/test-risk-map.json`. Its source inventory covers `app.py` and every Python
module under `obliteratus/`; every production file must belong to exactly one
named contract surface with an owner, contract types, and existing test files.
Adding a module without updating that graph fails `check_test_risk_map.py`.
Targeted risk-module entries add stricter conditional-gate and critical-path
requirements on top of the exhaustive contract ownership layer. They are
limited to the measured `obliteratus` coverage root so normalized CI evidence
can account for every targeted module; top-level `app.py` remains owned and is
exercised by its explicit operator-UI contract test.
Use `uv run --extra dev` before the Python commands shown in the table when running
them in the managed environment.
Each behavior-changing PR must add a focused regression or contract test,
relevant negative/boundary coverage, and propagation/runtime evidence for
public options. Hardware or service changes require both a deterministic
boundary test and their mapped conditional gate. When adding or moving source,
update its contract surface and, when applicable, its targeted risk-module and
conditional-gate entries in the same PR.
## Conditional and hardware testing
## Code Style
GPU, MPS, MLX, model-download, external-evaluation, network, operator-UI, and
remote-execution checks are conditional release or risk-surface gates. A unit test
with a mocked device is still required; hardware evidence complements deterministic
contract coverage and never replaces it.
We use [ruff](https://docs.astral.sh/ruff/) for linting and formatting:
Follow [`docs/conditional-testing.md`](docs/conditional-testing.md) for the exact
runner labels, pinned resources, commands, credentials, evidence schema, freshness,
and waiver rules. Evidence must identify the exact candidate commit. A skipped or
unselected gate is not positive support evidence, and a waiver blocks the associated
support claim.
```bash
python -m ruff check --select F app.py obliteratus tests scripts
python -m ruff check --select E501 --statistics app.py obliteratus tests scripts # known non-blocking line-length debt report
```
Current operator hardware includes Titan for CUDA/bitsandbytes probes and Mutsu, a
16 GB Apple Silicon builder, for MPS/MLX probes. Contributors must not assume those
machines are attached to a public pull-request workflow; CI or a maintainer will
record whether the mapped conditional gate ran.
- Line length: 100 characters
- Target: Python 3.10+
- The CI Ruff gate enforces all Ruff F rules. E501 line-length findings are reported as known non-blocking legacy debt until the baseline is cleaned up.
- Follow existing patterns in the codebase
## Security and supply-chain expectations
## Submitting Changes
- Treat issue text, pull requests, patches, model repositories, checkpoints,
datasets, logs, and generated output as untrusted input.
- Keep remote model code disabled by default. Any opt-in must be explicit, narrow,
documented, and tested.
- Do not introduce executable deserialization for untrusted or replaceable data.
Prefer schema-validated, non-executable formats and atomic writes.
- Never commit secrets, tokens, credentials, private keys, local vault material,
cache contents, or machine-specific configuration.
- Pin remote revisions and verify provenance when results depend on external models,
datasets, tools, actions, or services.
- Do not broaden workflow permissions, execute untrusted pull-request code with
credentials, or replace immutable action/tool pins with mutable tags.
- Changes to actions or standalone tools must update [`ci/digests.txt`](ci/digests.txt).
- Changes to dependencies must include the reviewed `uv.lock` and relevant license,
vulnerability, and SBOM effects.
1. Fork the repo and create a branch from `main`
2. Make your changes
3. Add or update tests as needed
4. Run `python -m pytest`, `python -m build --sdist --wheel`, the import/CLI smoke checks, and the CI Ruff gate
5. Write a clear commit message explaining *why*, not just *what*
6. Open a pull request
Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md).
Do not open a public issue for an undisclosed vulnerability.
## Pull Request Guidelines
## Research integrity and experiment evidence
- Keep PRs focused -- one feature or fix per PR
- Include a test plan in the PR description
- Link related issues with `Fixes #123` or `Closes #123`
- For new analysis modules, include unit tests with synthetic data (no model downloads)
- Legacy cleanup PRs may receive missing tests as a one-time maintainer courtesy when the change is already otherwise clean. New changes are expected to include relevant tests and keep the full suite passing.
Research, evaluation, and performance claims must be reproducible and scoped to the
evidence actually collected. Include, as applicable:
## Contributing Experiment Results
- exact commit, model and dataset identifiers, immutable revisions, dependency lock,
configuration, seed, hardware, and commands;
- raw or retained machine-readable output and hashes, not only a prose summary;
- baseline and comparison method, metric definition, uncertainty, failure cases, and
known limitations;
- citations to primary sources and provenance for imported claims or artifacts;
- a clear label for contributor-reported results that maintainers could not
independently reproduce.
Beyond code contributions, you can contribute abliteration experiment results to the community dataset used in the research paper. After running abliteration on any model:
Do not generalize a result beyond tested models, hardware, operating systems, or
backends. Performance claims need measured evidence; availability or a successful
import is not a performance result.
## Pull-request requirements
A merge-ready pull request:
- solves one coherent problem and links its canonical issue where one exists;
- explains user-visible behavior, risk surfaces, trust-boundary changes, and
compatibility impact;
- lists exact commands and outcomes, including checks not run and why;
- includes relevant code, tests, documentation, policy/risk-map updates, and lock or
digest changes in the same reviewable unit;
- contains no generated provider files, caches, unrelated cleanup, or drive-by
refactors;
- has signed commits, a current exact head, no unresolved review threads, no merge
conflicts, and every required CI check green.
Review is performed at an immutable head SHA. Pushing new commits invalidates prior
test and review conclusions until the new head is checked. Maintainers may split or
re-derive a stale legacy pull request and preserve its authorship, but new pull
requests are expected to arrive with their complete relevant test suite.
## Contributing experiment results
To contribute an abliteration result to the community dataset:
```bash
obliteratus obliterate <model> --method advanced --contribute \
--contribute-notes "Hardware: A100, prompt set: default"
--contribute-notes "Hardware: A100, prompt set: default"
```
This saves a structured JSON file to `community_results/`. To submit your results:
1. Run abliteration with `--contribute` on any model/method combination
2. Open a PR adding your `community_results/*.json` file(s)
3. The aggregation pipeline will incorporate your data into the paper tables
You can preview aggregated results locally:
Review the generated JSON for accidental sensitive data, add only the intended
`community_results/*.json` files, and include the provenance fields described above.
Preview aggregate output with:
```bash
obliteratus aggregate --format summary
obliteratus aggregate --format latex --min-runs 3
```
## Project Structure
## Reporting bugs
```
obliteratus/
abliterate.py # Core abliteration pipeline
informed_pipeline.py # Analysis-informed pipeline
community.py # Community contribution system
cli.py # CLI entry point
config.py # YAML config loading
interactive.py # Interactive mode
presets.py # Model presets (47 models)
runner.py # Ablation study runner
analysis/ # 15 analysis modules
evaluation/ # Metrics and benchmarks
models/ # Model loading utilities
reporting/ # Report generation
strategies/ # Ablation strategies (layer, head, FFN, embedding)
tests/ # 44 test files
paper/ # LaTeX paper
examples/ # YAML config examples
```
## Reporting Bugs
Open an issue with:
- What you expected to happen
- What actually happened
- Steps to reproduce
- Model name and hardware (GPU/CPU, VRAM)
## Security Issues
See [SECURITY.md](SECURITY.md) for responsible disclosure of security vulnerabilities.
Open an issue with expected and actual behavior, minimal reproduction, OBLITERATUS
version or commit, operating system, Python and dependency versions, model and
immutable revision where relevant, hardware/backend, and sanitized logs. Never post
credentials, private model data, or undisclosed security details.
## License
By contributing, you agree that your contributions will be licensed under the [AGPL-3.0](LICENSE).
By contributing, you agree that your contributions are licensed under
[`AGPL-3.0-or-later`](LICENSE).
WORKSPACE.md
+3 -2
View File
@@ -47,8 +47,9 @@ Canonical required checks:
coverage;
- at least 95% changed-line coverage plus no line or branch regression in any
touched production module, compared with coverage from the exact base commit;
- at least 92% line and 80% branch coverage for the documented mature
CPU-testable scope, plus a 75% selective mutation score and zero unexpected
- at least 80% line and 75% branch coverage for new production modules;
- at least 94% line and 84% branch coverage for the documented mature
CPU-testable scope, plus an 85% selective mutation score and zero unexpected
warnings;
- normalized per-test and per-marker duration evidence, owned slow-test
exceptions, fixed repeat-campaign budgets, and a ten-minute test-job cap;
tests/test_aiwg_workspace_contracts.py
+10
View File
@@ -78,6 +78,16 @@ def test_workspace_authority_and_delivery_permissions_remain_explicit():
assert config["delivery"] == {
"mode": "pr-required",
"default_branch": "main",
"committer": {
"name": "Joseph Magly",
"email": "1159087+jmagly@users.noreply.github.com",
},
"signing": {
"format": "openpgp",
"key": "62297562B1C7053088F405DB0117DAAA677A5BF2",
"enforce": "commits",
},
"require_signed_commits": True,
"require_ci_green": True,
"auto_close_issues": True,
"issue_comment_on_cycle": True,