Files
OBLITERATUS/CONTRIBUTING.md
T
Joseph Magly 37d008d462 test: establish Gate 1 quality baseline (#90)
Establishes the mandatory testing, coverage, repeatability, mutation, packaging, supply-chain, and AIWG workspace baseline before feature integration.
2026-08-15 01:07:47 -04:00

188 lines
8.0 KiB
Markdown

# Contributing to OBLITERATUS
Thanks for your interest in contributing. This document covers everything you need to get started.
## Development Setup
```bash
git clone https://github.com/elder-plinius/OBLITERATUS.git
cd OBLITERATUS
python -m pip install --index-url https://download.pytorch.org/whl/cpu "torch>=2.0"
python - <<'PY' > /tmp/torch-cpu-constraint.txt
import torch
print(f"torch=={torch.__version__}")
PY
python -m pip install -e ".[dev]" -c /tmp/torch-cpu-constraint.txt
```
This installs CPU PyTorch first, then installs the package in editable mode with pinned development tools while constraining PyTorch to the already-installed CPU build.
## Running Tests
```bash
python -m pytest # full suite with coverage
python -m pytest tests/test_abliterate.py # single file
python -m pytest -x # stop on first failure
python -m pytest -k "test_name" # run specific test
python - <<'PY'
import obliteratus
print(getattr(obliteratus, "__version__", "import ok"))
PY
python -m obliteratus --help
```
All tests must pass before submitting a PR. Tests are designed to run on CPU without downloading models.
The mandatory gate currently requires at least 70% repository statement coverage,
55% branch coverage, 90% coverage of changed executable lines, and 70% statement
coverage in the device, loader, architecture-profile, CLI, simulated MLX,
evaluation-metric, reporting, community-contribution, and telemetry boundary
modules. New changes should raise these floors rather than consume the existing
margin. A separately measured mature CPU-only scope must remain at or above 90%
statement and 78% branch coverage; its environment-bound exclusions and their
conditional-test ownership are versioned in `ci/test-quality-policy.json`.
Every changed production module is also compared with coverage generated from
the exact base commit. Line and branch coverage may not regress independently,
and new modules start at 80% line / 75% branch coverage in addition to the 90%
changed-line requirement.
The quality-depth job repeats the highest-consequence pure tests three times in
different file orders and with different deterministic hash seeds. It also runs
selective mutation testing over configuration, quality-policy, and pure
evaluation contracts. The mutation floor is 75%. Run these checks locally with:
```bash
uv sync --locked --extra dev --group quality
python scripts/run_repeat_gate.py --output test-results/repeat-gate.json
mutmut run --max-children 4
mutmut export-cicd-stats
python scripts/check_mutation_score.py mutants/mutmut-cicd-stats.json --minimum 75
```
The repeat and mutation selections are intentionally bounded; on the Wave A
calibration runner each completed in under one minute after environment setup.
CI retains normalized coverage/JUnit trends and repeat/mutation evidence for 90
days. Flake observations and time-bounded quarantines are governed in
`ci/test-quality-policy.json`; two observations within 30 days require an
owner, reason, repository issue, and expiry. Thresholds may move downward only
through a time-bounded exception linked to a reviewed repository issue.
Hardware, model-download, network-service, operator-UI, and remote-provider tests
run separately so the pull-request baseline stays offline and credential-free. See
[`docs/conditional-testing.md`](docs/conditional-testing.md) for manual commands,
runner labels, pinned resources, credential handling, costs, cadence, and evidence
freshness. Validate the mapping between those gates and CPU coverage exclusions with:
```bash
python scripts/check_conditional_policy.py
python scripts/check_test_risk_map.py
```
The machine-readable source-to-test ownership graph lives in
`ci/test-risk-map.json`. Its source inventory covers `app.py` and every Python
module under `obliteratus/`; every production file must belong to exactly one
named contract surface with an owner, contract types, and existing test files.
Adding a module without updating that graph fails `check_test_risk_map.py`.
Targeted risk-module entries add stricter conditional-gate and critical-path
requirements on top of the exhaustive contract ownership layer. They are
limited to the measured `obliteratus` coverage root so normalized CI evidence
can account for every targeted module; top-level `app.py` remains owned and is
exercised by its explicit operator-UI contract test.
Each behavior-changing PR must add a focused regression or contract test,
relevant negative/boundary coverage, and propagation/runtime evidence for
public options. Hardware or service changes require both a deterministic
boundary test and their mapped conditional gate. When adding or moving source,
update its contract surface and, when applicable, its targeted risk-module and
conditional-gate entries in the same PR.
## Code Style
We use [ruff](https://docs.astral.sh/ruff/) for linting and formatting:
```bash
python -m ruff check --select F app.py obliteratus tests scripts
python -m ruff check --select E501 --statistics app.py obliteratus tests scripts # known non-blocking line-length debt report
```
- Line length: 100 characters
- Target: Python 3.10+
- The CI Ruff gate enforces all Ruff F rules. E501 line-length findings are reported as known non-blocking legacy debt until the baseline is cleaned up.
- Follow existing patterns in the codebase
## Submitting Changes
1. Fork the repo and create a branch from `main`
2. Make your changes
3. Add or update tests as needed
4. Run `python -m pytest`, `python -m build --sdist --wheel`, the import/CLI smoke checks, and the CI Ruff gate
5. Write a clear commit message explaining *why*, not just *what*
6. Open a pull request
## Pull Request Guidelines
- Keep PRs focused -- one feature or fix per PR
- Include a test plan in the PR description
- Link related issues with `Fixes #123` or `Closes #123`
- For new analysis modules, include unit tests with synthetic data (no model downloads)
- Legacy cleanup PRs may receive missing tests as a one-time maintainer courtesy when the change is already otherwise clean. New changes are expected to include relevant tests and keep the full suite passing.
## Contributing Experiment Results
Beyond code contributions, you can contribute abliteration experiment results to the community dataset used in the research paper. After running abliteration on any model:
```bash
obliteratus obliterate <model> --method advanced --contribute \
--contribute-notes "Hardware: A100, prompt set: default"
```
This saves a structured JSON file to `community_results/`. To submit your results:
1. Run abliteration with `--contribute` on any model/method combination
2. Open a PR adding your `community_results/*.json` file(s)
3. The aggregation pipeline will incorporate your data into the paper tables
You can preview aggregated results locally:
```bash
obliteratus aggregate --format summary
obliteratus aggregate --format latex --min-runs 3
```
## Project Structure
```
obliteratus/
abliterate.py # Core abliteration pipeline
informed_pipeline.py # Analysis-informed pipeline
community.py # Community contribution system
cli.py # CLI entry point
config.py # YAML config loading
interactive.py # Interactive mode
presets.py # Model presets (47 models)
runner.py # Ablation study runner
analysis/ # 15 analysis modules
evaluation/ # Metrics and benchmarks
models/ # Model loading utilities
reporting/ # Report generation
strategies/ # Ablation strategies (layer, head, FFN, embedding)
tests/ # 44 test files
paper/ # LaTeX paper
examples/ # YAML config examples
```
## Reporting Bugs
Open an issue with:
- What you expected to happen
- What actually happened
- Steps to reproduce
- Model name and hardware (GPU/CPU, VRAM)
## Security Issues
See [SECURITY.md](SECURITY.md) for responsible disclosure of security vulnerabilities.
## License
By contributing, you agree that your contributions will be licensed under the [AGPL-3.0](LICENSE).