diff --git a/.github/TESTING.md b/.github/TESTING.md new file mode 100644 index 0000000..b678364 --- /dev/null +++ b/.github/TESTING.md @@ -0,0 +1,341 @@ +# Testing Guide + +This document describes the testing infrastructure for Claude How To. + +## Overview + +The project uses GitHub Actions to automatically run tests on every push and pull request. Tests cover: + +- **Unit Tests**: Python tests using pytest +- **Code Quality**: Linting and formatting with Ruff +- **Security**: Vulnerability scanning with Bandit +- **Type Checking**: Static type analysis with mypy +- **Build Verification**: EPUB generation test + +## Running Tests Locally + +### Prerequisites + +```bash +# Install uv (fast Python package manager) +pip install uv + +# Or on macOS with Homebrew +brew install uv +``` + +### Setup Environment + +```bash +# Clone the repository +git clone https://github.com/luongnv89/claude-howto.git +cd claude-howto + +# Create virtual environment +uv venv + +# Activate it +source .venv/bin/activate # macOS/Linux +# or +.venv\Scripts\activate # Windows + +# Install development dependencies +uv pip install -r requirements-dev.txt +``` + +### Run Tests + +```bash +# Run all unit tests +pytest scripts/tests/ -v + +# Run tests with coverage +pytest scripts/tests/ -v --cov=scripts --cov-report=html + +# Run specific test file +pytest scripts/tests/test_build_epub.py -v + +# Run specific test function +pytest scripts/tests/test_build_epub.py::test_function_name -v + +# Run tests in watch mode (requires pytest-watch) +ptw scripts/tests/ +``` + +### Run Linting + +```bash +# Check code formatting +ruff format --check scripts/ + +# Auto-fix formatting issues +ruff format scripts/ + +# Run linter +ruff check scripts/ + +# Auto-fix linter issues +ruff check --fix scripts/ +``` + +### Run Security Scan + +```bash +# Run Bandit security scan +bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/ + +# Generate JSON report +bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/ -f json -o bandit-report.json +``` + +### Run Type Checking + +```bash +# Check types with mypy +mypy scripts/ --ignore-missing-imports --no-implicit-optional +``` + +## GitHub Actions Workflow + +### Triggered On + +- **Push** to `main` or `develop` branches (when scripts change) +- **Pull Request** to `main` (when scripts change) +- Manual workflow dispatch + +### Jobs + +#### 1. Unit Tests (pytest) + +- **Runs on**: Ubuntu latest +- **Python versions**: 3.10, 3.11, 3.12 +- **What it does**: + - Installs dependencies from `requirements-dev.txt` + - Runs pytest with coverage reporting + - Uploads coverage to Codecov + - Archives test results and coverage HTML + +**Outcome**: If any test fails, the workflow fails (critical) + +#### 2. Code Quality (Ruff) + +- **Runs on**: Ubuntu latest +- **Python version**: 3.11 +- **What it does**: + - Checks code formatting with `ruff format` + - Runs linter with `ruff check` + - Reports issues but doesn't fail the workflow + +**Outcome**: Non-blocking (warning only) + +#### 3. Security Scan (Bandit) + +- **Runs on**: Ubuntu latest +- **Python version**: 3.11 +- **What it does**: + - Scans for security vulnerabilities + - Generates JSON report + - Uploads report as artifact + +**Outcome**: Non-blocking (warning only) + +#### 4. Type Checking (mypy) + +- **Runs on**: Ubuntu latest +- **Python version**: 3.11 +- **What it does**: + - Performs static type analysis + - Reports type mismatches + - Helps catch bugs early + +**Outcome**: Non-blocking (warning only) + +#### 5. Build EPUB + +- **Runs on**: Ubuntu latest +- **Depends on**: pytest, lint, security (all must pass) +- **What it does**: + - Builds the EPUB file using `scripts/build_epub.py` + - Verifies the EPUB was created successfully + - Uploads EPUB as artifact + +**Outcome**: If build fails, the workflow fails (critical) + +#### 6. Summary + +- **Runs on**: Ubuntu latest +- **Depends on**: All other jobs +- **What it does**: + - Generates workflow summary + - Lists all artifacts + - Reports overall status + +## Writing Tests + +### Test Structure + +Tests should be placed in `scripts/tests/` with names like `test_*.py`: + +```python +# scripts/tests/test_example.py +import pytest +from scripts.example_module import some_function + +def test_basic_functionality(): + """Test that some_function works correctly.""" + result = some_function("input") + assert result == "expected_output" + +def test_error_handling(): + """Test that some_function handles errors gracefully.""" + with pytest.raises(ValueError): + some_function("invalid_input") + +@pytest.mark.asyncio +async def test_async_function(): + """Test async functions.""" + result = await async_function() + assert result is not None +``` + +### Test Best Practices + +- **Use descriptive names**: `test_function_returns_correct_value()` +- **One assertion per test** (when possible): Easier to debug failures +- **Use fixtures** for reusable setup: See `scripts/tests/conftest.py` +- **Mock external services**: Use `unittest.mock` or `pytest-mock` +- **Test edge cases**: Empty inputs, None values, errors +- **Keep tests fast**: Avoid sleep() and external I/O +- **Use pytest markers**: `@pytest.mark.slow` for slow tests + +### Fixtures + +Common fixtures are defined in `scripts/tests/conftest.py`: + +```python +# Use fixtures in your tests +def test_something(tmp_path): + """tmp_path fixture provides temporary directory.""" + test_file = tmp_path / "test.txt" + test_file.write_text("content") + assert test_file.read_text() == "content" +``` + +## Coverage Reports + +### Local Coverage + +```bash +# Generate coverage report +pytest scripts/tests/ --cov=scripts --cov-report=html + +# Open the coverage report in your browser +open htmlcov/index.html +``` + +### Coverage Goals + +- **Minimum coverage**: 80% +- **Branch coverage**: Enabled +- **Focus areas**: Core functionality and error paths + +## Pre-commit Hooks + +The project uses pre-commit hooks to run checks automatically before commits: + +```bash +# Install pre-commit hooks +pre-commit install + +# Run hooks manually +pre-commit run --all-files + +# Skip hooks for a commit (not recommended) +git commit --no-verify +``` + +Configured hooks in `.pre-commit-config.yaml`: +- Ruff formatter +- Ruff linter +- Bandit security scanner +- YAML validation +- File size checks +- Merge conflict detection + +## Troubleshooting + +### Tests Pass Locally but Fail in CI + +Common causes: +1. **Python version difference**: CI uses 3.10, 3.11, 3.12 +2. **Missing dependencies**: Update `requirements-dev.txt` +3. **Platform differences**: Path separators, environment variables +4. **Flaky tests**: Tests that depend on timing or order + +Solution: +```bash +# Test with the same Python versions +uv python install 3.10 3.11 3.12 + +# Test with clean environment +rm -rf .venv +uv venv +uv pip install -r requirements-dev.txt +pytest scripts/tests/ +``` + +### Bandit Reports False Positives + +Some security warnings may be false positives. Configure in `pyproject.toml`: + +```toml +[tool.bandit] +exclude_dirs = ["scripts/tests"] +skips = ["B101"] # Skip assert_used warning +``` + +### Type Checking Too Strict + +Relax type checking for specific files: + +```python +# Add at the top of file +# type: ignore + +# Or for specific lines +some_dynamic_code() # type: ignore +``` + +## Continuous Integration Best Practices + +1. **Keep tests fast**: Each test should complete in <1 second +2. **Don't test external APIs**: Mock external services +3. **Test in isolation**: Each test should be independent +4. **Use clear assertions**: `assert x == 5` not `assert x` +5. **Handle async tests**: Use `@pytest.mark.asyncio` +6. **Generate reports**: Coverage, security, type checking + +## Resources + +- [pytest Documentation](https://docs.pytest.org/) +- [Ruff Documentation](https://docs.astral.sh/ruff/) +- [Bandit Documentation](https://bandit.readthedocs.io/) +- [mypy Documentation](https://mypy.readthedocs.io/) +- [GitHub Actions Documentation](https://docs.github.com/en/actions) + +## Contributing Tests + +When submitting a PR: + +1. **Write tests** for new functionality +2. **Run tests locally**: `pytest scripts/tests/ -v` +3. **Check coverage**: `pytest scripts/tests/ --cov=scripts` +4. **Run linting**: `ruff check scripts/` +5. **Security scan**: `bandit -r scripts/ --exclude scripts/tests/` +6. **Update documentation** if tests change + +Tests are required for all PRs! ๐Ÿงช + +--- + +For questions or issues with testing, open a GitHub issue or discussion. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ec767a7..79bc921 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -58,8 +58,42 @@ jobs: run: uv run bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/ test: - name: Tests + name: Unit Tests (Python ${{ matrix.python-version }}) runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ['3.10', '3.11', '3.12'] + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + + - name: Set up Python ${{ matrix.python-version }} + run: uv python install ${{ matrix.python-version }} + + - name: Create venv and install dependencies + run: | + uv venv + uv pip install -r requirements-dev.txt + + - name: Run pytest with coverage + run: uv run pytest scripts/tests/ -v --tb=short --cov=scripts --cov-report=xml --cov-report=term-missing + + - name: Upload coverage to Codecov + uses: codecov/codecov-action@v3 + with: + files: ./coverage.xml + flags: unittests + name: codecov-python-${{ matrix.python-version }} + fail_ci_if_error: false + + build: + name: Build EPUB + runs-on: ubuntu-latest + needs: [lint, security, test] steps: - name: Checkout code uses: actions/checkout@v4 @@ -73,22 +107,7 @@ jobs: - name: Create venv and install dependencies run: | uv venv - uv pip install -r requirements.txt - uv pip install pytest pytest-asyncio - - - name: Run Tests - run: uv run pytest scripts/tests/ -v --tb=short - - build: - name: Build EPUB - runs-on: ubuntu-latest - needs: [lint, security, test] - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Install uv - uses: astral-sh/setup-uv@v4 + uv pip install -r requirements-dev.txt - name: Build EPUB run: uv run scripts/build_epub.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..3b0a9ef --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,227 @@ +name: Automated Testing + +on: + push: + branches: [main, develop] + paths: + - 'scripts/**' + - '.github/workflows/test.yml' + - 'pyproject.toml' + - 'requirements*.txt' + pull_request: + branches: [main] + paths: + - 'scripts/**' + - '.github/workflows/test.yml' + - 'pyproject.toml' + - 'requirements*.txt' + +# Cancel in-progress runs for the same branch +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + pytest: + name: Unit Tests (Python ${{ matrix.python-version }}) + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ['3.10', '3.11', '3.12'] + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + + - name: Set up Python ${{ matrix.python-version }} + run: uv python install ${{ matrix.python-version }} + + - name: Create venv and install dependencies + run: | + uv venv + uv pip install -r requirements-dev.txt + + - name: Run pytest + run: uv run pytest scripts/tests/ -v --tb=short --cov=scripts --cov-report=xml --cov-report=html + continue-on-error: false + + - name: Upload coverage to Codecov + uses: codecov/codecov-action@v3 + with: + files: ./coverage.xml + flags: unittests + name: codecov-python-${{ matrix.python-version }} + fail_ci_if_error: false + verbose: true + + - name: Archive test results + if: always() + uses: actions/upload-artifact@v4 + with: + name: pytest-results-${{ matrix.python-version }} + path: | + coverage.xml + htmlcov/ + retention-days: 7 + + lint: + name: Code Quality (Lint & Format) + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + + - name: Set up Python + run: uv python install 3.11 + + - name: Create venv and install Ruff + run: | + uv venv + uv pip install ruff + + - name: Ruff Format Check + run: uv run ruff format --check scripts/ + continue-on-error: true + + - name: Ruff Lint Check + run: uv run ruff check scripts/ + continue-on-error: true + + security: + name: Security Scan + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + + - name: Set up Python + run: uv python install 3.11 + + - name: Create venv and install Bandit + run: | + uv venv + uv pip install "bandit[toml]" + + - name: Run Bandit Security Scan + run: uv run bandit -c pyproject.toml -r scripts/ --exclude scripts/tests/ -f json -o bandit-report.json + continue-on-error: true + + - name: Upload security report + uses: actions/upload-artifact@v4 + if: always() + with: + name: bandit-security-report + path: bandit-report.json + retention-days: 7 + + type-check: + name: Type Checking + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + + - name: Set up Python + run: uv python install 3.11 + + - name: Create venv and install mypy + run: | + uv venv + uv pip install -r requirements-dev.txt mypy + + - name: Run mypy + run: uv run mypy scripts/ --ignore-missing-imports --no-implicit-optional + continue-on-error: true + + build-epub: + name: Build EPUB Artifact + runs-on: ubuntu-latest + needs: [pytest, lint, security] + if: success() + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + + - name: Set up Python + run: uv python install 3.11 + + - name: Create venv and install dependencies + run: | + uv venv + uv pip install -r requirements-dev.txt + + - name: Build EPUB + run: uv run scripts/build_epub.py + + - name: Verify EPUB Created + run: | + if [ -f claude-howto-guide.epub ]; then + echo "โœ… EPUB built successfully" + ls -lh claude-howto-guide.epub + else + echo "โŒ EPUB file not found!" + exit 1 + fi + + - name: Upload EPUB Artifact + uses: actions/upload-artifact@v4 + with: + name: claude-howto-guide-epub + path: claude-howto-guide.epub + retention-days: 7 + compression-level: 0 + + summary: + name: Test Summary + needs: [pytest, lint, security, type-check, build-epub] + runs-on: ubuntu-latest + if: always() + + steps: + - name: Download all artifacts + uses: actions/download-artifact@v4 + + - name: Generate Summary + run: | + echo "# Test Summary" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "## Results" >> $GITHUB_STEP_SUMMARY + echo "- **Unit Tests**: ${{ needs.pytest.result }}" >> $GITHUB_STEP_SUMMARY + echo "- **Code Quality**: ${{ needs.lint.result }}" >> $GITHUB_STEP_SUMMARY + echo "- **Security Scan**: ${{ needs.security.result }}" >> $GITHUB_STEP_SUMMARY + echo "- **Type Checking**: ${{ needs.type-check.result }}" >> $GITHUB_STEP_SUMMARY + echo "- **EPUB Build**: ${{ needs.build-epub.result }}" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "## Artifacts" >> $GITHUB_STEP_SUMMARY + echo "- pytest-results-* (coverage and test results)" >> $GITHUB_STEP_SUMMARY + echo "- bandit-security-report (security scan JSON)" >> $GITHUB_STEP_SUMMARY + echo "- claude-howto-guide-epub (built EPUB file)" >> $GITHUB_STEP_SUMMARY + + - name: Check if all tests passed + if: | + needs.pytest.result != 'success' || + needs.build-epub.result != 'success' + run: | + echo "โŒ Critical tests failed" + exit 1 + + - name: All tests passed + if: success() + run: echo "โœ… All tests passed successfully!"