From 3557d791f54e916fe30750afb9866b3642a665bd Mon Sep 17 00:00:00 2001 From: Luong NGUYEN Date: Fri, 15 May 2026 08:55:29 +0200 Subject: [PATCH] feat(scripts): add static website generator from markdown sources (#85) (#121) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(scripts): add static website generator from markdown sources (#85) Generate an elegant, mobile-friendly static site from the existing tutorial markdown files. The markdown remains the single source of truth — `scripts/build_website.py` reads from the same `.md` files the EPUB builder uses, rewrites cross-references to site URLs, and rewrites references to non-markdown repo files (`.json`, `.sh`, `.py`) to GitHub blob URLs so users can jump to the source on github.com. Highlights: - Reuses the chapter ordering convention from `build_epub.py` - Anchor algorithm mirrors `check_cross_references.heading_to_anchor` for parity with the validator - Mermaid renders client-side via `mermaid.js` (no pre-render step) - Tailwind CSS via CDN; light/dark theme toggle; sidebar nav; in-page TOC; prev/next page navigation; mobile responsive - 27 unit + smoke tests covering anchors, link rewriting (including `` inside ``), Mermaid handling, and a full end-to-end build - GitHub Pages deploy workflow at `.github/workflows/pages.yml` Closes #85 * fix(website): use relative URLs in sidebar nav and avoid INDEX.html collision Two bugs found by local browser dogfooding: 1. **Sidebar nav broke from deep pages.** `build_navigation` emitted raw `output_url` values (site-root-relative) which made every sidebar link 404 from any page below the root. Moved the call inside the per-page render loop so each page gets nav links computed relative to its own URL — `01-slash-commands/index.html` from the root, `../01-slash-commands/...` from a depth-1 page, `../../01-slash-commands/...` from depth-2. 2. **`INDEX.md` overwrote `index.html`.** On case-insensitive filesystems (macOS/Windows), `INDEX.html` and `index.html` are the same file, so `INDEX.md` clobbered the rendered `README.md`. Added `_disambiguate_url` that detects case-insensitive collisions and suffixes the colliding page with its source stem (`INDEX-index.html`). Added 2 tests; full suite stays at 83 passed. * fix(scripts): skip URLs with port in localhost/127.0.0.1 skip list `check_links.is_skipped()` did an exact-match comparison against the host, so `http://localhost:8080` (used in scripts/README.md as a preview example) was not skipped and CI's link check tried to fetch it, which fails on the GitHub runner. Strip the port before comparing. * chore(scripts): drop vestigial mypy ignore_errors for build_website The override silenced all mypy errors for build_website, making the "mypy: clean" claim technically vacuous. Removing it shows mypy is actually clean — 0 issues on build_website after type annotations were added during PR review. * feat(website): self-host Tailwind, Mermaid, and Inter fonts Drop all third-party CDN dependencies from rendered pages. The site previously loaded Tailwind from cdn.tailwindcss.com (Play CDN — JIT compile in browser, marked not-for-production), Mermaid from cdn.jsdelivr.net, and Inter/JetBrains Mono from fonts.googleapis.com. Replace with a vendored toolchain: - scripts/vendor_assets.py downloads the Tailwind standalone CLI (Go binary, no Node toolchain), Mermaid's UMD bundle, and Google Fonts CSS + WOFF2 files. Cached under scripts/.vendor-cache/ (gitignored), refetched only when missing. - Tailwind compiles a per-build site/assets/tailwind.css with only the utility classes actually used by the rendered HTML. - Mermaid and font files land in site/assets/vendor/ and load via relative URLs. - Tailwind config + entry CSS live in scripts/website_templates/ alongside the Jinja template. - build_website grows a skip_vendor flag so the smoke test runs offline. - pre-commit mypy hook gets types-Markdown so it can resolve the same imports as the project venv. Verification: 86/86 pytest pass, ruff/mypy/bandit clean, full build produces a working site with zero external requests (verified in a headless browser — no console errors, no failed network calls, Mermaid diagrams render). * fix(website): use tree URLs for repo directory links (#85) * fix(website): include additional top-level docs (#85) --- .github/workflows/pages.yml | 52 ++ .gitignore | 5 + .pre-commit-config.yaml | 2 + scripts/README.md | 90 ++ scripts/build_website.py | 902 +++++++++++++++++++ scripts/check_links.py | 2 +- scripts/pyproject.toml | 1 + scripts/requirements.txt | 3 +- scripts/tests/test_build_website.py | 520 +++++++++++ scripts/vendor_assets.py | 209 +++++ scripts/website_templates/page.html.j2 | 253 ++++++ scripts/website_templates/site.css | 104 +++ scripts/website_templates/tailwind.config.js | 35 + scripts/website_templates/tailwind.input.css | 3 + 14 files changed, 2179 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/pages.yml create mode 100644 scripts/build_website.py create mode 100644 scripts/tests/test_build_website.py create mode 100644 scripts/vendor_assets.py create mode 100644 scripts/website_templates/page.html.j2 create mode 100644 scripts/website_templates/site.css create mode 100644 scripts/website_templates/tailwind.config.js create mode 100644 scripts/website_templates/tailwind.input.css diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 0000000..2aeed56 --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,52 @@ +name: Deploy Website to GitHub Pages + +on: + push: + branches: + - main + paths: + - '**/*.md' + - 'scripts/build_website.py' + - 'scripts/vendor_assets.py' + - 'scripts/website_templates/**' + - '.github/workflows/pages.yml' + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + name: Build static site + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + + - name: Build website + run: uv run scripts/build_website.py --verbose + + - name: Upload Pages artifact + uses: actions/upload-pages-artifact@v3 + with: + path: site + + deploy: + name: Deploy to GitHub Pages + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/.gitignore b/.gitignore index 054fbe0..cc14bf2 100644 --- a/.gitignore +++ b/.gitignore @@ -90,3 +90,8 @@ promo-video/ *.mp4 update-plan*.md +site/ +.gstack/ + +# Cached vendor binaries/assets for the website build (Tailwind CLI, Mermaid, fonts). +scripts/.vendor-cache/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 9b7f81a..8be75aa 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -60,6 +60,8 @@ repos: args: [--config-file, scripts/pyproject.toml] files: ^scripts/ exclude: ^scripts/tests/ + additional_dependencies: + - types-Markdown # Local doc quality hooks (mirrors CI checks — CI is a 2nd pass of these) - repo: local diff --git a/scripts/README.md b/scripts/README.md index 141b466..6016fb4 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -3,6 +3,19 @@ Claude How To +# Build Scripts + +This directory contains two generators that turn the tutorial markdown files +into distributable formats: + +- [**EPUB Builder**](#epub-builder-script) — `build_epub.py` +- [**Static Website Builder**](#static-website-builder) — `build_website.py` + +Both treat the `.md` files as the single source of truth — re-run the relevant +script after editing markdown to regenerate the output. + +--- + # EPUB Builder Script Build an EPUB ebook from the Claude How-To markdown files. @@ -118,3 +131,80 @@ Managed via PEP 723 inline script metadata: **Rate limiting**: Reduce concurrent requests with `--max-concurrent 3`. **Missing logo**: The script generates a text-only cover if `claude-howto-logo.png` is not found. + +--- + +# Static Website Builder + +Generate an elegant, mobile-friendly static website from the same markdown +files used by the EPUB build. The website is the rendered view; the `.md` +files remain the single source of truth. + +## Features + +- One HTML page per markdown source — internal `.md` links are rewritten to + the corresponding pages on the site +- References to non-markdown repo files (templates, scripts, JSON) become + GitHub blob URLs that open the source on github.com +- Mermaid diagrams render client-side via `mermaid.min.js`, served from the + built site (no CDN at runtime) +- Tailwind CSS compiled with the standalone CLI (Go binary, no Node.js) and + served from the built site — responsive layout with sidebar nav, in-page + TOC, dark mode toggle, and prev/next page navigation +- Inter + JetBrains Mono fonts are self-hosted alongside the CSS — no + third-party requests at page load +- Mirrors the EPUB curriculum order (`01-` … `10-` plus top-level docs) +- Hostable as plain static files — designed to deploy to GitHub Pages + +## Quick Start + +```bash +# Build the English website into ./site/ +uv run scripts/build_website.py + +# Preview locally +python -m http.server --directory site 8080 +# then open http://localhost:8080 +``` + +## Command-Line Options + +``` +usage: build_website.py [-h] [--root ROOT] [--output OUTPUT] + [--lang {en,vi,zh,ja,uk}] [--repo-url REPO_URL] + [--branch BRANCH] [--verbose] + +options: + --root, -r ROOT Source root (default: repo root) + --output, -o OUTPUT Output directory (default: /site) + --lang LANG Language to build: en | vi | zh | ja | uk + --repo-url URL GitHub repo for blob links (default: luongnv89/claude-howto) + --branch BRANCH Branch for blob links (default: main) + --verbose, -v Enable verbose logging +``` + +## GitHub Pages Deploy + +The repo ships a workflow at `.github/workflows/pages.yml` that builds the +site on every push to `main` (when any `.md` or generator file changes) and +publishes via `actions/deploy-pages`. Enable GitHub Pages in repo settings +with **Source: GitHub Actions** to activate it. + +## Architecture + +`build_website.py` reuses the chapter-ordering logic from `build_epub.py` and +ships HTML templates under `scripts/website_templates/`: + +- `page.html.j2` — per-page Jinja2 template with sidebar nav, TOC, prev/next +- `tailwind.config.js`, `tailwind.input.css` — config + entry CSS for the + Tailwind standalone CLI; the CLI scans the built HTML and produces + `site/assets/tailwind.css` with just the utilities actually used +- `site.css` — small layer of site-specific styles plus Pygments theme + +The Tailwind CLI binary, Mermaid bundle, and font files are downloaded on +first build into `scripts/.vendor-cache/` (gitignored) — see +`scripts/vendor_assets.py`. + +Heading anchors are generated using the exact algorithm in +`check_cross_references.heading_to_anchor`, so `#anchor` links validated by +the pre-commit hook resolve correctly on the rendered site. diff --git a/scripts/build_website.py b/scripts/build_website.py new file mode 100644 index 0000000..83abe3a --- /dev/null +++ b/scripts/build_website.py @@ -0,0 +1,902 @@ +#!/usr/bin/env -S uv run --script +# /// script +# dependencies = ["markdown", "beautifulsoup4", "jinja2"] +# /// +""" +Build a static website from the Claude How-To markdown files. + +Usage: + uv run scripts/build_website.py + uv run scripts/build_website.py --lang vi + uv run scripts/build_website.py --output site/ --verbose + +The website renders the existing markdown files as the single source of truth. +No content is duplicated — re-running this script regenerates the entire site +from the current state of the `.md` files. + +Output: + Creates `site/` (or the path passed to `--output`) containing one HTML page + per markdown source plus `assets/` with logos and copied images. + +Features: + - Renders the same chapter order as the EPUB build (curriculum order). + - Rewrites internal `.md` links to corresponding HTML pages on the site. + - Rewrites repo-file/folder references (`.json`, `.sh`, `.py`, etc.) to + GitHub source URLs so users can jump to the source on github.com. + - Self-hosted Tailwind CSS (compiled via standalone CLI), Inter font, and + `mermaid.min.js` — no third-party CDN scripts at runtime. + - Light/dark theme toggle, mobile-friendly responsive layout, sidebar. + - Hostable as plain static files (e.g. GitHub Pages). + +Vendor assets (Tailwind CLI binary, Mermaid bundle, font files) are downloaded +on first build and cached under `scripts/.vendor-cache/`. See +`scripts/vendor_assets.py` for details. +""" + +from __future__ import annotations + +import argparse +import html +import logging +import re +import shutil +import sys +from dataclasses import dataclass, field +from pathlib import Path + +import markdown +from bs4 import BeautifulSoup +from jinja2 import Environment, FileSystemLoader, select_autoescape + +# Make sibling script modules importable regardless of cwd. +sys.path.insert(0, str(Path(__file__).parent)) + +from vendor_assets import ( + build_tailwind_css, + fetch_fonts, + fetch_mermaid, + write_vendor_manifest, +) + +# ============================================================================= +# Configuration +# ============================================================================= + +REPO_URL = "https://github.com/luongnv89/claude-howto" +DEFAULT_BRANCH = "main" + +# Files/dirs that exist in the repo but should not appear on the site. +EXCLUDE_DIRS = { + ".git", + ".github", + ".venv", + "venv", + "env", + "node_modules", + "__pycache__", + ".pytest_cache", + ".ruff_cache", + "blog-posts", + "openspec", + "prompts", + ".agents", + "archive", + "local-progress", + "promo-video", + "slides", + ".gitissue", + ".asm-improver", + ".codex", + ".opencode", + ".claude", + "site", + "scripts", + "vi", + "zh", + "ja", + "uk", +} + +# Top-level markdown files that should not be rendered as standalone pages. +EXCLUDE_TOP_LEVEL = { + "CLAUDE.md", + "README.backup.md", +} + +EXCLUDE_TOP_LEVEL_PREFIXES = ("update-plan",) + +# Match the EPUB chapter ordering. +CHAPTER_ORDER: list[tuple[str, str]] = [ + ("README.md", "Introduction"), + ("LEARNING-ROADMAP.md", "Learning Roadmap"), + ("QUICK_REFERENCE.md", "Quick Reference"), + ("claude_concepts_guide.md", "Claude Concepts Guide"), + ("01-slash-commands", "Slash Commands"), + ("02-memory", "Memory"), + ("03-skills", "Skills"), + ("04-subagents", "Subagents"), + ("05-mcp", "MCP Protocol"), + ("06-hooks", "Hooks"), + ("07-plugins", "Plugins"), + ("08-checkpoints", "Checkpoints"), + ("09-advanced-features", "Advanced Features"), + ("10-cli", "CLI Reference"), + ("CATALOG.md", "Feature Catalog"), + ("INDEX.md", "Index"), + ("STYLE_GUIDE.md", "Style Guide"), + ("resources.md", "Resources"), +] + + +@dataclass +class WebsiteConfig: + """Configuration for the website builder.""" + + root_path: Path + output_path: Path + repo_url: str = REPO_URL + branch: str = DEFAULT_BRANCH + site_title: str = "Claude Code How-To Guide" + site_subtitle: str = "Master Claude Code in a Weekend" + language: str = "en" + + +@dataclass +class PageInfo: + """A single page generated for the site.""" + + source: Path # absolute path to the markdown source + rel_source: str # path relative to repo root (POSIX style) + output_url: str # site-relative URL, e.g. `01-slash-commands/index.html` + title: str + section: str # the chapter group name (e.g. "Slash Commands") + is_section_index: bool = False + + +@dataclass +class BuildState: + """Build-time state shared across page rendering.""" + + pages: list[PageInfo] = field(default_factory=list) + source_to_url: dict[str, str] = field(default_factory=dict) + + +# ============================================================================= +# Logging +# ============================================================================= + + +def setup_logging(verbose: bool = False) -> logging.Logger: + """Configure logging for the website builder.""" + level = logging.DEBUG if verbose else logging.INFO + logging.basicConfig( + level=level, + format="%(asctime)s - %(levelname)s - %(message)s", + datefmt="%H:%M:%S", + ) + return logging.getLogger("website_builder") + + +# ============================================================================= +# Anchor algorithm (mirrors check_cross_references.heading_to_anchor) +# ============================================================================= + + +def heading_to_anchor(heading: str) -> str: + """Convert a heading to a GitHub-style anchor. + + Mirrors `scripts/check_cross_references.heading_to_anchor` so the website + resolves the same `#anchor` references the validator accepts. + """ + heading = re.sub( + r"[\U0001F000-\U0001FFFF" + r"\U00002702-\U000027B0" + r"\U0000FE00-\U0000FE0F" + r"\U0000200D" + r"\U000000A9\U000000AE" + r"\U00002000-\U0000206F" + r"]", + "", + heading, + ) + anchor = re.sub(r"[^\w\s-]", "", heading.lower(), flags=re.UNICODE) + anchor = anchor.replace(" ", "-") + return anchor.rstrip("-") + + +# ============================================================================= +# Source discovery +# ============================================================================= + + +def is_excluded_dir(name: str) -> bool: + return name.startswith(".") or name in EXCLUDE_DIRS + + +def collect_folder_markdown(folder: Path) -> list[Path]: + """Return markdown files inside a chapter folder. + + Order: README.md first (if present), then other top-level markdown files + sorted alphabetically, then markdown files from non-hidden subfolders. + """ + files: list[Path] = [] + readme = folder / "README.md" + if readme.exists(): + files.append(readme) + + files.extend(md for md in sorted(folder.glob("*.md")) if md.name != "README.md") + + for sub in sorted(folder.iterdir()): + if sub.is_dir() and not is_excluded_dir(sub.name): + files.extend(collect_folder_markdown(sub)) + + return files + + +def is_excluded_top_level_markdown(name: str) -> bool: + return name in EXCLUDE_TOP_LEVEL or any( + name.startswith(prefix) for prefix in EXCLUDE_TOP_LEVEL_PREFIXES + ) + + +def derive_page_title(md_path: Path, default: str) -> str: + """Pick a page title from the first H1, falling back to filename / default.""" + try: + content = md_path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + return default + match = re.search(r"^#\s+(.+?)\s*$", content, flags=re.MULTILINE) + if match: + return match.group(1).strip() + return default + + +def source_to_site_url(rel_source: str) -> str: + """Map `01-slash-commands/README.md` → `01-slash-commands/index.html`.""" + if rel_source == "README.md": + return "index.html" + if rel_source.endswith("/README.md"): + return rel_source[: -len("README.md")] + "index.html" + if rel_source.endswith(".md"): + return rel_source[:-3] + ".html" + return rel_source + + +def _disambiguate_url(url: str, used_lower: set[str], rel_source: str) -> str: + """Avoid case-insensitive filesystem collisions (e.g. INDEX.html ↔ index.html). + + macOS/Windows treat `INDEX.html` and `index.html` as the same file. When + two source files would resolve to URLs that differ only in case, suffix + the second one with the source stem so both pages survive the build. + """ + if url.lower() not in used_lower: + return url + parent, sep, leaf = url.rpartition("/") + stem, dot, ext = leaf.rpartition(".") + src_stem = Path(rel_source).stem.lower() + candidate = f"{parent}{sep}{stem}-{src_stem}{dot}{ext}" + suffix = 2 + while candidate.lower() in used_lower: + candidate = f"{parent}{sep}{stem}-{src_stem}-{suffix}{dot}{ext}" + suffix += 1 + return candidate + + +def collect_pages(config: WebsiteConfig, logger: logging.Logger) -> BuildState: + """Walk the configured chapter order and produce a flat list of pages.""" + state = BuildState() + seen: set[str] = set() + used_urls: set[str] = set() + + for item, display_name in CHAPTER_ORDER: + item_path = config.root_path / item + if not item_path.exists(): + logger.debug(f"Skipping missing chapter target: {item}") + continue + + if item_path.is_file() and item_path.suffix == ".md": + if is_excluded_top_level_markdown(item) or item in seen: + continue + seen.add(item) + page_title = derive_page_title(item_path, display_name) + url = _disambiguate_url(source_to_site_url(item), used_urls, item) + used_urls.add(url.lower()) + state.pages.append( + PageInfo( + source=item_path, + rel_source=item, + output_url=url, + title=page_title, + section=display_name, + is_section_index=True, + ) + ) + elif item_path.is_dir(): + folder_files = collect_folder_markdown(item_path) + for md in folder_files: + rel = md.relative_to(config.root_path).as_posix() + if rel in seen: + continue + seen.add(rel) + is_index = md.name == "README.md" and md.parent == item_path + title = derive_page_title(md, display_name if is_index else md.stem) + url = _disambiguate_url(source_to_site_url(rel), used_urls, rel) + used_urls.add(url.lower()) + state.pages.append( + PageInfo( + source=md, + rel_source=rel, + output_url=url, + title=title, + section=display_name, + is_section_index=is_index, + ) + ) + else: + logger.warning(f"Chapter target is not a file or directory: {item}") + + for md in sorted(config.root_path.glob("*.md")): + rel = md.relative_to(config.root_path).as_posix() + if is_excluded_top_level_markdown(md.name) or rel in seen: + continue + seen.add(rel) + title_default = md.stem.replace("-", " ").replace("_", " ").title() + title = derive_page_title(md, title_default) + url = _disambiguate_url(source_to_site_url(rel), used_urls, rel) + used_urls.add(url.lower()) + state.pages.append( + PageInfo( + source=md, + rel_source=rel, + output_url=url, + title=title, + section="Additional Docs", + is_section_index=False, + ) + ) + + for page in state.pages: + state.source_to_url[page.rel_source] = page.output_url + + logger.info(f"Collected {len(state.pages)} pages across the curriculum") + return state + + +# ============================================================================= +# Link rewriting +# ============================================================================= + + +def is_external(href: str) -> bool: + return href.startswith(("http://", "https://", "mailto:", "tel:")) + + +def relative_link(from_url: str, to_url: str, anchor: str = "") -> str: + """Build a relative URL from `from_url` to `to_url` (both site-relative).""" + if from_url == to_url: + return anchor or "" + from_parts = from_url.split("/")[:-1] + to_parts = to_url.split("/") + + common = 0 + for a, b in zip(from_parts, to_parts, strict=False): + if a != b: + break + common += 1 + + ups = [".."] * (len(from_parts) - common) + downs = to_parts[common:] + parts = ups + downs + rel = "/".join(parts) if parts else to_parts[-1] + return rel + anchor + + +def _resolve_repo_relative(href: str, page_dir: Path, root_path: Path) -> str | None: + """Resolve `href` relative to `page_dir` and return its repo-relative path.""" + resolved = (page_dir / href).resolve() + try: + rel_to_root = resolved.relative_to(root_path) + except ValueError: + return None + return rel_to_root.as_posix() + + +def _github_source_url( + config: WebsiteConfig, rel_str: str, *, is_dir: bool, anchor: str = "" +) -> str: + kind = "tree" if is_dir else "blob" + if rel_str == ".": + return f"{config.repo_url}/{kind}/{config.branch}{anchor}" + return f"{config.repo_url}/{kind}/{config.branch}/{rel_str}{anchor}" + + +def _rewrite_anchor( + a: object, + page: PageInfo, + state: BuildState, + config: WebsiteConfig, + logger: logging.Logger, +) -> None: + """Rewrite a single `` to its site URL or GitHub source URL.""" + href = a.get("href", "") # type: ignore[attr-defined] + if not href or is_external(href) or href.startswith("#"): + return + + anchor = "" + if "#" in href: + href, anchor_part = href.split("#", 1) + anchor = "#" + anchor_part + if not href: + return + + rel_str = _resolve_repo_relative(href, page.source.parent, config.root_path) + if rel_str is None: + logger.debug(f"Link outside repo skipped: {href} (in {page.rel_source})") + return + + candidates = [rel_str] + resolved = (page.source.parent / href).resolve() + if not rel_str.endswith(".md") and resolved.is_dir(): + candidates.append(rel_str + "/README.md") + + for candidate in candidates: + if candidate in state.source_to_url: + a["href"] = relative_link( # type: ignore[index] + page.output_url, state.source_to_url[candidate], anchor + ) + return + + github_url = _github_source_url( + config, rel_str, is_dir=resolved.is_dir(), anchor=anchor + ) + a["href"] = github_url # type: ignore[index] + a["target"] = "_blank" # type: ignore[index] + a["rel"] = "noopener noreferrer" # type: ignore[index] + + +def _rewrite_asset_ref( + element: object, + attr: str, + raw_value: str, + page: PageInfo, + config: WebsiteConfig, +) -> None: + """Rewrite a single asset reference (img.src / source.srcset) to assets/.""" + if not raw_value or is_external(raw_value): + return + first = raw_value.split(",", 1)[0].strip().split(" ", 1)[0] + if not first or is_external(first): + return + rel_str = _resolve_repo_relative(first, page.source.parent, config.root_path) + if rel_str is None: + return + target = "assets/" + rel_str + element[attr] = relative_link(page.output_url, target) # type: ignore[index] + + +def rewrite_links( + html_content: str, + page: PageInfo, + state: BuildState, + config: WebsiteConfig, + logger: logging.Logger, +) -> str: + """Rewrite anchor/image hrefs so links resolve correctly on the site.""" + soup = BeautifulSoup(html_content, "html.parser") + + for a in soup.find_all("a"): + _rewrite_anchor(a, page, state, config, logger) + + for img in soup.find_all("img"): + _rewrite_asset_ref(img, "src", img.get("src", ""), page, config) + if img.get("src"): + img["loading"] = "lazy" + + for source in soup.find_all("source"): + _rewrite_asset_ref(source, "srcset", source.get("srcset", ""), page, config) + + return str(soup) + + +# ============================================================================= +# Mermaid handling +# ============================================================================= + + +MERMAID_BLOCK_RE = re.compile(r"```mermaid\n(.*?)```", re.DOTALL) + + +def replace_mermaid_blocks(md_content: str) -> str: + """Replace ```mermaid``` fences with `
` for client-side render."""
+
+    def _replace(match: re.Match[str]) -> str:
+        code = match.group(1)
+        return f'
{html.escape(code)}
' + + return MERMAID_BLOCK_RE.sub(_replace, md_content) + + +# ============================================================================= +# Markdown rendering +# ============================================================================= + + +def normalise_heading_ids(html_content: str) -> str: + """Force GitHub-style anchor ids on every heading. + + `python-markdown`'s `toc` extension generates its own slug; we re-write them + so they match `check_cross_references.heading_to_anchor`. + """ + soup = BeautifulSoup(html_content, "html.parser") + used: dict[str, int] = {} + for level in ("h1", "h2", "h3", "h4", "h5", "h6"): + for h in soup.find_all(level): + text = h.get_text(strip=True) + anchor = heading_to_anchor(text) + if not anchor: + continue + count = used.get(anchor, 0) + final = anchor if count == 0 else f"{anchor}-{count}" + used[anchor] = count + 1 + h["id"] = final + return str(soup) + + +def extract_toc(html_content: str) -> list[dict[str, str]]: + """Pull H2/H3 headings into a flat list for in-page navigation.""" + soup = BeautifulSoup(html_content, "html.parser") + toc: list[dict[str, str]] = [] + for h in soup.find_all(["h2", "h3"]): + anchor = h.get("id") + if not anchor: + continue + toc.append( + { + "level": h.name, + "text": h.get_text(strip=True), + "anchor": anchor, + } + ) + return toc + + +def render_markdown(md_content: str) -> str: + """Convert markdown to HTML using the same extensions as the EPUB build.""" + md_content = replace_mermaid_blocks(md_content) + html_content = markdown.markdown( + md_content, + extensions=["tables", "fenced_code", "codehilite", "toc"], + extension_configs={"codehilite": {"guess_lang": False}}, + ) + return normalise_heading_ids(html_content) + + +# ============================================================================= +# Asset copying +# ============================================================================= + + +ASSET_EXTENSIONS = { + ".svg", + ".png", + ".jpg", + ".jpeg", + ".gif", + ".webp", + ".ico", +} + + +def copy_assets( + config: WebsiteConfig, state: BuildState, logger: logging.Logger +) -> None: + """Copy images referenced by any rendered page into `/assets/`.""" + assets_dir = config.output_path / "assets" + assets_dir.mkdir(parents=True, exist_ok=True) + copied: set[Path] = set() + + for page in state.pages: + try: + content = page.source.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + continue + for match in re.finditer(r"!\[[^\]]*\]\(([^)]+)\)", content): + src = match.group(1).split(" ", 1)[0] + if is_external(src): + continue + resolved = (page.source.parent / src).resolve() + if not resolved.exists() or not resolved.is_file(): + continue + if resolved.suffix.lower() not in ASSET_EXTENSIONS: + continue + try: + rel = resolved.relative_to(config.root_path) + except ValueError: + continue + target = assets_dir / rel + if target in copied: + continue + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(resolved, target) + copied.add(target) + + # Always copy the logo set so the site header can reuse it. + logo_dir = config.root_path / "resources" / "logos" + if logo_dir.exists(): + for logo in logo_dir.glob("*.svg"): + try: + rel = logo.relative_to(config.root_path) + except ValueError: + continue + target = assets_dir / rel + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(logo, target) + copied.add(target) + + logger.info(f"Copied {len(copied)} asset file(s) into {assets_dir}") + + +# ============================================================================= +# Page rendering +# ============================================================================= + + +def build_navigation(state: BuildState, current_url: str) -> list[dict[str, object]]: + """Group pages into the sidebar navigation tree, rooted at `current_url`. + + Each item's `url` is a relative URL from `current_url` so the same nav + structure works from any page depth (top-level vs nested chapter folders). + """ + sections: list[dict[str, object]] = [] + section_map: dict[str, dict[str, object]] = {} + + for page in state.pages: + section = section_map.get(page.section) + if section is None: + section = { + "name": page.section, + "items": [], + } + section_map[page.section] = section + sections.append(section) + + items = section["items"] + assert isinstance(items, list) + items.append( + { + "title": page.title, + "url": relative_link(current_url, page.output_url), + "is_current": page.output_url == current_url, + "is_index": page.is_section_index, + } + ) + + return sections + + +def render_pages( + config: WebsiteConfig, + state: BuildState, + env: Environment, + logger: logging.Logger, +) -> None: + """Render each markdown page into `/`.""" + template = env.get_template("page.html.j2") + total = len(state.pages) + + for idx, page in enumerate(state.pages): + nav = build_navigation(state, page.output_url) + try: + md_content = page.source.read_text(encoding="utf-8") + except UnicodeDecodeError as e: + raise RuntimeError(f"Failed to read {page.source}: {e}") from e + + html_content = render_markdown(md_content) + html_content = rewrite_links(html_content, page, state, config, logger) + toc = extract_toc(html_content) + + prev_page = state.pages[idx - 1] if idx > 0 else None + next_page = state.pages[idx + 1] if idx < total - 1 else None + + rendered = template.render( + site_title=config.site_title, + site_subtitle=config.site_subtitle, + page_title=page.title, + section=page.section, + content=html_content, + toc=toc, + nav=nav, + current_url=page.output_url, + base_path=relative_link(page.output_url, "index.html").rsplit( + "index.html", 1 + )[0], + assets_prefix=relative_link(page.output_url, "assets/").rsplit( + "assets/", 1 + )[0] + + "assets/", + prev_page=( + { + "title": prev_page.title, + "url": relative_link(page.output_url, prev_page.output_url), + } + if prev_page + else None + ), + next_page=( + { + "title": next_page.title, + "url": relative_link(page.output_url, next_page.output_url), + } + if next_page + else None + ), + github_source_url=f"{config.repo_url}/blob/{config.branch}/{page.rel_source}", + repo_url=config.repo_url, + ) + + out_file = config.output_path / page.output_url + out_file.parent.mkdir(parents=True, exist_ok=True) + out_file.write_text(rendered, encoding="utf-8") + logger.debug(f"Wrote {out_file}") + + logger.info(f"Rendered {total} HTML page(s) into {config.output_path}") + + +# ============================================================================= +# Build orchestration +# ============================================================================= + + +def build_website( + config: WebsiteConfig, + logger: logging.Logger, + *, + skip_vendor: bool = False, +) -> Path: + """Generate the full static site at `config.output_path`. + + ``skip_vendor=True`` skips the Tailwind CLI compile and the Mermaid/font + downloads — used by tests that don't need network access. + """ + if not config.root_path.is_dir(): + raise RuntimeError(f"Root path is not a directory: {config.root_path}") + + config.output_path.mkdir(parents=True, exist_ok=True) + + template_dir = Path(__file__).parent / "website_templates" + env = Environment( + loader=FileSystemLoader(str(template_dir)), + autoescape=select_autoescape(["html", "xml"]), + trim_blocks=True, + lstrip_blocks=True, + ) + + state = collect_pages(config, logger) + if not state.pages: + raise RuntimeError( + f"No markdown pages found under {config.root_path}; " + "check that the chapter directories exist." + ) + + render_pages(config, state, env, logger) + copy_assets(config, state, logger) + + # Self-hosted vendor assets — drop all CDN dependencies. + assets_dir = config.output_path / "assets" + css_source = template_dir / "site.css" + if css_source.exists(): + css_target = assets_dir / "site.css" + css_target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(css_source, css_target) + + if skip_vendor: + logger.info("Skipping vendor asset fetch (skip_vendor=True)") + else: + vendor_dir = assets_dir / "vendor" + fetch_mermaid(vendor_dir / "mermaid", logger) + fonts_css = fetch_fonts(vendor_dir / "fonts", logger) + # Run Tailwind LAST so it can scan the rendered HTML for class usage. + build_tailwind_css( + output_css=assets_dir / "tailwind.css", + template_dir=template_dir, + site_dir=config.output_path, + logger=logger, + ) + fonts_files = vendor_dir / "fonts" / "files" + write_vendor_manifest( + vendor_dir, + fonts_count=sum(1 for _ in fonts_files.iterdir()) + if fonts_files.exists() + else 0, + ) + logger.debug(f"Fonts CSS: {fonts_css}") + + logger.info(f"Website build complete: {config.output_path}") + return config.output_path + + +# ============================================================================= +# CLI +# ============================================================================= + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Build a static website from Claude How-To markdown files." + ) + parser.add_argument( + "--root", + "-r", + type=Path, + default=None, + help="Root directory containing markdown files (default: repo root)", + ) + parser.add_argument( + "--output", + "-o", + type=Path, + default=None, + help="Output directory for the generated site (default: /site)", + ) + parser.add_argument( + "--lang", + type=str, + default="en", + choices=["en", "vi", "zh", "ja", "uk"], + help="Language code for the source tree (default: en — root markdown)", + ) + parser.add_argument( + "--repo-url", + type=str, + default=REPO_URL, + help=f"GitHub repository URL for blob links (default: {REPO_URL})", + ) + parser.add_argument( + "--branch", + type=str, + default=DEFAULT_BRANCH, + help=f"Branch name for GitHub blob links (default: {DEFAULT_BRANCH})", + ) + parser.add_argument( + "--verbose", "-v", action="store_true", help="Enable verbose logging" + ) + + args = parser.parse_args() + + repo_root = (args.root or Path(__file__).parent.parent).resolve() + + lang_root_map = { + "en": repo_root, + "vi": repo_root / "vi", + "zh": repo_root / "zh", + "ja": repo_root / "ja", + "uk": repo_root / "uk", + } + source_root = lang_root_map[args.lang].resolve() + + default_output = repo_root / ("site" if args.lang == "en" else f"site-{args.lang}") + output_path = (args.output or default_output).resolve() + + logger = setup_logging(args.verbose) + config = WebsiteConfig( + root_path=source_root, + output_path=output_path, + repo_url=args.repo_url, + branch=args.branch, + language=args.lang, + ) + + try: + build_website(config, logger) + print(f"Successfully built website at: {output_path}") + return 0 + except (OSError, RuntimeError) as exc: + logger.error(f"Build failed: {exc}") + return 1 + except KeyboardInterrupt: + logger.warning("Build interrupted by user") + return 130 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/check_links.py b/scripts/check_links.py index 4424907..487d623 100644 --- a/scripts/check_links.py +++ b/scripts/check_links.py @@ -53,7 +53,7 @@ URL_RE = re.compile(r"https?://[a-zA-Z0-9][a-zA-Z0-9\-._~:/?#\[\]@!$&'()*+,;=%]+ def is_skipped(url: str) -> bool: try: - domain = url.split("/")[2] + domain = url.split("/", 3)[2].split(":", 1)[0] except IndexError: return True # malformed URL if any(skip == domain or domain.endswith("." + skip) for skip in SKIP_DOMAINS): diff --git a/scripts/pyproject.toml b/scripts/pyproject.toml index b1922d1..b8bcef3 100644 --- a/scripts/pyproject.toml +++ b/scripts/pyproject.toml @@ -12,6 +12,7 @@ dependencies = [ "httpx", "pillow", "tenacity", + "jinja2", ] [project.optional-dependencies] diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 2fc7f46..1b36857 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,7 +1,8 @@ -# Core dependencies for build_epub.py +# Core dependencies for build_epub.py and build_website.py ebooklib==0.18 markdown==3.7 beautifulsoup4==4.12.3 httpx==0.28.1 pillow==11.1.0 tenacity==9.0.0 +jinja2==3.1.4 diff --git a/scripts/tests/test_build_website.py b/scripts/tests/test_build_website.py new file mode 100644 index 0000000..5abad84 --- /dev/null +++ b/scripts/tests/test_build_website.py @@ -0,0 +1,520 @@ +"""Tests for the static website builder.""" + +from __future__ import annotations + +import logging +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from build_website import ( + BuildState, + PageInfo, + WebsiteConfig, + _disambiguate_url, + build_website, + collect_folder_markdown, + collect_pages, + derive_page_title, + heading_to_anchor, + is_excluded_dir, + is_excluded_top_level_markdown, + relative_link, + render_markdown, + replace_mermaid_blocks, + rewrite_links, + source_to_site_url, +) + +# ============================================================================= +# Fixtures +# ============================================================================= + + +@pytest.fixture +def site_root(tmp_path: Path) -> Path: + """Create a minimal repo-like tree the builder can render.""" + (tmp_path / "README.md").write_text( + "\n" + ' \n' + ' Claude How To\n' + "\n\n" + "# Home Page\n\nWelcome. See [Slash Commands](01-slash-commands/README.md).\n" + "Also check [script](scripts/build.sh) and the [logo](resources/logos/logo.svg).\n" + ) + (tmp_path / "LEARNING-ROADMAP.md").write_text( + "# Learning Roadmap\n\nLink back to [Home](README.md#home-page).\n" + ) + (tmp_path / "CONTRIBUTING.md").write_text("# Contributing\n\nHelp improve docs.") + (tmp_path / "CLAUDE.md").write_text("# Internal Agent Notes\n") + (tmp_path / "update-plan-2026-05-02.md").write_text("# Temporary Plan\n") + + sc = tmp_path / "01-slash-commands" + sc.mkdir() + (sc / "README.md").write_text( + "# Slash Commands\n\nMermaid time:\n\n```mermaid\nflowchart LR\nA-->B\n```\n\n" + "See [example](example.md).\n" + ) + (sc / "example.md").write_text("# Example\n\nGo back to [overview](README.md).\n") + + # Non-markdown repo files + scripts_dir = tmp_path / "scripts" + scripts_dir.mkdir() + (scripts_dir / "build.sh").write_text("#!/bin/bash\necho hi\n") + + logos = tmp_path / "resources" / "logos" + logos.mkdir(parents=True) + (logos / "logo.svg").write_text("") + (logos / "claude-howto-logo.svg").write_text("") + (logos / "claude-howto-logo-dark.svg").write_text("") + + return tmp_path + + +@pytest.fixture +def logger() -> logging.Logger: + return logging.getLogger("test_build_website") + + +# ============================================================================= +# heading_to_anchor parity with check_cross_references +# ============================================================================= + + +class TestHeadingToAnchor: + def test_simple_title(self) -> None: + assert heading_to_anchor("Hello World") == "hello-world" + + def test_punctuation_removed(self) -> None: + assert heading_to_anchor("What's Next?") == "whats-next" + + def test_unicode_preserved(self) -> None: + assert heading_to_anchor("Hướng dẫn") == "hướng-dẫn" + + def test_emoji_stripped(self) -> None: + # The validator strips emoji; we mirror that exactly. + assert heading_to_anchor("🔥 Trending") == "-trending" + + +# ============================================================================= +# source_to_site_url +# ============================================================================= + + +class TestSourceToSiteUrl: + def test_root_readme_maps_to_index(self) -> None: + assert source_to_site_url("README.md") == "index.html" + + def test_folder_readme_maps_to_folder_index(self) -> None: + assert ( + source_to_site_url("01-slash-commands/README.md") + == "01-slash-commands/index.html" + ) + + def test_other_markdown_uses_html_extension(self) -> None: + assert ( + source_to_site_url("01-slash-commands/example.md") + == "01-slash-commands/example.html" + ) + + +class TestDisambiguateUrl: + def test_no_collision_passes_through(self) -> None: + used: set[str] = {"foo.html"} + assert _disambiguate_url("bar.html", used, "bar.md") == "bar.html" + + def test_case_insensitive_collision_disambiguated(self) -> None: + # README → index.html lands first; INDEX.md (case-insensitive collision) + # must get a suffix on macOS / Windows filesystems. + used: set[str] = {"index.html"} + result = _disambiguate_url("INDEX.html", used, "INDEX.md") + assert result.lower() != "index.html" + assert result.endswith(".html") + + +# ============================================================================= +# relative_link +# ============================================================================= + + +class TestRelativeLink: + def test_same_directory(self) -> None: + assert relative_link("01/index.html", "01/example.html") == "example.html" + + def test_anchor_appended(self) -> None: + assert ( + relative_link("01/index.html", "02/index.html", "#intro") + == "../02/index.html#intro" + ) + + def test_self_link_returns_anchor_only(self) -> None: + assert relative_link("01/index.html", "01/index.html", "#section") == "#section" + + def test_parent_directory(self) -> None: + assert relative_link("01/index.html", "index.html") == "../index.html" + + +# ============================================================================= +# is_excluded_dir +# ============================================================================= + + +class TestIsExcludedDir: + def test_hidden_dirs_excluded(self) -> None: + assert is_excluded_dir(".git") is True + + def test_known_dir_excluded(self) -> None: + assert is_excluded_dir("node_modules") is True + + def test_chapter_dir_kept(self) -> None: + assert is_excluded_dir("01-slash-commands") is False + + +class TestIsExcludedTopLevelMarkdown: + def test_internal_agent_file_excluded(self) -> None: + assert is_excluded_top_level_markdown("CLAUDE.md") is True + + def test_temporary_update_plan_excluded(self) -> None: + assert is_excluded_top_level_markdown("update-plan-2026-05-02.md") is True + + def test_project_doc_included(self) -> None: + assert is_excluded_top_level_markdown("CONTRIBUTING.md") is False + + +# ============================================================================= +# collect_folder_markdown +# ============================================================================= + + +class TestCollectFolderMarkdown: + def test_readme_first(self, tmp_path: Path) -> None: + (tmp_path / "b.md").write_text("# B") + (tmp_path / "README.md").write_text("# Readme") + (tmp_path / "a.md").write_text("# A") + files = collect_folder_markdown(tmp_path) + assert [f.name for f in files] == ["README.md", "a.md", "b.md"] + + def test_skips_hidden_subdirs(self, tmp_path: Path) -> None: + (tmp_path / "README.md").write_text("# R") + hidden = tmp_path / ".cache" + hidden.mkdir() + (hidden / "junk.md").write_text("# junk") + files = collect_folder_markdown(tmp_path) + assert [f.name for f in files] == ["README.md"] + + +class TestCollectPages: + def test_additional_top_level_docs_are_collected( + self, site_root: Path, logger: logging.Logger + ) -> None: + state = collect_pages( + WebsiteConfig(root_path=site_root, output_path=site_root / "site"), logger + ) + assert "CONTRIBUTING.md" in state.source_to_url + assert state.source_to_url["CONTRIBUTING.md"] == "CONTRIBUTING.html" + assert "CLAUDE.md" not in state.source_to_url + assert "update-plan-2026-05-02.md" not in state.source_to_url + + +# ============================================================================= +# derive_page_title +# ============================================================================= + + +class TestDerivePageTitle: + def test_uses_h1(self, tmp_path: Path) -> None: + f = tmp_path / "f.md" + f.write_text("Some intro\n# The Title\nBody") + assert derive_page_title(f, "Default") == "The Title" + + def test_falls_back_when_no_h1(self, tmp_path: Path) -> None: + f = tmp_path / "f.md" + f.write_text("No heading here") + assert derive_page_title(f, "Default") == "Default" + + +# ============================================================================= +# replace_mermaid_blocks +# ============================================================================= + + +class TestReplaceMermaidBlocks: + def test_replaces_fence(self) -> None: + md = "Before\n\n```mermaid\nflowchart LR\nA-->B\n```\n\nAfter" + out = replace_mermaid_blocks(md) + assert '
' in out
+        assert "flowchart LR" in out
+        assert "```mermaid" not in out
+
+    def test_escapes_html(self) -> None:
+        md = "```mermaid\nA --> B\n```\n"
+        out = replace_mermaid_blocks(md)
+        assert "<C>" in out
+
+
+# =============================================================================
+# render_markdown
+# =============================================================================
+
+
+class TestRenderMarkdown:
+    def test_heading_gets_github_anchor(self) -> None:
+        html_content = render_markdown("# Hello World\n\nBody")
+        assert 'id="hello-world"' in html_content
+
+    def test_duplicate_headings_get_suffix(self) -> None:
+        html_content = render_markdown("# Hi\n\n# Hi\n")
+        assert 'id="hi"' in html_content
+        assert 'id="hi-1"' in html_content
+
+
+# =============================================================================
+# rewrite_links
+# =============================================================================
+
+
+class TestRewriteLinks:
+    def _state(self) -> BuildState:
+        state = BuildState()
+        state.source_to_url = {
+            "README.md": "index.html",
+            "01-slash-commands/README.md": "01-slash-commands/index.html",
+        }
+        return state
+
+    def _config(self, root: Path) -> WebsiteConfig:
+        return WebsiteConfig(
+            root_path=root,
+            output_path=root / "out",
+            repo_url="https://github.com/example/repo",
+            branch="main",
+        )
+
+    def test_internal_markdown_link_rewritten(
+        self, tmp_path: Path, logger: logging.Logger
+    ) -> None:
+        (tmp_path / "README.md").write_text("# Home")
+        (tmp_path / "01-slash-commands").mkdir()
+        (tmp_path / "01-slash-commands" / "README.md").write_text("# Slash")
+        page = PageInfo(
+            source=tmp_path / "README.md",
+            rel_source="README.md",
+            output_url="index.html",
+            title="Home",
+            section="Introduction",
+            is_section_index=True,
+        )
+        html_in = 'go'
+        out = rewrite_links(
+            html_in, page, self._state(), self._config(tmp_path), logger
+        )
+        assert "01-slash-commands/index.html" in out
+        assert ".md" not in out
+
+    def test_anchor_preserved(self, tmp_path: Path, logger: logging.Logger) -> None:
+        (tmp_path / "README.md").write_text("# Home")
+        (tmp_path / "01-slash-commands").mkdir()
+        (tmp_path / "01-slash-commands" / "README.md").write_text("# Slash")
+        page = PageInfo(
+            source=tmp_path / "README.md",
+            rel_source="README.md",
+            output_url="index.html",
+            title="Home",
+            section="Introduction",
+            is_section_index=True,
+        )
+        html_in = 'go'
+        out = rewrite_links(
+            html_in, page, self._state(), self._config(tmp_path), logger
+        )
+        assert "#run" in out
+
+    def test_non_markdown_link_uses_github_blob(
+        self, tmp_path: Path, logger: logging.Logger
+    ) -> None:
+        (tmp_path / "scripts").mkdir()
+        (tmp_path / "scripts" / "build.sh").write_text("#!/bin/bash")
+        (tmp_path / "README.md").write_text("# Home")
+        page = PageInfo(
+            source=tmp_path / "README.md",
+            rel_source="README.md",
+            output_url="index.html",
+            title="Home",
+            section="Introduction",
+            is_section_index=True,
+        )
+        html_in = 'script'
+        out = rewrite_links(
+            html_in, page, self._state(), self._config(tmp_path), logger
+        )
+        assert "github.com/example/repo/blob/main/scripts/build.sh" in out
+        assert 'target="_blank"' in out
+
+    def test_repo_directory_link_uses_github_tree(
+        self, tmp_path: Path, logger: logging.Logger
+    ) -> None:
+        (tmp_path / "scripts").mkdir()
+        (tmp_path / "README.md").write_text("# Home")
+        page = PageInfo(
+            source=tmp_path / "README.md",
+            rel_source="README.md",
+            output_url="index.html",
+            title="Home",
+            section="Introduction",
+            is_section_index=True,
+        )
+        html_in = 'scripts'
+        out = rewrite_links(
+            html_in, page, self._state(), self._config(tmp_path), logger
+        )
+        assert "github.com/example/repo/tree/main/scripts" in out
+        assert "github.com/example/repo/blob/main/scripts" not in out
+
+    def test_repo_root_link_uses_github_tree(
+        self, tmp_path: Path, logger: logging.Logger
+    ) -> None:
+        (tmp_path / "README.md").write_text("# Home")
+        page = PageInfo(
+            source=tmp_path / "README.md",
+            rel_source="README.md",
+            output_url="index.html",
+            title="Home",
+            section="Introduction",
+            is_section_index=True,
+        )
+        html_in = 'repo root'
+        out = rewrite_links(
+            html_in, page, self._state(), self._config(tmp_path), logger
+        )
+        assert "github.com/example/repo/tree/main" in out
+        assert "github.com/example/repo/blob/main/." not in out
+
+    def test_external_link_left_alone(
+        self, tmp_path: Path, logger: logging.Logger
+    ) -> None:
+        page = PageInfo(
+            source=tmp_path / "README.md",
+            rel_source="README.md",
+            output_url="index.html",
+            title="Home",
+            section="Introduction",
+            is_section_index=True,
+        )
+        (tmp_path / "README.md").write_text("# Home")
+        html_in = 'site'
+        out = rewrite_links(
+            html_in, page, self._state(), self._config(tmp_path), logger
+        )
+        assert 'href="https://anthropic.com"' in out
+
+
+# =============================================================================
+# Full build smoke test
+# =============================================================================
+
+
+class TestBuildWebsite:
+    def test_smoke_build(self, site_root: Path, logger: logging.Logger) -> None:
+        out_dir = site_root / "site"
+        config = WebsiteConfig(
+            root_path=site_root,
+            output_path=out_dir,
+            repo_url="https://github.com/example/repo",
+            branch="main",
+        )
+
+        build_website(config, logger, skip_vendor=True)
+
+        # Index page rendered
+        index = out_dir / "index.html"
+        assert index.exists()
+        index_html = index.read_text(encoding="utf-8")
+        assert "Home Page" in index_html
+        assert "01-slash-commands/index.html" in index_html
+        assert "CONTRIBUTING.html" in index_html
+        # Non-markdown link rewritten to GitHub
+        assert "github.com/example/repo/blob/main/scripts/build.sh" in index_html
+        #  inside  rewritten to point at assets/
+        assert 'srcset="assets/resources/logos/' in index_html
+
+        # Additional top-level docs rendered
+        assert (out_dir / "CONTRIBUTING.html").exists()
+        assert not (out_dir / "CLAUDE.html").exists()
+        assert not (out_dir / "update-plan-2026-05-02.html").exists()
+
+        # Folder index rendered
+        sc_index = out_dir / "01-slash-commands" / "index.html"
+        assert sc_index.exists()
+        sc_html = sc_index.read_text(encoding="utf-8")
+        # Mermaid block became a 
 block
+        assert '
' in sc_html
+        # Sibling markdown link rewritten
+        assert "example.html" in sc_html
+
+        # Example page rendered
+        example_page = out_dir / "01-slash-commands" / "example.html"
+        assert example_page.exists()
+        example_html = example_page.read_text(encoding="utf-8")
+        # Back-link to README rewrites to index.html
+        assert "index.html" in example_html
+
+        # Stylesheet copied into assets
+        assert (out_dir / "assets" / "site.css").exists()
+        # Logo copied into assets
+        assert (out_dir / "assets" / "resources" / "logos" / "logo.svg").exists()
+        # Template no longer references third-party CDNs.
+        for hostile in (
+            "cdn.tailwindcss.com",
+            "cdn.jsdelivr.net",
+            "fonts.googleapis.com",
+        ):
+            assert (
+                hostile not in index_html
+            ), f"Built HTML still references {hostile} — CDN should be self-hosted"
+
+
+# =============================================================================
+# vendor_assets module smoke test
+# =============================================================================
+
+
+class TestVendorAssets:
+    def test_module_exports(self) -> None:
+        """vendor_assets exposes the API build_website depends on."""
+        import vendor_assets
+
+        for attr in (
+            "build_tailwind_css",
+            "fetch_mermaid",
+            "fetch_fonts",
+            "write_vendor_manifest",
+            "ensure_tailwind_binary",
+            "TAILWIND_VERSION",
+            "MERMAID_VERSION",
+        ):
+            assert hasattr(vendor_assets, attr), f"missing {attr}"
+
+    def test_detect_tailwind_asset_name(self) -> None:
+        """Platform detection returns one of the known asset names."""
+        from vendor_assets import _detect_tailwind_asset_name
+
+        known = {
+            "tailwindcss-macos-arm64",
+            "tailwindcss-macos-x64",
+            "tailwindcss-linux-arm64",
+            "tailwindcss-linux-armv7",
+            "tailwindcss-linux-x64",
+            "tailwindcss-windows-x64.exe",
+        }
+        assert _detect_tailwind_asset_name() in known
+
+    def test_download_rejects_non_http_scheme(self, tmp_path: Path) -> None:
+        """The _download helper refuses file:/ftp:/etc. — defense-in-depth."""
+        from vendor_assets import _download
+
+        with pytest.raises(ValueError, match="non-HTTP URL"):
+            _download("file:///etc/passwd", tmp_path / "out.bin")
diff --git a/scripts/vendor_assets.py b/scripts/vendor_assets.py
new file mode 100644
index 0000000..df91796
--- /dev/null
+++ b/scripts/vendor_assets.py
@@ -0,0 +1,209 @@
+"""Download and self-host third-party assets (Tailwind, Mermaid, Inter font).
+
+Replaces the previous CDN-based approach so the rendered site has no external
+runtime dependencies. Assets are cached under `scripts/.vendor-cache/` and
+copied into the built site's `assets/vendor/` directory.
+
+The Tailwind standalone CLI (a Go binary, no Node.js required) is downloaded
+on first use and reused thereafter. We pin to Tailwind v3 because the templates
+use v3-style runtime configuration; Tailwind v4 deprecated that config format.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import platform
+import re
+import shutil
+import stat
+import subprocess  # nosec B404 — used only to invoke the bundled Tailwind CLI
+import urllib.request
+from pathlib import Path
+
+TAILWIND_VERSION = "v3.4.19"
+MERMAID_VERSION = "10"
+MERMAID_URL = (
+    f"https://cdn.jsdelivr.net/npm/mermaid@{MERMAID_VERSION}/dist/mermaid.min.js"
+)
+GOOGLE_FONTS_CSS_URL = (
+    "https://fonts.googleapis.com/css2?"
+    "family=Inter:wght@400;500;600;700;800"
+    "&family=JetBrains+Mono:wght@400;500;600"
+    "&display=swap"
+)
+GOOGLE_FONTS_UA = (
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+    "AppleWebKit/537.36 (KHTML, like Gecko) "
+    "Chrome/120.0.0.0 Safari/537.36"
+)
+
+VENDOR_CACHE_DIR_NAME = ".vendor-cache"
+TAILWIND_BIN_NAME = "tailwindcss"
+
+
+def _vendor_cache_dir() -> Path:
+    return Path(__file__).parent / VENDOR_CACHE_DIR_NAME
+
+
+def _detect_tailwind_asset_name() -> str:
+    system = platform.system().lower()
+    machine = platform.machine().lower()
+    if system == "darwin":
+        return (
+            "tailwindcss-macos-arm64" if machine == "arm64" else "tailwindcss-macos-x64"
+        )
+    if system == "linux":
+        if machine in ("aarch64", "arm64"):
+            return "tailwindcss-linux-arm64"
+        if machine.startswith("armv7"):
+            return "tailwindcss-linux-armv7"
+        return "tailwindcss-linux-x64"
+    if system == "windows":
+        return "tailwindcss-windows-x64.exe"
+    raise RuntimeError(f"Unsupported platform for Tailwind CLI: {system}/{machine}")
+
+
+def _download(url: str, dest: Path, headers: dict[str, str] | None = None) -> None:
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    if not url.startswith(("http://", "https://")):
+        raise ValueError(f"Refusing to fetch non-HTTP URL: {url}")
+    req = urllib.request.Request(url, headers=headers or {})
+    with urllib.request.urlopen(req) as resp, dest.open("wb") as out:  # nosec B310 — scheme validated above
+        shutil.copyfileobj(resp, out)
+
+
+def ensure_tailwind_binary(logger: logging.Logger) -> Path:
+    """Download the Tailwind standalone CLI if not already cached."""
+    cache = _vendor_cache_dir()
+    bin_path = cache / TAILWIND_BIN_NAME
+    if bin_path.exists() and os.access(bin_path, os.X_OK):
+        return bin_path
+
+    asset = _detect_tailwind_asset_name()
+    url = f"https://github.com/tailwindlabs/tailwindcss/releases/download/{TAILWIND_VERSION}/{asset}"
+    logger.info(f"Downloading Tailwind CLI {TAILWIND_VERSION} ({asset})")
+    _download(url, bin_path)
+    bin_path.chmod(bin_path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
+    return bin_path
+
+
+def build_tailwind_css(
+    output_css: Path,
+    template_dir: Path,
+    site_dir: Path,
+    logger: logging.Logger,
+) -> None:
+    """Compile a single CSS file using the Tailwind standalone CLI."""
+    bin_path = ensure_tailwind_binary(logger)
+    config = template_dir / "tailwind.config.js"
+    input_css = template_dir / "tailwind.input.css"
+    if not config.exists():
+        raise RuntimeError(f"Tailwind config not found: {config}")
+    if not input_css.exists():
+        raise RuntimeError(f"Tailwind input CSS not found: {input_css}")
+
+    output_css.parent.mkdir(parents=True, exist_ok=True)
+    cmd = [
+        str(bin_path),
+        "--config",
+        str(config),
+        "--input",
+        str(input_css),
+        "--output",
+        str(output_css),
+        "--minify",
+    ]
+    logger.info(f"Compiling Tailwind CSS → {output_css}")
+    # Run from the project root so relative `content` globs in the config
+    # resolve against the built site directory.
+    cwd = site_dir.parent if site_dir.parent.exists() else Path.cwd()
+    env = os.environ.copy()
+    env["TAILWIND_SITE_DIR"] = str(site_dir)
+    env["TAILWIND_TEMPLATE_DIR"] = str(template_dir)
+    result = subprocess.run(  # nosec B603 — cmd is a fixed argv list, no shell, no user input
+        cmd,
+        cwd=str(cwd),
+        env=env,
+        capture_output=True,
+        text=True,
+        check=False,
+    )
+    if result.returncode != 0:
+        raise RuntimeError(
+            f"Tailwind CLI failed (exit {result.returncode}):\n{result.stderr}"
+        )
+    if result.stderr.strip():
+        logger.debug(f"Tailwind CLI: {result.stderr.strip()}")
+
+
+def fetch_mermaid(target_dir: Path, logger: logging.Logger) -> Path:
+    """Download Mermaid's UMD bundle into `target_dir` and return its path."""
+    cache = _vendor_cache_dir() / "mermaid"
+    cache.mkdir(parents=True, exist_ok=True)
+    cached = cache / f"mermaid-{MERMAID_VERSION}.min.js"
+    if not cached.exists():
+        logger.info(f"Downloading Mermaid {MERMAID_VERSION}")
+        _download(MERMAID_URL, cached)
+
+    target_dir.mkdir(parents=True, exist_ok=True)
+    target = target_dir / "mermaid.min.js"
+    shutil.copy2(cached, target)
+    return target
+
+
+_FONT_URL_RE = re.compile(r"url\((https://fonts\.gstatic\.com/[^)]+)\)")
+
+
+def fetch_fonts(target_dir: Path, logger: logging.Logger) -> Path:
+    """Download Google Fonts CSS + WOFF2 files, rewrite URLs to relative paths."""
+    cache = _vendor_cache_dir() / "fonts"
+    cache.mkdir(parents=True, exist_ok=True)
+    css_cache = cache / "fonts.css"
+    files_cache = cache / "files"
+    files_cache.mkdir(parents=True, exist_ok=True)
+
+    if not css_cache.exists():
+        logger.info("Downloading Google Fonts CSS")
+        _download(
+            GOOGLE_FONTS_CSS_URL, css_cache, headers={"User-Agent": GOOGLE_FONTS_UA}
+        )
+
+    css = css_cache.read_text(encoding="utf-8")
+    url_map: dict[str, str] = {}
+    for url in _FONT_URL_RE.findall(css):
+        # Stable filename derived from the gstatic path so duplicates collapse.
+        name = url.rsplit("/", 1)[-1]
+        local = files_cache / name
+        if not local.exists():
+            logger.debug(f"Fetching font file: {name}")
+            _download(url, local)
+        url_map[url] = f"files/{name}"
+
+    rewritten = css
+    for url, rel in url_map.items():
+        rewritten = rewritten.replace(url, rel)
+
+    target_dir.mkdir(parents=True, exist_ok=True)
+    (target_dir / "fonts.css").write_text(rewritten, encoding="utf-8")
+
+    target_files = target_dir / "files"
+    target_files.mkdir(parents=True, exist_ok=True)
+    for f in files_cache.iterdir():
+        if f.is_file():
+            shutil.copy2(f, target_files / f.name)
+    return target_dir / "fonts.css"
+
+
+def write_vendor_manifest(target_dir: Path, fonts_count: int) -> None:
+    """Record what was vendored — useful for debugging."""
+    manifest = {
+        "tailwind": TAILWIND_VERSION,
+        "mermaid": MERMAID_VERSION,
+        "fonts": {"family": ["Inter", "JetBrains Mono"], "files": fonts_count},
+    }
+    target_dir.mkdir(parents=True, exist_ok=True)
+    (target_dir / "manifest.json").write_text(
+        json.dumps(manifest, indent=2), encoding="utf-8"
+    )
diff --git a/scripts/website_templates/page.html.j2 b/scripts/website_templates/page.html.j2
new file mode 100644
index 0000000..3175bb2
--- /dev/null
+++ b/scripts/website_templates/page.html.j2
@@ -0,0 +1,253 @@
+
+
+
+  
+  
+  {{ page_title }} — {{ site_title }}
+  
+  
+  
+  
+  
+  
+
+
+
+  
+
+ + + + + {{ site_title }} + +
+ + +
+
+
+ +
+ {# ---------- Sidebar navigation ---------- #} + + + {# Backdrop for mobile sidebar #} + + + {# ---------- Main content ---------- #} +
+ + +
+ {{ content | safe }} +
+ + {% if prev_page or next_page %} +
+ {% if prev_page %} + +
← Previous
+
+ {{ prev_page.title }} +
+
+ {% else %} + + {% endif %} + {% if next_page %} + +
Next →
+
+ {{ next_page.title }} +
+
+ {% endif %} +
+ {% endif %} + +
+

+ Content rendered from + + {{ page_title }} + + on GitHub. Markdown is the single source of truth — re-run + scripts/build_website.py + after editing to refresh the site. +

+
+
+ + {# ---------- In-page TOC ---------- #} + {% if toc %} + + {% endif %} +
+ + + + + + diff --git a/scripts/website_templates/site.css b/scripts/website_templates/site.css new file mode 100644 index 0000000..b2d2296 --- /dev/null +++ b/scripts/website_templates/site.css @@ -0,0 +1,104 @@ +/* Site-specific styles layered on top of Tailwind. */ + +html { + font-family: 'Inter', ui-sans-serif, system-ui, sans-serif; +} + +code, +pre, +kbd, +samp { + font-family: 'JetBrains Mono', ui-monospace, monospace; +} + +/* Mermaid diagrams: centre and constrain width so they read on mobile. */ +.mermaid { + background: transparent !important; + padding: 1rem 0; + margin: 1.5rem 0; + text-align: center; + overflow-x: auto; +} + +.mermaid svg { + max-width: 100%; + height: auto; +} + +/* Prevent prose pre/code from inheriting the Tailwind dark prose background + which can clash with our slate-900 preference. */ +.prose pre { + border-radius: 0.5rem; + padding: 1rem 1.25rem; + overflow-x: auto; + line-height: 1.55; + font-size: 0.875rem; +} + +.prose pre code { + background: transparent; + padding: 0; + color: inherit; + font-size: inherit; +} + +/* Inline code gets a softer treatment than fenced blocks. */ +.prose :where(p, li, h1, h2, h3, h4) code { + font-size: 0.875em; + font-weight: 500; +} + +/* Tables wrap on mobile rather than forcing horizontal scroll on every page. */ +.prose table { + display: block; + overflow-x: auto; + white-space: nowrap; +} + +@media (min-width: 768px) { + .prose table { + display: table; + white-space: normal; + } +} + +/* Anchor offset so sticky header doesn't cover heading targets. */ +.prose :where(h1, h2, h3, h4, h5, h6) { + scroll-margin-top: 6rem; +} + +/* Hide focus rings on mouse, keep for keyboard. */ +:focus:not(:focus-visible) { + outline: none; +} + +/* Pygments syntax highlighting (paired with `markdown` codehilite extension). */ +.codehilite { background: transparent; } +.codehilite .hll { background-color: #ffffcc } +.codehilite .c, .codehilite .c1, .codehilite .cm { color: #999988; font-style: italic } +.codehilite .k, .codehilite .kc, .codehilite .kd, .codehilite .kn, .codehilite .kp, +.codehilite .kr, .codehilite .kt { color: #f0883e; font-weight: 600 } +.codehilite .s, .codehilite .s1, .codehilite .s2 { color: #a5d6ff } +.codehilite .nb { color: #79c0ff } +.codehilite .nf { color: #d2a8ff } +.codehilite .o, .codehilite .p { color: #ff7b72 } +.codehilite .m, .codehilite .mi, .codehilite .mf { color: #79c0ff } +.codehilite .nv, .codehilite .vc, .codehilite .vg, .codehilite .vi { color: #ffa657 } +.codehilite .err { color: inherit } + +/* Light-theme readability tweaks for code colours. */ +:root:not(.dark) .codehilite .k, +:root:not(.dark) .codehilite .kc, +:root:not(.dark) .codehilite .kd, +:root:not(.dark) .codehilite .kn, +:root:not(.dark) .codehilite .kp, +:root:not(.dark) .codehilite .kr, +:root:not(.dark) .codehilite .kt { color: #cf222e; } +:root:not(.dark) .codehilite .s, +:root:not(.dark) .codehilite .s1, +:root:not(.dark) .codehilite .s2 { color: #0a3069; } +:root:not(.dark) .codehilite .nb { color: #0550ae; } +:root:not(.dark) .codehilite .nf { color: #6f42c1; } +:root:not(.dark) .codehilite .m, +:root:not(.dark) .codehilite .mi, +:root:not(.dark) .codehilite .mf { color: #0550ae; } diff --git a/scripts/website_templates/tailwind.config.js b/scripts/website_templates/tailwind.config.js new file mode 100644 index 0000000..41e164b --- /dev/null +++ b/scripts/website_templates/tailwind.config.js @@ -0,0 +1,35 @@ +/** @type {import('tailwindcss').Config} */ +// +// Used by the Tailwind standalone CLI (invoked from scripts/vendor_assets.py). +// The `content` globs are passed as absolute paths via env vars so the build +// works from any cwd. +// +const path = require('path'); +const siteDir = process.env.TAILWIND_SITE_DIR || path.join(__dirname, '..', '..', 'site'); +const templateDir = process.env.TAILWIND_TEMPLATE_DIR || __dirname; + +module.exports = { + content: [ + path.join(siteDir, '**/*.html'), + path.join(templateDir, '**/*.j2'), + ], + darkMode: 'class', + theme: { + extend: { + fontFamily: { + sans: ['Inter', 'ui-sans-serif', 'system-ui', 'sans-serif'], + mono: ['"JetBrains Mono"', 'ui-monospace', 'monospace'], + }, + colors: { + brand: { + 50: '#eef2ff', + 100: '#e0e7ff', + 500: '#6366f1', + 600: '#4f46e5', + 700: '#4338ca', + }, + }, + }, + }, + plugins: [require('@tailwindcss/typography')], +}; diff --git a/scripts/website_templates/tailwind.input.css b/scripts/website_templates/tailwind.input.css new file mode 100644 index 0000000..b5c61c9 --- /dev/null +++ b/scripts/website_templates/tailwind.input.css @@ -0,0 +1,3 @@ +@tailwind base; +@tailwind components; +@tailwind utilities;