PentestPilot/HOWTO.md

# PentestPilot - HOWTO

## Table of Contents
- [Overview](#overview)
- [Install & Setup](#install--setup)
- [Core Env Vars](#core-env-vars)
- [Target Workflow](#target-workflow)
- [Automation & Orchestration](#automation--orchestration)
- [Dashboard (Status & Evidence)](#dashboard-status--evidence)
- [Manifest (State & Resume)](#manifest-state--resume)
- [AI Integrations](#ai-integrations)
- [Web Recon & Routing](#web-recon--routing)
- [Active Directory & SMB](#active-directory--smb)
- [Passwords & Wordlists](#passwords--wordlists)
- [Shells, Transfers, Privesc](#shells-transfers-privesc)
- [Tunnels & Port Forwards](#tunnels--port-forwards)
- [QoL Utilities](#qol-utilities)
- [Post-Exploitation & Reporting](#post-exploitation--reporting)
- [Safety Notes](#safety-notes)
- [End-to-End Example](#end-to-end-example)
- [Troubleshooting](#troubleshooting)
- [Customization](#customization)
- [Appendix - Common Command Recipes](#appendix--common-command-recipes)

## Overview
- This toolkit streamlines OSCP/HTB workflows: discovery, web recon, AD, credential hygiene, shells, tunnels, transfers, privilege escalation, post‑exploitation, reporting, and AI‑assisted orchestration.
- Everything is CLI‑first, idempotent when possible, and resume‑aware via a per‑target manifest.
- See: README.md:1 for the quick summary and TOOLKIT.md:1 for the command catalog.
 - Tips and conventions below assume a Linux attacker VM (Kali/Parrot/Ubuntu). Adjust paths for your OS.

## Install & Setup
1) Place the repo in your working directory (e.g., `~/hax/htb`).
2) Load the shell profile so aliases and PATH work:
   echo "source $(pwd)/.zshrc.htb" >> ~/.zshrc
   exec zsh
3) Optional AI setup:
   - OpenAI: export OPENAI_API_KEY=sk-... (and optionally OPENAI_MODEL)
   - Ollama: install+run, optionally export OLLAMA_MODEL=llama3.1 (default) and OLLAMA_HOST

Recommended Tools
- Install commonly used tools up‑front (Debian/Ubuntu examples):
  sudo apt update && sudo apt install -y nmap curl jq ripgrep python3 python3-pip tmux
  sudo apt install -y gobuster seclists ffuf sqlmap
  sudo apt install -y smbclient ldap-utils snmp snmp-mibs-downloader
  pipx install httpx-toolkit nuclei gowitness || true
  pipx runpip nuclei install -U nuclei || true
  pipx install "impacket" || true
  gem install wpscan || true
  pipx install droopescan || true
  apt install joomscan || true
  snap install magescan || true
  # optional: chisel, socat, naabu, masscan, subfinder/amass, crackmapexec

Notes:
- Some tools (httpx/nuclei) are provided by multiple packages; ensure they are in PATH.
- If a wrapper says a tool is missing, either install or skip that specific step.
- Use `pipx` (or venv) for Python‑based tools to avoid site‑packages collisions.

## Core Env Vars
- `HTB_ROOT` (default: current repo path) - base for targets and scripts.
- `TARGET` - a current target convenience var set by `settarget`.
- `OUTDIR` - output directory for scans in the current target (set by `settarget`).
- Proxies: `HTTP_PROXY`/`HTTPS_PROXY` can be toggled via `proxy_toggle.sh on|off`.

## Target Workflow
1) Create a target workspace:
   settarget 10.10.10.10
   This creates `targets/<target>/{scans,loot,www,exploits}` and sets `OUTDIR`.
2) Notes:
   - `notesinit` scaffolds `notes.md` in the target directory.
   - `notesattach` appends a scan artifacts summary to notes.
3) Directories:
   - `targets/<target>/scans` - scanner logs, json, summaries
   - `targets/<target>/loot`  - collected artifacts
   - `targets/<target>/notes.md` - your engagement notes
   - `targets/<target>/manifest.json` - per-target state (see Manifest below)
4) Common recipes (see Appendix for more):
   - Quick nmap: nq → review `scans/*_quick_*.nmap`
   - Full TCP then service: nf → review `phase1`/`phase2` outputs
   - UDP quick check: nu → review common UDP services
   - Web checks: headers/methods/tech → dirbuster/param_fuzz → sqli_quick
   - SMB/LDAP: smb_enum.sh / ldap_enum.sh - save listings in `scans/`

Quick Aliases
- Nmap: `nq` (quick), `nf` (full TCP), `nu` (UDP top)
- Web: `webrecon` (current TARGET), `wideweb <hosts.txt>` (lists)
- Full pipeline: `fullpipe <domain|hosts.txt>` (DNS→httpx→nuclei→tech route, resume‑aware)
- AI agents: `agent` (multi‑task), `aiweb`, `aiplan`, `aireview`
- Dashboard: `dashboard` (status), `resumeall`, `techactions`
- QoL: `proxyon`, `proxyoff`, `cleanupscans`, `tmux_init.sh`

## Automation & Orchestration
- Minimal recon: `auto_recon.sh <target>`
- Web recon (current TARGET): `web_recon.sh <target|--url URL>` → headers/tech/backup/dirb (+screenshots if `gowitness`)
- Wide recon (list of hosts): `wide_web_recon.sh <hosts.txt>` → httpx + nuclei + screenshots
- One‑click pipeline: `full_pipeline.sh <domain|hosts.txt> [--resume|--force]`
  - DNS subenum (if domain) → httpx (balanced) → nuclei (auto severity) → tech route → optional WPScan
  - Resume (default) consults `manifest.json` and skips completed steps.
  - Writes evidence JSON + summaries (httpx/nuclei) into OUTDIR and manifest.
- Agents (AI-aware): `bin/ai/agent_orchestrator.py:1`
  - `agent full <domain|hosts.txt>` - small state machine for the full pipeline; retries resume passes, then runs `tech_actions.py --run`.
  - `agent web <hosts.txt> [--force]` - httpx -> nuclei -> screenshots -> AI plan (resume-aware subtasks)
  - `agent ad <host> [--force]` - enum4linux/smbmap/rpc (resume-aware)
  - `agent notes <target> [--force]` - notes init + attach (resume-aware)
  - `agent post <target> [--force]` - linux_loot + report pack (resume-aware)
- Resume all targets: `resume_all.py` - loops over targets/* and resumes incomplete `full_pipeline` runs.

Advanced: Pipeline Semantics
- `--resume` (default) skips steps whose manifest task status is `ok`.
- `--force` reruns steps and overwrites evidence (new timestamps/files).
- Each phase records elapsed seconds and evidence file paths in manifest meta.
- If a run fails midway, you can re‑invoke with `--resume` to continue where you left off.

## Dashboard (Status & Evidence)
- Command: `dashboard`  Options: `--no-color`, `--compact`, `--json`
- Columns:
  - target, created, last (timestamp of last pipeline), urls (count)
  - dns, httpx, nuclei, tech, wp - per-phase status with elapsed seconds
  - sev - severity counts (e.g., c:1 h:3 m:2)
  - top-techs - top techs from httpx tech summary (e.g., wordpress:3, drupal:1)
  - bar - colorized severity proportion bar (critical/high/medium/low)
- Evidence sources (auto‑persisted by pipeline):
  - httpx JSON: `OUTDIR/httpx_<ts>.json` and `httpx_<ts>.summary.json`
  - nuclei JSON: `OUTDIR/httpx2nuclei_<ts>/nuclei.json` and `summary.json`

## Manifest (State & Resume)
- Path: `targets/<target>/manifest.json`
- Schema (high‑level):
  {
    "target": "<name>",
    "created_at": "YYYY-MM-DD HH:MM:SS",
    "last_pipeline": "<ts>",
    "urls": [ ... ],
    "tasks": {
      "dns": {"status":"ok|running|fail","started_at":"...","finished_at":"...","meta":{"subs_file":"...","elapsed_sec":N}},
      "httpx": {"meta":{"urls_file":"...","httpx_json":"...","httpx_summary":"...","elapsed_sec":N}},
      "nuclei": {"meta":{"log":"...","nuclei_json":"...","nuclei_summary":"...","elapsed_sec":N}},
      "techroute": {"meta":{"log":"...","elapsed_sec":N}},
      "wpscan": {"meta":{"log":"...","elapsed_sec":N}},
      "web_httpx|web_nuclei|web_shots|web_plan": {"meta":{"elapsed_sec":N}},
      "ad_*", "notes_*", "post_*": {"meta":{"elapsed_sec":N}}
    }
  }
- CLI: `bin/automation/manifest.py:1`
- `init <target>` - create manifest
- `set|get <target> <key> [value]` - set or read top-level values
- `addlist <target> <key> <file|a,b,c>` - append to a list
- `show <target>` - print JSON
- `task <target> <name> start|ok|fail [meta-json]` - update tasks (status, timestamps, meta)
- `taskstatus <target> <name>` - prints status; exit 0 if ok, 2 if running, 1 otherwise
- `taskreset <target> <name>` - remove/reset a task entry

## AI Integrations
- Providers: OpenAI (OPENAI_API_KEY) or local Ollama (defaults chosen automatically).
- Robust helpers: `bin/ai/_ai_utils.py:1` (retries, timeouts, prompt truncation)
- Tools:
- `ask.py` - quick prompts
- `commands_planner.py` - converts a goal/context into ready-to-run toolkit commands
- `orchestrate_web.py` - probes (httpx) and asks AI for a recon plan
- `review_findings.py` - summarizes notes into risks + next steps
  - `agent_orchestrator.py` - orchestrates web/full/ad/notes/post tasks and updates manifest
Troubleshooting AI:
- If calls fail, `_ai_utils.py` retries with exponential backoff.
- If no OPENAI_API_KEY is set, the system falls back to Ollama (ensure it’s running).
- You can reduce output size by setting smaller prompts and using `--compact` when calling dashboard.

## Web Recon & Routing
- Pipeline: `httpx_to_nuclei.sh` → httpx alive list → nuclei with auto severity (based on URL count) → produces `.txt`, `.json`, `summary.json`.
- Tech routing: `httpx_tech_route.py` flags:
  - `--tech` filter; `--severity` list; `--wpscan [--wpscan-limit N]`; `--extra [--extra-limit N]`; `--dry-run`
  - Presets: wordpress, drupal, joomla, laravel, aspnet, spring, tomcat, iis, exchange, sharepoint, grafana, kibana, gitlab, confluence, jupyter, jenkins, magento, sonarqube, jira, confluence
  - With `--extra`, auto‑runs quick wrappers when present (e.g., WPScan, Droopescan, Joomscan, Jenkins/SonarQube/Magento/Jira/Confluence quick checks)
- Extras:
  - `httpx_presets.sh`: concurrency profiles; `httpx_probe.sh` for fast probes
  - `gobuster_dir.sh`, `gobuster_vhost.sh`; `dirbuster.sh` (ffuf); backup hunters, CORS/methods/TLS, LFI tester

## Active Directory & SMB
- Impacket wrappers: `getnpusers_wrapper.sh`, `getspns_wrapper.sh`
- `kerbrute_wrapper.sh` (user enum), `cme_quick.sh` (shares/sessions/loggedon), `rpc_quick.sh`
- SMB `smbmap_quick.sh` and `smb_check_write.sh`

## Passwords & Wordlists
- `mutate_words.py`, `merge_dedupe.sh`, `wordlist_cleanup.sh` - build/clean wordlists
- `spray_http_basic.sh` - cautious HTTP Basic Auth spray (respect lockout policies)

## Shells, Transfers, Privesc
- Shells: reverse one‑liners (`revsh.py`), listener (`listener.sh`), TTY upgrade tips
- Transfers: `http_serve.sh` or `serve.py` (with web upload), `smb_server.sh`, `dl_oneshots.sh`, `push_http.sh`
- Linux privesc: `linux_quick_enum.sh`, `suid_scan.sh`, `caps_scan.sh`
- Windows privesc: `privesc_quick.ps1`, `find_unquoted_services.ps1`, `find_path_writable.ps1`, `win_share_enum.ps1`

## Tunnels & Port Forwards
- `chisel_server.sh` / `chisel_client.sh` - reverse tunnels
- `autossh_socks.sh` - resilient SOCKS proxy
- `socat_forward.sh` and `port_forward.sh` - local/remote forwards

## QoL Utilities
- `cleanup_scans.sh` - prune old scan files
- `proxy_toggle.sh` - set/unset HTTP(S) proxy env vars
- `tmux_init.sh` - quick tmux workspace
- `extract_urls.py` - harvest URLs from files (logs/notes)

## Post‑Exploitation & Reporting
- `linux_loot.sh` - safe, size-capped artifacts collector (config via env: `MAX_SIZE`, `INCLUDE_*`)
- `windows_loot.ps1` - conservative Windows loot collector (zip fallback)
- `pack_report.sh` - compiles a markdown with summaries and file listings

## Safety Notes
- Use only with explicit authorization.
- Many steps are safe by default (no brute force). Be mindful of account lockout policies when using auth‑related tooling.
- For “unsafe” or exploit‑heavy checks, consider separate gated wrappers and explicit flags.

## End‑to‑End Example
1) Set up target and notes:
   settarget target.htb
   notesinit
2) Run full autonomous recon (resume‑aware):
   agent full target.htb
3) Review dashboard:
   dashboard --compact
4) Let AI suggest next steps from tech:
   techactions $TARGET
5) Post‑exploitation:
   agent post $TARGET
6) Resume across multiple targets later:
   resumeall

## Troubleshooting
- Tool missing: wrappers fail gracefully and log hints. Install optional tools (httpx, nuclei, gobuster, gowitness, wpscan, droopescan, joomscan, magescan, impacket).
- Manifest stuck in running: `manifest.py taskreset <target> <name>`.
- No colors in dashboard: add `--no-color` or your terminal might not support ANSI.

## Customization
- Adjust tags/severity in `httpx_to_nuclei.sh:1` and `httpx_tech_route.py:1`.
- Extend tech presets and quick wrappers in `bin/web/`.
- Tweak agent behaviors in `bin/ai/agent_orchestrator.py:1`.
 - Add your own manifest keys via `manifest.py set <target> key value` for custom dashboards.

## Appendix - Common Command Recipes
- Directory brute (gobuster): gobuster_dir.sh http://$TARGET/ /usr/share/wordlists/dirb/common.txt php,txt 50
- Vhost brute: gobuster_vhost.sh http://$TARGET/ subdomains-top1million-5000.txt 100
- Probe techs: httpx_probe.sh hosts.txt > live.txt
- Route by tech (with extras): httpx_tech_route.py live.txt --tech wordpress,drupal --extra --wpscan
- Nuclei quick: nuclei_quick.sh live.txt cves,exposures
- SMB write check: smb_check_write.sh $TARGET sharename
- LDAP quick users: ldap_quick_users.sh $TARGET 'DC=target,DC=htb'
- Secrets scan: scan_secrets.sh .

Legend:
- DNS/httpx/nuclei/tech/wp: status + elapsed time `(OK(12s))`.
- sev: short counts (`c:2 h:3 m:5`), bar: █ blocks colored per severity.
- --compact removes dates and shows essentials when terminal space is tight.
- --json lets you script your own dashboards.
Example manifest snippet:
{
  "target": "target.htb",
  "tasks": {
    "httpx": {
      "status": "ok",
      "started_at": "2025-10-08 10:21:00",
      "finished_at": "2025-10-08 10:21:08",
      "meta": {
        "urls": 34,
        "urls_file": "targets/target.htb/scans/urls_20251008_1021.txt",
        "httpx_json": ".../httpx_20251008_1021.json",
        "httpx_summary": ".../httpx_20251008_1021.summary.json",
        "elapsed_sec": 8
      }
    }
  }
}
Customizing Tech Routes:
- Edit `httpx_tech_route.py` to add or adjust presets in the `presets` map.
- To auto‑launch additional quick wrappers, update the `--extra` handler.

Auto Severity Tuning (nuclei):
- `httpx_to_nuclei.sh` sets nuclei severity via `--severity auto` mapping:
  - >500 URLs → `high,critical`; >100 → `medium,high,critical`; else `low,medium,high,critical`.
- Override with explicit `--severity` or adjust logic in the script.