PentestPilot/HOWTO.md

PentestPilot — HOWTO

Table of Contents

• Overview — #overview
• Install & Setup — #install--setup
• Core Env Vars — #core-env-vars
• Target Workflow — #target-workflow
• Automation & Orchestration — #automation--orchestration
• Dashboard (Status & Evidence) — #dashboard-status--evidence
• Manifest (State & Resume) — #manifest-state--resume
• AI Integrations — #ai-integrations
• Web Recon & Routing — #web-recon--routing
• Active Directory & SMB — #active-directory--smb
• Passwords & Wordlists — #passwords--wordlists
• Shells, Transfers, Privesc — #shells-transfers-privesc
• Tunnels & Port Forwards — #tunnels--port-forwards
• QoL Utilities — #qol-utilities
• Post‑Exploitation & Reporting — #post-exploitation--reporting
• Safety Notes — #safety-notes
• End‑to‑End Example — #end-to-end-example
• Troubleshooting — #troubleshooting
• Customization — #customization
• Appendix — Common Command Recipes — #appendix--common-command-recipes

Overview

• This toolkit streamlines OSCP/HTB workflows: discovery, web recon, AD, credential hygiene, shells, tunnels, transfers, privilege escalation, post‑exploitation, reporting, and AI‑assisted orchestration.
• Everything is CLI‑first, idempotent when possible, and resume‑aware via a per‑target manifest.
• See: README.md:1 for the quick summary and TOOLKIT.md:1 for the command catalog.
• Tips and conventions below assume a Linux attacker VM (Kali/Parrot/Ubuntu). Adjust paths for your OS.

Install & Setup

1. Place the repo in your working directory (e.g., ~/hax/htb).
2. Load the shell profile so aliases and PATH work: echo "source $(pwd)/.zshrc.htb" >> ~/.zshrc exec zsh
3. Optional AI setup:
   • OpenAI: export OPENAI_API_KEY=sk-... (and optionally OPENAI_MODEL)
   • Ollama: install+run, optionally export OLLAMA_MODEL=llama3.1 (default) and OLLAMA_HOST

Recommended Tools

• Install commonly used tools up‑front (Debian/Ubuntu examples): sudo apt update && sudo apt install -y nmap curl jq ripgrep python3 python3-pip tmux sudo apt install -y gobuster seclists ffuf sqlmap sudo apt install -y smbclient ldap-utils snmp snmp-mibs-downloader pipx install httpx-toolkit nuclei gowitness || true pipx runpip nuclei install -U nuclei || true pipx install "impacket" || true gem install wpscan || true pipx install droopescan || true apt install joomscan || true snap install magescan || true

  optional: chisel, socat, naabu, masscan, subfinder/amass, crackmapexec

Notes:

• Some tools (httpx/nuclei) are provided by multiple packages; ensure they are in PATH.
• If a wrapper says a tool is missing, either install or skip that specific step.
• Use pipx (or venv) for Python‑based tools to avoid site‑packages collisions.

Core Env Vars

• HTB_ROOT (default: current repo path) — base for targets and scripts.
• TARGET — a current target convenience var set by settarget.
• OUTDIR — output directory for scans in the current target (set by settarget).
• Proxies: HTTP_PROXY/HTTPS_PROXY can be toggled via proxy_toggle.sh on|off.

Target Workflow

1. Create a target workspace: settarget 10.10.10.10 This creates targets/<target>/{scans,loot,www,exploits} and sets OUTDIR.
2. Notes:
   • notesinit scaffolds notes.md in the target directory.
   • notesattach appends a scan artifacts summary to notes.
3. Directories:
   • targets/<target>/scans — scanner logs, json, summaries
   • targets/<target>/loot — collected artifacts
   • targets/<target>/notes.md — your engagement notes
   • targets/<target>/manifest.json — per‑target state (see Manifest below)
4. Common recipes (see Appendix for more):
   • Quick nmap: nq → review scans/*_quick_*.nmap
   • Full TCP then service: nf → review phase1/phase2 outputs
   • UDP quick check: nu → review common UDP services
   • Web checks: headers/methods/tech → dirbuster/param_fuzz → sqli_quick
   • SMB/LDAP: smb_enum.sh / ldap_enum.sh — save listings in scans/

Quick Aliases

• Nmap: nq (quick), nf (full TCP), nu (UDP top)
• Web: webrecon (current TARGET), wideweb <hosts.txt> (lists)
• Full pipeline: fullpipe <domain|hosts.txt> (DNS→httpx→nuclei→tech route, resume‑aware)
• AI agents: agent (multi‑task), aiweb, aiplan, aireview
• Dashboard: dashboard (status), resumeall, techactions
• QoL: proxyon, proxyoff, cleanupscans, tmux_init.sh

Automation & Orchestration

• Minimal recon: auto_recon.sh <target>
• Web recon (current TARGET): web_recon.sh <target|--url URL> → headers/tech/backup/dirb (+screenshots if gowitness)
• Wide recon (list of hosts): wide_web_recon.sh <hosts.txt> → httpx + nuclei + screenshots
• One‑click pipeline: full_pipeline.sh <domain|hosts.txt> [--resume|--force]
  • DNS subenum (if domain) → httpx (balanced) → nuclei (auto severity) → tech route → optional WPScan
  • Resume (default) consults manifest.json and skips completed steps.
  • Writes evidence JSON + summaries (httpx/nuclei) into OUTDIR and manifest.
• Agents (AI‑aware): bin/ai/agent_orchestrator.py:1
  • agent full <domain|hosts.txt> — small state machine for the full pipeline; retries resume passes, then runs tech_actions.py --run.
  • agent web <hosts.txt> [--force] — httpx → nuclei → screenshots → AI plan (resume‑aware subtasks)
  • agent ad <host> [--force] — enum4linux/smbmap/rpc (resume‑aware)
  • agent notes <target> [--force] — notes init + attach (resume‑aware)
  • agent post <target> [--force] — linux_loot + report pack (resume‑aware)
• Resume all targets: resume_all.py — loops over targets/* and resumes incomplete full_pipeline runs.

Advanced: Pipeline Semantics

• --resume (default) skips steps whose manifest task status is ok.
• --force reruns steps and overwrites evidence (new timestamps/files).
• Each phase records elapsed seconds and evidence file paths in manifest meta.
• If a run fails midway, you can re‑invoke with --resume to continue where you left off.

Dashboard (Status & Evidence)

• Command: dashboard Options: --no-color, --compact, --json
• Columns:
  • target, created, last (timestamp of last pipeline), urls (count)
  • dns, httpx, nuclei, tech, wp — per‑phase status with elapsed seconds
  • sev — severity counts (e.g., c:1 h:3 m:2)
  • top‑techs — top techs from httpx tech summary (e.g., wordpress:3, drupal:1)
  • bar — colorized severity proportion bar (critical/high/medium/low)
• Evidence sources (auto‑persisted by pipeline):
  • httpx JSON: OUTDIR/httpx_<ts>.json and httpx_<ts>.summary.json
  • nuclei JSON: OUTDIR/httpx2nuclei_<ts>/nuclei.json and summary.json

Manifest (State & Resume)

• Path: targets/<target>/manifest.json
• Schema (high‑level): { "target": "", "created_at": "YYYY-MM-DD HH:MM:SS", "last_pipeline": "", "urls": [ ... ], "tasks": { "dns": {"status":"ok|running|fail","started_at":"...","finished_at":"...","meta":{"subs_file":"...","elapsed_sec":N}}, "httpx": {"meta":{"urls_file":"...","httpx_json":"...","httpx_summary":"...","elapsed_sec":N}}, "nuclei": {"meta":{"log":"...","nuclei_json":"...","nuclei_summary":"...","elapsed_sec":N}}, "techroute": {"meta":{"log":"...","elapsed_sec":N}}, "wpscan": {"meta":{"log":"...","elapsed_sec":N}}, "web_httpx|web_nuclei|web_shots|web_plan": {"meta":{"elapsed_sec":N}}, "ad_", "notes_", "post_*": {"meta":{"elapsed_sec":N}} } }
• CLI: bin/automation/manifest.py:1
  • init <target> — create manifest
  • set|get <target> <key> [value] — set or read top‑level values
  • addlist <target> <key> <file|a,b,c> — append to a list
  • show <target> — print JSON
  • task <target> <name> start|ok|fail [meta-json] — update tasks (status, timestamps, meta)
  • taskstatus <target> <name> — prints status; exit 0 if ok, 2 if running, 1 otherwise
  • taskreset <target> <name> — remove/reset a task entry

AI Integrations

• Providers: OpenAI (OPENAI_API_KEY) or local Ollama (defaults chosen automatically).
• Robust helpers: bin/ai/_ai_utils.py:1 (retries, timeouts, prompt truncation)
• Tools:
  • ask.py — quick prompts
  • commands_planner.py — converts a goal/context into ready‑to‑run toolkit commands
  • orchestrate_web.py — probes (httpx) and asks AI for a recon plan
  • review_findings.py — summarizes notes into risks + next steps
  • agent_orchestrator.py — orchestrates web/full/ad/notes/post tasks and updates manifest Troubleshooting AI:
• If calls fail, _ai_utils.py retries with exponential backoff.
• If no OPENAI_API_KEY is set, the system falls back to Ollama (ensure it’s running).
• You can reduce output size by setting smaller prompts and using --compact when calling dashboard.

Web Recon & Routing

• Pipeline: httpx_to_nuclei.sh → httpx alive list → nuclei with auto severity (based on URL count) → produces .txt, .json, summary.json.
• Tech routing: httpx_tech_route.py flags:
  • --tech filter; --severity list; --wpscan [--wpscan-limit N]; --extra [--extra-limit N]; --dry-run
  • Presets: wordpress, drupal, joomla, laravel, aspnet, spring, tomcat, iis, exchange, sharepoint, grafana, kibana, gitlab, confluence, jupyter, jenkins, magento, sonarqube, jira, confluence
  • With --extra, auto‑runs quick wrappers when present (e.g., WPScan, Droopescan, Joomscan, Jenkins/SonarQube/Magento/Jira/Confluence quick checks)
• Extras:
  • httpx_presets.sh: concurrency profiles; httpx_probe.sh for fast probes
  • gobuster_dir.sh, gobuster_vhost.sh; dirbuster.sh (ffuf); backup hunters, CORS/methods/TLS, LFI tester

Active Directory & SMB

• Impacket wrappers: getnpusers_wrapper.sh, getspns_wrapper.sh
• kerbrute_wrapper.sh (user enum), cme_quick.sh (shares/sessions/loggedon), rpc_quick.sh
• SMB smbmap_quick.sh and smb_check_write.sh

Passwords & Wordlists

• mutate_words.py, merge_dedupe.sh, wordlist_cleanup.sh — build/clean wordlists
• spray_http_basic.sh — cautious HTTP Basic Auth spray (respect lockout policies)

Shells, Transfers, Privesc

• Shells: reverse one‑liners (revsh.py), listener (listener.sh), TTY upgrade tips
• Transfers: http_serve.sh or serve.py (with web upload), smb_server.sh, dl_oneshots.sh, push_http.sh
• Linux privesc: linux_quick_enum.sh, suid_scan.sh, caps_scan.sh
• Windows privesc: privesc_quick.ps1, find_unquoted_services.ps1, find_path_writable.ps1, win_share_enum.ps1

Tunnels & Port Forwards

• chisel_server.sh / chisel_client.sh — reverse tunnels
• autossh_socks.sh — resilient SOCKS proxy
• socat_forward.sh and port_forward.sh — local/remote forwards

QoL Utilities

• cleanup_scans.sh — prune old scan files
• proxy_toggle.sh — set/unset HTTP(S) proxy env vars
• tmux_init.sh — quick tmux workspace
• extract_urls.py — harvest URLs from files (logs/notes)

Post‑Exploitation & Reporting

• linux_loot.sh — safe, size‑capped artifacts collector (config via env: MAX_SIZE, INCLUDE_*)
• windows_loot.ps1 — conservative Windows loot collector (zip fallback)
• pack_report.sh — compiles a markdown with summaries and file listings

Safety Notes

• Use only with explicit authorization.
• Many steps are safe by default (no brute force). Be mindful of account lockout policies when using auth‑related tooling.
• For “unsafe” or exploit‑heavy checks, consider separate gated wrappers and explicit flags.

End‑to‑End Example

1. Set up target and notes: settarget target.htb notesinit
2. Run full autonomous recon (resume‑aware): agent full target.htb
3. Review dashboard: dashboard --compact
4. Let AI suggest next steps from tech: techactions $TARGET
5. Post‑exploitation: agent post $TARGET
6. Resume across multiple targets later: resumeall

Troubleshooting

• Tool missing: wrappers fail gracefully and log hints. Install optional tools (httpx, nuclei, gobuster, gowitness, wpscan, droopescan, joomscan, magescan, impacket).
• Manifest stuck in running: manifest.py taskreset <target> <name>.
• No colors in dashboard: add --no-color or your terminal might not support ANSI.

Customization

• Adjust tags/severity in httpx_to_nuclei.sh:1 and httpx_tech_route.py:1.
• Extend tech presets and quick wrappers in bin/web/.
• Tweak agent behaviors in bin/ai/agent_orchestrator.py:1.
• Add your own manifest keys via manifest.py set <target> key value for custom dashboards.

Appendix — Common Command Recipes

• Directory brute (gobuster): gobuster_dir.sh http://$TARGET/ /usr/share/wordlists/dirb/common.txt php,txt 50
• Vhost brute: gobuster_vhost.sh http://$TARGET/ subdomains-top1million-5000.txt 100
• Probe techs: httpx_probe.sh hosts.txt > live.txt
• Route by tech (with extras): httpx_tech_route.py live.txt --tech wordpress,drupal --extra --wpscan
• Nuclei quick: nuclei_quick.sh live.txt cves,exposures
• SMB write check: smb_check_write.sh $TARGET sharename
• LDAP quick users: ldap_quick_users.sh $TARGET 'DC=target,DC=htb'
• Secrets scan: scan_secrets.sh .

Legend:

• DNS/httpx/nuclei/tech/wp: status + elapsed time (OK(12s)).
• sev: short counts (c:2 h:3 m:5), bar: █ blocks colored per severity.
• --compact removes dates and shows essentials when terminal space is tight.
• --json lets you script your own dashboards. Example manifest snippet: { "target": "target.htb", "tasks": { "httpx": { "status": "ok", "started_at": "2025-10-08 10:21:00", "finished_at": "2025-10-08 10:21:08", "meta": { "urls": 34, "urls_file": "targets/target.htb/scans/urls_20251008_1021.txt", "httpx_json": ".../httpx_20251008_1021.json", "httpx_summary": ".../httpx_20251008_1021.summary.json", "elapsed_sec": 8 } } } } Customizing Tech Routes:
• Edit httpx_tech_route.py to add or adjust presets in the presets map.
• To auto‑launch additional quick wrappers, update the --extra handler.

Auto Severity Tuning (nuclei):

• httpx_to_nuclei.sh sets nuclei severity via --severity auto mapping:

  • 500 URLs → high,critical; >100 → medium,high,critical; else low,medium,high,critical.

• Override with explicit --severity or adjust logic in the script.