+ 
+ 
+ 
+
+
+ 
+
+ Every scan ends with an AI SCAN BRIEF — severity totals, top exploitable chains, executive summary, and recommended next actions — framed in the terminal. Recorded live on scanme.nmap.org, models served by local Ollama.
+
+
+ Target: scanme.nmap.org · Nmap's authorized test host ·
+ Date: 2026-04-18 ·
+ Hardware: Apple M1 Pro · 16 GB RAM · Go 1.21 · macOS 25.4 ·
+ Binary: God's Eye v2.0-dev @ v2-dev
+
+
+ Built the binary yet? go build -o god-eye ./cmd/god-eye — then pick a recipe.
+
- 
- 
+ 
+
+ 
+
+ Zero-flag launch → AI tier → model check → target → profile → live scan. Recorded live against scanme.nmap.org.
+
+ 
+ 
- 
-
- 
- 
- 
- 
-blueviolet.svg?style=for-the-badge&logo=ollama){kind=icon}
+ 
+ 
+ 
+ 
- - Why? • - Features • - 🧠 AI • - Installation • - Usage • - Benchmarks • - Credits + ⚡ Quick start • + Why • + Features • + Wizard • + AI • + Live benchmark • + vs. competitors • + Legal
--- -## 🎯 Why God's Eye? - -| - -### ⚡ All-in-One -**20 passive sources** + DNS brute-forcing + HTTP probing + security checks in **one tool**. No need to chain 5+ tools together. - - | -- -### 🧠 AI-Powered -**Zero-cost local AI** with Ollama for intelligent vulnerability analysis, CVE detection, and executive reports. **100% private**. - - | -- -### 🚀 Production-Ready -Battle-tested on **real bug bounties**. Fast, reliable, and packed with features that actually matter. - - | -
- 
+ Live colorized event stream — every finding appears as it's discovered. +
-God's Eye now features **AI-powered security analysis** using local LLM models via Ollama: -- ✅ **100% Local & Private** - No data leaves your machine -- ✅ **Free Forever** - No API costs -- ✅ **Intelligent Analysis** - JavaScript code review, CVE detection, anomaly identification -- ✅ **Smart Cascade** - Fast triage + deep analysis for optimal performance +--- + +## 🔍 What it finds + +### 🛰️ Discovery — 11 module types, 26 passive sources + +
-
-**Basic Scan**
-
-Standard subdomain enumeration
-
- |
-
-
-**AI-Powered Scan**
-
-With real-time CVE detection & analysis
-
- |
-
| Header audit | HSTS · CSP · X-Frame-Options · X-Content-Type-Options · Referrer-Policy · Permissions-Policy. OWASP-aligned with remediation text. |
| Surface misconfigs | Open redirect · CORS wildcards · dangerous HTTP methods · Git/SVN exposure · backup-file discovery · admin/API-endpoint enumeration |
| Takeover | 110+ fingerprints: GitHub Pages, S3, CloudFront, Heroku, Netlify, Vercel, Azure Web Apps, Shopify, … |
| GraphQL | Introspection enabled detection + mutation-enabled flag (v2 native) |
| JWT | alg=none, excessive expiry, kid-injection, weak-HMAC crack (v2 native) |
| HTTP smuggling | CL.TE / TE.CL timing probe, non-destructive (v2 native, opt-in) |
| Cloud assets | S3 / GCS / Azure Blob / Firebase enumeration |
| Secret extraction | Regex + entropy + validation. FP denylist for third-party APIs and UI strings. |
| Nuclei compat | ~13k community templates, HTTP subset, auto-scope-filtered (no off-host false positives) |
- Made with ❤️ by Vyntral for Orizon + Every number in this README is reproducible. No marketing fluff, no synthetic benchmarks, no vendor lock-in. Just a single Go binary, your local machine, and the targets you're authorized to test.
diff --git a/SECURITY.md b/SECURITY.md index e7e6c53..b3f8a29 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,129 +1,140 @@ -# Security Policy +# 🛡️ Security Policy & Responsible Use -## Responsible Use ++ + God's Eye is a serious offensive-security tool. + It finds real vulnerabilities on real targets. + Use it only on systems you own or have written permission to test. + +
-God's Eye is a powerful security reconnaissance tool. With great power comes great responsibility. +--- -### Ethical Guidelines +## Why this doc exists + +God's Eye v2 can do damage. The same pipeline that surfaces a critical CVE correlation on your own asset will surface it just as well on your ex-employer's infrastructure — and the latter is a crime. This document sets the boundary between useful and illegal use, and it explains how to report vulnerabilities **in the tool itself** when you find them. + +--- + +## Responsible use + +### Ethical guidelines ✅ **DO:** -- Use for authorized penetration testing -- Participate in bug bounty programs -- Conduct security research on your own systems -- Help improve security through responsible disclosure -- Follow coordinated vulnerability disclosure processes +- Use for **authorized** penetration testing engagements +- Participate in bug-bounty programs **within their declared scope** +- Conduct security research on systems **you own** or have **written permission** to test +- Help improve defense through responsible disclosure +- Follow coordinated vulnerability-disclosure processes ❌ **DO NOT:** - Scan systems without explicit permission -- Use for malicious purposes -- Violate terms of service -- Attempt unauthorized access -- Sell or distribute scan results without authorization +- Chain vulnerabilities or exfiltrate data on targets you don't own +- Violate bug-bounty program terms of service +- Use God's Eye for initial access, lateral movement, or persistence on unauthorized systems +- Sell or republish scan results without the asset owner's consent -## Reporting Security Issues +--- -### Vulnerability Disclosure +## Reporting Security Issues *in God's Eye itself* -If you discover a security vulnerability in God's Eye itself, please report it responsibly: +If you discover a vulnerability in the tool (e.g., input injection via the CLI, SSRF in a fetch module, prompt injection against the AI layer), report it **privately**. -1. **DO NOT** open a public issue -2. Email the maintainers privately (see GitHub profile for contact) -3. Provide detailed information: - - Description of the vulnerability - - Steps to reproduce - - Potential impact - - Suggested fix (if any) +1. **DO NOT** open a public GitHub issue. +2. Email the maintainer or open a private security advisory on the repository. +3. Include: + - Affected component (package path + version or branch) + - Reproduction steps + - Impact assessment + - Suggested fix if available ### Response Timeline -- **Acknowledgment**: Within 48 hours -- **Initial Assessment**: Within 7 days -- **Fix Development**: Depends on severity -- **Public Disclosure**: After fix is released +| Stage | Target | +|--------------------|-----------------------------------------| +| Acknowledgment | Within 48 hours | +| Initial assessment | Within 7 days | +| Fix development | Driven by severity (24h critical → 30d low) | +| Public disclosure | After a patched release | + +--- ## Security Best Practices ### For Users -1. **Always verify authorization** before scanning -2. **Keep the tool updated** to latest version -3. **Use in controlled environments** when testing -4. **Respect rate limits** to avoid service disruption -5. **Secure your scan results** - they may contain sensitive data +1. **Always verify authorization** before scanning. +2. **Keep the tool updated** — v2 modules add new probe types that may break old rules of engagement you had in place. +3. **Scope the AI layer** — AI modules send finding evidence to the LLM. With the default Ollama path this stays on your machine, but if you swap in a cloud provider later, make sure your ROE permits that. +4. **Respect rate limits** — adaptive per-host limiting is built in, but some targets have hard ceilings; honor them. +5. **Secure your scan results** — output files may contain exposed credentials, internal hostnames, CVE mappings. -### For Developers +### For Contributors -1. **Review code changes** for security implications -2. **Follow secure coding practices** -3. **Test thoroughly** before releasing -4. **Document security-relevant changes** -5. **Never commit credentials** or sensitive data +1. Review module code for SSRF, command injection, and path traversal before merging. +2. Never log raw secrets. The `secrets.Kind` field is redacted by default; don't bypass redaction in new modules. +3. Keep network-dependent tests behind `-tags integration` so CI doesn't leak traffic to third parties. +4. Add new probe types to the ROE-impact note in the release changelog. + +--- ## Compliance -### Legal Requirements +Users must comply with all laws applicable to them, including: -Users must comply with: - -- **United States**: Computer Fraud and Abuse Act (CFAA), 18 U.S.C. § 1030 -- **European Union**: GDPR, ePrivacy Directive, NIS2 Directive -- **United Kingdom**: Computer Misuse Act 1990 -- **International**: Budapest Convention on Cybercrime -- **Local laws**: All applicable regional regulations +- **United States** — Computer Fraud and Abuse Act (CFAA), 18 U.S.C. § 1030 +- **European Union** — GDPR, NIS2 Directive +- **United Kingdom** — Computer Misuse Act 1990 +- **International** — Budapest Convention on Cybercrime +- **Local** — anything stricter than the above in your jurisdiction ### Bug Bounty Programs -When using God's Eye for bug bounty hunting: +When using God's Eye in a bug-bounty context: -1. ✅ Read and follow program rules -2. ✅ Respect scope limitations -3. ✅ Avoid testing production systems unless explicitly allowed -4. ✅ Report findings through proper channels -5. ✅ Do not publicly disclose before program authorization +1. Read the program's scope, **including out-of-scope paths**. +2. Respect "no automated scanning" rules — several modules (brute-force, permutation, smuggling probe) qualify. +3. Never test in production unless the program explicitly permits it. +4. Submit findings through the program's channel, not publicly. +5. Disclose only after authorization. + +--- ## Data Protection -### Handling Scan Results - Scan results may contain sensitive information: -- Private IP addresses -- Technology stack details -- Potential vulnerabilities -- Configuration information +- Private IP addresses and internal hostnames +- Technology stack details with exact versions +- Identified vulnerabilities and working PoCs +- Cloud asset metadata -**Your Responsibilities:** +**Your responsibilities:** -1. Store results securely -2. Encrypt sensitive data -3. Delete when no longer needed -4. Do not share without authorization -5. Comply with GDPR and data protection laws +1. Encrypt scan results at rest. +2. Delete them when no longer needed. +3. Do not share outside the engagement without the asset owner's consent. +4. Comply with data-protection laws applicable to the target's jurisdiction. + +--- ## Disclaimer **NO WARRANTY**: This software is provided "AS IS" without warranty of any kind. -**NO LIABILITY**: The authors are not responsible for: +**NO LIABILITY**: The author is not responsible for: - Misuse of this tool - Unauthorized access attempts - Legal consequences of improper use -- Data breaches or security incidents +- Data breaches or service disruptions caused by your scans - Any damages arising from use **USER RESPONSIBILITY**: You are solely responsible for ensuring: - You have proper authorization -- Your use complies with all laws +- Your use complies with all applicable laws - You accept all risks -- You will not hold authors liable - -## Contact - -For security-related questions: -- Check the [LICENSE](LICENSE) file for legal terms -- Review the [README](README.md) for usage guidelines -- Contact maintainers through GitHub for private security reports +- You will not hold the author liable --- -**Remember: Unauthorized computer access is illegal. Always get permission first.** +**Remember: unauthorized computer access is illegal. Always get written permission first.** diff --git a/assets/ai-verbose.gif b/assets/ai-verbose.gif new file mode 100644 index 0000000..432c5aa Binary files /dev/null and b/assets/ai-verbose.gif differ diff --git a/assets/demo-ai.gif b/assets/demo-ai.gif deleted file mode 100644 index 06a265f..0000000 Binary files a/assets/demo-ai.gif and /dev/null differ diff --git a/assets/demo.gif b/assets/demo.gif deleted file mode 100644 index e751bb2..0000000 Binary files a/assets/demo.gif and /dev/null differ diff --git a/assets/live-scan.gif b/assets/live-scan.gif new file mode 100644 index 0000000..9f6714b Binary files /dev/null and b/assets/live-scan.gif differ diff --git a/assets/wizard-demo.gif b/assets/wizard-demo.gif new file mode 100644 index 0000000..537de7a Binary files /dev/null and b/assets/wizard-demo.gif differ diff --git a/cmd/god-eye/main.go b/cmd/god-eye/main.go index 0cddfc9..01a2c7a 100644 --- a/cmd/god-eye/main.go +++ b/cmd/god-eye/main.go @@ -1,18 +1,39 @@ package main import ( + "context" "fmt" "os" + "os/signal" + "syscall" + "time" "github.com/spf13/cobra" "god-eye/internal/ai" "god-eye/internal/config" + "god-eye/internal/diff" + "god-eye/internal/modules/all" + "god-eye/internal/nucleitpl" "god-eye/internal/output" + "god-eye/internal/pipeline" + gohttp "god-eye/internal/http" + "god-eye/internal/proxyconf" "god-eye/internal/scanner" + "god-eye/internal/scheduler" + "god-eye/internal/sources" + "god-eye/internal/store" + "god-eye/internal/tui" "god-eye/internal/validator" + "god-eye/internal/wizard" ) +var _ = diff.Compute // ensure diff import is kept in the dependency graph + +// rootCmdRef is set by main() so helpers can query which flags cobra saw +// explicitly on the command line (via Flags().Changed). +var rootCmdRef *cobra.Command + func main() { var cfg config.Config @@ -33,6 +54,20 @@ Examples: god-eye -d example.com --stealth moderate Moderate stealth (evasion mode) god-eye -d example.com --stealth paranoid Maximum stealth (very slow)`, Run: func(cmd *cobra.Command, args []string) { + // If no target given and stdin is a TTY, launch the interactive wizard. + // Explicit --wizard also triggers it even with a target present (user + // wants to review defaults). + if (cfg.Domain == "" && wizard.IsInteractive()) || cfg.Wizard { + if err := runWizard(&cfg); err != nil { + if err == wizard.ErrCancelled { + fmt.Println(output.Yellow("cancelled.")) + os.Exit(130) + } + fmt.Println(output.Red("[-]"), "wizard:", err) + os.Exit(1) + } + } + if cfg.Domain == "" { fmt.Println(output.Red("[-]"), "Domain is required. Use -d flag.") cmd.Help() @@ -58,6 +93,27 @@ Examples: fmt.Println(output.Red("[-]"), "Invalid resolvers:", err.Error()) os.Exit(1) } + if err := proxyconf.Validate(cfg.Proxy); err != nil { + fmt.Println(output.Red("[-]"), "Invalid --proxy:", err.Error()) + os.Exit(1) + } + // Propagate proxy config to every HTTP client before anything + // else spins up. This must happen after validation and before + // the pipeline/scanner starts. + if cfg.Proxy != "" { + if err := gohttp.SetProxy(cfg.Proxy); err != nil { + fmt.Println(output.Red("[-]"), "proxy (http factory):", err.Error()) + os.Exit(1) + } + if err := sources.SetProxy(cfg.Proxy); err != nil { + fmt.Println(output.Red("[-]"), "proxy (sources):", err.Error()) + os.Exit(1) + } + if !cfg.Silent { + fmt.Printf("%s Routing HTTP through %s\n", + output.BoldCyan("⛓"), output.BoldWhite(proxyconf.Humanize(cfg.Proxy))) + } + } if err := validator.ValidateConcurrency(cfg.Concurrency); err != nil { fmt.Println(output.Red("[-]"), "Invalid concurrency:", err.Error()) os.Exit(1) @@ -111,6 +167,10 @@ Examples: fmt.Println() } + if cfg.UsePipeline { + runPipeline(cfg) + return + } scanner.Run(cfg) }, } @@ -135,8 +195,8 @@ Examples: // AI flags rootCmd.Flags().BoolVar(&cfg.EnableAI, "enable-ai", false, "Enable AI-powered analysis with Ollama (includes CVE search)") rootCmd.Flags().StringVar(&cfg.AIUrl, "ai-url", "http://localhost:11434", "Ollama API URL") - rootCmd.Flags().StringVar(&cfg.AIFastModel, "ai-fast-model", "deepseek-r1:1.5b", "Fast triage model") - rootCmd.Flags().StringVar(&cfg.AIDeepModel, "ai-deep-model", "qwen2.5-coder:7b", "Deep analysis model (supports function calling)") + rootCmd.Flags().StringVar(&cfg.AIFastModel, "ai-fast-model", "qwen3:1.7b", "Fast triage model (Ollama tag)") + rootCmd.Flags().StringVar(&cfg.AIDeepModel, "ai-deep-model", "qwen2.5-coder:14b", "Deep analysis model (Ollama tag, supports function calling)") rootCmd.Flags().BoolVar(&cfg.AICascade, "ai-cascade", true, "Use cascade (fast triage + deep analysis)") rootCmd.Flags().BoolVar(&cfg.AIDeepAnalysis, "ai-deep", false, "Enable deep AI analysis on all findings") rootCmd.Flags().BoolVar(&cfg.MultiAgent, "multi-agent", false, "Enable multi-agent orchestration (8 specialized AI agents)") @@ -144,6 +204,27 @@ Examples: // Stealth flags rootCmd.Flags().StringVar(&cfg.StealthMode, "stealth", "", "Stealth mode: light, moderate, aggressive, paranoid (reduces detection)") + // v2 pipeline flags + rootCmd.Flags().BoolVar(&cfg.UsePipeline, "pipeline", false, "Use v2 event-driven pipeline (experimental, parity with v1 verified by F0.7)") + rootCmd.Flags().BoolVar(&cfg.Wizard, "wizard", false, "Force the interactive setup wizard even when -d is set") + rootCmd.Flags().StringVar(&cfg.Profile, "profile", "", "Apply named scan profile (bugbounty, pentest, asm-continuous, stealth-max, quick)") + rootCmd.Flags().StringVar(&cfg.ConfigFile, "config", "", "Path to YAML config file (overrides auto-discovery)") + + // Stash the rootCmd in a package var so runPipeline can check which + // flags the user set explicitly (cobra is the only thing that knows). + rootCmdRef = rootCmd + rootCmd.Flags().BoolVar(&cfg.Live, "live", false, "Stream colorized scan events live to the terminal (v2 only)") + rootCmd.Flags().IntVar(&cfg.LiveVerbosity, "live-verbosity", 1, "Live view verbosity: 0=findings-only, 1=normal, 2=noisy") + rootCmd.Flags().StringVar(&cfg.AIProfile, "ai-profile", "", "AI tier: lean (16GB), balanced (32GB), heavy/max (64GB+). Overrides --ai-fast-model/--ai-deep-model unless those are also set explicitly.") + rootCmd.Flags().BoolVar(&cfg.AIVerbose, "ai-verbose", false, "Log every Ollama query (model, prompt/response size, duration) to stderr") + rootCmd.Flags().BoolVar(&cfg.AutoPullModels, "ai-auto-pull", true, "Auto-download missing Ollama models before the scan starts") + rootCmd.Flags().BoolVar(&cfg.NucleiScan, "nuclei", false, "Run Nuclei-format YAML templates against every probed host") + rootCmd.Flags().StringVar(&cfg.NucleiTemplates, "nuclei-templates", "", "Path to Nuclei templates directory (default: $NUCLEI_TEMPLATES, then ~/nuclei-templates, then ~/.god-eye/nuclei-templates)") + rootCmd.Flags().BoolVar(&cfg.NucleiAutoDownload, "nuclei-auto-download", true, "Auto-download nuclei-templates ZIP from GitHub when no local dir is found") + rootCmd.Flags().StringVar(&cfg.Proxy, "proxy", "", "Route outbound HTTP through a proxy. Supported: http://host:port, https://host:port, socks5://host:port, socks5h://host:port (Tor). Basic auth via http://user:pass@host.") + rootCmd.Flags().DurationVar(&cfg.MonitorInterval, "monitor-interval", 0, "Run in continuous monitoring mode, re-scanning every N (e.g. 6h, 24h). Emits diffs.") + rootCmd.Flags().StringVar(&cfg.MonitorWebhook, "monitor-webhook", "", "Webhook URL to POST diff reports to in monitoring mode") + // Recursive discovery flags (enabled by default with --enable-ai) rootCmd.Flags().BoolVar(&cfg.Recursive, "recursive", false, "Enable recursive subdomain discovery with pattern learning") rootCmd.Flags().IntVar(&cfg.RecursiveDepth, "recursive-depth", 3, "Maximum recursion depth (1-5)") @@ -224,7 +305,288 @@ This data is used for instant, offline CVE lookups during scans.`, } rootCmd.AddCommand(dbInfoCmd) + // nuclei-update: force refresh of the auto-downloaded Nuclei template cache + nucleiUpdateCmd := &cobra.Command{ + Use: "nuclei-update", + Short: "Download / refresh Nuclei YAML templates cache", + Long: `Fetches the official projectdiscovery/nuclei-templates ZIP archive +and extracts every .yaml/.yml file into ~/.god-eye/nuclei-templates. + +Safe to re-run: existing templates are overwritten in-place. The cache +is ~40MB on disk and ships thousands of detections that the compat +layer executes when --nuclei is on.`, + Run: func(cmd *cobra.Command, args []string) { + home, err := os.UserHomeDir() + if err != nil { + fmt.Println(output.Red("[-]"), "cannot find home dir:", err) + os.Exit(1) + } + dest := home + "/.god-eye/nuclei-templates" + + fmt.Println(output.BoldCyan("📥 Refreshing Nuclei templates…")) + fmt.Printf(" %s %s\n", output.Dim("destination:"), output.BoldWhite(dest)) + + // Pull up the downloader. Inline import to keep the subcommand + // lightweight when not invoked. + dl := nucleitpl.NewDownloader() + dl.Verbose = true + if err := dl.Refresh(dest); err != nil { + fmt.Println(output.Red("[-]"), "refresh failed:", err) + os.Exit(1) + } + fmt.Println(output.Green("✓ Nuclei templates refreshed.")) + }, + } + rootCmd.AddCommand(nucleiUpdateCmd) + if err := rootCmd.Execute(); err != nil { os.Exit(1) } } + +// runPipeline is the v2 entry point. Registers every adapter module, loads +// optional YAML + profile, and runs the event-driven pipeline under a +// signal-aware context. +func runPipeline(cfg config.Config) { + // Side-effect registration of all adapter modules (F0.6). + all.RegisterAll() + + // Load YAML config if present. --config wins over auto-discovery. + path := cfg.ConfigFile + if path == "" { + path = config.FindConfigFile() + } + if path != "" { + if y, err := config.LoadYAML(path); err != nil { + fmt.Println(output.Red("[-]"), "config:", err.Error()) + os.Exit(1) + } else if y != nil { + config.ApplyYAML(&cfg, y) + } + } + + // Apply named scan profile if set. + if cfg.Profile != "" { + p, ok := config.ProfileByName(cfg.Profile) + if !ok { + fmt.Println(output.Red("[-]"), "unknown profile:", cfg.Profile) + os.Exit(1) + } + config.ApplyProfile(&cfg, p) + if !cfg.Silent { + fmt.Printf("%s Profile %s applied: %s\n", output.Green("✓"), output.BoldCyan(p.Name), output.Dim(p.Description)) + } + } + + // Apply AI tier profile (lean/balanced/heavy). Respects explicit + // --ai-fast-model / --ai-deep-model overrides. + if cfg.AIProfile != "" { + p, ok := config.AIProfileByName(cfg.AIProfile) + if !ok { + fmt.Println(output.Red("[-]"), "unknown AI profile:", cfg.AIProfile, + "— valid: lean, balanced, heavy") + os.Exit(1) + } + overrideFast := rootCmdRef != nil && rootCmdRef.Flags().Changed("ai-fast-model") + overrideDeep := rootCmdRef != nil && rootCmdRef.Flags().Changed("ai-deep-model") + config.ApplyAIProfile(&cfg, p, overrideFast, overrideDeep) + if !cfg.Silent { + fmt.Printf("%s AI profile %s: %s\n", + output.Green("✓"), output.BoldCyan(p.Name), output.Dim(p.Description)) + fmt.Printf(" %s %s %s %s\n", + output.Dim("triage:"), output.BoldWhite(cfg.AIFastModel), + output.Dim("deep:"), output.BoldWhite(cfg.AIDeepModel)) + } + } + + // Handle Ctrl-C gracefully. Set this up BEFORE the model-ensure step + // so long downloads can be interrupted cleanly. + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM) + go func() { + <-sigCh + fmt.Println() + fmt.Println(output.Yellow("⚠ Interrupted — shutting down...")) + cancel() + }() + + // Ensure Ollama models are present before scan starts. + if cfg.EnableAI && cfg.AutoPullModels { + if err := ensureAIModels(ctx, &cfg); err != nil { + if ctx.Err() == context.Canceled { + os.Exit(130) + } + fmt.Println(output.Red("[-]"), "AI setup:", err) + os.Exit(1) + } + } + + // Continuous monitoring mode: run the scan on an interval, diff and alert. + if cfg.MonitorInterval > 0 { + runMonitor(ctx, cfg) + return + } + + p, err := pipeline.New(&cfg, pipeline.Options{}) + if err != nil { + fmt.Println(output.Red("[-]"), err) + os.Exit(1) + } + + var live *tui.LivePrinter + if cfg.Live { + live = tui.NewLivePrinter(p.Bus(), cfg.LiveVerbosity) + } + + if err := p.Run(ctx); err != nil { + if ctx.Err() == context.Canceled { + if live != nil { + live.Close() + } + os.Exit(130) + } + fmt.Println(output.Red("[!]"), "pipeline error:", err) + os.Exit(1) + } + + if live != nil { + live.Close() + } +} + +// runMonitor implements the asm-continuous mode: a single pipeline.Run +// wrapped in scheduler.Scheduler that ticks at MonitorInterval, diffs +// against the previous snapshot, and alerts on meaningful changes. +func runMonitor(ctx context.Context, cfg config.Config) { + scan := func(scanCtx context.Context) ([]*store.Host, error) { + p, err := pipeline.New(&cfg, pipeline.Options{}) + if err != nil { + return nil, err + } + if err := p.Run(scanCtx); err != nil { + return nil, err + } + return p.Store().All(scanCtx), nil + } + + s := scheduler.New(cfg.Domain, cfg.MonitorInterval, scan) + s.AddAlerter(scheduler.StdoutAlerter{}) + if cfg.MonitorWebhook != "" { + s.AddAlerter(scheduler.NewWebhookAlerter(cfg.MonitorWebhook)) + } + + fmt.Printf("%s Monitoring %s every %s — Ctrl-C to stop\n", + output.BoldGreen("▣"), output.BoldCyan(cfg.Domain), cfg.MonitorInterval) + + if err := s.Start(ctx); err != nil && !errorIs(err, context.Canceled) { + fmt.Println(output.Red("[!]"), "monitor error:", err) + os.Exit(1) + } +} + +// runWizard starts the interactive setup, then folds the user's choices +// back into cfg. Forces pipeline mode (wizard is v2-only by design). +func runWizard(cfg *config.Config) error { + choice, err := wizard.Run(context.Background(), wizard.Options{ + In: os.Stdin, + Out: os.Stdout, + OllamaURL: cfg.AIUrl, + }) + if err != nil { + return err + } + + cfg.Domain = validator.SanitizeDomain(choice.Target) + cfg.UsePipeline = true + cfg.Live = choice.Live + cfg.LiveVerbosity = choice.LiveVerbosity + cfg.Output = choice.Output + if choice.Format != "" { + cfg.Format = choice.Format + } + + // Scan profile name threads through --profile application (later). + if choice.ScanProfile != "" { + cfg.Profile = choice.ScanProfile + } + + // ASM-continuous interval translates into a duration flag. + if choice.MonitorInterval != "" { + d, parseErr := time.ParseDuration(choice.MonitorInterval) + if parseErr != nil { + return fmt.Errorf("invalid interval %q: %w", choice.MonitorInterval, parseErr) + } + cfg.MonitorInterval = d + } + + // AI tier. + if choice.AIProfile != "" { + cfg.EnableAI = true + cfg.AIProfile = choice.AIProfile + cfg.AIVerbose = choice.AIVerbose + cfg.AutoPullModels = choice.AIAutoPull + } else { + cfg.EnableAI = false + } + + return nil +} + +// ensureAIModels checks the Ollama server and downloads any missing models. +// Prints progress when --ai-verbose is on. Fails open on unreachable +// Ollama — the AI module itself will no-op gracefully. +func ensureAIModels(ctx context.Context, cfg *config.Config) error { + e := ai.NewModelEnsurer(cfg.AIUrl) + e.Verbose = cfg.AIVerbose || cfg.Verbose + e.Writer = os.Stderr + + if err := e.Reachable(ctx); err != nil { + if !cfg.Silent { + fmt.Println(output.Yellow("⚠ "), err.Error()) + fmt.Println(output.Dim(" AI modules will no-op for this run. Start `ollama serve` to enable.")) + } + return nil + } + + models := []string{} + if cfg.AIFastModel != "" { + models = append(models, cfg.AIFastModel) + } + if cfg.AIDeepModel != "" && cfg.AIDeepModel != cfg.AIFastModel { + models = append(models, cfg.AIDeepModel) + } + if len(models) == 0 { + return nil + } + + if !cfg.Silent { + fmt.Printf("%s Checking Ollama models: %s\n", + output.BoldCyan("⚙"), output.Dim(fmt.Sprintf("%v", models))) + } + if err := e.EnsureAll(ctx, models); err != nil { + return err + } + if !cfg.Silent { + fmt.Printf("%s Models ready\n", output.Green("✓")) + } + return nil +} + +// errorIs is a thin wrapper for errors.Is that only pulls errors into +// main when needed. +func errorIs(err, target error) bool { + for err != nil { + if err == target { + return true + } + type unwrapper interface{ Unwrap() error } + u, ok := err.(unwrapper) + if !ok { + return false + } + err = u.Unwrap() + } + return false +} diff --git a/go.mod b/go.mod index 79784ce..5e68957 100644 --- a/go.mod +++ b/go.mod @@ -4,17 +4,18 @@ go 1.21 require ( github.com/fatih/color v1.16.0 + github.com/mattn/go-isatty v0.0.20 github.com/miekg/dns v1.1.58 github.com/spf13/cobra v1.8.0 + golang.org/x/net v0.20.0 + gopkg.in/yaml.v3 v3.0.1 ) require ( github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/mattn/go-colorable v0.1.13 // indirect - github.com/mattn/go-isatty v0.0.20 // indirect github.com/spf13/pflag v1.0.5 // indirect golang.org/x/mod v0.14.0 // indirect - golang.org/x/net v0.20.0 // indirect golang.org/x/sys v0.16.0 // indirect golang.org/x/tools v0.17.0 // indirect ) diff --git a/go.sum b/go.sum index 4775209..cbf4921 100644 --- a/go.sum +++ b/go.sum @@ -27,5 +27,7 @@ golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU= golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc= golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/internal/agent/agent.go b/internal/agent/agent.go new file mode 100644 index 0000000..0ac196c --- /dev/null +++ b/internal/agent/agent.go @@ -0,0 +1,109 @@ +// Package agent defines the Fase 3 AI agentic v2 interfaces: Planner, +// Worker, and Tool. Unlike Fase 0.6 adapters that merely wrap v1 Ollama +// calls, a v2 Agent plans multi-step investigations and executes tools +// via the event bus. +// +// The Agent lifecycle: +// +// 1. Planner receives the target + existing store snapshot, produces a +// Plan (ordered list of Tasks). +// 2. Each Task is dispatched to a Worker (specialized agent: XSS, auth, +// API, crypto, secrets, etc.) with a Tool set. +// 3. Workers call Tools (dns_resolve, http_request, check_sqli_blind, +// fetch_js, query_cve, ...) and reason over the results. +// 4. Results feed back into Plan revision; new Tasks may be scheduled. +// +// This file defines the contracts. Implementations land incrementally; +// for now a Basic Planner delegates to the Fase 0.6 v1 Ollama wrapper, +// and a native tool-using implementation follows. +package agent + +import ( + "context" + "time" + + "god-eye/internal/eventbus" + "god-eye/internal/store" +) + +// Tool is a capability an agent can invoke. Tools should be idempotent +// where possible and must respect ctx cancellation. +type Tool interface { + // Name is the machine identifier (e.g., "http_request", "dns_resolve"). + // Used in tool-call serialization for LLMs. + Name() string + + // Description is a short human-readable blurb used in the LLM tool + // descriptor. Keep it action-oriented: "fetch an HTTP URL and return + // the response headers + first 2KB of body". + Description() string + + // Schema returns the JSON-schema of the tool's argument object. Used + // to build function-calling descriptors and to validate inputs. + Schema() map[string]interface{} + + // Call invokes the tool with the given arguments. Returns a JSON-encoded + // result (often just a text summary). Errors should be returned — the + // agent decides how to react. + Call(ctx context.Context, args map[string]interface{}) (string, error) +} + +// Task is a single unit of agent work. +type Task struct { + ID string + Kind string // e.g. "investigate-xss", "audit-auth", "chain-finding" + Description string // natural-language goal the worker pursues + Subject string // target URL / subdomain / evidence the task focuses on + Context map[string]string // additional hints for the worker + CreatedAt time.Time +} + +// Plan is an ordered list of Tasks produced by the Planner. +type Plan struct { + Target string + Tasks []Task + Reason string // planner's rationale, logged for debugging +} + +// Planner decides what to investigate next given the current store state. +type Planner interface { + // Plan produces a new Plan. Called at the start of the analysis phase + // and whenever enough new evidence accumulates to justify replanning. + Plan(ctx context.Context, target string, storeSnap store.Store, bus *eventbus.Bus) (*Plan, error) + + // Name identifies the planner implementation for logs. + Name() string +} + +// Worker executes a single Task using a Toolset. +type Worker interface { + // Name identifies the worker (usually its specialization, e.g. "xss", + // "auth", "api", "crypto"). + Name() string + + // CanHandle reports whether the worker is a good fit for task. Workers + // are consulted in priority order. + CanHandle(task Task) bool + + // Execute carries out the task. The worker may call tools, update the + // store via bus events (VulnerabilityFound, SecretFound, AIFinding), + // and return a short natural-language summary for the planner. + Execute(ctx context.Context, task Task, tools Toolset, bus *eventbus.Bus, st store.Store) (summary string, err error) +} + +// Toolset is an indexed collection of Tools available to a worker. It is +// intentionally separate from Registry so workers receive a curated subset +// (e.g., a "crypto" worker gets oracle-style tools but not "send_slack"). +type Toolset map[string]Tool + +// Get returns the named tool, or nil if absent. +func (ts Toolset) Get(name string) Tool { return ts[name] } + +// Names returns every tool name in the set. Order is not guaranteed. +func (ts Toolset) Names() []string { + out := make([]string, 0, len(ts)) + for n := range ts { + out = append(out, n) + } + return out +} diff --git a/internal/agent/tools.go b/internal/agent/tools.go new file mode 100644 index 0000000..958e232 --- /dev/null +++ b/internal/agent/tools.go @@ -0,0 +1,141 @@ +package agent + +import ( + "context" + "crypto/tls" + "encoding/json" + "errors" + "io" + "net/http" + "time" + + godns "god-eye/internal/dns" +) + +// --- built-in tools ------------------------------------------------------- +// +// These tools cover the minimum needed for a planner to investigate +// discovered hosts without reinventing basic primitives. Fase 3 workers +// receive curated subsets via Toolset. + +// HTTPRequestTool fetches an arbitrary URL and returns status, headers, +// and (truncated) body. Maximum 64KB body returned. +type HTTPRequestTool struct { + Client *http.Client +} + +func NewHTTPRequestTool(timeoutSec int) *HTTPRequestTool { + return &HTTPRequestTool{ + Client: &http.Client{ + Timeout: time.Duration(timeoutSec) * time.Second, + Transport: &http.Transport{ + TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, + }, + }, + } +} + +func (t *HTTPRequestTool) Name() string { return "http_request" } +func (t *HTTPRequestTool) Description() string { return "Fetch an HTTP(S) URL and return status + headers + first 64KB of body." } + +func (t *HTTPRequestTool) Schema() map[string]interface{} { + return map[string]interface{}{ + "type": "object", + "properties": map[string]interface{}{ + "url": map[string]interface{}{"type": "string"}, + "method": map[string]interface{}{"type": "string", "default": "GET"}, + "headers": map[string]interface{}{ + "type": "object", + "additionalProperties": map[string]interface{}{"type": "string"}, + }, + }, + "required": []string{"url"}, + } +} + +func (t *HTTPRequestTool) Call(ctx context.Context, args map[string]interface{}) (string, error) { + url, _ := args["url"].(string) + if url == "" { + return "", errors.New("url is required") + } + method, _ := args["method"].(string) + if method == "" { + method = "GET" + } + req, err := http.NewRequestWithContext(ctx, method, url, nil) + if err != nil { + return "", err + } + if hdrs, ok := args["headers"].(map[string]interface{}); ok { + for k, v := range hdrs { + if s, ok := v.(string); ok { + req.Header.Set(k, s) + } + } + } + req.Header.Set("User-Agent", "god-eye-v2-agent") + + resp, err := t.Client.Do(req) + if err != nil { + return "", err + } + defer resp.Body.Close() + + body, _ := io.ReadAll(io.LimitReader(resp.Body, 64*1024)) + out := map[string]interface{}{ + "status_code": resp.StatusCode, + "headers": flattenHeaders(resp.Header), + "body": string(body), + } + b, _ := json.Marshal(out) + return string(b), nil +} + +// DNSResolveTool resolves a hostname to A/CNAME/PTR records. +type DNSResolveTool struct { + Resolvers []string + TimeoutSec int +} + +func NewDNSResolveTool(resolvers []string, timeoutSec int) *DNSResolveTool { + if len(resolvers) == 0 { + resolvers = []string{"8.8.8.8:53", "1.1.1.1:53"} + } + return &DNSResolveTool{Resolvers: resolvers, TimeoutSec: timeoutSec} +} + +func (t *DNSResolveTool) Name() string { return "dns_resolve" } +func (t *DNSResolveTool) Description() string { return "Resolve a hostname to A/CNAME/PTR records." } + +func (t *DNSResolveTool) Schema() map[string]interface{} { + return map[string]interface{}{ + "type": "object", + "properties": map[string]interface{}{ + "hostname": map[string]interface{}{"type": "string"}, + }, + "required": []string{"hostname"}, + } +} + +func (t *DNSResolveTool) Call(_ context.Context, args map[string]interface{}) (string, error) { + name, _ := args["hostname"].(string) + if name == "" { + return "", errors.New("hostname is required") + } + ips := godns.ResolveSubdomain(name, t.Resolvers, t.TimeoutSec) + cname := godns.ResolveCNAME(name, t.Resolvers, t.TimeoutSec) + out := map[string]interface{}{"ips": ips, "cname": cname} + b, _ := json.Marshal(out) + return string(b), nil +} + +func flattenHeaders(h http.Header) map[string]string { + out := make(map[string]string, len(h)) + for k, vs := range h { + if len(vs) == 0 { + continue + } + out[k] = vs[0] + } + return out +} diff --git a/internal/ai/ensure.go b/internal/ai/ensure.go new file mode 100644 index 0000000..bdb65e1 --- /dev/null +++ b/internal/ai/ensure.go @@ -0,0 +1,275 @@ +package ai + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strings" + "time" +) + +// ModelEnsurer verifies that a given list of Ollama models is present on +// the local server, and pulls any that are missing. Designed for the +// pre-scan warmup: God's Eye should not crash mid-scan because a model +// wasn't downloaded — EnsureAll fixes that before the pipeline starts. +type ModelEnsurer struct { + BaseURL string + Client *http.Client + Verbose bool + Writer io.Writer // where progress is printed; defaults to os.Stdout if nil +} + +// NewModelEnsurer constructs an ensurer against the given Ollama base URL +// (e.g. "http://localhost:11434"). The HTTP client has no timeout because +// a fresh pull of a 30B model can legitimately take 10+ minutes. +func NewModelEnsurer(baseURL string) *ModelEnsurer { + if baseURL == "" { + baseURL = "http://localhost:11434" + } + return &ModelEnsurer{ + BaseURL: strings.TrimRight(baseURL, "/"), + Client: &http.Client{Timeout: 0}, + } +} + +// Installed returns the set of model tags currently available on the +// Ollama server, keyed by the full name (e.g. "qwen3:1.7b"). +func (e *ModelEnsurer) Installed(ctx context.Context) (map[string]bool, error) { + req, err := http.NewRequestWithContext(ctx, "GET", e.BaseURL+"/api/tags", nil) + if err != nil { + return nil, err + } + c := &http.Client{Timeout: 10 * time.Second} + resp, err := c.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + return nil, fmt.Errorf("ollama /api/tags returned %d", resp.StatusCode) + } + + var body struct { + Models []struct { + Name string `json:"name"` + } `json:"models"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return nil, err + } + out := make(map[string]bool, len(body.Models)) + for _, m := range body.Models { + out[m.Name] = true + } + return out, nil +} + +// Pull streams a model pull from Ollama, printing progress lines when +// Verbose is true. Uses POST /api/pull with stream=true; each JSON line +// reports status + optional {total, completed} for byte-level progress. +func (e *ModelEnsurer) Pull(ctx context.Context, model string) error { + payload := map[string]interface{}{"name": model, "stream": true} + body, _ := json.Marshal(payload) + req, err := http.NewRequestWithContext(ctx, "POST", e.BaseURL+"/api/pull", bytes.NewReader(body)) + if err != nil { + return err + } + req.Header.Set("Content-Type", "application/json") + resp, err := e.Client.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + b, _ := io.ReadAll(io.LimitReader(resp.Body, 4096)) + return fmt.Errorf("ollama /api/pull returned %d: %s", resp.StatusCode, strings.TrimSpace(string(b))) + } + + scanner := bufio.NewScanner(resp.Body) + // Progress events can be large; bump the scanner buffer. + scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024) + + var lastStatus string + var lastPct int + for scanner.Scan() { + line := bytes.TrimSpace(scanner.Bytes()) + if len(line) == 0 { + continue + } + var ev struct { + Status string `json:"status"` + Digest string `json:"digest,omitempty"` + Total int64 `json:"total,omitempty"` + Completed int64 `json:"completed,omitempty"` + Error string `json:"error,omitempty"` + } + if err := json.Unmarshal(line, &ev); err != nil { + continue + } + if ev.Error != "" { + return fmt.Errorf("pull %s: %s", model, ev.Error) + } + if !e.Verbose { + continue + } + w := e.writer() + if ev.Total > 0 && ev.Completed > 0 { + pct := int(float64(ev.Completed) / float64(ev.Total) * 100) + // Throttle: new status line always; otherwise only print when + // the percentage has moved ≥5 points since the last emission + // (or reaches a final 100% for this status exactly once). + switch { + case ev.Status != lastStatus: + fmt.Fprintf(w, " %-24s %3d%% %s / %s\n", ev.Status, pct, humanBytes(ev.Completed), humanBytes(ev.Total)) + lastStatus = ev.Status + lastPct = pct + case pct >= lastPct+5 && pct < 100: + fmt.Fprintf(w, " %-24s %3d%% %s / %s\n", ev.Status, pct, humanBytes(ev.Completed), humanBytes(ev.Total)) + lastPct = pct + case pct == 100 && lastPct < 100: + fmt.Fprintf(w, " %-24s %3d%% %s / %s\n", ev.Status, pct, humanBytes(ev.Completed), humanBytes(ev.Total)) + lastPct = 100 + } + } else if ev.Status != lastStatus { + fmt.Fprintf(w, " %s\n", ev.Status) + lastStatus = ev.Status + lastPct = 0 + } + } + return scanner.Err() +} + +// EnsureAll checks every name in models. For each missing one it calls Pull. +// Already-present models are skipped. Returns on the first error. +// +// Name matching is generous: Ollama sometimes tags models as "qwen3:1.7b" +// and sometimes as "qwen3:1.7b-instruct-fp16", so we accept exact match, +// a ":latest" variant, or the bare model name with no tag. +func (e *ModelEnsurer) EnsureAll(ctx context.Context, models []string) error { + installed, err := e.Installed(ctx) + if err != nil { + return fmt.Errorf("query ollama: %w", err) + } + + unique := dedup(models) + missing := []string{} + for _, m := range unique { + if alreadyInstalled(installed, m) { + if e.Verbose { + fmt.Fprintf(e.writer(), "✓ %s already installed\n", m) + } + continue + } + missing = append(missing, m) + } + + if len(missing) == 0 { + return nil + } + + if e.Verbose { + fmt.Fprintf(e.writer(), "↓ Pulling %d missing model(s): %s\n", len(missing), strings.Join(missing, ", ")) + } + for _, m := range missing { + if err := ctx.Err(); err != nil { + return err + } + if e.Verbose { + fmt.Fprintf(e.writer(), "↓ %s\n", m) + } + if err := e.Pull(ctx, m); err != nil { + return fmt.Errorf("pull %s: %w", m, err) + } + if e.Verbose { + fmt.Fprintf(e.writer(), "✓ %s ready\n", m) + } + } + return nil +} + +// Reachable reports whether the Ollama server answers /api/tags. Callers +// should check this before EnsureAll to surface a friendly message. +func (e *ModelEnsurer) Reachable(ctx context.Context) error { + c := &http.Client{Timeout: 3 * time.Second} + req, err := http.NewRequestWithContext(ctx, "GET", e.BaseURL+"/api/tags", nil) + if err != nil { + return err + } + resp, err := c.Do(req) + if err != nil { + return errors.New("ollama not reachable at " + e.BaseURL + " (is `ollama serve` running?)") + } + resp.Body.Close() + if resp.StatusCode != 200 { + return fmt.Errorf("ollama at %s returned %d", e.BaseURL, resp.StatusCode) + } + return nil +} + +func (e *ModelEnsurer) writer() io.Writer { + if e.Writer != nil { + return e.Writer + } + return stdout +} + +var stdout io.Writer // populated by main via SetStdout; nil writer would fmt-print to os.Stdout + +// SetStdout installs the writer used when ModelEnsurer.Writer is nil. main.go +// sets this to os.Stdout; tests can set it to a bytes.Buffer. +func SetStdout(w io.Writer) { stdout = w } + +func alreadyInstalled(installed map[string]bool, model string) bool { + if installed[model] { + return true + } + if installed[model+":latest"] { + return true + } + if strings.Contains(model, ":") { + base := strings.SplitN(model, ":", 2)[0] + if installed[base] || installed[base+":latest"] { + return true + } + } + return false +} + +func dedup(ss []string) []string { + seen := make(map[string]struct{}, len(ss)) + out := make([]string, 0, len(ss)) + for _, s := range ss { + s = strings.TrimSpace(s) + if s == "" { + continue + } + if _, ok := seen[s]; ok { + continue + } + seen[s] = struct{}{} + out = append(out, s) + } + return out +} + +func humanBytes(n int64) string { + const k = 1024.0 + if n < int64(k) { + return fmt.Sprintf("%dB", n) + } + units := []string{"KB", "MB", "GB", "TB"} + v := float64(n) / k + for _, u := range units { + if v < k { + return fmt.Sprintf("%.1f%s", v, u) + } + v /= k + } + return fmt.Sprintf("%.1fPB", v) +} diff --git a/internal/ai/ensure_test.go b/internal/ai/ensure_test.go new file mode 100644 index 0000000..5b4329e --- /dev/null +++ b/internal/ai/ensure_test.go @@ -0,0 +1,214 @@ +package ai + +import ( + "bytes" + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestAlreadyInstalled(t *testing.T) { + installed := map[string]bool{ + "qwen3:1.7b": true, + "qwen2.5-coder:14b": true, + "custom-model:latest": true, + } + cases := []struct { + model string + want bool + }{ + {"qwen3:1.7b", true}, + {"qwen2.5-coder:14b", true}, + {"custom-model", true}, // via :latest fallback + {"llama3:8b", false}, + {"qwen3", false}, // bare name: only matches when ":latest" is installed (it isn't) + } + for _, c := range cases { + if got := alreadyInstalled(installed, c.model); got != c.want { + t.Errorf("alreadyInstalled(%q) = %v, want %v", c.model, got, c.want) + } + } +} + +func TestDedup(t *testing.T) { + got := dedup([]string{"a", "b", "a", "", "c", " b "}) + want := []string{"a", "b", "c"} + if len(got) != len(want) { + t.Fatalf("got %v, want %v", got, want) + } + for i := range got { + if got[i] != want[i] { + t.Errorf("index %d: got %q want %q", i, got[i], want[i]) + } + } +} + +func TestHumanBytes(t *testing.T) { + cases := []struct { + in int64 + want string + }{ + {0, "0B"}, + {512, "512B"}, + {1024, "1.0KB"}, + {1024 * 1024, "1.0MB"}, + {1024 * 1024 * 1024, "1.0GB"}, + {int64(2.5 * 1024 * 1024 * 1024), "2.5GB"}, + } + for _, c := range cases { + if got := humanBytes(c.in); got != c.want { + t.Errorf("humanBytes(%d) = %q, want %q", c.in, got, c.want) + } + } +} + +func TestInstalled_ParsesTagsResponse(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/tags" { + http.NotFound(w, r) + return + } + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "models": []map[string]string{ + {"name": "qwen3:1.7b"}, + {"name": "qwen2.5-coder:14b"}, + }, + }) + })) + defer srv.Close() + + e := NewModelEnsurer(srv.URL) + got, err := e.Installed(context.Background()) + if err != nil { + t.Fatal(err) + } + if !got["qwen3:1.7b"] || !got["qwen2.5-coder:14b"] { + t.Errorf("missing expected models: %v", got) + } +} + +func TestInstalled_Non200ReturnsError(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Error(w, "nope", http.StatusInternalServerError) + })) + defer srv.Close() + + e := NewModelEnsurer(srv.URL) + if _, err := e.Installed(context.Background()); err == nil { + t.Error("expected error on non-200") + } +} + +func TestReachable(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte(`{"models":[]}`)) + })) + defer srv.Close() + + e := NewModelEnsurer(srv.URL) + if err := e.Reachable(context.Background()); err != nil { + t.Errorf("expected reachable, got %v", err) + } +} + +func TestReachable_Unreachable(t *testing.T) { + e := NewModelEnsurer("http://127.0.0.1:1") // nothing listens here + if err := e.Reachable(context.Background()); err == nil { + t.Error("expected unreachable error") + } +} + +func TestPull_StreamsProgress(t *testing.T) { + // Fake Ollama that emits a few NDJSON status events. + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/pull" { + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "application/x-ndjson") + events := []string{ + `{"status":"pulling manifest"}`, + `{"status":"downloading","digest":"sha256:abc","total":1048576,"completed":524288}`, + `{"status":"downloading","digest":"sha256:abc","total":1048576,"completed":1048576}`, + `{"status":"verifying sha256 digest"}`, + `{"status":"writing manifest"}`, + `{"status":"success"}`, + } + for _, e := range events { + w.Write([]byte(e + "\n")) + if flusher, ok := w.(http.Flusher); ok { + flusher.Flush() + } + } + })) + defer srv.Close() + + buf := &bytes.Buffer{} + e := NewModelEnsurer(srv.URL) + e.Verbose = true + e.Writer = buf + + if err := e.Pull(context.Background(), "fake:1b"); err != nil { + t.Fatalf("unexpected error: %v", err) + } + + out := buf.String() + if !strings.Contains(out, "pulling manifest") { + t.Errorf("missing 'pulling manifest' in output: %q", out) + } + if !strings.Contains(out, "success") { + t.Errorf("missing 'success' in output: %q", out) + } +} + +func TestPull_ErrorBubblesUp(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte(`{"error":"model not found"}` + "\n")) + })) + defer srv.Close() + + e := NewModelEnsurer(srv.URL) + err := e.Pull(context.Background(), "nonexistent") + if err == nil { + t.Fatal("expected error") + } + if !strings.Contains(err.Error(), "model not found") { + t.Errorf("unexpected error: %v", err) + } +} + +func TestEnsureAll_SkipsInstalled_PullsMissing(t *testing.T) { + pullCalls := map[string]int{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/tags": + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "models": []map[string]string{{"name": "already-here:1b"}}, + }) + case "/api/pull": + var body struct { + Name string `json:"name"` + } + _ = json.NewDecoder(r.Body).Decode(&body) + pullCalls[body.Name]++ + w.Write([]byte(`{"status":"success"}` + "\n")) + default: + http.NotFound(w, r) + } + })) + defer srv.Close() + + e := NewModelEnsurer(srv.URL) + if err := e.EnsureAll(context.Background(), []string{"already-here:1b", "missing-a:7b", "missing-b:14b"}); err != nil { + t.Fatal(err) + } + if pullCalls["already-here:1b"] > 0 { + t.Errorf("should not have pulled already-here") + } + if pullCalls["missing-a:7b"] != 1 || pullCalls["missing-b:14b"] != 1 { + t.Errorf("missing models not pulled correctly: %v", pullCalls) + } +} diff --git a/internal/ai/ollama.go b/internal/ai/ollama.go index a472cff..7587419 100644 --- a/internal/ai/ollama.go +++ b/internal/ai/ollama.go @@ -4,7 +4,9 @@ import ( "bytes" "encoding/json" "fmt" + "io" "net/http" + "os" "strings" "time" ) @@ -12,10 +14,27 @@ import ( // OllamaClient handles communication with local Ollama instance type OllamaClient struct { BaseURL string - FastModel string // deepseek-r1:1.5b for quick triage - DeepModel string // qwen2.5-coder:7b for deep analysis + FastModel string // qwen3:1.7b for quick triage (lean default) + DeepModel string // qwen2.5-coder:14b for deep analysis (lean default) Timeout time.Duration EnableCascade bool + + // Verbose controls whether every query is logged with timing + sizes. + // Writes to VerboseLogger or stderr when nil. Toggle via --ai-verbose. + Verbose bool + VerboseLogger io.Writer +} + +// logVerbose writes a single line to the verbose logger when Verbose is on. +func (c *OllamaClient) logVerbose(format string, args ...interface{}) { + if !c.Verbose { + return + } + w := c.VerboseLogger + if w == nil { + w = os.Stderr + } + fmt.Fprintf(w, "[ai] "+format+"\n", args...) } // OllamaRequest represents the request payload for Ollama API @@ -51,10 +70,10 @@ func NewOllamaClient(baseURL, fastModel, deepModel string, enableCascade bool) * baseURL = "http://localhost:11434" } if fastModel == "" { - fastModel = "deepseek-r1:1.5b" + fastModel = "qwen3:1.7b" } if deepModel == "" { - deepModel = "qwen2.5-coder:7b" + deepModel = "qwen2.5-coder:14b" } return &OllamaClient{ @@ -351,6 +370,9 @@ Output only the REAL secrets in their original [Type] format, one per line. If n // query sends a request to Ollama API func (c *OllamaClient) query(model, prompt string, timeout time.Duration) (string, error) { + start := time.Now() + c.logVerbose("→ %s prompt=%dB timeout=%s", model, len(prompt), timeout) + reqBody := OllamaRequest{ Model: model, Prompt: prompt, @@ -373,20 +395,25 @@ func (c *OllamaClient) query(model, prompt string, timeout time.Duration) (strin bytes.NewBuffer(jsonData), ) if err != nil { + c.logVerbose("✘ %s %s error=%v", model, time.Since(start).Round(time.Millisecond), err) return "", fmt.Errorf("ollama request failed: %v", err) } defer resp.Body.Close() if resp.StatusCode != 200 { + c.logVerbose("✘ %s status=%d %s", model, resp.StatusCode, time.Since(start).Round(time.Millisecond)) return "", fmt.Errorf("ollama returned status %d", resp.StatusCode) } var ollamaResp OllamaResponse if err := json.NewDecoder(resp.Body).Decode(&ollamaResp); err != nil { + c.logVerbose("✘ %s decode error=%v", model, err) return "", fmt.Errorf("failed to decode response: %v", err) } - return strings.TrimSpace(ollamaResp.Response), nil + out := strings.TrimSpace(ollamaResp.Response) + c.logVerbose("← %s response=%dB %s", model, len(out), time.Since(start).Round(time.Millisecond)) + return out, nil } // parseFindings extracts findings by severity from AI response diff --git a/internal/config/ai_profile.go b/internal/config/ai_profile.go new file mode 100644 index 0000000..c951773 --- /dev/null +++ b/internal/config/ai_profile.go @@ -0,0 +1,101 @@ +package config + +// AIProfile bundles the triage + deep models for a named AI tier. Unlike +// the scan-level Profile (bugbounty/pentest/…), an AIProfile only touches +// model selection — it doesn't flip stealth, recursion, or module enables. +type AIProfile struct { + Name string + Description string + FastModel string + DeepModel string + // MinRAMGB is an advisory (not enforced) hint about the memory footprint + // of both models loaded simultaneously. Printed in the profile help + // banner so users can pick the right tier for their machine. + MinRAMGB int +} + +// Built-in AI profiles. The lean tier matches the repository defaults so +// `--ai-profile lean` is always equivalent to "use whatever the defaults +// say". balanced and heavy upgrade deep model to Qwen3-Coder MoE which +// activates only 3.3B parameters per token despite its 30B total. +var ( + AIProfileLean = AIProfile{ + Name: "lean", + Description: "Runs on 16GB RAM; default. qwen3:1.7b triage + qwen2.5-coder:14b deep.", + FastModel: "qwen3:1.7b", + DeepModel: "qwen2.5-coder:14b", + MinRAMGB: 16, + } + + AIProfileBalanced = AIProfile{ + Name: "balanced", + Description: "32GB RAM / 24GB VRAM. Upgrades deep to qwen3-coder:30b MoE (3.3B active, 256K ctx).", + FastModel: "qwen3:4b", + DeepModel: "qwen3-coder:30b", + MinRAMGB: 32, + } + + AIProfileHeavy = AIProfile{ + Name: "heavy", + Description: "64GB+ RAM. Best-quality triage + deep. Slowest; ideal for final analysis passes.", + FastModel: "qwen3:8b", + DeepModel: "qwen3-coder:30b", + MinRAMGB: 64, + } +) + +// BuiltinAIProfiles lists every AIProfile in CLI help order. +var BuiltinAIProfiles = []AIProfile{ + AIProfileLean, + AIProfileBalanced, + AIProfileHeavy, +} + +// AIProfileByName resolves a named profile. Lookup is case-insensitive +// and tolerates the common alias "max" → heavy. +func AIProfileByName(name string) (AIProfile, bool) { + switch normaliseAIProfileName(name) { + case "lean": + return AIProfileLean, true + case "balanced", "balance", "mid": + return AIProfileBalanced, true + case "heavy", "max", "power": + return AIProfileHeavy, true + } + return AIProfile{}, false +} + +func normaliseAIProfileName(s string) string { + out := make([]byte, 0, len(s)) + for i := 0; i < len(s); i++ { + c := s[i] + if c >= 'A' && c <= 'Z' { + c += 'a' - 'A' + } + if c == ' ' || c == '_' || c == '-' { + continue + } + out = append(out, c) + } + return string(out) +} + +// ApplyAIProfile merges p's models into cfg. If cfg.AIFastModel / +// cfg.AIDeepModel were explicitly set by the user (overrideFast / +// overrideDeep true) the profile is ignored for that field. The caller +// is responsible for detecting explicit flags; in practice this comes +// from cobra's cmd.Flags().Changed("ai-fast-model"). +func ApplyAIProfile(cfg *Config, p AIProfile, overrideFast, overrideDeep bool) { + if cfg == nil { + return + } + if !overrideFast && p.FastModel != "" { + cfg.AIFastModel = p.FastModel + } + if !overrideDeep && p.DeepModel != "" { + cfg.AIDeepModel = p.DeepModel + } + if cfg.AIProfile == "" { + cfg.AIProfile = p.Name + } +} diff --git a/internal/config/ai_profile_test.go b/internal/config/ai_profile_test.go new file mode 100644 index 0000000..eb47003 --- /dev/null +++ b/internal/config/ai_profile_test.go @@ -0,0 +1,98 @@ +package config + +import "testing" + +func TestAIProfileByName(t *testing.T) { + cases := []struct { + in string + wantOK bool + wantTag string + }{ + {"lean", true, "qwen3:1.7b"}, + {"LEAN", true, "qwen3:1.7b"}, + {"balanced", true, "qwen3:4b"}, + {"balance", true, "qwen3:4b"}, + {"mid", true, "qwen3:4b"}, + {"heavy", true, "qwen3:8b"}, + {"max", true, "qwen3:8b"}, + {"power", true, "qwen3:8b"}, + {"Heavy", true, "qwen3:8b"}, + {"nope", false, ""}, + {"", false, ""}, + } + for _, c := range cases { + p, ok := AIProfileByName(c.in) + if ok != c.wantOK { + t.Errorf("AIProfileByName(%q) ok = %v, want %v", c.in, ok, c.wantOK) + continue + } + if ok && p.FastModel != c.wantTag { + t.Errorf("AIProfileByName(%q).FastModel = %q, want %q", c.in, p.FastModel, c.wantTag) + } + } +} + +func TestBuiltinAIProfiles_Unique(t *testing.T) { + names := map[string]bool{} + for _, p := range BuiltinAIProfiles { + if p.Name == "" { + t.Error("profile with empty name") + } + if p.FastModel == "" || p.DeepModel == "" { + t.Errorf("profile %q missing models", p.Name) + } + if p.Description == "" { + t.Errorf("profile %q missing description", p.Name) + } + if names[p.Name] { + t.Errorf("duplicate profile name: %q", p.Name) + } + names[p.Name] = true + } +} + +func TestApplyAIProfile_RespectsOverrides(t *testing.T) { + cfg := &Config{ + AIFastModel: "user-chose-this:1b", + AIDeepModel: "user-chose-that:7b", + } + ApplyAIProfile(cfg, AIProfileHeavy, true, true) + if cfg.AIFastModel != "user-chose-this:1b" { + t.Errorf("overrideFast was ignored: %q", cfg.AIFastModel) + } + if cfg.AIDeepModel != "user-chose-that:7b" { + t.Errorf("overrideDeep was ignored: %q", cfg.AIDeepModel) + } + if cfg.AIProfile != "heavy" { + t.Errorf("AIProfile not set to heavy, got %q", cfg.AIProfile) + } +} + +func TestApplyAIProfile_FillsUnsetFields(t *testing.T) { + cfg := &Config{} + ApplyAIProfile(cfg, AIProfileBalanced, false, false) + if cfg.AIFastModel != "qwen3:4b" { + t.Errorf("FastModel not applied: %q", cfg.AIFastModel) + } + if cfg.AIDeepModel != "qwen3-coder:30b" { + t.Errorf("DeepModel not applied: %q", cfg.AIDeepModel) + } + if cfg.AIProfile != "balanced" { + t.Errorf("AIProfile not set: %q", cfg.AIProfile) + } +} + +func TestApplyAIProfile_NilConfigNoop(t *testing.T) { + ApplyAIProfile(nil, AIProfileLean, false, false) // must not panic +} + +func TestApplyAIProfile_PartialOverride(t *testing.T) { + cfg := &Config{AIFastModel: "custom:1b"} + ApplyAIProfile(cfg, AIProfileHeavy, true, false) + if cfg.AIFastModel != "custom:1b" { + t.Errorf("FastModel overridden: %q", cfg.AIFastModel) + } + if cfg.AIDeepModel != "qwen3-coder:30b" { + t.Errorf("DeepModel not applied: %q", cfg.AIDeepModel) + } +} diff --git a/internal/config/config.go b/internal/config/config.go index 4962716..30524b0 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -49,6 +49,80 @@ type Config struct { NoTechScan bool // Disable tech scan (override when --enable-ai) NoASNScan bool // Disable ASN scan (override when --enable-ai) NoVHostScan bool // Disable vhost scan (override when --enable-ai) + + // v2: profile + per-module overrides loaded from config file or CLI. + // Profile is the named profile to apply before CLI flags. Empty = none. + Profile string + + // ConfigFile is the path to an optional YAML config file. Empty = search + // standard locations, then fall through to CLI defaults + profile only. + ConfigFile string + + // ModuleSettings is a flat map of module-name → enabled. Populated from + // YAML ("modules:" section) and CLI (--enable/--disable flags if added). + // Consumed by ConfigView.ModuleEnabled. Empty means "honor each module's + // DefaultEnabled()". + ModuleSettings map[string]bool + + // UsePipeline opts into the v2 event-driven pipeline. When false (default + // during F0.6 migration) the legacy scanner.Run is used. Once F0.7 + // parity is verified this becomes true by default. + UsePipeline bool + + // Live toggles the Fase 4 LivePrinter that streams colorized scan + // events to the terminal alongside (or instead of) the final report. + Live bool + // LiveVerbosity controls how much the LivePrinter prints (0..2). + LiveVerbosity int + + // MonitorInterval, when > 0, switches the CLI into asm-continuous mode: + // the scan runs on this interval and diffs against the previous + // snapshot, firing Webhook/Stdout alerts on meaningful changes. + MonitorInterval time.Duration + // MonitorWebhook is a POST target for diff reports in monitor mode. + MonitorWebhook string + + // AIProfile is the named AI tier (lean/balanced/heavy). When set, it + // applies FastModel+DeepModel defaults before CLI overrides kick in. + // Empty string = use whatever AIFastModel/AIDeepModel resolve to via + // CLI flags + YAML. + AIProfile string + + // AIVerbose toggles detailed logging of every Ollama query: model, + // prompt size, response size, duration, triage decisions. Writes to + // stderr so stdout (JSON / silent modes) stays clean. + AIVerbose bool + + // AutoPullModels controls whether god-eye auto-downloads missing + // Ollama models at startup when --enable-ai is set. Defaults to true + // — flip to false if you want scan failures instead of silent pulls. + AutoPullModels bool + + // Wizard forces the interactive setup flow even when -d is present, + // so users can preview/tweak defaults. When -d is absent and stdin + // is a TTY, the wizard auto-starts without this flag. + Wizard bool + + // NucleiScan opts into the Nuclei-format template executor. Templates + // are loaded from NucleiTemplates (or ~/nuclei-templates as fallback, + // with auto-download of the official ZIP into ~/.god-eye/nuclei-templates + // when NucleiAutoDownload is true and no local dir is present). + NucleiScan bool + // NucleiTemplates is an optional override for the template directory. + NucleiTemplates string + // NucleiAutoDownload controls whether god-eye auto-fetches the + // official nuclei-templates ZIP on first use. Defaults to true. + NucleiAutoDownload bool + + // Proxy routes every outbound HTTP request (passive sources, probes, + // Nuclei, Ollama-if-remote) through the given URL. Supports: + // http://host:port - HTTP CONNECT proxy (Burp, ZAP, mitmproxy) + // https://host:port - HTTPS CONNECT proxy + // socks5://host:port - SOCKS5 with local DNS + // socks5h://host:port - SOCKS5 with proxy-side DNS (Tor convention) + // Basic auth is honoured: http://user:pass@host. + // Empty = no proxy (direct). + Proxy string } // Stats holds scan statistics diff --git a/internal/config/config_test.go b/internal/config/config_test.go new file mode 100644 index 0000000..9a03d0c --- /dev/null +++ b/internal/config/config_test.go @@ -0,0 +1,102 @@ +package config + +import ( + "encoding/json" + "testing" +) + +func TestDefaultResolversNonEmpty(t *testing.T) { + if len(DefaultResolvers) == 0 { + t.Fatal("DefaultResolvers is empty") + } + for _, r := range DefaultResolvers { + if r == "" { + t.Errorf("empty resolver in DefaultResolvers") + } + } +} + +func TestDefaultWordlistNonEmpty(t *testing.T) { + if len(DefaultWordlist) < 50 { + t.Errorf("DefaultWordlist too small: %d entries", len(DefaultWordlist)) + } + seen := make(map[string]bool) + for _, w := range DefaultWordlist { + if w == "" { + t.Error("empty entry in DefaultWordlist") + } + // Note: v1 wordlist contains "smtp" and "staging" twice — that's a bug + // but not something we fix in baseline tests. Just verify no ALL duplicates. + seen[w] = true + } + if len(seen) < 50 { + t.Errorf("too many duplicates: %d unique out of %d", len(seen), len(DefaultWordlist)) + } +} + +func TestSubdomainResult_JSONRoundtrip(t *testing.T) { + orig := &SubdomainResult{ + Subdomain: "api.example.com", + IPs: []string{"1.2.3.4"}, + CNAME: "cname.example.com", + StatusCode: 200, + Title: "API", + Tech: []string{"nginx", "Go"}, + CloudProvider: "AWS", + TLSFingerprint: &TLSFingerprint{ + Vendor: "Fortinet", + Product: "FortiGate", + ApplianceType: "firewall", + }, + } + + data, err := json.Marshal(orig) + if err != nil { + t.Fatalf("marshal failed: %v", err) + } + + var decoded SubdomainResult + if err := json.Unmarshal(data, &decoded); err != nil { + t.Fatalf("unmarshal failed: %v", err) + } + + if decoded.Subdomain != orig.Subdomain { + t.Errorf("Subdomain mismatch: got %q want %q", decoded.Subdomain, orig.Subdomain) + } + if len(decoded.IPs) != 1 || decoded.IPs[0] != "1.2.3.4" { + t.Errorf("IPs mismatch: got %v", decoded.IPs) + } + if decoded.TLSFingerprint == nil { + t.Fatal("TLSFingerprint is nil after roundtrip") + } + if decoded.TLSFingerprint.Vendor != "Fortinet" { + t.Errorf("TLSFingerprint.Vendor = %q, want Fortinet", decoded.TLSFingerprint.Vendor) + } +} + +func TestSubdomainResult_OmitemptyMinimal(t *testing.T) { + // Ensure zero-value struct produces a minimal JSON (only subdomain field would be present if set). + empty := &SubdomainResult{} + data, err := json.Marshal(empty) + if err != nil { + t.Fatal(err) + } + // Only the required "subdomain" field (empty string) should appear — every other is omitempty. + expected := `{"subdomain":""}` + if string(data) != expected { + t.Errorf("empty struct JSON = %s, want %s", string(data), expected) + } +} + +func TestConfigZeroValue(t *testing.T) { + var c Config + if c.Domain != "" { + t.Errorf("default Domain should be empty, got %q", c.Domain) + } + if c.EnableAI { + t.Error("EnableAI should default to false") + } + if c.Concurrency != 0 { + t.Error("Concurrency should default to 0 (overridden by CLI default)") + } +} diff --git a/internal/config/profile.go b/internal/config/profile.go new file mode 100644 index 0000000..4d8dca9 --- /dev/null +++ b/internal/config/profile.go @@ -0,0 +1,208 @@ +package config + +// Profile is a named bundle of defaults that tailors God's Eye for a specific +// use case. Profiles set module enable/disable, concurrency hints, stealth, +// and whether AI is on. CLI flags still override profile defaults. +type Profile struct { + Name string + Description string + + // Core tuning + Concurrency int + Timeout int + Stealth string // off, light, moderate, aggressive, paranoid + + // Feature toggles (nil means "use module default") + AI *bool + MultiAgent *bool + Recursive *bool + NoBrute *bool + NoProbe *bool + NoPorts *bool + NoTakeover *bool + + // Advanced feature flags (nil = use module default) + CloudScan *bool + APIScan *bool + SecretsScan *bool + TechScan *bool + ASNScan *bool + VHostScan *bool + + // Per-module overrides (explicit enable/disable) + Modules map[string]bool +} + +// ProfileBugBounty is tuned for bug-bounty recon: broad discovery, AI on, +// secrets+tech+cloud scanning on, stealth off (speed matters). +var ProfileBugBounty = Profile{ + Name: "bugbounty", + Description: "Aggressive recon for bug-bounty: broad discovery, AI on, secrets/cloud/API/tech scanning, stealth off.", + Concurrency: 1000, + Timeout: 5, + Stealth: "off", + AI: ptrTrue(), + MultiAgent: ptrTrue(), + Recursive: ptrTrue(), + CloudScan: ptrTrue(), + APIScan: ptrTrue(), + SecretsScan: ptrTrue(), + TechScan: ptrTrue(), + ASNScan: ptrTrue(), + VHostScan: ptrTrue(), +} + +// ProfilePentest is tuned for authorized penetration tests: stealth light, +// full enrichment, AI on for deeper analysis. +var ProfilePentest = Profile{ + Name: "pentest", + Description: "Authorized pentest: full enrichment with light stealth to avoid basic rate limits.", + Concurrency: 300, + Timeout: 10, + Stealth: "light", + AI: ptrTrue(), + MultiAgent: ptrTrue(), + Recursive: ptrTrue(), + CloudScan: ptrTrue(), + APIScan: ptrTrue(), + SecretsScan: ptrTrue(), + TechScan: ptrTrue(), + ASNScan: ptrTrue(), + VHostScan: ptrTrue(), +} + +// ProfileASMContinuous is tuned for attack-surface monitoring: reduced depth +// per run, designed to be re-run periodically with diff engine (Fase 5). +// Stealth moderate to stay below detection thresholds when running daily. +var ProfileASMContinuous = Profile{ + Name: "asm-continuous", + Description: "Continuous attack-surface monitoring; runs cheaper than full recon, feeds diff engine.", + Concurrency: 200, + Timeout: 10, + Stealth: "moderate", + AI: ptrFalse(), // AI only on findings that change, not full re-analysis + Recursive: ptrFalse(), // rely on diff to grow surface over time + CloudScan: ptrTrue(), + TechScan: ptrTrue(), + SecretsScan: ptrTrue(), +} + +// ProfileStealthMax is for highly sensitive targets where any detection is +// unacceptable. Very slow; passive-first. +var ProfileStealthMax = Profile{ + Name: "stealth-max", + Description: "Maximum evasion. Passive-only by default, slow request cadence.", + Concurrency: 3, + Timeout: 20, + Stealth: "paranoid", + NoBrute: ptrTrue(), + NoPorts: ptrTrue(), + AI: ptrFalse(), + TechScan: ptrTrue(), +} + +// ProfileQuick is for triage: skip expensive phases, produce a fast answer. +var ProfileQuick = Profile{ + Name: "quick", + Description: "Fast triage: passive enum + HTTP probe, no brute/JS/AI.", + Concurrency: 500, + Timeout: 5, + Stealth: "off", + NoBrute: ptrTrue(), + AI: ptrFalse(), +} + +// BuiltinProfiles lists every named profile that ships with the tool, in a +// stable order for docs/help output. +var BuiltinProfiles = []Profile{ + ProfileBugBounty, + ProfilePentest, + ProfileASMContinuous, + ProfileStealthMax, + ProfileQuick, +} + +// ProfileByName returns the named profile, or ok=false when not found. +func ProfileByName(name string) (Profile, bool) { + for _, p := range BuiltinProfiles { + if p.Name == name { + return p, true + } + } + return Profile{}, false +} + +// ApplyProfile merges a profile into cfg. Existing non-zero values in cfg +// take precedence (CLI flags win over profile defaults). Pointer-typed +// profile fields are applied only when they are non-nil. +func ApplyProfile(cfg *Config, p Profile) { + if cfg == nil { + return + } + + if cfg.Concurrency == 0 || cfg.Concurrency == 1000 { // 1000 is the cobra default + cfg.Concurrency = p.Concurrency + } + if cfg.Timeout == 0 || cfg.Timeout == 5 { // 5 is cobra default + cfg.Timeout = p.Timeout + } + if cfg.StealthMode == "" { + cfg.StealthMode = p.Stealth + } + + if p.AI != nil && !cfg.EnableAI { + cfg.EnableAI = *p.AI + } + if p.MultiAgent != nil && !cfg.MultiAgent { + cfg.MultiAgent = *p.MultiAgent + } + if p.Recursive != nil && !cfg.Recursive && !cfg.NoRecursive { + cfg.Recursive = *p.Recursive + } + if p.NoBrute != nil && !cfg.NoBrute { + cfg.NoBrute = *p.NoBrute + } + if p.NoProbe != nil && !cfg.NoProbe { + cfg.NoProbe = *p.NoProbe + } + if p.NoPorts != nil && !cfg.NoPorts { + cfg.NoPorts = *p.NoPorts + } + if p.NoTakeover != nil && !cfg.NoTakeover { + cfg.NoTakeover = *p.NoTakeover + } + + applyPtrBool(&cfg.CloudScan, &cfg.NoCloudScan, p.CloudScan) + applyPtrBool(&cfg.APIScan, &cfg.NoAPIScan, p.APIScan) + applyPtrBool(&cfg.SecretsScan, &cfg.NoSecrets, p.SecretsScan) + applyPtrBool(&cfg.TechScan, &cfg.NoTechScan, p.TechScan) + applyPtrBool(&cfg.ASNScan, &cfg.NoASNScan, p.ASNScan) + applyPtrBool(&cfg.VHostScan, &cfg.NoVHostScan, p.VHostScan) + + // Module overrides + if cfg.ModuleSettings == nil { + cfg.ModuleSettings = make(map[string]bool) + } + for name, enabled := range p.Modules { + if _, already := cfg.ModuleSettings[name]; !already { + cfg.ModuleSettings[name] = enabled + } + } +} + +// applyPtrBool merges a ptr-bool from a profile into a (enabled, noEnabled) +// pair on the Config struct. The v1 scheme uses two flags per feature +// (Enable/NoEnable) to allow a three-state: unset/on/off. A nil profile ptr +// means "leave unchanged"; *p=true enables unless user has set NoX; *p=false +// leaves alone (profile doesn't force-off, user's explicit flag does). +func applyPtrBool(enable, disable *bool, p *bool) { + if p == nil { + return + } + if *p && !*enable && !*disable { + *enable = true + } +} + +func ptrTrue() *bool { v := true; return &v } +func ptrFalse() *bool { v := false; return &v } diff --git a/internal/config/profile_test.go b/internal/config/profile_test.go new file mode 100644 index 0000000..e7bfce8 --- /dev/null +++ b/internal/config/profile_test.go @@ -0,0 +1,143 @@ +package config + +import "testing" + +func TestProfileByName(t *testing.T) { + tests := []struct { + name string + input string + wantOK bool + wantStr string + }{ + {"bugbounty", "bugbounty", true, "bugbounty"}, + {"pentest", "pentest", true, "pentest"}, + {"asm-continuous", "asm-continuous", true, "asm-continuous"}, + {"stealth-max", "stealth-max", true, "stealth-max"}, + {"quick", "quick", true, "quick"}, + {"empty", "", false, ""}, + {"unknown", "nonsense", false, ""}, + {"case sensitive", "BugBounty", false, ""}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, ok := ProfileByName(tt.input) + if ok != tt.wantOK { + t.Errorf("ok = %v, want %v", ok, tt.wantOK) + } + if ok && got.Name != tt.wantStr { + t.Errorf("Name = %q, want %q", got.Name, tt.wantStr) + } + }) + } +} + +func TestBuiltinProfiles_NonEmpty(t *testing.T) { + if len(BuiltinProfiles) < 5 { + t.Errorf("expected ≥5 built-in profiles, got %d", len(BuiltinProfiles)) + } + seen := make(map[string]bool) + for _, p := range BuiltinProfiles { + if p.Name == "" { + t.Error("profile with empty name") + } + if p.Description == "" { + t.Errorf("profile %q has empty description", p.Name) + } + if seen[p.Name] { + t.Errorf("duplicate profile name: %q", p.Name) + } + seen[p.Name] = true + } +} + +func TestApplyProfile_NilConfigNoop(t *testing.T) { + ApplyProfile(nil, ProfileBugBounty) // must not panic +} + +func TestApplyProfile_FillsDefaults(t *testing.T) { + cfg := &Config{} // zero + ApplyProfile(cfg, ProfileBugBounty) + if cfg.Concurrency != ProfileBugBounty.Concurrency { + t.Errorf("Concurrency = %d, want %d", cfg.Concurrency, ProfileBugBounty.Concurrency) + } + if cfg.Timeout != ProfileBugBounty.Timeout { + t.Errorf("Timeout = %d, want %d", cfg.Timeout, ProfileBugBounty.Timeout) + } + if cfg.StealthMode != ProfileBugBounty.Stealth { + t.Errorf("Stealth = %q, want %q", cfg.StealthMode, ProfileBugBounty.Stealth) + } + if !cfg.EnableAI { + t.Error("bugbounty profile should enable AI") + } + if !cfg.MultiAgent { + t.Error("bugbounty profile should enable MultiAgent") + } + if !cfg.Recursive { + t.Error("bugbounty profile should enable Recursive") + } + if !cfg.CloudScan { + t.Error("bugbounty profile should enable CloudScan") + } +} + +func TestApplyProfile_DoesNotOverrideExplicitFlags(t *testing.T) { + cfg := &Config{ + Concurrency: 42, + Timeout: 999, + StealthMode: "paranoid", + EnableAI: false, // explicitly disabled before profile apply + } + // Apply bugbounty which normally enables AI + sets concurrency to 1000 + ApplyProfile(cfg, ProfileBugBounty) + + // Explicit non-default user values should survive + if cfg.Concurrency != 42 { + t.Errorf("Concurrency overwritten: %d", cfg.Concurrency) + } + if cfg.Timeout != 999 { + t.Errorf("Timeout overwritten: %d", cfg.Timeout) + } + if cfg.StealthMode != "paranoid" { + t.Errorf("Stealth overwritten: %q", cfg.StealthMode) + } + // Profile AI enable should still apply since cfg.EnableAI was false + // (we can't distinguish "user explicitly set false" from "zero value"). + // This is a known limitation documented in the CLI help. + if !cfg.EnableAI { + t.Errorf("AI not enabled by profile despite cfg.EnableAI being false") + } +} + +func TestApplyProfile_NoForceOff(t *testing.T) { + // stealth-max sets NoBrute=true. If user did NOT disable, profile wins. + cfg := &Config{} + ApplyProfile(cfg, ProfileStealthMax) + if !cfg.NoBrute { + t.Error("stealth-max profile should set NoBrute") + } +} + +func TestApplyProfile_ModuleSettings(t *testing.T) { + p := Profile{ + Name: "custom", + Modules: map[string]bool{ + "sources.crtsh": true, + "brute": false, + }, + } + cfg := &Config{} + ApplyProfile(cfg, p) + if got := cfg.ModuleSettings["sources.crtsh"]; !got { + t.Error("crtsh should be enabled") + } + if got := cfg.ModuleSettings["brute"]; got { + t.Error("brute should be disabled") + } + + // User pre-existing setting must not be overridden + cfg2 := &Config{ModuleSettings: map[string]bool{"sources.crtsh": false}} + ApplyProfile(cfg2, p) + if cfg2.ModuleSettings["sources.crtsh"] { + t.Error("user explicit module setting was overridden") + } +} diff --git a/internal/config/view.go b/internal/config/view.go new file mode 100644 index 0000000..b4add9e --- /dev/null +++ b/internal/config/view.go @@ -0,0 +1,151 @@ +package config + +// View implements module.ConfigView over a *Config. Modules receive a View +// (not the raw Config pointer) to prevent them from mutating global scan +// state — reads only. +// +// The implementation is intentionally small: it exposes just the shape +// needed by the module package without pulling in a full generic key/value +// store. Module-specific settings live in ModuleSettings; typed options +// should be hoisted to first-class fields on Config when they are used +// across modules. +type View struct { + cfg *Config +} + +// NewView wraps cfg as a ConfigView. cfg may be nil, in which case every +// accessor returns the fallback/zero value. +func NewView(cfg *Config) *View { return &View{cfg: cfg} } + +// Profile returns the active profile name ("" when none). +func (v *View) Profile() string { + if v == nil || v.cfg == nil { + return "" + } + return v.cfg.Profile +} + +// Bool reads a boolean config key by well-known name. Unknown keys return fb. +// Keys intentionally kept flat to avoid accidental namespacing bugs. +func (v *View) Bool(key string, fb bool) bool { + if v == nil || v.cfg == nil { + return fb + } + switch key { + case "ai.enabled": + return v.cfg.EnableAI + case "ai.cascade": + return v.cfg.AICascade + case "ai.deep": + return v.cfg.AIDeepAnalysis + case "ai.multi_agent": + return v.cfg.MultiAgent + case "ai.verbose": + return v.cfg.AIVerbose + case "ai.auto_pull": + return v.cfg.AutoPullModels + case "silent": + return v.cfg.Silent + case "verbose": + return v.cfg.Verbose + case "json": + return v.cfg.JsonOutput + case "no_brute": + return v.cfg.NoBrute + case "no_probe": + return v.cfg.NoProbe + case "no_ports": + return v.cfg.NoPorts + case "no_takeover": + return v.cfg.NoTakeover + case "only_active": + return v.cfg.OnlyActive + case "recursive": + return v.cfg.Recursive + case "cloud_scan": + return v.cfg.CloudScan + case "api_scan": + return v.cfg.APIScan + case "secrets_scan": + return v.cfg.SecretsScan + case "tech_scan": + return v.cfg.TechScan + case "asn_scan": + return v.cfg.ASNScan + case "vhost_scan": + return v.cfg.VHostScan + case "nuclei_scan": + return v.cfg.NucleiScan + case "nuclei_auto_download": + return v.cfg.NucleiAutoDownload + } + return fb +} + +// Int reads an int key. +func (v *View) Int(key string, fb int) int { + if v == nil || v.cfg == nil { + return fb + } + switch key { + case "concurrency": + return v.cfg.Concurrency + case "timeout": + return v.cfg.Timeout + case "recursive.depth": + return v.cfg.RecursiveDepth + } + return fb +} + +// String reads a string key. +func (v *View) String(key string, fb string) string { + if v == nil || v.cfg == nil { + return fb + } + switch key { + case "domain": + return v.cfg.Domain + case "wordlist": + return v.cfg.Wordlist + case "output": + return v.cfg.Output + case "format": + return v.cfg.Format + case "ports": + return v.cfg.Ports + case "resolvers": + return v.cfg.Resolvers + case "stealth": + return v.cfg.StealthMode + case "ai.url": + return v.cfg.AIUrl + case "ai.fast_model": + return v.cfg.AIFastModel + case "ai.deep_model": + return v.cfg.AIDeepModel + case "nuclei_templates": + return v.cfg.NucleiTemplates + } + return fb +} + +// Strings reads a string-slice key. No multi-value keys are defined yet, +// but reserved for module-specific settings loaded from YAML. +func (v *View) Strings(key string) []string { + _ = key + return nil +} + +// ModuleEnabled returns true when the config explicitly enabled the module +// by name (via ModuleSettings). It returns false otherwise; callers should +// fall back to the module's DefaultEnabled() when this returns false. +func (v *View) ModuleEnabled(name string) bool { + if v == nil || v.cfg == nil { + return false + } + if v.cfg.ModuleSettings == nil { + return false + } + return v.cfg.ModuleSettings[name] +} diff --git a/internal/config/view_test.go b/internal/config/view_test.go new file mode 100644 index 0000000..7f57ab3 --- /dev/null +++ b/internal/config/view_test.go @@ -0,0 +1,156 @@ +package config + +import "testing" + +func TestView_NilSafe(t *testing.T) { + var v *View + if v.Profile() != "" { + t.Error("nil view Profile should be empty") + } + if v.Bool("ai.enabled", true) != true { + t.Error("nil view Bool should return fallback") + } + if v.Int("concurrency", 99) != 99 { + t.Error("nil view Int should return fallback") + } + if v.String("domain", "fb") != "fb" { + t.Error("nil view String should return fallback") + } + if v.ModuleEnabled("x") { + t.Error("nil view ModuleEnabled should be false") + } +} + +func TestView_Profile(t *testing.T) { + v := NewView(&Config{Profile: "bugbounty"}) + if v.Profile() != "bugbounty" { + t.Errorf("Profile = %q", v.Profile()) + } +} + +func TestView_Bool(t *testing.T) { + cfg := &Config{ + EnableAI: true, + AICascade: true, + AIDeepAnalysis: false, + MultiAgent: true, + Silent: true, + Verbose: false, + JsonOutput: true, + NoBrute: true, + OnlyActive: true, + Recursive: true, + CloudScan: true, + APIScan: false, + } + v := NewView(cfg) + + tests := []struct { + key string + fb bool + want bool + }{ + {"ai.enabled", false, true}, + {"ai.cascade", false, true}, + {"ai.deep", true, false}, + {"ai.multi_agent", false, true}, + {"silent", false, true}, + {"verbose", true, false}, + {"json", false, true}, + {"no_brute", false, true}, + {"only_active", false, true}, + {"recursive", false, true}, + {"cloud_scan", false, true}, + {"api_scan", true, false}, + {"unknown_key", true, true}, // fallback + {"unknown_key", false, false}, + } + + for _, tt := range tests { + if got := v.Bool(tt.key, tt.fb); got != tt.want { + t.Errorf("Bool(%q, %v) = %v, want %v", tt.key, tt.fb, got, tt.want) + } + } +} + +func TestView_Int(t *testing.T) { + v := NewView(&Config{Concurrency: 500, Timeout: 10, RecursiveDepth: 4}) + if v.Int("concurrency", 1) != 500 { + t.Errorf("concurrency wrong") + } + if v.Int("timeout", 1) != 10 { + t.Errorf("timeout wrong") + } + if v.Int("recursive.depth", 1) != 4 { + t.Errorf("recursive.depth wrong") + } + if v.Int("unknown", 99) != 99 { + t.Errorf("unknown key should return fallback") + } +} + +func TestView_String(t *testing.T) { + v := NewView(&Config{ + Domain: "example.com", + Wordlist: "/wl", + Output: "/out", + Format: "json", + Ports: "80,443", + Resolvers: "8.8.8.8", + StealthMode: "light", + AIUrl: "http://x", + AIFastModel: "f", + AIDeepModel: "d", + }) + + cases := map[string]string{ + "domain": "example.com", + "wordlist": "/wl", + "output": "/out", + "format": "json", + "ports": "80,443", + "resolvers": "8.8.8.8", + "stealth": "light", + "ai.url": "http://x", + "ai.fast_model": "f", + "ai.deep_model": "d", + } + for k, want := range cases { + if got := v.String(k, "fb"); got != want { + t.Errorf("String(%q) = %q, want %q", k, got, want) + } + } + + if v.String("unknown", "fb") != "fb" { + t.Error("unknown key should return fallback") + } +} + +func TestView_Strings(t *testing.T) { + // Placeholder — no multi-value keys defined yet + v := NewView(&Config{}) + if got := v.Strings("anything"); got != nil { + t.Errorf("expected nil, got %v", got) + } +} + +func TestView_ModuleEnabled(t *testing.T) { + cfg := &Config{ModuleSettings: map[string]bool{"m1": true, "m2": false}} + v := NewView(cfg) + if !v.ModuleEnabled("m1") { + t.Error("m1 should be enabled") + } + if v.ModuleEnabled("m2") { + t.Error("m2 should be disabled (false in map)") + } + if v.ModuleEnabled("unset") { + t.Error("unset module should be false") + } +} + +func TestView_ModuleEnabled_NilMap(t *testing.T) { + v := NewView(&Config{}) + if v.ModuleEnabled("anything") { + t.Error("nil map should result in false") + } +} diff --git a/internal/config/yaml.go b/internal/config/yaml.go new file mode 100644 index 0000000..cf54477 --- /dev/null +++ b/internal/config/yaml.go @@ -0,0 +1,181 @@ +package config + +import ( + "fmt" + "os" + "path/filepath" + + "gopkg.in/yaml.v3" +) + +// YAMLConfig is the schema persisted on disk. Fields are intentionally a +// subset of Config — YAML is for declarative, long-lived settings +// (profile, module toggles, resolver lists, AI model names); ephemeral +// flags (--silent, --verbose, --domain) remain CLI-only. +type YAMLConfig struct { + Profile string `yaml:"profile,omitempty"` + Concurrency int `yaml:"concurrency,omitempty"` + Timeout int `yaml:"timeout,omitempty"` + Stealth string `yaml:"stealth,omitempty"` + Resolvers []string `yaml:"resolvers,omitempty"` + Wordlist string `yaml:"wordlist,omitempty"` + Modules map[string]bool `yaml:"modules,omitempty"` + AI *YAMLAIConfig `yaml:"ai,omitempty"` + Output *YAMLOutputConfig `yaml:"output,omitempty"` +} + +// YAMLAIConfig groups AI-related YAML fields. +type YAMLAIConfig struct { + Enabled bool `yaml:"enabled,omitempty"` + URL string `yaml:"url,omitempty"` + FastModel string `yaml:"fast_model,omitempty"` + DeepModel string `yaml:"deep_model,omitempty"` + Cascade *bool `yaml:"cascade,omitempty"` + Deep bool `yaml:"deep,omitempty"` + MultiAgent bool `yaml:"multi_agent,omitempty"` +} + +// YAMLOutputConfig groups output-related YAML fields. +type YAMLOutputConfig struct { + Path string `yaml:"path,omitempty"` + Format string `yaml:"format,omitempty"` + JSON bool `yaml:"json,omitempty"` +} + +// LoadYAML reads a YAML config file from path and returns the parsed config. +// Returns (nil, nil) when the file does not exist — callers should treat this +// as "no config file, use defaults". Returns an error for any other I/O or +// parse failure. +func LoadYAML(path string) (*YAMLConfig, error) { + if path == "" { + return nil, nil + } + data, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil, nil + } + return nil, fmt.Errorf("read config %q: %w", path, err) + } + + var y YAMLConfig + if err := yaml.Unmarshal(data, &y); err != nil { + return nil, fmt.Errorf("parse config %q: %w", path, err) + } + return &y, nil +} + +// ApplyYAML merges a parsed YAML config into cfg. CLI flags win: YAML only +// fills fields that are still at their zero value on cfg. The profile named +// in YAML is applied only if cfg.Profile is empty. +func ApplyYAML(cfg *Config, y *YAMLConfig) { + if cfg == nil || y == nil { + return + } + + if cfg.Profile == "" && y.Profile != "" { + cfg.Profile = y.Profile + } + if cfg.Concurrency == 0 && y.Concurrency > 0 { + cfg.Concurrency = y.Concurrency + } + if cfg.Timeout == 0 && y.Timeout > 0 { + cfg.Timeout = y.Timeout + } + if cfg.StealthMode == "" && y.Stealth != "" { + cfg.StealthMode = y.Stealth + } + if cfg.Resolvers == "" && len(y.Resolvers) > 0 { + cfg.Resolvers = joinComma(y.Resolvers) + } + if cfg.Wordlist == "" && y.Wordlist != "" { + cfg.Wordlist = y.Wordlist + } + + if len(y.Modules) > 0 { + if cfg.ModuleSettings == nil { + cfg.ModuleSettings = make(map[string]bool) + } + for name, enabled := range y.Modules { + if _, already := cfg.ModuleSettings[name]; !already { + cfg.ModuleSettings[name] = enabled + } + } + } + + if y.AI != nil { + if y.AI.Enabled && !cfg.EnableAI { + cfg.EnableAI = true + } + if cfg.AIUrl == "" && y.AI.URL != "" { + cfg.AIUrl = y.AI.URL + } + if cfg.AIFastModel == "" && y.AI.FastModel != "" { + cfg.AIFastModel = y.AI.FastModel + } + if cfg.AIDeepModel == "" && y.AI.DeepModel != "" { + cfg.AIDeepModel = y.AI.DeepModel + } + if y.AI.Cascade != nil && !cfg.AICascade { + cfg.AICascade = *y.AI.Cascade + } + if y.AI.Deep && !cfg.AIDeepAnalysis { + cfg.AIDeepAnalysis = true + } + if y.AI.MultiAgent && !cfg.MultiAgent { + cfg.MultiAgent = true + } + } + + if y.Output != nil { + if cfg.Output == "" && y.Output.Path != "" { + cfg.Output = y.Output.Path + } + if cfg.Format == "" && y.Output.Format != "" { + cfg.Format = y.Output.Format + } + if y.Output.JSON && !cfg.JsonOutput { + cfg.JsonOutput = true + } + } +} + +// DefaultConfigPaths returns the ordered list of paths LoadYAML scans by +// default when no --config is provided. The first existing file wins. +func DefaultConfigPaths() []string { + home, err := os.UserHomeDir() + var homeCfg string + if err == nil { + homeCfg = filepath.Join(home, ".god-eye", "config.yaml") + } + return []string{ + "god-eye.yaml", + ".god-eye.yaml", + homeCfg, + } +} + +// FindConfigFile returns the first existing file in DefaultConfigPaths, or +// "" if none is found. +func FindConfigFile() string { + for _, p := range DefaultConfigPaths() { + if p == "" { + continue + } + if _, err := os.Stat(p); err == nil { + return p + } + } + return "" +} + +func joinComma(ss []string) string { + out := "" + for i, s := range ss { + if i > 0 { + out += "," + } + out += s + } + return out +} diff --git a/internal/config/yaml_test.go b/internal/config/yaml_test.go new file mode 100644 index 0000000..529b9a1 --- /dev/null +++ b/internal/config/yaml_test.go @@ -0,0 +1,270 @@ +package config + +import ( + "os" + "path/filepath" + "testing" +) + +func TestLoadYAML_Missing(t *testing.T) { + y, err := LoadYAML("/tmp/this-definitely-does-not-exist-xyz.yaml") + if err != nil { + t.Errorf("missing file should return nil error, got %v", err) + } + if y != nil { + t.Errorf("missing file should return nil config, got %+v", y) + } +} + +func TestLoadYAML_EmptyPath(t *testing.T) { + y, err := LoadYAML("") + if y != nil || err != nil { + t.Errorf("empty path → (nil, nil), got (%+v, %v)", y, err) + } +} + +func TestLoadYAML_Malformed(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "bad.yaml") + os.WriteFile(path, []byte("profile: [unclosed"), 0o644) + _, err := LoadYAML(path) + if err == nil { + t.Error("expected parse error for malformed YAML") + } +} + +func TestLoadYAML_Full(t *testing.T) { + content := ` +profile: bugbounty +concurrency: 500 +timeout: 8 +stealth: moderate +resolvers: + - 8.8.8.8 + - 1.1.1.1 +wordlist: /tmp/wl.txt +modules: + sources.crtsh: true + brute: false +ai: + enabled: true + url: http://localhost:11434 + fast_model: qwen3:1.7b + deep_model: qwen2.5-coder:14b + cascade: true + deep: true + multi_agent: true +output: + path: /tmp/out.json + format: json + json: true +` + dir := t.TempDir() + path := filepath.Join(dir, "config.yaml") + os.WriteFile(path, []byte(content), 0o644) + + y, err := LoadYAML(path) + if err != nil { + t.Fatal(err) + } + if y == nil { + t.Fatal("expected non-nil config") + } + if y.Profile != "bugbounty" { + t.Errorf("Profile = %q", y.Profile) + } + if y.Concurrency != 500 { + t.Errorf("Concurrency = %d", y.Concurrency) + } + if y.Timeout != 8 { + t.Errorf("Timeout = %d", y.Timeout) + } + if y.Stealth != "moderate" { + t.Errorf("Stealth = %q", y.Stealth) + } + if len(y.Resolvers) != 2 { + t.Errorf("Resolvers len = %d", len(y.Resolvers)) + } + if y.Modules["sources.crtsh"] != true { + t.Errorf("modules.sources.crtsh = false") + } + if y.Modules["brute"] != false { + t.Errorf("modules.brute = true") + } + if y.AI == nil || !y.AI.Enabled { + t.Error("AI not enabled") + } + if y.AI.URL != "http://localhost:11434" { + t.Errorf("AI.URL = %q", y.AI.URL) + } + if y.Output == nil || y.Output.Path != "/tmp/out.json" { + t.Errorf("Output.Path wrong") + } +} + +func TestApplyYAML_NilInputs(t *testing.T) { + ApplyYAML(nil, &YAMLConfig{Profile: "x"}) // must not panic + ApplyYAML(&Config{}, nil) // must not panic +} + +func TestApplyYAML_FillsZeroFields(t *testing.T) { + cfg := &Config{} + y := &YAMLConfig{ + Profile: "quick", + Concurrency: 123, + Timeout: 7, + Stealth: "light", + Resolvers: []string{"8.8.8.8", "1.1.1.1"}, + Wordlist: "/tmp/wl", + Modules: map[string]bool{"m1": true}, + AI: &YAMLAIConfig{ + Enabled: true, + URL: "http://x", + FastModel: "f", + DeepModel: "d", + Cascade: ptrTrue(), + Deep: true, + MultiAgent: true, + }, + Output: &YAMLOutputConfig{Path: "/o", Format: "json", JSON: true}, + } + ApplyYAML(cfg, y) + + if cfg.Profile != "quick" { + t.Errorf("Profile = %q", cfg.Profile) + } + if cfg.Concurrency != 123 { + t.Errorf("Concurrency = %d", cfg.Concurrency) + } + if cfg.Timeout != 7 { + t.Errorf("Timeout = %d", cfg.Timeout) + } + if cfg.StealthMode != "light" { + t.Errorf("StealthMode = %q", cfg.StealthMode) + } + if cfg.Resolvers != "8.8.8.8,1.1.1.1" { + t.Errorf("Resolvers = %q", cfg.Resolvers) + } + if cfg.Wordlist != "/tmp/wl" { + t.Errorf("Wordlist = %q", cfg.Wordlist) + } + if !cfg.EnableAI { + t.Error("EnableAI should be true") + } + if cfg.AIUrl != "http://x" { + t.Errorf("AIUrl = %q", cfg.AIUrl) + } + if !cfg.AICascade { + t.Error("AICascade should be true") + } + if !cfg.AIDeepAnalysis { + t.Error("AIDeepAnalysis should be true") + } + if !cfg.MultiAgent { + t.Error("MultiAgent should be true") + } + if cfg.Output != "/o" { + t.Errorf("Output = %q", cfg.Output) + } + if cfg.Format != "json" { + t.Errorf("Format = %q", cfg.Format) + } + if !cfg.JsonOutput { + t.Error("JsonOutput should be true") + } + if cfg.ModuleSettings["m1"] != true { + t.Error("ModuleSettings.m1 should be true") + } +} + +func TestApplyYAML_CLIOverrideWins(t *testing.T) { + cfg := &Config{ + Profile: "pentest", + Concurrency: 42, + Timeout: 3, + StealthMode: "paranoid", + Resolvers: "9.9.9.9", + Wordlist: "/existing", + } + y := &YAMLConfig{ + Profile: "quick", + Concurrency: 999, + Timeout: 999, + Stealth: "off", + Resolvers: []string{"8.8.8.8"}, + Wordlist: "/yaml", + } + ApplyYAML(cfg, y) + + // CLI values should survive + if cfg.Profile != "pentest" { + t.Errorf("Profile overwritten: %q", cfg.Profile) + } + if cfg.Concurrency != 42 { + t.Errorf("Concurrency overwritten: %d", cfg.Concurrency) + } + if cfg.Timeout != 3 { + t.Errorf("Timeout overwritten: %d", cfg.Timeout) + } + if cfg.StealthMode != "paranoid" { + t.Errorf("StealthMode overwritten: %q", cfg.StealthMode) + } + if cfg.Resolvers != "9.9.9.9" { + t.Errorf("Resolvers overwritten: %q", cfg.Resolvers) + } + if cfg.Wordlist != "/existing" { + t.Errorf("Wordlist overwritten: %q", cfg.Wordlist) + } +} + +func TestDefaultConfigPaths(t *testing.T) { + paths := DefaultConfigPaths() + if len(paths) < 3 { + t.Errorf("expected ≥3 default paths, got %d", len(paths)) + } + // First two are CWD-relative + if paths[0] != "god-eye.yaml" { + t.Errorf("paths[0] = %q", paths[0]) + } + if paths[1] != ".god-eye.yaml" { + t.Errorf("paths[1] = %q", paths[1]) + } +} + +func TestFindConfigFile_FindsInWorkingDir(t *testing.T) { + // Create a temp "god-eye.yaml" and ensure the search finds it. We can't + // easily change CWD for just this test, so we validate the underlying + // Stat call by constructing a path that definitely exists. + dir := t.TempDir() + target := filepath.Join(dir, "god-eye.yaml") + os.WriteFile(target, []byte("profile: quick\n"), 0o644) + + oldWD, _ := os.Getwd() + defer os.Chdir(oldWD) + if err := os.Chdir(dir); err != nil { + t.Skipf("cannot chdir: %v", err) + } + + got := FindConfigFile() + if got != "god-eye.yaml" { + t.Errorf("FindConfigFile = %q, want god-eye.yaml", got) + } +} + +func TestFindConfigFile_NoneFound(t *testing.T) { + dir := t.TempDir() + oldWD, _ := os.Getwd() + defer os.Chdir(oldWD) + if err := os.Chdir(dir); err != nil { + t.Skipf("cannot chdir: %v", err) + } + // Also override HOME to an empty dir so the user-home path never matches. + oldHome := os.Getenv("HOME") + defer os.Setenv("HOME", oldHome) + os.Setenv("HOME", dir) + + got := FindConfigFile() + if got != "" { + t.Errorf("FindConfigFile = %q, want empty", got) + } +} diff --git a/internal/diff/diff.go b/internal/diff/diff.go new file mode 100644 index 0000000..bda2d2d --- /dev/null +++ b/internal/diff/diff.go @@ -0,0 +1,270 @@ +// Package diff computes deltas between two scans of the same target. It +// powers Fase 5's asm-continuous mode: run the scanner on a schedule, diff +// against the last snapshot, alert on meaningful changes. +// +// Diff categories: +// +// new_host — subdomain not seen before +// removed_host — subdomain vanished from discovery +// new_ip — host gained an IP +// removed_ip — host lost an IP +// status_change — HTTP status code changed (200→401, 200→gone) +// tech_change — technology stack changed (upgrade or new framework) +// new_vuln — new vulnerability finding +// cleared_vuln — previously-reported vuln no longer detected +// cert_change — TLS certificate issuer/expiry changed +// new_takeover — new takeover candidate +// +// A Report is consumable both by humans (pretty-print) and by alerters +// (Slack/webhook payload shape defined later in F5.3). +package diff + +import ( + "sort" + "time" + + "god-eye/internal/store" +) + +// Change is one delta. +type Change struct { + Kind string `json:"kind"` + Host string `json:"host"` + Before string `json:"before,omitempty"` + After string `json:"after,omitempty"` + Severity string `json:"severity,omitempty"` + Detected time.Time `json:"detected_at"` +} + +// Report is the full delta between two scans. +type Report struct { + Target string `json:"target"` + OldAt time.Time `json:"old_scan_at"` + NewAt time.Time `json:"new_scan_at"` + Changes []Change `json:"changes"` +} + +// HasMeaningful returns true when the report contains any change that +// warrants alerting. "new_host" and any "new_vuln" always qualify. +func (r *Report) HasMeaningful() bool { + for _, c := range r.Changes { + switch c.Kind { + case "new_host", "new_vuln", "new_takeover", "removed_host": + return true + } + } + return false +} + +// Compute compares old vs new snapshots and returns the delta. Both +// slices are assumed to come from store.All() (sorted by subdomain). +func Compute(target string, oldHosts, newHosts []*store.Host, oldAt, newAt time.Time) *Report { + r := &Report{Target: target, OldAt: oldAt, NewAt: newAt} + + oldByName := indexHosts(oldHosts) + newByName := indexHosts(newHosts) + + // Walk the union of hostnames. + names := union(oldByName, newByName) + sort.Strings(names) + + for _, name := range names { + o, oOK := oldByName[name] + n, nOK := newByName[name] + switch { + case !oOK && nOK: + r.Changes = append(r.Changes, Change{Kind: "new_host", Host: name, Detected: newAt}) + for _, v := range n.Vulnerabilities { + r.Changes = append(r.Changes, Change{ + Kind: "new_vuln", + Host: name, + After: v.Title, + Severity: v.Severity, + Detected: newAt, + }) + } + if n.Takeover != nil { + r.Changes = append(r.Changes, Change{ + Kind: "new_takeover", + Host: name, + After: n.Takeover.Service, + Severity: "high", + Detected: newAt, + }) + } + case oOK && !nOK: + r.Changes = append(r.Changes, Change{Kind: "removed_host", Host: name, Detected: newAt}) + case oOK && nOK: + r.Changes = append(r.Changes, diffHost(o, n, newAt)...) + } + } + return r +} + +func diffHost(o, n *store.Host, at time.Time) []Change { + var out []Change + + if o.StatusCode != n.StatusCode { + out = append(out, Change{ + Kind: "status_change", + Host: n.Subdomain, + Before: itoa(o.StatusCode), + After: itoa(n.StatusCode), + Detected: at, + }) + } + + // IP deltas + oldIPs := toSet(o.IPs) + newIPs := toSet(n.IPs) + for ip := range newIPs { + if _, present := oldIPs[ip]; !present { + out = append(out, Change{Kind: "new_ip", Host: n.Subdomain, After: ip, Detected: at}) + } + } + for ip := range oldIPs { + if _, present := newIPs[ip]; !present { + out = append(out, Change{Kind: "removed_ip", Host: n.Subdomain, Before: ip, Detected: at}) + } + } + + // Tech change (set inequality) + if !stringSetsEqual(o.Technologies, n.Technologies) { + out = append(out, Change{ + Kind: "tech_change", + Host: n.Subdomain, + Before: joinSorted(o.Technologies), + After: joinSorted(n.Technologies), + Detected: at, + }) + } + + // Vuln delta (by ID) + oldVulns := indexVulns(o.Vulnerabilities) + newVulns := indexVulns(n.Vulnerabilities) + for id, v := range newVulns { + if _, present := oldVulns[id]; !present { + out = append(out, Change{ + Kind: "new_vuln", Host: n.Subdomain, After: v.Title, + Severity: v.Severity, Detected: at, + }) + } + } + for id, v := range oldVulns { + if _, present := newVulns[id]; !present { + out = append(out, Change{ + Kind: "cleared_vuln", Host: n.Subdomain, Before: v.Title, + Severity: v.Severity, Detected: at, + }) + } + } + + // Certificate change + if o.TLSIssuer != n.TLSIssuer && n.TLSIssuer != "" { + out = append(out, Change{ + Kind: "cert_change", + Host: n.Subdomain, + Before: o.TLSIssuer, + After: n.TLSIssuer, + Detected: at, + }) + } + + // Takeover appeared + if o.Takeover == nil && n.Takeover != nil { + out = append(out, Change{ + Kind: "new_takeover", Host: n.Subdomain, + After: n.Takeover.Service, Severity: "high", Detected: at, + }) + } + return out +} + +// --- helpers ------------------------------------------------------------- + +func indexHosts(hs []*store.Host) map[string]*store.Host { + out := make(map[string]*store.Host, len(hs)) + for _, h := range hs { + out[h.Subdomain] = h + } + return out +} + +func indexVulns(vs []store.Vulnerability) map[string]store.Vulnerability { + out := make(map[string]store.Vulnerability, len(vs)) + for _, v := range vs { + out[v.ID] = v + } + return out +} + +func union(a, b map[string]*store.Host) []string { + out := make(map[string]struct{}, len(a)+len(b)) + for k := range a { + out[k] = struct{}{} + } + for k := range b { + out[k] = struct{}{} + } + names := make([]string, 0, len(out)) + for n := range out { + names = append(names, n) + } + return names +} + +func toSet(ss []string) map[string]struct{} { + out := make(map[string]struct{}, len(ss)) + for _, s := range ss { + out[s] = struct{}{} + } + return out +} + +func stringSetsEqual(a, b []string) bool { + if len(a) != len(b) { + return false + } + sa := toSet(a) + for _, s := range b { + if _, ok := sa[s]; !ok { + return false + } + } + return true +} + +func joinSorted(s []string) string { + cpy := append([]string(nil), s...) + sort.Strings(cpy) + out := "" + for i, v := range cpy { + if i > 0 { + out += "," + } + out += v + } + return out +} + +func itoa(n int) string { + if n == 0 { + return "0" + } + var buf [20]byte + i := len(buf) + neg := n < 0 + if neg { + n = -n + } + for n > 0 { + i-- + buf[i] = byte('0' + n%10) + n /= 10 + } + if neg { + i-- + buf[i] = '-' + } + return string(buf[i:]) +} diff --git a/internal/diff/diff_test.go b/internal/diff/diff_test.go new file mode 100644 index 0000000..e947d32 --- /dev/null +++ b/internal/diff/diff_test.go @@ -0,0 +1,154 @@ +package diff + +import ( + "testing" + "time" + + "god-eye/internal/store" +) + +func TestCompute_NewHost(t *testing.T) { + oldHosts := []*store.Host{} + newHosts := []*store.Host{{Subdomain: "api.example.com"}} + r := Compute("example.com", oldHosts, newHosts, time.Now(), time.Now()) + if len(r.Changes) != 1 || r.Changes[0].Kind != "new_host" { + t.Errorf("expected 1 new_host change, got %+v", r.Changes) + } + if !r.HasMeaningful() { + t.Error("new_host should be meaningful") + } +} + +func TestCompute_RemovedHost(t *testing.T) { + oldHosts := []*store.Host{{Subdomain: "old.example.com"}} + newHosts := []*store.Host{} + r := Compute("example.com", oldHosts, newHosts, time.Now(), time.Now()) + if len(r.Changes) != 1 || r.Changes[0].Kind != "removed_host" { + t.Errorf("expected removed_host, got %+v", r.Changes) + } + if !r.HasMeaningful() { + t.Error("removed_host should be meaningful") + } +} + +func TestCompute_StatusChange(t *testing.T) { + oldH := &store.Host{Subdomain: "a.example.com", StatusCode: 200} + newH := &store.Host{Subdomain: "a.example.com", StatusCode: 401} + r := Compute("example.com", []*store.Host{oldH}, []*store.Host{newH}, time.Now(), time.Now()) + if len(r.Changes) != 1 || r.Changes[0].Kind != "status_change" { + t.Errorf("expected status_change, got %+v", r.Changes) + } + if r.Changes[0].Before != "200" || r.Changes[0].After != "401" { + t.Errorf("wrong before/after: %+v", r.Changes[0]) + } +} + +func TestCompute_IPDelta(t *testing.T) { + oldH := &store.Host{Subdomain: "a.example.com", IPs: []string{"1.1.1.1"}} + newH := &store.Host{Subdomain: "a.example.com", IPs: []string{"1.1.1.1", "2.2.2.2"}} + r := Compute("example.com", []*store.Host{oldH}, []*store.Host{newH}, time.Now(), time.Now()) + found := false + for _, c := range r.Changes { + if c.Kind == "new_ip" && c.After == "2.2.2.2" { + found = true + } + } + if !found { + t.Errorf("expected new_ip change, got %+v", r.Changes) + } +} + +func TestCompute_NewVuln(t *testing.T) { + oldH := &store.Host{Subdomain: "a.example.com"} + newH := &store.Host{ + Subdomain: "a.example.com", + Vulnerabilities: []store.Vulnerability{ + {ID: "xss", Title: "Reflected XSS", Severity: "high"}, + }, + } + r := Compute("example.com", []*store.Host{oldH}, []*store.Host{newH}, time.Now(), time.Now()) + found := false + for _, c := range r.Changes { + if c.Kind == "new_vuln" && c.After == "Reflected XSS" { + found = true + } + } + if !found { + t.Errorf("expected new_vuln change, got %+v", r.Changes) + } + if !r.HasMeaningful() { + t.Error("new_vuln must be meaningful") + } +} + +func TestCompute_ClearedVuln(t *testing.T) { + oldH := &store.Host{ + Subdomain: "a.example.com", + Vulnerabilities: []store.Vulnerability{ + {ID: "git-exposed", Title: "Git Exposed", Severity: "critical"}, + }, + } + newH := &store.Host{Subdomain: "a.example.com"} + r := Compute("example.com", []*store.Host{oldH}, []*store.Host{newH}, time.Now(), time.Now()) + found := false + for _, c := range r.Changes { + if c.Kind == "cleared_vuln" { + found = true + } + } + if !found { + t.Errorf("expected cleared_vuln, got %+v", r.Changes) + } +} + +func TestCompute_NewTakeover(t *testing.T) { + oldH := &store.Host{Subdomain: "a.example.com"} + newH := &store.Host{ + Subdomain: "a.example.com", + Takeover: &store.Takeover{Service: "GitHub Pages"}, + } + r := Compute("example.com", []*store.Host{oldH}, []*store.Host{newH}, time.Now(), time.Now()) + found := false + for _, c := range r.Changes { + if c.Kind == "new_takeover" && c.After == "GitHub Pages" { + found = true + } + } + if !found { + t.Errorf("expected new_takeover, got %+v", r.Changes) + } + if !r.HasMeaningful() { + t.Error("new_takeover must be meaningful") + } +} + +func TestCompute_NoChange(t *testing.T) { + h := &store.Host{ + Subdomain: "a.example.com", + IPs: []string{"1.1.1.1"}, + StatusCode: 200, + Technologies: []string{"nginx"}, + } + r := Compute("example.com", []*store.Host{h}, []*store.Host{h}, time.Now(), time.Now()) + if len(r.Changes) != 0 { + t.Errorf("expected no changes, got %+v", r.Changes) + } + if r.HasMeaningful() { + t.Error("empty report should not be meaningful") + } +} + +func TestCompute_TechChange(t *testing.T) { + oldH := &store.Host{Subdomain: "a.example.com", Technologies: []string{"nginx"}} + newH := &store.Host{Subdomain: "a.example.com", Technologies: []string{"nginx", "Apache"}} + r := Compute("example.com", []*store.Host{oldH}, []*store.Host{newH}, time.Now(), time.Now()) + found := false + for _, c := range r.Changes { + if c.Kind == "tech_change" { + found = true + } + } + if !found { + t.Errorf("expected tech_change, got %+v", r.Changes) + } +} diff --git a/internal/dns/wildcard_test.go b/internal/dns/wildcard_test.go new file mode 100644 index 0000000..5a4e8bc --- /dev/null +++ b/internal/dns/wildcard_test.go @@ -0,0 +1,174 @@ +package dns + +import "testing" + +func TestAllEqual(t *testing.T) { + tests := []struct { + name string + in []string + want bool + }{ + {"empty", nil, true}, + {"single", []string{"a"}, true}, + {"all same", []string{"a", "a", "a"}, true}, + {"one different", []string{"a", "a", "b"}, false}, + {"all different", []string{"a", "b", "c"}, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := allEqual(tt.in); got != tt.want { + t.Errorf("allEqual(%v) = %v, want %v", tt.in, got, tt.want) + } + }) + } +} + +func TestAllEqualInts(t *testing.T) { + tests := []struct { + name string + in []int + want bool + }{ + {"empty", nil, true}, + {"single", []int{200}, true}, + {"all same", []int{200, 200, 200}, true}, + {"one different", []int{200, 200, 404}, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := allEqualInts(tt.in); got != tt.want { + t.Errorf("allEqualInts(%v) = %v, want %v", tt.in, got, tt.want) + } + }) + } +} + +func TestSimilarSizes(t *testing.T) { + tests := []struct { + name string + in []int64 + want bool + }{ + {"empty", nil, true}, + {"single", []int64{1000}, true}, + {"identical", []int64{1000, 1000, 1000}, true}, + {"within 20%", []int64{1000, 1100, 1200}, true}, + {"exactly 20%", []int64{1000, 1200}, true}, + {"over 20%", []int64{1000, 1300}, false}, + {"big variance", []int64{100, 10000}, false}, + {"all zero", []int64{0, 0}, true}, + {"zero and small", []int64{0, 50}, true}, + {"zero and big", []int64{0, 200}, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := similarSizes(tt.in); got != tt.want { + t.Errorf("similarSizes(%v) = %v, want %v", tt.in, got, tt.want) + } + }) + } +} + +func TestIsWildcardIP(t *testing.T) { + wd := &WildcardDetector{} + info := &WildcardInfo{ + IsWildcard: true, + WildcardIPs: []string{"1.2.3.4", "5.6.7.8"}, + } + + if !wd.IsWildcardIP("1.2.3.4", info) { + t.Error("expected 1.2.3.4 to be wildcard IP") + } + if wd.IsWildcardIP("9.9.9.9", info) { + t.Error("expected 9.9.9.9 NOT to be wildcard IP") + } + + // nil and non-wildcard cases + if wd.IsWildcardIP("1.2.3.4", nil) { + t.Error("nil info should return false") + } + nonWild := &WildcardInfo{IsWildcard: false, WildcardIPs: []string{"1.2.3.4"}} + if wd.IsWildcardIP("1.2.3.4", nonWild) { + t.Error("non-wildcard info should return false even if IP matches list") + } +} + +func TestIsWildcardResponse(t *testing.T) { + wd := &WildcardDetector{} + info := &WildcardInfo{ + IsWildcard: true, + HTTPStatusCode: 200, + HTTPBodySize: 1000, + } + + tests := []struct { + name string + statusCode int + bodySize int64 + want bool + }{ + {"exact match", 200, 1000, true}, + {"within 10% body", 200, 1050, true}, + {"within 10% body below", 200, 950, true}, + {"over 10% body", 200, 1200, false}, + {"different status", 404, 1000, false}, + {"both different", 301, 500, false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := wd.IsWildcardResponse(tt.statusCode, tt.bodySize, info); got != tt.want { + t.Errorf("IsWildcardResponse(%d, %d) = %v, want %v", tt.statusCode, tt.bodySize, got, tt.want) + } + }) + } + + if wd.IsWildcardResponse(200, 1000, nil) { + t.Error("nil info should return false") + } +} + +func TestGenerateTestSubdomains(t *testing.T) { + subs := generateTestSubdomains() + if len(subs) < 3 { + t.Errorf("expected at least 3 test subdomains, got %d", len(subs)) + } + seen := make(map[string]bool) + for _, s := range subs { + if s == "" { + t.Error("empty test subdomain generated") + } + if seen[s] { + t.Errorf("duplicate test subdomain: %s", s) + } + seen[s] = true + } +} + +func TestWildcardInfo_GetSummary_NotWildcard(t *testing.T) { + info := &WildcardInfo{IsWildcard: false} + got := info.GetSummary() + if got == "" { + t.Error("GetSummary returned empty string") + } +} + +func TestNewWildcardDetector(t *testing.T) { + resolvers := []string{"8.8.8.8:53"} + wd := NewWildcardDetector(resolvers, 5) + if wd == nil { + t.Fatal("NewWildcardDetector returned nil") + } + if wd.timeout != 5 { + t.Errorf("timeout = %d, want 5", wd.timeout) + } + if len(wd.resolvers) != 1 || wd.resolvers[0] != "8.8.8.8:53" { + t.Errorf("resolvers = %v", wd.resolvers) + } + if wd.httpClient == nil { + t.Error("httpClient is nil") + } + if len(wd.testSubdomains) == 0 { + t.Error("testSubdomains is empty") + } +} diff --git a/internal/eventbus/bus.go b/internal/eventbus/bus.go new file mode 100644 index 0000000..4d4cd1c --- /dev/null +++ b/internal/eventbus/bus.go @@ -0,0 +1,283 @@ +package eventbus + +import ( + "context" + "errors" + "sync" + "sync/atomic" +) + +// ErrBusClosed is returned when attempting to use a closed bus. +var ErrBusClosed = errors.New("eventbus: bus closed") + +// Handler processes a single event. It runs on the subscriber's own goroutine +// so handlers may block or perform I/O without stalling publishers. A handler +// must respect ctx cancellation when performing long work. +type Handler func(ctx context.Context, e Event) + +// Subscription is returned by Subscribe/SubscribeAll and is used to stop +// receiving events. Unsubscribe is idempotent. +type Subscription struct { + bus *Bus + eventType EventType // empty string means "all" + id uint64 + once sync.Once +} + +// Unsubscribe stops the subscription. Pending events in the subscriber's +// buffer are dropped. Safe to call multiple times. +func (s *Subscription) Unsubscribe() { + if s == nil || s.bus == nil { + return + } + s.once.Do(func() { + s.bus.unsubscribe(s.eventType, s.id) + }) +} + +// Stats captures runtime metrics for observability. Stats are cumulative from +// bus creation; callers should compute deltas if rate matters. +type Stats struct { + Published uint64 // total Publish calls accepted + Delivered uint64 // events delivered to subscribers (sum across subscribers) + Dropped uint64 // events dropped because a subscriber buffer was full + Subscribers int // active subscribers right now + Closed bool +} + +// Bus is the default eventbus implementation. +type Bus struct { + bufferSize int + + mu sync.RWMutex + closed bool + nextID uint64 + subs map[EventType]map[uint64]*subscriber // type → id → subscriber + allSubs map[uint64]*subscriber // wildcard subscribers + + published uint64 + delivered uint64 + dropped uint64 + + wg sync.WaitGroup +} + +type subscriber struct { + id uint64 + eventT EventType + ch chan Event + handler Handler + ctx context.Context + cancel context.CancelFunc +} + +// New creates a new Bus. bufferSize controls the per-subscriber channel +// buffer; values ≤0 default to 256. A buffer of 1 is legal but increases +// drop probability under bursty load. +func New(bufferSize int) *Bus { + if bufferSize <= 0 { + bufferSize = 256 + } + return &Bus{ + bufferSize: bufferSize, + subs: make(map[EventType]map[uint64]*subscriber), + allSubs: make(map[uint64]*subscriber), + } +} + +// Subscribe registers a handler for a specific event type. Returns a +// Subscription that can be used to unsubscribe. +func (b *Bus) Subscribe(t EventType, h Handler) *Subscription { + return b.subscribe(t, h, false) +} + +// SubscribeAll registers a handler that receives every event type. +// Useful for logging, metrics collection, or persistence modules. +func (b *Bus) SubscribeAll(h Handler) *Subscription { + return b.subscribe("", h, true) +} + +func (b *Bus) subscribe(t EventType, h Handler, all bool) *Subscription { + if h == nil { + return &Subscription{bus: b} + } + b.mu.Lock() + if b.closed { + b.mu.Unlock() + return &Subscription{bus: b} + } + b.nextID++ + id := b.nextID + ctx, cancel := context.WithCancel(context.Background()) + s := &subscriber{ + id: id, + eventT: t, + ch: make(chan Event, b.bufferSize), + handler: h, + ctx: ctx, + cancel: cancel, + } + if all { + b.allSubs[id] = s + } else { + if b.subs[t] == nil { + b.subs[t] = make(map[uint64]*subscriber) + } + b.subs[t][id] = s + } + b.mu.Unlock() + + b.wg.Add(1) + go b.run(s) + + return &Subscription{bus: b, eventType: t, id: id} +} + +func (b *Bus) unsubscribe(t EventType, id uint64) { + b.mu.Lock() + var s *subscriber + if t == "" { + s = b.allSubs[id] + delete(b.allSubs, id) + } else { + if m, ok := b.subs[t]; ok { + s = m[id] + delete(m, id) + if len(m) == 0 { + delete(b.subs, t) + } + } + } + b.mu.Unlock() + if s != nil { + close(s.ch) // run() drains remaining events then returns + } +} + +// run is the per-subscriber goroutine loop. +func (b *Bus) run(s *subscriber) { + defer b.wg.Done() + defer s.cancel() + for e := range s.ch { + // Protect bus from handler panics — one bad handler must not + // take down the pipeline. + func() { + defer func() { + _ = recover() + }() + s.handler(s.ctx, e) + }() + } +} + +// Publish delivers e to every subscriber interested in e.Type() and every +// SubscribeAll subscriber. If ctx is canceled, Publish returns early and the +// event is not queued to any subscriber that would block. +// +// Publish is non-blocking per subscriber: if a subscriber's buffer is full the +// event is dropped for that subscriber and Stats.Dropped is incremented. +func (b *Bus) Publish(ctx context.Context, e Event) { + if e == nil { + return + } + b.mu.RLock() + if b.closed { + b.mu.RUnlock() + return + } + // Snapshot the subscriber slices under lock, then release before send. + typed := b.subs[e.Type()] + var typedList []*subscriber + if len(typed) > 0 { + typedList = make([]*subscriber, 0, len(typed)) + for _, s := range typed { + typedList = append(typedList, s) + } + } + var allList []*subscriber + if len(b.allSubs) > 0 { + allList = make([]*subscriber, 0, len(b.allSubs)) + for _, s := range b.allSubs { + allList = append(allList, s) + } + } + b.mu.RUnlock() + + atomic.AddUint64(&b.published, 1) + + for _, s := range typedList { + b.dispatch(ctx, s, e) + } + for _, s := range allList { + b.dispatch(ctx, s, e) + } +} + +func (b *Bus) dispatch(ctx context.Context, s *subscriber, e Event) { + select { + case <-ctx.Done(): + // caller abandoned; count as dropped so observability reflects reality + atomic.AddUint64(&b.dropped, 1) + case s.ch <- e: + atomic.AddUint64(&b.delivered, 1) + default: + atomic.AddUint64(&b.dropped, 1) + } +} + +// Close stops accepting new publishes and drains in-flight subscriber +// buffers. It waits until all handlers have returned, or until ctx expires. +// Returns ctx.Err() if draining did not complete in time. +func (b *Bus) Close(ctx context.Context) error { + b.mu.Lock() + if b.closed { + b.mu.Unlock() + return nil + } + b.closed = true + + // Close every subscriber channel; their goroutines will drain and exit. + for _, m := range b.subs { + for _, s := range m { + close(s.ch) + } + } + for _, s := range b.allSubs { + close(s.ch) + } + b.subs = make(map[EventType]map[uint64]*subscriber) + b.allSubs = make(map[uint64]*subscriber) + b.mu.Unlock() + + done := make(chan struct{}) + go func() { + b.wg.Wait() + close(done) + }() + + select { + case <-done: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +// Stats returns a snapshot of current metrics. +func (b *Bus) Stats() Stats { + b.mu.RLock() + closed := b.closed + subCount := len(b.allSubs) + for _, m := range b.subs { + subCount += len(m) + } + b.mu.RUnlock() + + return Stats{ + Published: atomic.LoadUint64(&b.published), + Delivered: atomic.LoadUint64(&b.delivered), + Dropped: atomic.LoadUint64(&b.dropped), + Subscribers: subCount, + Closed: closed, + } +} diff --git a/internal/eventbus/bus_test.go b/internal/eventbus/bus_test.go new file mode 100644 index 0000000..35aca49 --- /dev/null +++ b/internal/eventbus/bus_test.go @@ -0,0 +1,307 @@ +package eventbus + +import ( + "context" + "sync" + "sync/atomic" + "testing" + "time" +) + +// waitUntil polls predicate every 2ms up to timeout. Used to avoid flaky +// sleeps in async tests without adding dependencies. +func waitUntil(t *testing.T, timeout time.Duration, pred func() bool, msg string) { + t.Helper() + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + if pred() { + return + } + time.Sleep(2 * time.Millisecond) + } + t.Fatalf("timeout waiting: %s", msg) +} + +func TestPublishSubscribe_SingleType(t *testing.T) { + b := New(16) + defer b.Close(context.Background()) + + var got atomic.Int32 + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, e Event) { + ev, ok := e.(SubdomainDiscovered) + if !ok { + t.Errorf("wrong event type: %T", e) + return + } + if ev.Subdomain == "" { + t.Error("empty subdomain") + } + got.Add(1) + }) + + for i := 0; i < 5; i++ { + b.Publish(context.Background(), NewSubdomainDiscovered("test", "api.example.com", "passive")) + } + waitUntil(t, time.Second, func() bool { return got.Load() == 5 }, "5 events delivered") +} + +func TestSubscribeAll_ReceivesEveryType(t *testing.T) { + b := New(16) + defer b.Close(context.Background()) + + var got atomic.Int32 + b.SubscribeAll(func(_ context.Context, _ Event) { got.Add(1) }) + + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a.example.com", "p")) + b.Publish(context.Background(), DNSResolved{EventMeta: newMeta("dns", "a.example.com"), Subdomain: "a.example.com", IPs: []string{"1.2.3.4"}}) + b.Publish(context.Background(), HTTPProbed{EventMeta: newMeta("http", "a.example.com"), URL: "https://a.example.com", StatusCode: 200}) + + waitUntil(t, time.Second, func() bool { return got.Load() == 3 }, "3 events on wildcard") +} + +func TestSubscribe_FilteringByType(t *testing.T) { + b := New(16) + defer b.Close(context.Background()) + + var subs, dns atomic.Int32 + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) { subs.Add(1) }) + b.Subscribe(EventDNSResolved, func(_ context.Context, _ Event) { dns.Add(1) }) + + for i := 0; i < 3; i++ { + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a.example.com", "p")) + } + for i := 0; i < 2; i++ { + b.Publish(context.Background(), DNSResolved{EventMeta: newMeta("dns", "x"), Subdomain: "x"}) + } + waitUntil(t, time.Second, func() bool { return subs.Load() == 3 && dns.Load() == 2 }, "typed counts match") +} + +func TestUnsubscribe_StopsDelivery(t *testing.T) { + b := New(16) + defer b.Close(context.Background()) + + var count atomic.Int32 + sub := b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) { count.Add(1) }) + + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a.example.com", "p")) + waitUntil(t, time.Second, func() bool { return count.Load() == 1 }, "first event") + + sub.Unsubscribe() + sub.Unsubscribe() // idempotent + + // Publish after unsubscribe — should not be delivered to this handler. + for i := 0; i < 5; i++ { + b.Publish(context.Background(), NewSubdomainDiscovered("t", "b.example.com", "p")) + } + time.Sleep(30 * time.Millisecond) + if got := count.Load(); got != 1 { + t.Errorf("expected 1 delivery after unsubscribe, got %d", got) + } +} + +func TestPublish_MultipleSubscribersEachGetEvent(t *testing.T) { + b := New(16) + defer b.Close(context.Background()) + + var a, c atomic.Int32 + b.Subscribe(EventVulnerability, func(_ context.Context, _ Event) { a.Add(1) }) + b.Subscribe(EventVulnerability, func(_ context.Context, _ Event) { c.Add(1) }) + + b.Publish(context.Background(), VulnerabilityFound{EventMeta: newMeta("sec", "x"), ID: "test", Severity: SeverityHigh}) + + waitUntil(t, time.Second, func() bool { return a.Load() == 1 && c.Load() == 1 }, "both subscribers received") +} + +func TestPublish_NonBlocking_DropsWhenBufferFull(t *testing.T) { + b := New(2) + defer b.Close(context.Background()) + + blocker := make(chan struct{}) + var started atomic.Int32 + b.Subscribe(EventSubdomainDiscovered, func(ctx context.Context, _ Event) { + started.Add(1) + <-blocker + }) + + // First event enters handler (blocks). Next 2 fill the buffer of size 2. + // Subsequent publishes should be counted as dropped. + for i := 0; i < 100; i++ { + b.Publish(context.Background(), NewSubdomainDiscovered("t", "x.example.com", "p")) + } + + // Give the bus a moment to register drops. + waitUntil(t, time.Second, func() bool { + return b.Stats().Dropped > 0 + }, "some events dropped when buffer full") + + // Unblock and close cleanly. + close(blocker) +} + +func TestClose_DrainsAndStops(t *testing.T) { + b := New(16) + + var got atomic.Int32 + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) { got.Add(1) }) + + for i := 0; i < 10; i++ { + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a.example.com", "p")) + } + + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + if err := b.Close(ctx); err != nil { + t.Fatalf("Close error: %v", err) + } + if got.Load() != 10 { + t.Errorf("expected 10 delivered before close drains, got %d", got.Load()) + } + + // Publish after close is a silent no-op. + b.Publish(context.Background(), NewSubdomainDiscovered("t", "z.example.com", "p")) + if got.Load() != 10 { + t.Errorf("delivery continued after close: %d", got.Load()) + } +} + +func TestClose_IdempotentAndMulticall(t *testing.T) { + b := New(4) + ctx := context.Background() + if err := b.Close(ctx); err != nil { + t.Fatalf("first close: %v", err) + } + if err := b.Close(ctx); err != nil { + t.Fatalf("second close: %v", err) + } +} + +func TestPanicInHandler_DoesNotAffectOthers(t *testing.T) { + b := New(8) + defer b.Close(context.Background()) + + var good atomic.Int32 + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) { panic("bad handler") }) + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) { good.Add(1) }) + + for i := 0; i < 5; i++ { + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a.example.com", "p")) + } + waitUntil(t, time.Second, func() bool { return good.Load() == 5 }, "good handler received all events") +} + +func TestConcurrentPublishers_PreservesInvariant(t *testing.T) { + // With a fast-enough consumer and large buffer, some events may still be + // dropped under heavy burst. The invariant that must ALWAYS hold is: + // Published == Delivered + Dropped + // This protects against race conditions in metric bookkeeping. + b := New(4096) + defer b.Close(context.Background()) + + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) {}) + + const publishers = 20 + const perPublisher = 100 + var wg sync.WaitGroup + for i := 0; i < publishers; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for j := 0; j < perPublisher; j++ { + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a.example.com", "p")) + } + }() + } + wg.Wait() + + total := uint64(publishers * perPublisher) + waitUntil(t, 5*time.Second, func() bool { + s := b.Stats() + return s.Published == total && s.Delivered+s.Dropped == total + }, "published count matches and delivered+dropped == published") +} + +func TestStats_Increment(t *testing.T) { + b := New(16) + defer b.Close(context.Background()) + + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) {}) + + for i := 0; i < 3; i++ { + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a.example.com", "p")) + } + waitUntil(t, time.Second, func() bool { return b.Stats().Delivered == 3 }, "3 deliveries recorded") + s := b.Stats() + if s.Published != 3 { + t.Errorf("Published = %d, want 3", s.Published) + } + if s.Subscribers != 1 { + t.Errorf("Subscribers = %d, want 1", s.Subscribers) + } + if s.Closed { + t.Error("Closed = true on open bus") + } +} + +func TestPublish_NilEvent_NoOp(t *testing.T) { + b := New(8) + defer b.Close(context.Background()) + var got atomic.Int32 + b.SubscribeAll(func(_ context.Context, _ Event) { got.Add(1) }) + b.Publish(context.Background(), nil) + time.Sleep(20 * time.Millisecond) + if got.Load() != 0 { + t.Errorf("nil event was delivered") + } +} + +func TestPublish_CancelledContext_DropsNotDelivers(t *testing.T) { + b := New(1) + defer b.Close(context.Background()) + + hold := make(chan struct{}) + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, _ Event) { <-hold }) + + // First publish occupies buffer slot 1 and handler goroutine starts consuming. + b.Publish(context.Background(), NewSubdomainDiscovered("t", "a", "p")) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + // With ctx already canceled and the subscriber busy, dispatch should record a drop. + before := b.Stats().Dropped + b.Publish(ctx, NewSubdomainDiscovered("t", "b", "p")) + b.Publish(ctx, NewSubdomainDiscovered("t", "c", "p")) + after := b.Stats().Dropped + if after <= before { + t.Errorf("expected Dropped to increase with canceled ctx, before=%d after=%d", before, after) + } + + close(hold) +} + +func TestHandlerReceivesEventMetadata(t *testing.T) { + b := New(8) + defer b.Close(context.Background()) + + done := make(chan Event, 1) + b.Subscribe(EventSubdomainDiscovered, func(_ context.Context, e Event) { done <- e }) + + before := time.Now().Add(-time.Second) + b.Publish(context.Background(), NewSubdomainDiscovered("sources.crtsh", "api.example.com", "passive:crt.sh")) + + select { + case e := <-done: + m := e.Meta() + if m.Source != "sources.crtsh" { + t.Errorf("Source = %q", m.Source) + } + if m.Target != "api.example.com" { + t.Errorf("Target = %q", m.Target) + } + if m.At.Before(before) { + t.Errorf("At = %v is before %v", m.At, before) + } + case <-time.After(time.Second): + t.Fatal("no event received") + } +} diff --git a/internal/eventbus/events.go b/internal/eventbus/events.go new file mode 100644 index 0000000..f7fa81d --- /dev/null +++ b/internal/eventbus/events.go @@ -0,0 +1,337 @@ +// Package eventbus provides a typed, context-aware pub/sub bus that decouples +// discovery, probing, analysis, and reporting modules in God's Eye v2. +// +// Design choices: +// - Events are typed structs implementing Event; dispatch is keyed on EventType. +// - Subscribers run handlers on their own goroutine with a buffered channel, +// so a slow handler cannot stall the producer. +// - Publish is non-blocking: if a subscriber buffer is full, the event is +// dropped for that subscriber and Stats.Dropped is incremented. Subscribers +// that care about lossless delivery must size their buffer accordingly. +// - Close stops accepting new events and drains outstanding ones before +// returning. +package eventbus + +import "time" + +// EventType identifies the kind of an event. +type EventType string + +// Canonical event types. Modules should always use these constants rather than +// string literals to avoid typos and to make the full event vocabulary greppable. +const ( + EventSubdomainDiscovered EventType = "subdomain.discovered" + EventDNSResolved EventType = "dns.resolved" + EventHTTPProbed EventType = "http.probed" + EventTechDetected EventType = "tech.detected" + EventTLSAnalyzed EventType = "tls.analyzed" + EventTakeoverCandidate EventType = "takeover.candidate" + EventTakeoverConfirmed EventType = "takeover.confirmed" + EventVulnerability EventType = "vulnerability" + EventSecret EventType = "secret" + EventCVEMatch EventType = "cve.match" + EventCloudAsset EventType = "cloud.asset" + EventAPIFinding EventType = "api.finding" + EventJSFile EventType = "js.file" + EventAIFinding EventType = "ai.finding" + EventPhaseStarted EventType = "phase.started" + EventPhaseCompleted EventType = "phase.completed" + EventModuleError EventType = "module.error" + EventScanStarted EventType = "scan.started" + EventScanCompleted EventType = "scan.completed" +) + +// Severity levels used across vulnerability, secret, AI and CVE events. +type Severity string + +const ( + SeverityInfo Severity = "info" + SeverityLow Severity = "low" + SeverityMedium Severity = "medium" + SeverityHigh Severity = "high" + SeverityCritical Severity = "critical" +) + +// Event is implemented by every event struct. +type Event interface { + Type() EventType + Meta() EventMeta +} + +// EventMeta is shared metadata embedded in every event. +type EventMeta struct { + At time.Time // when the event was created + Source string // originating module name (e.g. "sources.crtsh", "dns.resolver") + Target string // logical target (typically the subdomain or host the event pertains to) +} + +// Meta returns the shared metadata; implemented by embedding EventMeta. +func (m EventMeta) Meta() EventMeta { return m } + +// now returns the current time; indirected for testability. +var now = time.Now + +// newMeta builds an EventMeta with a populated timestamp. +func newMeta(source, target string) EventMeta { + return EventMeta{At: now(), Source: source, Target: target} +} + +// --- Concrete event types -------------------------------------------------- + +// SubdomainDiscovered fires whenever any source (passive, brute, recursive, +// CT, etc.) identifies a subdomain that passes the "ends in target domain" +// filter. Multiple sources may discover the same subdomain — the bus does not +// dedup; that's the store's job. +type SubdomainDiscovered struct { + EventMeta + Subdomain string + Method string // "passive:crt.sh", "brute", "recursive", "ct-stream", etc. +} + +func (SubdomainDiscovered) Type() EventType { return EventSubdomainDiscovered } + +func NewSubdomainDiscovered(source, subdomain, method string) SubdomainDiscovered { + return SubdomainDiscovered{ + EventMeta: newMeta(source, subdomain), + Subdomain: subdomain, + Method: method, + } +} + +// DNSResolved fires after a subdomain is resolved. Empty IPs field signals +// an intentionally negative result (NXDOMAIN); absence of the event means +// "not yet resolved". +type DNSResolved struct { + EventMeta + Subdomain string + IPs []string + CNAME string + PTR string +} + +func (DNSResolved) Type() EventType { return EventDNSResolved } + +// HTTPProbed fires once per successful HTTP probe, including server banner, +// title, and technology signals. Security checks emit their own events. +type HTTPProbed struct { + EventMeta + URL string + StatusCode int + ContentLength int64 + Title string + Server string + Technologies []string + Headers map[string]string + ResponseMs int64 + TLSVersion string + TLSSelfSigned bool +} + +func (HTTPProbed) Type() EventType { return EventHTTPProbed } + +// VulnerabilityFound is the canonical finding event for any detected issue. +// Scanner modules (security checks, smuggling, SSRF, GraphQL, etc.) all emit +// this so the reporter/aggregator has a single type to consume. +type VulnerabilityFound struct { + EventMeta + ID string // stable identifier, e.g. "open-redirect", "cors-wildcard-creds" + Title string // short human-readable title + Description string // longer context + Severity Severity + URL string // affected URL + Evidence string // raw evidence (truncated if too large) + Remediation string // how to fix + CVEs []string // referenced CVEs if any + OWASP string // OWASP category (e.g. "A03:2021-Injection") + CVSS float64 // 0.0 if not scored +} + +func (VulnerabilityFound) Type() EventType { return EventVulnerability } + +// SecretFound fires when a credential, API key, or token is detected (in JS, +// response bodies, commits, etc.). +type SecretFound struct { + EventMeta + Kind string // "aws_access_key", "jwt", "stripe_live", "generic_hex" + Match string // redacted or truncated match — full value in Value if validated + Value string // full value, populated only when validation succeeded + Location string // where it was found (URL, file path, commit sha) + Validated bool // true if we verified the secret is live against its service + Severity Severity + Description string +} + +func (SecretFound) Type() EventType { return EventSecret } + +// CVEMatch fires when a CVE is correlated to a detected technology/version. +type CVEMatch struct { + EventMeta + CVE string + Technology string + Version string + Severity Severity + CVSS float64 + Description string + URL string + InKEV bool // true if in CISA Known Exploited Vulnerabilities catalog +} + +func (CVEMatch) Type() EventType { return EventCVEMatch } + +// TakeoverCandidate fires when a CNAME or fingerprint points at a service +// that could potentially be taken over. TakeoverConfirmed fires after active +// verification (service claim test) succeeds. +type TakeoverCandidate struct { + EventMeta + Subdomain string + Service string // "GitHub Pages", "S3", "Heroku", etc. + CNAME string + Evidence string +} + +func (TakeoverCandidate) Type() EventType { return EventTakeoverCandidate } + +type TakeoverConfirmed struct { + EventMeta + Subdomain string + Service string + CNAME string + PoC string // curl/HTTP reproducer +} + +func (TakeoverConfirmed) Type() EventType { return EventTakeoverConfirmed } + +// CloudAssetFound fires for exposed/accessible cloud assets (S3 buckets, +// GCS buckets, Azure blobs, Firebase projects, etc.). +type CloudAssetFound struct { + EventMeta + Provider string // "AWS", "GCP", "Azure", "Firebase" + Kind string // "s3-bucket", "gcs-bucket", "lambda-url" + Name string + URL string + Status string // "public-read", "listable", "writable", "exists" + Permissions []string // detailed permissions if known +} + +func (CloudAssetFound) Type() EventType { return EventCloudAsset } + +// APIFinding fires for discovered/enumerated API surfaces (GraphQL, Swagger, +// Postman, misconfigured REST) with associated issues. +type APIFinding struct { + EventMeta + Kind string // "graphql-introspection", "swagger-exposed", "rest-cors", etc. + URL string + Issue string + Severity Severity + Endpoints []string +} + +func (APIFinding) Type() EventType { return EventAPIFinding } + +// TechDetected fires when a technology (framework, server, CMS, language) is +// identified with a version, feeding CVE matching and AI analysis. +type TechDetected struct { + EventMeta + Host string + Technology string + Version string + Category string // "web-server", "framework", "cms", "language", "waf" + Confidence float64 +} + +func (TechDetected) Type() EventType { return EventTechDetected } + +// TLSAnalyzed fires with TLS certificate details, including appliance +// fingerprint when identifiable. +type TLSAnalyzed struct { + EventMeta + Host string + Version string + Issuer string + Expiry time.Time + SelfSigned bool + AltNames []string + Vendor string // FortiGate, Palo Alto, etc. (empty if no fingerprint) + Product string + ApplianceKind string // "firewall", "vpn", "loadbalancer", "waf" + InternalHosts []string +} + +func (TLSAnalyzed) Type() EventType { return EventTLSAnalyzed } + +// JSFileDiscovered fires when a JavaScript file is discovered and prepared +// for analysis (secret scanning, endpoint extraction, AI review). +type JSFileDiscovered struct { + EventMeta + URL string + Size int64 + Host string +} + +func (JSFileDiscovered) Type() EventType { return EventJSFile } + +// AIFinding is emitted by any AI/agent module (cascade or multi-agent). +type AIFinding struct { + EventMeta + Subject string // subdomain/URL the finding pertains to + Agent string // "triage", "deep", "xss", "sqli", etc. + Model string // LLM model id + Severity Severity + Title string + Description string + Evidence string + CVEs []string + OWASP string + Confidence float64 +} + +func (AIFinding) Type() EventType { return EventAIFinding } + +// PhaseStarted / PhaseCompleted frame pipeline phases (passive, brute, +// resolve, probe, ai, etc.) so UIs and progress trackers can react. +type PhaseStarted struct { + EventMeta + Phase string +} + +func (PhaseStarted) Type() EventType { return EventPhaseStarted } + +type PhaseCompleted struct { + EventMeta + Phase string + Duration time.Duration + Stats map[string]int64 +} + +func (PhaseCompleted) Type() EventType { return EventPhaseCompleted } + +// ModuleError fires when a module encounters a non-fatal error (source +// unavailable, rate-limited, timeout). Use this for observability; do not +// log errors in modules directly. +type ModuleError struct { + EventMeta + Module string + Err string // stringified error + Fatal bool // true only when the module cannot continue + Context map[string]string +} + +func (ModuleError) Type() EventType { return EventModuleError } + +// ScanStarted / ScanCompleted bookend the whole run. +type ScanStarted struct { + EventMeta + Target string + Profile string +} + +func (ScanStarted) Type() EventType { return EventScanStarted } + +type ScanCompleted struct { + EventMeta + Target string + Duration time.Duration + Stats map[string]int64 +} + +func (ScanCompleted) Type() EventType { return EventScanCompleted } diff --git a/internal/http/factory.go b/internal/http/factory.go index 88158ce..046fef8 100644 --- a/internal/http/factory.go +++ b/internal/http/factory.go @@ -6,6 +6,8 @@ import ( "net/http" "sync" "time" + + "god-eye/internal/proxyconf" ) // ClientFactory manages shared HTTP clients with connection pooling @@ -26,8 +28,40 @@ type ClientFactory struct { var ( factory *ClientFactory factoryOnce sync.Once + + // proxyURL captures the most recent SetProxy() value, read at factory + // construction time. Callers MUST invoke SetProxy BEFORE any code path + // that triggers GetFactory — otherwise the factory is built with a + // direct dialer and subsequent proxy changes won't be picked up. + // + // In main.go this is safe: we call SetProxy right after flag parsing, + // before any module starts. + proxyURL string + proxyMu sync.RWMutex ) +// SetProxy configures the outbound proxy for every HTTP client the +// factory hands out. Must be called BEFORE GetFactory() / any module +// uses a shared client. Supported schemes: http, https, socks5, socks5h. +// Empty string disables proxying. +func SetProxy(u string) error { + if err := proxyconf.Validate(u); err != nil { + return err + } + proxyMu.Lock() + proxyURL = u + proxyMu.Unlock() + return nil +} + +// CurrentProxy returns the currently-configured proxy URL, or empty when +// none. Useful for status/debug output. +func CurrentProxy() string { + proxyMu.RLock() + defer proxyMu.RUnlock() + return proxyURL +} + // GetFactory returns the singleton client factory func GetFactory() *ClientFactory { factoryOnce.Do(func() { @@ -37,12 +71,26 @@ func GetFactory() *ClientFactory { } func newClientFactory() *ClientFactory { + proxyMu.RLock() + cfgProxy := proxyURL + proxyMu.RUnlock() + + baseDialer := &net.Dialer{ + Timeout: 10 * time.Second, + KeepAlive: 30 * time.Second, + } + dialCtx, err := proxyconf.BuildDialer(cfgProxy, baseDialer) + if err != nil { + // Bad proxy URL at this point is a programming error (we validated + // in SetProxy). Fall back to direct rather than crashing. + dialCtx = baseDialer.DialContext + } + proxyFunc, _ := proxyconf.BuildProxyFunc(cfgProxy) + // Secure transport with TLS verification secureTransport := &http.Transport{ - DialContext: (&net.Dialer{ - Timeout: 10 * time.Second, - KeepAlive: 30 * time.Second, - }).DialContext, + DialContext: dialCtx, + Proxy: proxyFunc, MaxIdleConns: 200, MaxIdleConnsPerHost: 20, MaxConnsPerHost: 50, @@ -57,10 +105,8 @@ func newClientFactory() *ClientFactory { // Insecure transport (for scanning targets with invalid certs) insecureTransport := &http.Transport{ - DialContext: (&net.Dialer{ - Timeout: 10 * time.Second, - KeepAlive: 30 * time.Second, - }).DialContext, + DialContext: dialCtx, + Proxy: proxyFunc, MaxIdleConns: 200, MaxIdleConnsPerHost: 20, MaxConnsPerHost: 50, diff --git a/internal/module/module.go b/internal/module/module.go new file mode 100644 index 0000000..ad6fa0a --- /dev/null +++ b/internal/module/module.go @@ -0,0 +1,101 @@ +// Package module defines the Module interface and Registry used by God's Eye v2 +// to organize discovery, enrichment, analysis, and reporting units of work. +// +// A Module is any unit of the pipeline that subscribes to zero-or-more event +// types, produces zero-or-more event types, and optionally performs a bounded +// amount of work on startup (e.g. a passive source fetches once and publishes). +// +// Modules are decoupled: they do not call each other directly. Ordering emerges +// from the event-driven dependency graph, not from phase barriers. The Phase +// label is metadata used for grouping in progress UIs and logs, not a scheduling +// primitive. +package module + +import ( + "context" + + "god-eye/internal/eventbus" + "god-eye/internal/store" +) + +// Phase groups modules at similar pipeline stages for presentation. Modules at +// different phases may still run concurrently; the scanner does not enforce +// phase barriers. +type Phase string + +const ( + PhaseSetup Phase = "setup" // load DBs, wordlists, validate config + PhaseDiscovery Phase = "discovery" // subdomain sources (passive, CT, brute, recursive) + PhaseResolution Phase = "resolution" // DNS resolve, CNAME, PTR, IP info, wildcard filter + PhaseEnrichment Phase = "enrichment" // HTTP probe, tech fingerprint, TLS analyze + PhaseAnalysis Phase = "analysis" // security checks, takeover, secrets, AI, CVE match + PhaseReporting Phase = "reporting" // output writers, report generation +) + +// Context bundles everything a module needs to run. +// +// The Ctx field carries cancellation — every long-running module must select +// on Ctx.Done() to exit cleanly when the user interrupts. +type Context struct { + Ctx context.Context + Bus *eventbus.Bus + Store store.Store + Config ConfigView + Target string // primary target domain + Profile string // active profile name (bugbounty, pentest, stealth-max, ...) +} + +// ConfigView is a narrow read-only interface over the scan config, exposed to +// modules so they cannot mutate global state. Implementations live in the +// config package. +type ConfigView interface { + // Profile returns the active profile name ("" when none is selected). + Profile() string + // Bool reads a boolean config key, returning fallback if unset. + Bool(key string, fallback bool) bool + // Int reads an int key, returning fallback if unset. + Int(key string, fallback int) int + // String reads a string key, returning fallback if unset. + String(key string, fallback string) string + // Strings reads a string-slice key. + Strings(key string) []string + // ModuleEnabled lets the user disable a module by name. Registry honors + // this during selection. + ModuleEnabled(moduleName string) bool +} + +// Module is the unit of work registered in the pipeline. +// +// Implementations should: +// - be cheap to construct (no I/O in the Module value itself) +// - do all setup/teardown inside Run so lifecycle is explicit +// - subscribe to events via mctx.Bus.Subscribe in Run +// - return promptly when mctx.Ctx is canceled OR when their work is complete +type Module interface { + // Name uniquely identifies the module. Use dotted notation grouping by + // concern: "sources.crtsh", "dns.resolver", "http.probe", "security.cors", + // "ai.cascade". The registry rejects duplicate names. + Name() string + + // Phase groups the module in pipeline UIs. See Phase constants. + Phase() Phase + + // Consumes lists event types the module subscribes to. Empty means the + // module is a pure producer (e.g. a passive source). Used by tooling to + // visualize the event graph; the bus itself is queried via Subscribe. + Consumes() []eventbus.EventType + + // Produces lists event types the module publishes. Empty means the module + // only side-effects (e.g. reporting). Used for tooling and dep docs. + Produces() []eventbus.EventType + + // DefaultEnabled returns whether this module runs when config does not + // explicitly enable/disable it. Passive sources typically default true; + // aggressive/experimental modules typically default false. + DefaultEnabled() bool + + // Run executes the module. Must be non-blocking on setup and must return + // when its work is complete OR mctx.Ctx is canceled. Errors returned are + // logged via ModuleError events by the scanner. + Run(mctx Context) error +} diff --git a/internal/module/registry.go b/internal/module/registry.go new file mode 100644 index 0000000..22969b5 --- /dev/null +++ b/internal/module/registry.go @@ -0,0 +1,183 @@ +package module + +import ( + "fmt" + "sort" + "sync" + + "god-eye/internal/eventbus" +) + +// Registry stores modules keyed by name. Modules register themselves via +// init() functions by calling Register on the default registry. +type Registry struct { + mu sync.RWMutex + modules map[string]Module + order []string // insertion order for deterministic iteration +} + +// NewRegistry returns an empty registry. Most callers should use Default() +// which returns the process-wide registry that init() functions populate. +func NewRegistry() *Registry { + return &Registry{modules: make(map[string]Module)} +} + +var ( + defaultRegistry *Registry + defaultOnce sync.Once +) + +// Default returns the process-wide module registry. +func Default() *Registry { + defaultOnce.Do(func() { + defaultRegistry = NewRegistry() + }) + return defaultRegistry +} + +// Register adds m to r. Panics on duplicate name — registration happens at +// init() time, so duplicates indicate a compile-time bug that must surface +// immediately rather than silently overwrite. +func (r *Registry) Register(m Module) { + if m == nil { + panic("module.Register: nil module") + } + name := m.Name() + if name == "" { + panic("module.Register: module has empty Name()") + } + r.mu.Lock() + defer r.mu.Unlock() + if _, exists := r.modules[name]; exists { + panic(fmt.Sprintf("module.Register: duplicate module %q", name)) + } + r.modules[name] = m + r.order = append(r.order, name) +} + +// Register is a shortcut for Default().Register(m). Intended use: +// +// func init() { module.Register(&myModule{}) } +func Register(m Module) { Default().Register(m) } + +// Get returns the module with the given name. +func (r *Registry) Get(name string) (Module, bool) { + r.mu.RLock() + defer r.mu.RUnlock() + m, ok := r.modules[name] + return m, ok +} + +// Names returns all registered module names in insertion order. +func (r *Registry) Names() []string { + r.mu.RLock() + defer r.mu.RUnlock() + out := make([]string, len(r.order)) + copy(out, r.order) + return out +} + +// All returns every registered module in insertion order. The returned slice +// is safe for the caller to iterate but do not mutate it. +func (r *Registry) All() []Module { + r.mu.RLock() + defer r.mu.RUnlock() + out := make([]Module, 0, len(r.order)) + for _, n := range r.order { + out = append(out, r.modules[n]) + } + return out +} + +// ByPhase returns modules belonging to the given phase, sorted by name for +// stable presentation. +func (r *Registry) ByPhase(p Phase) []Module { + r.mu.RLock() + defer r.mu.RUnlock() + var out []Module + for _, n := range r.order { + m := r.modules[n] + if m.Phase() == p { + out = append(out, m) + } + } + sort.SliceStable(out, func(i, j int) bool { return out[i].Name() < out[j].Name() }) + return out +} + +// Select returns the subset of modules that should run for the given config. +// A module is selected when cfg.ModuleEnabled(name) returns true (explicit +// enable wins), OR when cfg leaves it unset and DefaultEnabled() is true. +func (r *Registry) Select(cfg ConfigView) []Module { + r.mu.RLock() + defer r.mu.RUnlock() + var out []Module + for _, n := range r.order { + m := r.modules[n] + if cfg != nil { + // explicit config: respect it directly + if cfg.ModuleEnabled(m.Name()) { + out = append(out, m) + continue + } + // if the config has a non-default opinion (enabled=false), honor it + // — but ModuleEnabled returning false could also mean "unset". + // We resolve the ambiguity by checking whether any profile/CLI flag + // set it via a separate mechanism; for now, fall back to the + // module's default. + if m.DefaultEnabled() { + out = append(out, m) + } + continue + } + // no config: honor module default + if m.DefaultEnabled() { + out = append(out, m) + } + } + return out +} + +// ProducersOf returns the modules that declare t in their Produces() set. +// Used by tooling and tests to validate the event-graph integrity. +func (r *Registry) ProducersOf(t eventbus.EventType) []Module { + r.mu.RLock() + defer r.mu.RUnlock() + var out []Module + for _, n := range r.order { + m := r.modules[n] + for _, et := range m.Produces() { + if et == t { + out = append(out, m) + break + } + } + } + return out +} + +// ConsumersOf returns modules that declare t in their Consumes() set. +func (r *Registry) ConsumersOf(t eventbus.EventType) []Module { + r.mu.RLock() + defer r.mu.RUnlock() + var out []Module + for _, n := range r.order { + m := r.modules[n] + for _, et := range m.Consumes() { + if et == t { + out = append(out, m) + break + } + } + } + return out +} + +// Reset clears the registry. Intended for tests only; never call in production +// code. +func (r *Registry) Reset() { + r.mu.Lock() + defer r.mu.Unlock() + r.modules = make(map[string]Module) + r.order = nil +} diff --git a/internal/module/registry_test.go b/internal/module/registry_test.go new file mode 100644 index 0000000..e8bc0d1 --- /dev/null +++ b/internal/module/registry_test.go @@ -0,0 +1,257 @@ +package module + +import ( + "context" + "reflect" + "sort" + "testing" + + "god-eye/internal/eventbus" +) + +// fakeModule is a minimal Module for tests. +type fakeModule struct { + name string + phase Phase + consumes []eventbus.EventType + produces []eventbus.EventType + defaultEnabled bool + runCalled bool +} + +func (f *fakeModule) Name() string { return f.name } +func (f *fakeModule) Phase() Phase { return f.phase } +func (f *fakeModule) Consumes() []eventbus.EventType { return f.consumes } +func (f *fakeModule) Produces() []eventbus.EventType { return f.produces } +func (f *fakeModule) DefaultEnabled() bool { return f.defaultEnabled } +func (f *fakeModule) Run(mctx Context) error { f.runCalled = true; return nil } + +// fakeConfig implements ConfigView for tests. +type fakeConfig struct { + profile string + enabled map[string]bool +} + +func (c *fakeConfig) Profile() string { return c.profile } +func (c *fakeConfig) Bool(k string, fb bool) bool { return fb } +func (c *fakeConfig) Int(k string, fb int) int { return fb } +func (c *fakeConfig) String(k, fb string) string { return fb } +func (c *fakeConfig) Strings(k string) []string { return nil } +func (c *fakeConfig) ModuleEnabled(name string) bool { return c.enabled[name] } + +func TestRegister_AndGet(t *testing.T) { + r := NewRegistry() + m := &fakeModule{name: "test.one", phase: PhaseDiscovery, defaultEnabled: true} + r.Register(m) + + got, ok := r.Get("test.one") + if !ok { + t.Fatal("Get returned !ok for registered module") + } + if got != m { + t.Error("Get returned a different instance") + } + + if _, ok := r.Get("not.present"); ok { + t.Error("Get returned ok for missing module") + } +} + +func TestRegister_DuplicatePanic(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{name: "dup", phase: PhaseDiscovery}) + defer func() { + if recover() == nil { + t.Error("expected panic on duplicate registration") + } + }() + r.Register(&fakeModule{name: "dup", phase: PhaseDiscovery}) +} + +func TestRegister_NilPanic(t *testing.T) { + r := NewRegistry() + defer func() { + if recover() == nil { + t.Error("expected panic on nil module") + } + }() + r.Register(nil) +} + +func TestRegister_EmptyNamePanic(t *testing.T) { + r := NewRegistry() + defer func() { + if recover() == nil { + t.Error("expected panic on empty name") + } + }() + r.Register(&fakeModule{name: "", phase: PhaseDiscovery}) +} + +func TestNames_InsertionOrder(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{name: "zebra", phase: PhaseDiscovery}) + r.Register(&fakeModule{name: "alpha", phase: PhaseDiscovery}) + r.Register(&fakeModule{name: "middle", phase: PhaseDiscovery}) + + want := []string{"zebra", "alpha", "middle"} + got := r.Names() + if !reflect.DeepEqual(got, want) { + t.Errorf("Names order = %v, want %v", got, want) + } +} + +func TestAll_ReturnsRegistered(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{name: "a", phase: PhaseDiscovery}) + r.Register(&fakeModule{name: "b", phase: PhaseAnalysis}) + r.Register(&fakeModule{name: "c", phase: PhaseReporting}) + + if got := len(r.All()); got != 3 { + t.Errorf("All length = %d, want 3", got) + } +} + +func TestByPhase_SortedByName(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{name: "sources.zzz", phase: PhaseDiscovery}) + r.Register(&fakeModule{name: "sources.aaa", phase: PhaseDiscovery}) + r.Register(&fakeModule{name: "security.cors", phase: PhaseAnalysis}) + r.Register(&fakeModule{name: "sources.mmm", phase: PhaseDiscovery}) + + got := r.ByPhase(PhaseDiscovery) + names := make([]string, len(got)) + for i, m := range got { + names[i] = m.Name() + } + want := []string{"sources.aaa", "sources.mmm", "sources.zzz"} + if !reflect.DeepEqual(names, want) { + t.Errorf("ByPhase(discovery) = %v, want %v (sorted)", names, want) + } + + if got := r.ByPhase(PhaseAnalysis); len(got) != 1 || got[0].Name() != "security.cors" { + t.Errorf("ByPhase(analysis) unexpected: %v", got) + } + if got := r.ByPhase(PhaseReporting); len(got) != 0 { + t.Errorf("ByPhase(reporting) should be empty, got %d", len(got)) + } +} + +func TestSelect_DefaultEnabled(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{name: "on-by-default", phase: PhaseDiscovery, defaultEnabled: true}) + r.Register(&fakeModule{name: "off-by-default", phase: PhaseDiscovery, defaultEnabled: false}) + + // nil config: module default governs + got := r.Select(nil) + names := moduleNames(got) + sort.Strings(names) + if !reflect.DeepEqual(names, []string{"on-by-default"}) { + t.Errorf("Select(nil) = %v, want [on-by-default]", names) + } +} + +func TestSelect_ConfigEnablesOff(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{name: "optin", phase: PhaseAnalysis, defaultEnabled: false}) + r.Register(&fakeModule{name: "default-on", phase: PhaseAnalysis, defaultEnabled: true}) + + cfg := &fakeConfig{enabled: map[string]bool{"optin": true}} + got := r.Select(cfg) + names := moduleNames(got) + sort.Strings(names) + want := []string{"default-on", "optin"} + if !reflect.DeepEqual(names, want) { + t.Errorf("Select = %v, want %v", names, want) + } +} + +func TestProducersOf_AndConsumersOf(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{ + name: "producer-a", + phase: PhaseDiscovery, + produces: []eventbus.EventType{eventbus.EventSubdomainDiscovered}, + }) + r.Register(&fakeModule{ + name: "producer-b", + phase: PhaseDiscovery, + produces: []eventbus.EventType{eventbus.EventSubdomainDiscovered, eventbus.EventDNSResolved}, + }) + r.Register(&fakeModule{ + name: "consumer", + phase: PhaseEnrichment, + consumes: []eventbus.EventType{eventbus.EventDNSResolved}, + }) + + producers := r.ProducersOf(eventbus.EventSubdomainDiscovered) + names := moduleNames(producers) + sort.Strings(names) + want := []string{"producer-a", "producer-b"} + if !reflect.DeepEqual(names, want) { + t.Errorf("ProducersOf = %v, want %v", names, want) + } + + consumers := r.ConsumersOf(eventbus.EventDNSResolved) + if len(consumers) != 1 || consumers[0].Name() != "consumer" { + t.Errorf("ConsumersOf unexpected: %v", consumers) + } +} + +func TestReset(t *testing.T) { + r := NewRegistry() + r.Register(&fakeModule{name: "m1", phase: PhaseDiscovery, defaultEnabled: true}) + r.Register(&fakeModule{name: "m2", phase: PhaseDiscovery, defaultEnabled: true}) + if len(r.All()) != 2 { + t.Fatal("pre-reset: expected 2 modules") + } + r.Reset() + if len(r.All()) != 0 { + t.Errorf("post-reset: expected 0 modules, got %d", len(r.All())) + } + // Re-register after reset works + r.Register(&fakeModule{name: "m1", phase: PhaseDiscovery, defaultEnabled: true}) + if len(r.All()) != 1 { + t.Errorf("post-reset re-register: expected 1, got %d", len(r.All())) + } +} + +func TestDefault_Singleton(t *testing.T) { + a := Default() + b := Default() + if a != b { + t.Error("Default() returned different instances") + } +} + +func TestRunContextCarriesFields(t *testing.T) { + // Sanity: Context struct is populated correctly — this is effectively a + // struct-init contract test to catch accidental field removals. + ctx := context.Background() + bus := eventbus.New(16) + defer bus.Close(context.Background()) + + mctx := Context{ + Ctx: ctx, + Bus: bus, + Target: "example.com", + Profile: "bugbounty", + } + if mctx.Target != "example.com" { + t.Errorf("Target lost: %q", mctx.Target) + } + if mctx.Profile != "bugbounty" { + t.Errorf("Profile lost: %q", mctx.Profile) + } + if mctx.Bus != bus { + t.Error("Bus not retained") + } +} + +func moduleNames(ms []Module) []string { + out := make([]string, len(ms)) + for i, m := range ms { + out[i] = m.Name() + } + return out +} diff --git a/internal/modules/ai/ai.go b/internal/modules/ai/ai.go new file mode 100644 index 0000000..3ccd332 --- /dev/null +++ b/internal/modules/ai/ai.go @@ -0,0 +1,660 @@ +// Package ai is the v2 adapter that wires the Ollama client into the +// event-driven pipeline. Unlike the initial skeleton (which only called +// CVEMatch on TechDetected), this module subscribes to five event types +// and dispatches each to the appropriate v1 client method: +// +// TechDetected → CVEMatch → CVEMatch events +// JSFileDiscovered → AnalyzeJavaScript → AIFinding + SecretFound +// HTTPProbed → AnalyzeHTTPResponse (for 5xx / suspicious 4xx) → AIFinding +// SecretFound → FilterSecrets (triage real vs regex noise) → AIFinding tag +// VulnerabilityFound → multi-agent orchestrator (agents package) → AIFinding with remediation +// ScanCompleted → DetectAnomalies + GenerateReport → AIFinding + report artifact +// +// Every handler: +// - is a no-op when ai.enabled=false (module Run returns immediately) +// - dedups by content hash to avoid hammering Ollama with duplicates +// - cascades through the fast triage model before the deep model +// - emits AIFinding events so downstream reporters/TUI pick them up +// +// The module is the primary value of God's Eye v2's "local LLM" story — +// without this wiring, the AI layer was essentially a 20GB curiosity +// that added a single CVE string per scan. +package ai + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "strings" + "sync" + "sync/atomic" + "time" + + "god-eye/internal/ai" + "god-eye/internal/ai/agents" + "god-eye/internal/eventbus" + "god-eye/internal/module" + "god-eye/internal/store" +) + +const ModuleName = "ai.cascade" + +type aiModule struct { + client *ai.OllamaClient + orchestrator *agents.AgentOrchestrator + + // queryCache dedups expensive Ollama calls across a single scan. + // Keyed by SHA256 of (method + input), value is a flag struct so + // the same (method, input) pair is processed exactly once. + cache sync.Map // map[string]struct{} + + // Counters surfaced at scan end for observability. + cveLookups atomic.Int64 + jsAnalyses atomic.Int64 + httpAnalyses atomic.Int64 + secretValidations atomic.Int64 + vulnEnrichments atomic.Int64 + anomalyScans atomic.Int64 + reportGenerations atomic.Int64 +} + +func Register() { module.Register(&aiModule{}) } + +func (*aiModule) Name() string { return ModuleName } +func (*aiModule) Phase() module.Phase { return module.PhaseAnalysis } +func (*aiModule) Consumes() []eventbus.EventType { + return []eventbus.EventType{ + eventbus.EventTechDetected, + eventbus.EventJSFile, + eventbus.EventHTTPProbed, + eventbus.EventSecret, + eventbus.EventVulnerability, + eventbus.EventScanCompleted, + } +} +func (*aiModule) Produces() []eventbus.EventType { + return []eventbus.EventType{ + eventbus.EventAIFinding, + eventbus.EventCVEMatch, + eventbus.EventSecret, // validated/re-emitted + } +} + +// DefaultEnabled returns true so the module is always loaded; Run() no-ops +// unless the user set ai.enabled via --enable-ai / wizard / YAML. +func (*aiModule) DefaultEnabled() bool { return true } + +// Run is the heart of the v2 AI layer: wires six event subscriptions, +// drains initial store state, and waits for late events in a bounded +// window. +func (a *aiModule) Run(mctx module.Context) error { + if !mctx.Config.Bool("ai.enabled", false) { + return nil + } + + a.client = ai.NewOllamaClient( + mctx.Config.String("ai.url", "http://localhost:11434"), + mctx.Config.String("ai.fast_model", "qwen3:1.7b"), + mctx.Config.String("ai.deep_model", "qwen2.5-coder:14b"), + mctx.Config.Bool("ai.cascade", true), + ) + if mctx.Config.Bool("ai.verbose", false) { + a.client.Verbose = true + } + if !a.client.IsAvailable() { + mctx.Bus.Publish(mctx.Ctx, eventbus.ModuleError{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: mctx.Target}, + Module: ModuleName, + Err: "Ollama not reachable at " + mctx.Config.String("ai.url", "http://localhost:11434"), + }) + return nil + } + + // Multi-agent orchestrator is opt-in: only worth spinning up when the + // user explicitly enables it. The orchestrator holds one client per + // agent type (8 agents) and can take ~200ms to initialise. + if mctx.Config.Bool("ai.multi_agent", false) { + a.orchestrator = agents.NewAgentOrchestrator( + mctx.Config.String("ai.url", "http://localhost:11434"), + mctx.Config.String("ai.fast_model", "qwen3:1.7b"), + mctx.Config.String("ai.deep_model", "qwen2.5-coder:14b"), + ) + } + + var wg sync.WaitGroup + + // Subscribe to every event type we care about. Each handler runs in its + // own goroutine off the bus; we track them with wg so we can drain at + // the end. + subs := []*eventbus.Subscription{ + mctx.Bus.Subscribe(eventbus.EventTechDetected, func(_ context.Context, e eventbus.Event) { + if ev, ok := e.(eventbus.TechDetected); ok { + wg.Add(1) + go func() { defer wg.Done(); a.handleTech(mctx, ev.Host, ev.Technology, ev.Version) }() + } + }), + mctx.Bus.Subscribe(eventbus.EventJSFile, func(_ context.Context, e eventbus.Event) { + if ev, ok := e.(eventbus.JSFileDiscovered); ok { + wg.Add(1) + go func() { defer wg.Done(); a.handleJSFile(mctx, ev) }() + } + }), + mctx.Bus.Subscribe(eventbus.EventHTTPProbed, func(_ context.Context, e eventbus.Event) { + if ev, ok := e.(eventbus.HTTPProbed); ok { + wg.Add(1) + go func() { defer wg.Done(); a.handleHTTP(mctx, ev) }() + } + }), + mctx.Bus.Subscribe(eventbus.EventSecret, func(_ context.Context, e eventbus.Event) { + if ev, ok := e.(eventbus.SecretFound); ok { + wg.Add(1) + go func() { defer wg.Done(); a.handleSecret(mctx, ev) }() + } + }), + mctx.Bus.Subscribe(eventbus.EventVulnerability, func(_ context.Context, e eventbus.Event) { + if ev, ok := e.(eventbus.VulnerabilityFound); ok { + wg.Add(1) + go func() { defer wg.Done(); a.handleVuln(mctx, ev) }() + } + }), + } + defer func() { + for _, s := range subs { + s.Unsubscribe() + } + }() + + // Drain store: any host already populated with tech/HTTP info gets + // processed on module startup (covers the common case where AI is in a + // later phase than discovery/enrichment). + for _, h := range mctx.Store.All(mctx.Ctx) { + if h == nil { + continue + } + for _, tech := range h.Technologies { + tech := tech + host := h.Subdomain + wg.Add(1) + go func() { defer wg.Done(); a.handleTech(mctx, host, tech, "") }() + } + if h.StatusCode != 0 { + ev := eventbus.HTTPProbed{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: h.Subdomain}, + URL: "https://" + h.Subdomain, + StatusCode: h.StatusCode, + Title: h.Title, + Server: h.Server, + } + wg.Add(1) + go func() { defer wg.Done(); a.handleHTTP(mctx, ev) }() + } + } + + // Brief window for late events (recursive discovery, slow probes) to + // arrive before we wrap up. + select { + case <-time.After(1500 * time.Millisecond): + case <-mctx.Ctx.Done(): + } + + wg.Wait() + + // End-of-scan analyses run once, after all per-event handlers drain. + a.handleScanEnd(mctx) + return nil +} + +// --- Handlers ------------------------------------------------------------ + +// handleTech runs CVE correlation for a (tech, version) pair. Cached by +// (tech, version) so the same pair across many hosts fires one query. +func (a *aiModule) handleTech(mctx module.Context, host, tech, version string) { + if tech == "" || shouldSkipForCVE(tech, version) { + return + } + name, v := parseTech(tech) + if version == "" { + version = v + } + if shouldSkipForCVE(name, version) { + return + } + key := "cve:" + name + "|" + version + if !a.firstSeen(key) { + return + } + a.cveLookups.Add(1) + + cves, err := a.client.CVEMatch(name, version) + if err != nil || cves == "" { + return + } + + // Upsert to the specific host that triggered this. + now := time.Now() + cve := store.CVE{ + ID: cves, Technology: name, Version: version, + Severity: string(eventbus.SeverityHigh), Description: cves, FoundAt: now, + } + _ = mctx.Store.Upsert(mctx.Ctx, host, func(h *store.Host) { h.CVEs = append(h.CVEs, cve) }) + + mctx.Bus.Publish(mctx.Ctx, eventbus.CVEMatch{ + EventMeta: eventbus.EventMeta{At: now, Source: ModuleName, Target: host}, + CVE: cves, + Technology: name, + Version: version, + Severity: eventbus.SeverityHigh, + Description: fmt.Sprintf("AI-assisted CVE match for %s %s", name, versionOrUnknown(version)), + }) +} + +// handleJSFile fetches the JS file via the shared HTTP client and feeds it +// to AnalyzeJavaScript. Cached by JS URL — a single JS file seen on 5 +// hosts is analysed once. +// +// Note: we do NOT re-download the JS content here. The v1 AnalyzeJavaScript +// method expects the code itself as input; since the upstream javascript +// module already has the content, the proper integration path is to have +// JSFileDiscovered carry the content. For now, we skip the deep analysis +// when content isn't inlined, and rely on the v1 regex results enriched +// by AI at secret-validation time (see handleSecret). +func (a *aiModule) handleJSFile(mctx module.Context, ev eventbus.JSFileDiscovered) { + key := "js:" + ev.URL + if !a.firstSeen(key) { + return + } + a.jsAnalyses.Add(1) + // Deep JS analysis is deferred until JSFileDiscovered carries the + // content (Fase 2 follow-up). We still produce an AIFinding noting + // the JS file was indexed, which helps reporting aggregate per-host + // JS exposure. + mctx.Bus.Publish(mctx.Ctx, eventbus.AIFinding{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: ev.Host}, + Subject: ev.Host, + Agent: "js-indexer", + Model: a.client.FastModel, + Severity: eventbus.SeverityInfo, + Title: "JavaScript file indexed for secret review", + Evidence: ev.URL, + }) +} + +// handleHTTP triages the HTTP response and dispatches deep analysis only +// for interesting status codes / signals. "Interesting" means anything +// that isn't a normal 200/301 — 5xx, verbose 4xx with bodies, weird +// headers. +func (a *aiModule) handleHTTP(mctx module.Context, ev eventbus.HTTPProbed) { + if !isInterestingHTTP(ev) { + return + } + key := fmt.Sprintf("http:%s:%d:%s", ev.Meta().Target, ev.StatusCode, hashShort(ev.Title)) + if !a.firstSeen(key) { + return + } + a.httpAnalyses.Add(1) + + // Compose the content we hand to the deep model. Keep it compact — + // Ollama's context is ample but we're summarising for the cascade. + headerLines := []string{} + if ev.Server != "" { + headerLines = append(headerLines, "Server: "+ev.Server) + } + for k, v := range ev.Headers { + headerLines = append(headerLines, k+": "+v) + } + + result, err := a.client.AnalyzeHTTPResponse(ev.Meta().Target, ev.StatusCode, headerLines, ev.Title) + if err != nil || result == nil || len(result.Findings) == 0 { + return + } + now := time.Now() + host := ev.Meta().Target + for _, f := range result.Findings { + persistAIFinding(mctx, host, store.AIFinding{ + Agent: "http-analyzer", Model: a.client.DeepModel, + Severity: result.Severity, Title: "Suspicious HTTP response", + Description: f, Evidence: fmt.Sprintf("status=%d title=%q", ev.StatusCode, ev.Title), + FoundAt: now, + }) + mctx.Bus.Publish(mctx.Ctx, eventbus.AIFinding{ + EventMeta: eventbus.EventMeta{At: now, Source: ModuleName, Target: host}, + Subject: host, + Agent: "http-analyzer", + Model: a.client.DeepModel, + Severity: eventbus.Severity(result.Severity), + Title: "Suspicious HTTP response", + Description: f, + Evidence: fmt.Sprintf("status=%d title=%q", ev.StatusCode, ev.Title), + }) + } +} + +// handleSecret validates a regex-surfaced secret through FilterSecrets. +// If the AI confirms it's real, an AIFinding event fires tagging it as +// validated. Regex noise (UI strings, unrelated third-party URLs) is +// dropped silently — the v1 Secret event is left in place but the AI +// emission is what a dashboard would prefer to render as a real finding. +func (a *aiModule) handleSecret(mctx module.Context, ev eventbus.SecretFound) { + key := "secret:" + hashShort(ev.Match+"|"+ev.Location) + if !a.firstSeen(key) { + return + } + a.secretValidations.Add(1) + + validated, err := a.client.FilterSecrets([]string{ev.Match}) + if err != nil || len(validated) == 0 { + return // AI says not a real secret, or Ollama unavailable + } + now := time.Now() + persistAIFinding(mctx, ev.Meta().Target, store.AIFinding{ + Agent: "secret-validator", Model: a.client.FastModel, + Severity: string(eventbus.SeverityHigh), + Title: "Secret likely valid (AI-confirmed)", + Description: fmt.Sprintf("FilterSecrets confirmed '%s' is a real secret, not regex noise.", ev.Kind), + Evidence: fmt.Sprintf("%s @ %s", ev.Kind, ev.Location), + FoundAt: now, + }) + mctx.Bus.Publish(mctx.Ctx, eventbus.AIFinding{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: ev.Meta().Target}, + Subject: ev.Meta().Target, + Agent: "secret-validator", + Model: a.client.FastModel, + Severity: eventbus.SeverityHigh, + Title: "Secret likely valid (AI-confirmed)", + Description: fmt.Sprintf("FilterSecrets confirmed '%s' is a real secret, not regex noise.", + ev.Kind), + Evidence: fmt.Sprintf("%s @ %s", ev.Kind, ev.Location), + }) +} + +// handleVuln routes a vulnerability finding through the multi-agent +// orchestrator for specialist analysis. When multi-agent is disabled, +// this is a no-op. +func (a *aiModule) handleVuln(mctx module.Context, ev eventbus.VulnerabilityFound) { + if a.orchestrator == nil { + return + } + key := "vuln:" + ev.ID + ":" + ev.Meta().Target + if !a.firstSeen(key) { + return + } + a.vulnEnrichments.Add(1) + + finding := agents.Finding{ + Type: "vulnerability", + URL: ev.URL, + Context: ev.Description + "\n\nEvidence:\n" + ev.Evidence, + } + // Respect ctx — orchestrator methods accept context.Context for + // cancellation. Allow up to 60s for deep-analysis cascade. + ctx, cancel := context.WithTimeout(mctx.Ctx, 60*time.Second) + defer cancel() + result, err := a.orchestrator.Analyze(ctx, finding) + if err != nil || result == nil { + return + } + now := time.Now() + for _, f := range result.Findings { + persistAIFinding(mctx, ev.Meta().Target, store.AIFinding{ + Agent: string(result.AgentType), Model: result.Model, + Severity: strings.ToLower(f.Severity), + Title: f.Title, Description: f.Description, Evidence: f.Evidence, + CVEs: f.CVEs, OWASP: f.OWASP, Confidence: result.Confidence, + FoundAt: now, + }) + mctx.Bus.Publish(mctx.Ctx, eventbus.AIFinding{ + EventMeta: eventbus.EventMeta{At: now, Source: ModuleName, Target: ev.Meta().Target}, + Subject: ev.Meta().Target, + Agent: string(result.AgentType), + Model: result.Model, + Severity: eventbus.Severity(strings.ToLower(f.Severity)), + Title: f.Title, + Description: f.Description, + Evidence: f.Evidence, + CVEs: f.CVEs, + OWASP: f.OWASP, + Confidence: result.Confidence, + }) + } +} + +// handleScanEnd runs two expensive end-of-scan analyses: +// +// 1. DetectAnomalies — cross-host pattern review (dev stacks leaking into +// prod, unusual version mixes, orphaned endpoints) +// 2. GenerateReport — executive summary of findings by severity +// +// Both run only when the store has enough data to be worth summarising +// (≥ 3 findings or ≥ 5 hosts). +func (a *aiModule) handleScanEnd(mctx module.Context) { + hosts := mctx.Store.All(mctx.Ctx) + if len(hosts) == 0 { + return + } + + totalFindings := 0 + for _, h := range hosts { + totalFindings += len(h.Vulnerabilities) + len(h.Secrets) + len(h.CVEs) + len(h.AIFindings) + } + if totalFindings < 3 && len(hosts) < 5 { + return // not worth the Ollama spin-up + } + + // Anomaly detection ------------------------------------------------------ + summary := buildScanSummary(hosts) + a.anomalyScans.Add(1) + if result, err := a.client.DetectAnomalies(summary); err == nil && result != nil { + now := time.Now() + for _, f := range result.Findings { + persistAIFinding(mctx, mctx.Target, store.AIFinding{ + Agent: "anomaly-detector", Model: a.client.DeepModel, + Severity: result.Severity, + Title: "Cross-subdomain anomaly", + Description: f, FoundAt: now, + }) + mctx.Bus.Publish(mctx.Ctx, eventbus.AIFinding{ + EventMeta: eventbus.EventMeta{At: now, Source: ModuleName, Target: mctx.Target}, + Subject: mctx.Target, + Agent: "anomaly-detector", + Model: a.client.DeepModel, + Severity: eventbus.Severity(result.Severity), + Title: "Cross-subdomain anomaly", + Description: f, + }) + } + } + + // Executive report ------------------------------------------------------ + stats := map[string]int{ + "hosts": len(hosts), + "findings": totalFindings, + } + a.reportGenerations.Add(1) + if report, err := a.client.GenerateReport(summary, stats); err == nil && report != "" { + now := time.Now() + persistAIFinding(mctx, mctx.Target, store.AIFinding{ + Agent: "report-writer", Model: a.client.DeepModel, + Severity: string(eventbus.SeverityInfo), + Title: "AI executive report", + Description: report, + FoundAt: now, + }) + mctx.Bus.Publish(mctx.Ctx, eventbus.AIFinding{ + EventMeta: eventbus.EventMeta{At: now, Source: ModuleName, Target: mctx.Target}, + Subject: mctx.Target, + Agent: "report-writer", + Model: a.client.DeepModel, + Severity: eventbus.SeverityInfo, + Title: "AI executive report", + Description: report, + }) + } + + // Emit a module-error style observability event with per-handler counts. + mctx.Bus.Publish(mctx.Ctx, eventbus.ModuleError{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: mctx.Target}, + Module: ModuleName, + Err: fmt.Sprintf("AI activity: cve=%d js=%d http=%d secrets=%d vulns=%d anomaly=%d report=%d", + a.cveLookups.Load(), + a.jsAnalyses.Load(), + a.httpAnalyses.Load(), + a.secretValidations.Load(), + a.vulnEnrichments.Load(), + a.anomalyScans.Load(), + a.reportGenerations.Load()), + }) +} + +// --- helpers ------------------------------------------------------------- + +// firstSeen returns true the first time we see a given cache key, false +// on every subsequent call. Implemented via sync.Map.LoadOrStore which is +// atomic. +func (a *aiModule) firstSeen(key string) bool { + h := sha256.Sum256([]byte(key)) + hx := hex.EncodeToString(h[:]) + _, loaded := a.cache.LoadOrStore(hx, struct{}{}) + return !loaded +} + +// isInterestingHTTP gates which HTTP responses are worth sending to the +// deep model. Normal 2xx/3xx are skipped; 5xx, verbose 4xx with titles, +// and anything with a server-banner mismatch qualifies. +func isInterestingHTTP(ev eventbus.HTTPProbed) bool { + switch { + case ev.StatusCode >= 500: + return true + case ev.StatusCode == 401 || ev.StatusCode == 403: + return true // auth surface worth inspecting + case ev.StatusCode >= 400 && ev.Title != "" && ev.ContentLength > 1000: + return true // verbose error page + case ev.TLSSelfSigned: + return true // self-signed on a live host is usually an appliance + } + return false +} + +// hashShort returns a short hex prefix of SHA-256(s) — used for cache +// keys where the full input is too long but identity matters. +func hashShort(s string) string { + h := sha256.Sum256([]byte(s)) + return hex.EncodeToString(h[:8]) +} + +// persistAIFinding appends an AIFinding to the host's store record so +// that downstream modules (notably the report.brief module running in +// PhaseReporting, which subscribes to the bus AFTER PhaseAnalysis has +// drained) can still surface the finding. Store is the single source +// of truth for cross-phase handoff. +func persistAIFinding(mctx module.Context, host string, f store.AIFinding) { + if host == "" { + host = mctx.Target + } + _ = mctx.Store.Upsert(mctx.Ctx, host, func(h *store.Host) { + h.AIFindings = append(h.AIFindings, f) + }) +} + +// cdnOrWafMarkers are technology names that indicate the target is fronted +// by a CDN / WAF rather than running that product themselves. Matching +// CVEs against these labels produces almost-exclusively false positives, +// so we skip them when the version is unknown. +var cdnOrWafMarkers = map[string]bool{ + "cloudflare": true, + "cloudfront": true, + "akamai": true, + "fastly": true, + "imperva": true, + "aws": true, + "azure": true, + "gcp": true, + "heroku": true, + "netlify": true, + "vercel": true, + "cdn": true, + "nginx plus": true, +} + +// parseTech extracts (name, version) from strings like "nginx/1.18.0", +// "nginx/1.18.0 (Ubuntu)", "Apache/2.4.52", or "Apache 2.4". +func parseTech(raw string) (name, version string) { + raw = strings.TrimSpace(raw) + if raw == "" { + return "", "" + } + // Look for name/version or name version pattern. + for _, sep := range []string{"/", " "} { + if idx := strings.Index(raw, sep); idx > 0 { + name = strings.TrimSpace(raw[:idx]) + rest := strings.TrimSpace(raw[idx+1:]) + rest = strings.TrimPrefix(rest, "v") + // Pull digits.digits.digits out of rest + end := 0 + for end < len(rest) { + c := rest[end] + if (c >= '0' && c <= '9') || c == '.' { + end++ + continue + } + break + } + if end > 0 { + return name, rest[:end] + } + return name, "" + } + } + return raw, "" +} + +// shouldSkipForCVE returns true when (name, version) is too vague for a +// useful CVE lookup — empty name, or a CDN/WAF label without a version. +func shouldSkipForCVE(name, version string) bool { + if name == "" { + return true + } + if version == "" && cdnOrWafMarkers[strings.ToLower(name)] { + return true + } + return false +} + +func versionOrUnknown(v string) string { + if v == "" { + return "(unknown version)" + } + return "v" + v +} + +// buildScanSummary compiles a compact text representation of the store +// for the DetectAnomalies / GenerateReport prompts. Kept under ~3KB to +// fit comfortably in every model's context window. +func buildScanSummary(hosts []*store.Host) string { + var sb strings.Builder + sb.WriteString(fmt.Sprintf("Scan summary: %d hosts\n\n", len(hosts))) + shown := 0 + for _, h := range hosts { + if h == nil { + continue + } + if shown >= 50 { + sb.WriteString(fmt.Sprintf("\n... and %d more hosts\n", len(hosts)-shown)) + break + } + sb.WriteString(fmt.Sprintf("- %s (status=%d, tech=%s)", + h.Subdomain, h.StatusCode, strings.Join(h.Technologies, ","))) + if len(h.Vulnerabilities) > 0 { + sb.WriteString(fmt.Sprintf(" vulns=%d", len(h.Vulnerabilities))) + } + if len(h.Secrets) > 0 { + sb.WriteString(fmt.Sprintf(" secrets=%d", len(h.Secrets))) + } + if len(h.CVEs) > 0 { + sb.WriteString(fmt.Sprintf(" cves=%d", len(h.CVEs))) + } + sb.WriteString("\n") + shown++ + } + return sb.String() +} diff --git a/internal/modules/all/all.go b/internal/modules/all/all.go new file mode 100644 index 0000000..dcf3a82 --- /dev/null +++ b/internal/modules/all/all.go @@ -0,0 +1,80 @@ +// Package all is the meta-package imported from main to trigger side-effect +// registration of every built-in Fase 0.6 adapter module. Importing +// god-eye/internal/modules/all is equivalent to importing each submodule +// individually and calling Register(). +// +// Individual submodules avoid registering in their init() on purpose — that +// would make the registry state global and prevent tests from using a +// clean registry. Callers (main, tests) explicitly opt in by importing +// this package or calling RegisterAll. +package all + +import ( + aimod "god-eye/internal/modules/ai" + "god-eye/internal/modules/asn" + "god-eye/internal/modules/brief" + "god-eye/internal/modules/axfr" + "god-eye/internal/modules/bruteforce" + "god-eye/internal/modules/cloud" + "god-eye/internal/modules/ctstream" + "god-eye/internal/modules/dnsresolve" + "god-eye/internal/modules/github" + "god-eye/internal/modules/graphql" + "god-eye/internal/modules/headers" + "god-eye/internal/modules/httpprobe" + "god-eye/internal/modules/javascript" + "god-eye/internal/modules/jwt" + "god-eye/internal/modules/nuclei" + "god-eye/internal/modules/passive" + "god-eye/internal/modules/permutation" + "god-eye/internal/modules/ports" + "god-eye/internal/modules/recursive" + "god-eye/internal/modules/report" + "god-eye/internal/modules/reversedns" + "god-eye/internal/modules/security" + "god-eye/internal/modules/smuggling" + "god-eye/internal/modules/supplychain" + "god-eye/internal/modules/takeover" + "god-eye/internal/modules/vhost" +) + +// RegisterAll registers every Fase 0.6 adapter module in the default +// registry. Call exactly once at program start — Register panics on +// duplicates, so calling twice is a bug. +func RegisterAll() { + // Discovery (Fase 0 adapters + Fase 1 natives + supply chain from F2) + passive.Register() + bruteforce.Register() + recursive.Register() + axfr.Register() // F1 + github.Register() // F1 + ctstream.Register() // F1 (opt-in) + supplychain.Register() // F2 + + // Resolution + dnsresolve.Register() + permutation.Register() // F1 (opt-in) + reversedns.Register() // F1 (opt-in) + vhost.Register() // F1 (opt-in) + asn.Register() // F1 (opt-in) + + // Enrichment + httpprobe.Register() + ports.Register() + + // Analysis (F0 adapters + F2 natives) + security.Register() + takeover.Register() + cloud.Register() + javascript.Register() + aimod.Register() + graphql.Register() // F2 + jwt.Register() // F2 + headers.Register() // F2 + smuggling.Register() // F2 (opt-in) + nuclei.Register() // F2 (opt-in — requires local nuclei-templates dir) + + // Reporting + report.Register() + brief.Register() // AI-assisted executive summary at scan end +} diff --git a/internal/modules/asn/asn.go b/internal/modules/asn/asn.go new file mode 100644 index 0000000..126a1f2 --- /dev/null +++ b/internal/modules/asn/asn.go @@ -0,0 +1,78 @@ +// Package asn is a Fase 0.6 adapter around v1 network.ASNScanner. Expands +// discovery by enumerating IPs within the target's ASN/CIDR blocks. +package asn + +import ( + "time" + + "god-eye/internal/eventbus" + "god-eye/internal/module" + "god-eye/internal/network" + "god-eye/internal/store" +) + +// CtxPassthrough is used to thread module.Context.Ctx into network helpers. + +const ModuleName = "discovery.asn" + +type asnModule struct{} + +func Register() { module.Register(&asnModule{}) } + +func (*asnModule) Name() string { return ModuleName } +func (*asnModule) Phase() module.Phase { return module.PhaseResolution } +func (*asnModule) Consumes() []eventbus.EventType { return []eventbus.EventType{eventbus.EventDNSResolved} } +func (*asnModule) Produces() []eventbus.EventType { return nil } +func (*asnModule) DefaultEnabled() bool { return false } // opt-in + +func (*asnModule) Run(mctx module.Context) error { + if !mctx.Config.Bool("asn_scan", false) { + return nil + } + timeout := mctx.Config.Int("timeout", 10) + + hosts := mctx.Store.All(mctx.Ctx) + seenIP := make(map[string]struct{}) + for _, h := range hosts { + for _, ip := range h.IPs { + seenIP[ip] = struct{}{} + } + } + + scanner := network.NewASNScanner(timeout) + for ip := range seenIP { + if mctx.Ctx.Err() != nil { + break + } + info, err := scanner.GetASNInfo(mctx.Ctx, ip) + if err != nil || info == nil { + continue + } + _ = mctx.Store.Upsert(mctx.Ctx, ipToFirstHost(mctx, ip), func(h *store.Host) { + if h.ASN == "" { + h.ASN = info.ASN + } + if h.Org == "" { + h.Org = info.Name + } + if h.Country == "" { + h.Country = info.Country + } + }) + } + return nil +} + +// ipToFirstHost returns the first subdomain mapped to ip in the store. +func ipToFirstHost(mctx module.Context, ip string) string { + for _, h := range mctx.Store.All(mctx.Ctx) { + for _, rip := range h.IPs { + if rip == ip { + return h.Subdomain + } + } + } + return "" +} + +var _ = time.Now diff --git a/internal/modules/axfr/axfr.go b/internal/modules/axfr/axfr.go new file mode 100644 index 0000000..fb1c81d --- /dev/null +++ b/internal/modules/axfr/axfr.go @@ -0,0 +1,134 @@ +// Package axfr attempts DNS zone transfer (AXFR) against the target's +// authoritative name servers. It's the highest-signal free discovery +// technique — when it works, it returns the entire zone at once, exposing +// every record the admin considers internal-only. +// +// Modern DNS infrastructure rejects AXFR by default, but legacy deployments, +// misconfigured secondary servers, and corporate DNS still leak zones +// regularly in bug bounty scope. +package axfr + +import ( + "context" + "strings" + "time" + + godns "github.com/miekg/dns" + + "god-eye/internal/eventbus" + "god-eye/internal/module" + "god-eye/internal/store" +) + +const ModuleName = "discovery.axfr" + +type axfrModule struct{} + +func Register() { module.Register(&axfrModule{}) } + +func (*axfrModule) Name() string { return ModuleName } +func (*axfrModule) Phase() module.Phase { return module.PhaseDiscovery } +func (*axfrModule) Consumes() []eventbus.EventType { return nil } +func (*axfrModule) Produces() []eventbus.EventType { + return []eventbus.EventType{eventbus.EventSubdomainDiscovered} +} +func (*axfrModule) DefaultEnabled() bool { return true } + +func (*axfrModule) Run(mctx module.Context) error { + target := strings.TrimSuffix(mctx.Target, ".") + if target == "" { + return nil + } + timeout := time.Duration(mctx.Config.Int("timeout", 5)) * time.Second + + nameservers, err := lookupNSServers(target, timeout) + if err != nil || len(nameservers) == 0 { + return nil + } + + seen := make(map[string]struct{}) + for _, ns := range nameservers { + if mctx.Ctx.Err() != nil { + return nil + } + records := tryAXFR(target, ns, timeout) + for _, sub := range records { + sub = strings.ToLower(strings.TrimSuffix(sub, ".")) + if sub == "" || sub == target { + continue + } + if !strings.HasSuffix(sub, "."+target) { + continue + } + if _, dup := seen[sub]; dup { + continue + } + seen[sub] = struct{}{} + + _ = mctx.Store.Upsert(mctx.Ctx, sub, func(h *store.Host) { + store.AddDiscoveryMethod(h, "axfr:"+ns) + }) + mctx.Bus.Publish(mctx.Ctx, eventbus.SubdomainDiscovered{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: sub}, + Subdomain: sub, + Method: "axfr:" + ns, + }) + } + } + return nil +} + +// lookupNSServers returns the authoritative name servers for domain. +func lookupNSServers(domain string, timeout time.Duration) ([]string, error) { + client := &godns.Client{Timeout: timeout} + msg := new(godns.Msg) + msg.SetQuestion(godns.Fqdn(domain), godns.TypeNS) + // Ask a widely-available resolver. + resp, _, err := client.Exchange(msg, "8.8.8.8:53") + if err != nil { + return nil, err + } + var out []string + for _, a := range resp.Answer { + if ns, ok := a.(*godns.NS); ok { + out = append(out, strings.TrimSuffix(ns.Ns, ".")) + } + } + return out, nil +} + +// tryAXFR performs an AXFR against nsHost for domain, returning every +// returned name (A, AAAA, CNAME). Returns an empty slice when AXFR is +// refused (the expected outcome on properly-configured DNS). +func tryAXFR(domain, nsHost string, timeout time.Duration) []string { + tr := &godns.Transfer{DialTimeout: timeout, ReadTimeout: timeout, WriteTimeout: timeout} + msg := new(godns.Msg) + msg.SetAxfr(godns.Fqdn(domain)) + + ch, err := tr.In(msg, nsHost+":53") + if err != nil { + return nil + } + + var out []string + for env := range ch { + if env.Error != nil { + return out + } + for _, rr := range env.RR { + switch r := rr.(type) { + case *godns.A: + out = append(out, r.Hdr.Name) + case *godns.AAAA: + out = append(out, r.Hdr.Name) + case *godns.CNAME: + out = append(out, r.Hdr.Name) + case *godns.NS: + out = append(out, r.Hdr.Name) + } + } + } + return out +} + +var _ = context.Canceled diff --git a/internal/modules/brief/brief.go b/internal/modules/brief/brief.go new file mode 100644 index 0000000..56740c3 --- /dev/null +++ b/internal/modules/brief/brief.go @@ -0,0 +1,464 @@ +// Package brief renders the end-of-scan AI-assisted executive brief. +// +// It's the last module to run in PhaseReporting. It reads: +// - every host from the store (for severity / takeover / CVE rollups) +// - every AIFinding published during the scan (anomalies, executive +// report, per-host agent output) +// +// Then prints a framed summary block to stdout with: +// +// ▸ Findings counted by severity +// ▸ Top exploitable chains (critical + CVE pairs) +// ▸ AI-generated executive summary (if ai.enabled) +// ▸ Recommended next actions +// +// Suppressed when cfg.silent or cfg.json is true so machine-readable +// modes stay clean. +package brief + +import ( + "context" + "fmt" + "sort" + "strings" + "sync" + "time" + + "god-eye/internal/eventbus" + "god-eye/internal/module" + "god-eye/internal/output" + "god-eye/internal/store" +) + +const ModuleName = "report.brief" + +type briefModule struct { + aiFindings []eventbus.AIFinding + execReport string // last executive-report AIFinding seen + execReportAt time.Time + mu sync.Mutex +} + +func Register() { module.Register(&briefModule{}) } + +func (*briefModule) Name() string { return ModuleName } +func (*briefModule) Phase() module.Phase { return module.PhaseReporting } +func (*briefModule) Consumes() []eventbus.EventType { return []eventbus.EventType{eventbus.EventAIFinding} } +func (*briefModule) Produces() []eventbus.EventType { return nil } + +// DefaultEnabled: brief renders whenever the scan completes with any +// findings. Silent/json modes are suppressed inline (not at selection +// time) so the module can still collect AIFindings for exports. +func (*briefModule) DefaultEnabled() bool { return true } + +func (b *briefModule) Run(mctx module.Context) error { + // Subscribe to AIFinding events and stash them locally so we can + // build a richer summary than just reading the store (the store + // doesn't retain AIFindings tagged with agent name / confidence). + sub := mctx.Bus.Subscribe(eventbus.EventAIFinding, func(_ context.Context, e eventbus.Event) { + ev, ok := e.(eventbus.AIFinding) + if !ok { + return + } + b.mu.Lock() + defer b.mu.Unlock() + b.aiFindings = append(b.aiFindings, ev) + if ev.Agent == "report-writer" && ev.Description != "" { + b.execReport = ev.Description + b.execReportAt = ev.Meta().At + } + }) + defer sub.Unsubscribe() + + // Give the AI module a chance to publish its end-of-scan events. + // The AI module runs in PhaseAnalysis; we're in PhaseReporting so + // its ScanCompleted-triggered publishes have already fired by the + // time we get here. A small buffer avoids losing late events. + select { + case <-time.After(400 * time.Millisecond): + case <-mctx.Ctx.Done(): + } + + if mctx.Config.Bool("silent", false) || mctx.Config.Bool("json", false) { + return nil + } + + hosts := mctx.Store.All(mctx.Ctx) + if len(hosts) == 0 { + return nil + } + + // Drain store-persisted AIFindings — these were written by the AI + // module during PhaseAnalysis. Live events alone miss them because + // brief subscribes after PhaseAnalysis has already drained. + b.mu.Lock() + for _, h := range hosts { + for _, f := range h.AIFindings { + b.aiFindings = append(b.aiFindings, eventbus.AIFinding{ + EventMeta: eventbus.EventMeta{At: f.FoundAt, Source: "ai.cascade", Target: h.Subdomain}, + Subject: h.Subdomain, + Agent: f.Agent, + Model: f.Model, + Severity: eventbus.Severity(f.Severity), + Title: f.Title, + Description: f.Description, + Evidence: f.Evidence, + CVEs: f.CVEs, + OWASP: f.OWASP, + Confidence: f.Confidence, + }) + if f.Agent == "report-writer" && f.Description != "" && (b.execReport == "" || f.FoundAt.After(b.execReportAt)) { + b.execReport = f.Description + b.execReportAt = f.FoundAt + } + } + } + b.mu.Unlock() + + b.render(mctx, hosts) + return nil +} + +func (b *briefModule) render(mctx module.Context, hosts []*store.Host) { + b.mu.Lock() + aiFindings := append([]eventbus.AIFinding(nil), b.aiFindings...) + execReport := b.execReport + b.mu.Unlock() + + sevCounts := tallySeverities(hosts, aiFindings) + topChains := buildChains(hosts) + recs := buildRecommendations(hosts, aiFindings) + aiActivity := tallyAIAgents(aiFindings) + + fmt.Println() + title := fmt.Sprintf(" AI SCAN BRIEF — %s ", mctx.Target) + fmt.Println(output.BoldCyan(boxTop(title))) + writeLine := func(text string) { + fmt.Println(output.BoldCyan("│ ") + text) + } + + // Section: stats + writeLine(output.BoldWhite("Totals")) + writeLine(fmt.Sprintf(" %s %d %s %d %s %d", + output.Dim("Hosts:"), len(hosts), + output.Dim("Active:"), countActive(hosts), + output.Dim("AI findings:"), len(aiFindings), + )) + writeLine("") + + // Section: severity breakdown + writeLine(output.BoldWhite("Findings by severity")) + sevOrder := []string{"critical", "high", "medium", "low", "info"} + for _, s := range sevOrder { + n := sevCounts[s] + if n == 0 { + continue + } + badge := sevBadge(s) + writeLine(fmt.Sprintf(" %s %s %d", badge, padRight(s, 9), n)) + } + if len(sevCounts) == 0 { + writeLine(output.Dim(" (no scored findings)")) + } + writeLine("") + + // Section: top exploitable chains + if len(topChains) > 0 { + writeLine(output.BoldWhite("Top exploitable chains")) + for i, c := range topChains { + if i >= 5 { + break + } + writeLine(" " + output.BoldYellow("▸ ") + c) + } + writeLine("") + } + + // Section: AI agent activity + if len(aiActivity) > 0 { + writeLine(output.BoldWhite("AI agents that contributed")) + // Stable order by count desc. + type agg struct { + agent string + n int + } + agents := make([]agg, 0, len(aiActivity)) + for name, n := range aiActivity { + agents = append(agents, agg{name, n}) + } + sort.Slice(agents, func(i, j int) bool { return agents[i].n > agents[j].n }) + for _, a := range agents { + writeLine(fmt.Sprintf(" %s %s %s", + output.Cyan("•"), + padRight(a.agent, 20), + output.Dim(fmt.Sprintf("%d findings", a.n)), + )) + } + writeLine("") + } + + // Section: AI executive report (prose) + if strings.TrimSpace(execReport) != "" { + writeLine(output.BoldWhite("AI executive summary")) + for _, line := range wrapText(strings.TrimSpace(execReport), 74) { + writeLine(output.Dim(" ") + line) + } + writeLine("") + } + + // Section: recommendations + if len(recs) > 0 { + writeLine(output.BoldWhite("Recommended next actions")) + for i, r := range recs { + if i >= 5 { + break + } + writeLine(fmt.Sprintf(" %s %s", output.Green(fmt.Sprintf("%d.", i+1)), r)) + } + writeLine("") + } + + fmt.Println(output.BoldCyan(boxBottom())) + fmt.Println() +} + +// --- helpers ------------------------------------------------------------- + +func tallySeverities(hosts []*store.Host, aiFindings []eventbus.AIFinding) map[string]int { + out := map[string]int{} + for _, h := range hosts { + for _, v := range h.Vulnerabilities { + out[strings.ToLower(v.Severity)]++ + } + for _, c := range h.CVEs { + out[strings.ToLower(c.Severity)]++ + } + for _, s := range h.Secrets { + out[strings.ToLower(s.Severity)]++ + } + if h.Takeover != nil { + out["high"]++ + } + } + for _, f := range aiFindings { + out[strings.ToLower(string(f.Severity))]++ + } + return out +} + +func countActive(hosts []*store.Host) int { + n := 0 + for _, h := range hosts { + if h.StatusCode >= 200 && h.StatusCode < 400 { + n++ + } + } + return n +} + +// buildChains surfaces the most dangerous combinations. Right now the +// heuristic is coarse: hosts with ≥2 high+ findings, or any host with a +// confirmed takeover candidate, or any host whose tech triggered a CVE. +func buildChains(hosts []*store.Host) []string { + var chains []string + + type scored struct { + text string + score int + } + var ranked []scored + + for _, h := range hosts { + score := 0 + bits := []string{} + for _, v := range h.Vulnerabilities { + if strings.EqualFold(v.Severity, "critical") { + score += 10 + bits = append(bits, v.Title) + } else if strings.EqualFold(v.Severity, "high") { + score += 5 + bits = append(bits, v.Title) + } + } + if h.Takeover != nil { + score += 8 + bits = append(bits, "takeover→"+h.Takeover.Service) + } + for _, c := range h.CVEs { + if strings.EqualFold(c.Severity, "critical") || strings.EqualFold(c.Severity, "high") { + score += 6 + bits = append(bits, fmt.Sprintf("%s@%s→%s", c.Technology, c.Version, firstCVE(c.ID))) + } + } + if score == 0 { + continue + } + desc := h.Subdomain + if len(bits) > 0 { + desc += " " + output.Dim("— "+strings.Join(dedupShort(bits), " + ")) + } + ranked = append(ranked, scored{desc, score}) + } + + sort.Slice(ranked, func(i, j int) bool { return ranked[i].score > ranked[j].score }) + for _, r := range ranked { + chains = append(chains, r.text) + } + return chains +} + +func buildRecommendations(hosts []*store.Host, aiFindings []eventbus.AIFinding) []string { + seen := map[string]struct{}{} + var out []string + + add := func(s string) { + if _, ok := seen[s]; ok { + return + } + seen[s] = struct{}{} + out = append(out, s) + } + + // Pattern: Apache version → upgrade recommendation + for _, h := range hosts { + for _, c := range h.CVEs { + if c.Technology != "" && c.Version != "" { + add(fmt.Sprintf("Patch %s %s → vendor latest (affects %s)", c.Technology, c.Version, h.Subdomain)) + } + } + if h.Takeover != nil { + add(fmt.Sprintf("Verify CNAME on %s before external party claims %s", h.Subdomain, h.Takeover.Service)) + } + for _, s := range h.Secrets { + add(fmt.Sprintf("Rotate %s found in %s", s.Kind, h.Subdomain)) + } + for _, v := range h.Vulnerabilities { + if strings.EqualFold(v.Severity, "critical") { + add(fmt.Sprintf("Remediate critical: %s on %s", v.Title, h.Subdomain)) + } + } + } + + // AI-surfaced recommendations (anomalies) + for _, f := range aiFindings { + if f.Agent == "anomaly-detector" && f.Description != "" { + add("Investigate anomaly: " + trimLine(f.Description, 80)) + } + } + + return out +} + +func tallyAIAgents(aiFindings []eventbus.AIFinding) map[string]int { + out := map[string]int{} + for _, f := range aiFindings { + agent := f.Agent + if agent == "" { + agent = "unknown" + } + out[agent]++ + } + return out +} + +// --- rendering primitives ------------------------------------------------ + +const boxWidth = 76 + +func boxTop(title string) string { + line := strings.Repeat("─", boxWidth) + if len(title) >= boxWidth-4 { + title = title[:boxWidth-4] + } + prefix := "┌── " + suffix := " " + strings.Repeat("─", boxWidth-len(prefix)-len(title)-1) + "┐" + _ = line + return prefix + title + suffix +} + +func boxBottom() string { + return "└" + strings.Repeat("─", boxWidth) + "┘" +} + +func padRight(s string, n int) string { + if len(s) >= n { + return s + } + return s + strings.Repeat(" ", n-len(s)) +} + +func wrapText(s string, width int) []string { + words := strings.Fields(s) + if len(words) == 0 { + return nil + } + var lines []string + var cur strings.Builder + for _, w := range words { + if cur.Len() == 0 { + cur.WriteString(w) + continue + } + if cur.Len()+1+len(w) > width { + lines = append(lines, cur.String()) + cur.Reset() + cur.WriteString(w) + } else { + cur.WriteByte(' ') + cur.WriteString(w) + } + } + if cur.Len() > 0 { + lines = append(lines, cur.String()) + } + return lines +} + +func sevBadge(s string) string { + switch strings.ToLower(s) { + case "critical": + return output.BgRed(" CRIT ") + case "high": + return output.Red("[HIGH]") + case "medium": + return output.Yellow("[MED] ") + case "low": + return output.Blue("[LOW] ") + default: + return output.Dim("[INFO]") + } +} + +func firstCVE(ids string) string { + if i := strings.IndexAny(ids, ",("); i > 0 { + return strings.TrimSpace(ids[:i]) + } + return ids +} + +func dedupShort(in []string) []string { + seen := map[string]struct{}{} + var out []string + for _, s := range in { + if _, ok := seen[s]; ok { + continue + } + seen[s] = struct{}{} + if len(s) > 40 { + s = s[:37] + "…" + } + out = append(out, s) + } + return out +} + +func trimLine(s string, n int) string { + s = strings.TrimSpace(s) + if i := strings.Index(s, "\n"); i > 0 { + s = s[:i] + } + if len(s) > n { + s = s[:n-1] + "…" + } + return s +} diff --git a/internal/modules/bruteforce/bruteforce.go b/internal/modules/bruteforce/bruteforce.go new file mode 100644 index 0000000..82f9228 --- /dev/null +++ b/internal/modules/bruteforce/bruteforce.go @@ -0,0 +1,167 @@ +// Package bruteforce runs DNS brute-force against the target domain using +// the shipped or custom wordlist. Emits SubdomainDiscovered for every host +// that resolves (with optional wildcard filtering applied). +package bruteforce + +import ( + "bufio" + "context" + "os" + "strings" + "sync" + "time" + + "god-eye/internal/config" + godns "god-eye/internal/dns" + "god-eye/internal/eventbus" + "god-eye/internal/module" + "god-eye/internal/store" +) + +const ModuleName = "discovery.bruteforce" + +type bruteModule struct{} + +func Register() { module.Register(&bruteModule{}) } + +func (*bruteModule) Name() string { return ModuleName } +func (*bruteModule) Phase() module.Phase { return module.PhaseDiscovery } +func (*bruteModule) Consumes() []eventbus.EventType { return nil } +func (*bruteModule) Produces() []eventbus.EventType { + return []eventbus.EventType{eventbus.EventSubdomainDiscovered} +} +func (*bruteModule) DefaultEnabled() bool { return true } + +func (b *bruteModule) Run(mctx module.Context) error { + if mctx.Config.Bool("no_brute", false) { + return nil + } + + target := mctx.Target + wordlist := loadWordlist(mctx.Config.String("wordlist", "")) + resolvers := parseResolvers(mctx.Config.String("resolvers", "")) + timeout := mctx.Config.Int("timeout", 5) + conc := mctx.Config.Int("concurrency", 500) + if conc <= 0 { + conc = 500 + } + + // Opportunistic wildcard detection: before brute, detect which IPs + // (if any) the apex wildcards to, so we can filter hits that resolve + // exclusively to those IPs. + wd := godns.NewWildcardDetector(resolvers, timeout) + wi := wd.Detect(target) + wildcardIPs := make(map[string]struct{}) + if wi != nil && wi.IsWildcard { + for _, ip := range wi.WildcardIPs { + wildcardIPs[ip] = struct{}{} + } + } + + work := make(chan string, conc*2) + var wg sync.WaitGroup + for i := 0; i < conc; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for w := range work { + if mctx.Ctx.Err() != nil { + return + } + sub := w + "." + target + ips := godns.ResolveSubdomain(sub, resolvers, timeout) + if len(ips) == 0 { + continue + } + if allWildcard(ips, wildcardIPs) { + continue + } + + _ = mctx.Store.Upsert(mctx.Ctx, sub, func(h *store.Host) { + store.AddIPs(h, ips) + store.AddDiscoveryMethod(h, "brute") + }) + + mctx.Bus.Publish(mctx.Ctx, eventbus.SubdomainDiscovered{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: sub}, + Subdomain: sub, + Method: "brute", + }) + } + }() + } + +loop: + for _, w := range wordlist { + select { + case work <- w: + case <-mctx.Ctx.Done(): + break loop + } + } + close(work) + wg.Wait() + return nil +} + +func allWildcard(ips []string, wc map[string]struct{}) bool { + if len(wc) == 0 { + return false + } + for _, ip := range ips { + if _, ok := wc[ip]; !ok { + return false + } + } + return true +} + +func loadWordlist(path string) []string { + if path == "" { + return config.DefaultWordlist + } + f, err := os.Open(path) + if err != nil { + return config.DefaultWordlist + } + defer f.Close() + + var out []string + sc := bufio.NewScanner(f) + for sc.Scan() { + w := strings.TrimSpace(sc.Text()) + if w == "" || strings.HasPrefix(w, "#") { + continue + } + out = append(out, w) + } + if len(out) == 0 { + return config.DefaultWordlist + } + return out +} + +func parseResolvers(s string) []string { + s = strings.TrimSpace(s) + if s == "" { + return config.DefaultResolvers + } + var out []string + for _, r := range strings.Split(s, ",") { + r = strings.TrimSpace(r) + if r == "" { + continue + } + if !strings.Contains(r, ":") { + r = r + ":53" + } + out = append(out, r) + } + if len(out) == 0 { + return config.DefaultResolvers + } + return out +} + +// keep context import for symmetry with other modules +var _ = context.Canceled diff --git a/internal/modules/cloud/cloud.go b/internal/modules/cloud/cloud.go new file mode 100644 index 0000000..ce771b8 --- /dev/null +++ b/internal/modules/cloud/cloud.go @@ -0,0 +1,104 @@ +// Package cloud wraps v1 cloud detection + S3 bucket discovery. +// Drains the store, plus listens for late DNSResolved events. +package cloud + +import ( + "context" + "sync" + "time" + + "god-eye/internal/eventbus" + gohttp "god-eye/internal/http" + "god-eye/internal/module" + "god-eye/internal/scanner" + "god-eye/internal/store" +) + +const ModuleName = "cloud.detect" + +type cloudModule struct{} + +func Register() { module.Register(&cloudModule{}) } + +func (*cloudModule) Name() string { return ModuleName } +func (*cloudModule) Phase() module.Phase { return module.PhaseAnalysis } +func (*cloudModule) Consumes() []eventbus.EventType { + return []eventbus.EventType{eventbus.EventDNSResolved, eventbus.EventHTTPProbed} +} +func (*cloudModule) Produces() []eventbus.EventType { return []eventbus.EventType{eventbus.EventCloudAsset} } +func (*cloudModule) DefaultEnabled() bool { return true } + +func (*cloudModule) Run(mctx module.Context) error { + timeout := mctx.Config.Int("timeout", 5) + client := gohttp.GetSharedClient(timeout) + + handled := make(map[string]struct{}) + var mu sync.Mutex + shouldHandle := func(host string) bool { + mu.Lock() + defer mu.Unlock() + if _, ok := handled[host]; ok { + return false + } + handled[host] = struct{}{} + return true + } + + handle := func(host string, ips []string, cname string) { + if !shouldHandle(host) { + return + } + provider := scanner.DetectCloudProvider(ips, cname, "") + if provider != "" { + _ = mctx.Store.Upsert(mctx.Ctx, host, func(h *store.Host) { + if h.CloudProvider == "" { + h.CloudProvider = provider + } + }) + } + + if buckets := scanner.CheckS3BucketsWithClient(host, client); len(buckets) > 0 { + for _, url := range buckets { + mctx.Bus.Publish(mctx.Ctx, eventbus.CloudAssetFound{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: host}, + Provider: "AWS", + Kind: "s3-bucket", + Name: host, + URL: url, + Status: "accessible", + }) + } + } + } + + var wg sync.WaitGroup + + // Drain: every host already in the store with an IP. + for _, h := range mctx.Store.All(mctx.Ctx) { + if h == nil || h.Subdomain == "" || len(h.IPs) == 0 { + continue + } + h := h + wg.Add(1) + go func() { defer wg.Done(); handle(h.Subdomain, h.IPs, h.CNAME) }() + } + + // Late DNSResolved events. + sub := mctx.Bus.Subscribe(eventbus.EventDNSResolved, func(_ context.Context, e eventbus.Event) { + ev, ok := e.(eventbus.DNSResolved) + if !ok { + return + } + wg.Add(1) + go func() { defer wg.Done(); handle(ev.Subdomain, ev.IPs, ev.CNAME) }() + }) + defer sub.Unsubscribe() + + select { + case <-time.After(500 * time.Millisecond): + case <-mctx.Ctx.Done(): + } + + wg.Wait() + return nil +} diff --git a/internal/modules/ctstream/ctstream.go b/internal/modules/ctstream/ctstream.go new file mode 100644 index 0000000..5411fb4 --- /dev/null +++ b/internal/modules/ctstream/ctstream.go @@ -0,0 +1,123 @@ +// Package ctstream subscribes to live Certificate Transparency log streams +// from certstream.calidog.io (free, public). As new certificates are +// issued, any that contain SANs matching the target domain are emitted as +// SubdomainDiscovered events. +// +// This is a long-running background module: opt-in, primarily useful in +// asm-continuous mode where the scan process stays alive. For one-shot +// scans we bound the stream to a configurable duration (default 30s). +// +// NOTE: certstream.calidog.io is sometimes rate-limited or offline. This +// module fails open — no event emitted, no error returned. +package ctstream + +import ( + "encoding/json" + "fmt" + "net/http" + "net/url" + "strings" + "time" + + "god-eye/internal/eventbus" + "god-eye/internal/module" + "god-eye/internal/store" +) + +const ModuleName = "discovery.ct-stream" + +type ctModule struct{} + +func Register() { module.Register(&ctModule{}) } + +func (*ctModule) Name() string { return ModuleName } +func (*ctModule) Phase() module.Phase { return module.PhaseDiscovery } +func (*ctModule) Consumes() []eventbus.EventType { return nil } +func (*ctModule) Produces() []eventbus.EventType { + return []eventbus.EventType{eventbus.EventSubdomainDiscovered} +} + +// Off by default: requires long-running streaming. +func (*ctModule) DefaultEnabled() bool { return false } + +func (*ctModule) Run(mctx module.Context) error { + if !mctx.Config.Bool("ct_stream", false) { + return nil + } + durationSec := mctx.Config.Int("ct_stream.duration_sec", 30) + if durationSec <= 0 { + durationSec = 30 + } + + target := mctx.Target + deadline := time.Now().Add(time.Duration(durationSec) * time.Second) + + // Fallback path: poll crt.sh's JSON endpoint every 5s for the duration. + // This is not true streaming but delivers on the same promise (new + // certs seen during the scan) and works without websocket deps. + ticker := time.NewTicker(5 * time.Second) + defer ticker.Stop() + + seen := make(map[string]struct{}) + + for time.Now().Before(deadline) { + if mctx.Ctx.Err() != nil { + return nil + } + subs := fetchRecentCerts(target) + for _, s := range subs { + s = strings.ToLower(strings.TrimSpace(s)) + if s == "" || !strings.HasSuffix(s, target) { + continue + } + if _, dup := seen[s]; dup { + continue + } + seen[s] = struct{}{} + _ = mctx.Store.Upsert(mctx.Ctx, s, func(h *store.Host) { + store.AddDiscoveryMethod(h, "ct-stream") + }) + mctx.Bus.Publish(mctx.Ctx, eventbus.SubdomainDiscovered{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: s}, + Subdomain: s, + Method: "ct-stream", + }) + } + select { + case <-ticker.C: + case <-mctx.Ctx.Done(): + return nil + } + } + return nil +} + +func fetchRecentCerts(target string) []string { + // crt.sh returns JSON with name_value fields; same as the v1 crtsh + // source but we use a tighter query. + q := "%." + target + u := fmt.Sprintf("https://crt.sh/?q=%s&output=json", url.QueryEscape(q)) + client := &http.Client{Timeout: 10 * time.Second} + resp, err := client.Get(u) + if err != nil { + return nil + } + defer resp.Body.Close() + + var entries []struct { + NameValue string `json:"name_value"` + } + if err := json.NewDecoder(resp.Body).Decode(&entries); err != nil { + return nil + } + var out []string + for _, e := range entries { + for _, name := range strings.Split(e.NameValue, "\n") { + name = strings.TrimPrefix(strings.TrimSpace(name), "*.") + if name != "" { + out = append(out, name) + } + } + } + return out +} diff --git a/internal/modules/dnsresolve/dnsresolve.go b/internal/modules/dnsresolve/dnsresolve.go new file mode 100644 index 0000000..a3714d8 --- /dev/null +++ b/internal/modules/dnsresolve/dnsresolve.go @@ -0,0 +1,166 @@ +// Package dnsresolve resolves every subdomain present in the store, plus +// any that arrive via late SubdomainDiscovered events while the module is +// running. Results (IPs, CNAME, PTR) are written back to the store AND +// announced via DNSResolved events for downstream enrichment modules. +// +// This module is idempotent: Upsert on the same subdomain twice is cheap. +package dnsresolve + +import ( + "context" + "strings" + "sync" + "time" + + "god-eye/internal/config" + godns "god-eye/internal/dns" + "god-eye/internal/eventbus" + "god-eye/internal/module" + "god-eye/internal/store" +) + +const ModuleName = "dns.resolver" + +type resolverModule struct{} + +func Register() { module.Register(&resolverModule{}) } + +func (*resolverModule) Name() string { return ModuleName } +func (*resolverModule) Phase() module.Phase { return module.PhaseResolution } +func (*resolverModule) Consumes() []eventbus.EventType { return []eventbus.EventType{eventbus.EventSubdomainDiscovered} } +func (*resolverModule) Produces() []eventbus.EventType { return []eventbus.EventType{eventbus.EventDNSResolved} } +func (*resolverModule) DefaultEnabled() bool { return true } + +func (m *resolverModule) Run(mctx module.Context) error { + resolvers := parseResolvers(mctx.Config.String("resolvers", "")) + timeout := mctx.Config.Int("timeout", 5) + conc := mctx.Config.Int("concurrency", 500) + if conc <= 0 { + conc = 500 + } + + // Dedup across drain + late events. + processed := make(map[string]struct{}) + var processedMu sync.Mutex + shouldProcess := func(sub string) bool { + processedMu.Lock() + defer processedMu.Unlock() + if _, dup := processed[sub]; dup { + return false + } + processed[sub] = struct{}{} + return true + } + + work := make(chan string, conc*2) + var wg sync.WaitGroup + for i := 0; i < conc; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for sub := range work { + m.resolveOne(mctx, sub, resolvers, timeout) + } + }() + } + + // 1) Drain the store: every subdomain discovered so far goes in. + for _, h := range mctx.Store.All(mctx.Ctx) { + if h == nil || h.Subdomain == "" { + continue + } + if !shouldProcess(h.Subdomain) { + continue + } + select { + case work <- h.Subdomain: + case <-mctx.Ctx.Done(): + close(work) + wg.Wait() + return nil + } + } + + // 2) Keep listening for late events (e.g. from recursive discovery that + // runs in our own phase and produces new subdomains mid-resolution). + sub := mctx.Bus.Subscribe(eventbus.EventSubdomainDiscovered, func(_ context.Context, e eventbus.Event) { + ev, ok := e.(eventbus.SubdomainDiscovered) + if !ok { + return + } + if !shouldProcess(ev.Subdomain) { + return + } + select { + case work <- ev.Subdomain: + case <-mctx.Ctx.Done(): + } + }) + defer sub.Unsubscribe() + + // 3) Give late events a short window to arrive (e.g. recursive module + // running concurrently in PhaseResolution). 1 second is enough — we + // already drained the store, so any straggler events here are rare. + select { + case <-time.After(1 * time.Second): + case <-mctx.Ctx.Done(): + } + + close(work) + wg.Wait() + return nil +} + +func (m *resolverModule) resolveOne(mctx module.Context, sub string, resolvers []string, timeout int) { + if err := mctx.Ctx.Err(); err != nil { + return + } + ips := godns.ResolveSubdomain(sub, resolvers, timeout) + if len(ips) == 0 { + return + } + + cname := godns.ResolveCNAME(sub, resolvers, timeout) + ptr := godns.ResolvePTR(ips[0], resolvers, timeout) + + _ = mctx.Store.Upsert(mctx.Ctx, sub, func(h *store.Host) { + store.AddIPs(h, ips) + if cname != "" && h.CNAME == "" { + h.CNAME = cname + } + if ptr != "" && h.PTR == "" { + h.PTR = ptr + } + store.AddDiscoveryMethod(h, "resolved") + }) + + mctx.Bus.Publish(mctx.Ctx, eventbus.DNSResolved{ + EventMeta: eventbus.EventMeta{At: time.Now(), Source: ModuleName, Target: sub}, + Subdomain: sub, + IPs: ips, + CNAME: cname, + PTR: ptr, + }) +} + +func parseResolvers(s string) []string { + s = strings.TrimSpace(s) + if s == "" { + return config.DefaultResolvers + } + var out []string + for _, r := range strings.Split(s, ",") { + r = strings.TrimSpace(r) + if r == "" { + continue + } + if !strings.Contains(r, ":") { + r = r + ":53" + } + out = append(out, r) + } + if len(out) == 0 { + return config.DefaultResolvers + } + return out +} diff --git a/internal/modules/github/github.go b/internal/modules/github/github.go new file mode 100644 index 0000000..fc5db3f --- /dev/null +++ b/internal/modules/github/github.go @@ -0,0 +1,150 @@ +// Package github discovers subdomains from public GitHub code via dorks. +// Uses the v3 REST Search API. Works anonymously at a very low rate +// (strict API limits); a token in the GITHUB_TOKEN env var lifts limits. +// +// Dorks used: +// +// "