* docs(readme): reflect Safari, Windows ABE, and DuckDuckGo support
* docs(readme): name the ABE build output as hack-browser-data.exe
* docs(readme): address review feedback on ABE browsers and FDA wording
* feat(safari): localstorage extraction
Extracts Safari 17+ localStorage from WebKit's nested layout —
WebsiteDataStore/<uuid>/Origins/<top-hash>/<frame-hash>/LocalStorage/
localstorage.sqlite3 for named profiles, WebsiteData/Default for the
default profile. Parses the binary SecurityOrigin serialization
(length-prefixed scheme+host plus 0x00 default-port or 0x01 <uint16_le>
explicit-port section) and decodes UTF-16 LE ItemTable value BLOBs,
capping oversized values at 2048 bytes to match the Chromium extractor.
Reports the frame origin URL so partitioned third-party storage is
attributed to the iframe origin JavaScript actually sees.
Closes the remaining LocalStorage checkbox in #565.
* docs(safari): add RFC-011 data storage
Documents Safari's profile structure, per-category file layouts, and
storage formats including the Safari 17+ nested WebKit Origins
localStorage layout and binary SecurityOrigin serialization. Defers
Keychain credential extraction to RFC-006 §7 and notes the cross-browser
differences (plaintext cookies, plist bookmarks/downloads, Core Data
epoch timestamps, partitioned storage).
* fix(safari): latin-1 origin decoding, NULL key skip, count fast-path
- Decode originEncASCII via decodeLatin1 so high-byte records preserve
their ISO-8859-1 meaning instead of being interpreted as UTF-8.
Matches the pattern in chromium/extract_storage.go.
- Skip ItemTable rows where key is NULL — SQLite's UNIQUE constraint
permits multiple NULLs, and silently lowering them to empty strings
would collide with legitimate empty-string keys.
- countLocalStorage now walks origin dirs and runs SELECT COUNT(key)
per localstorage.sqlite3 instead of fully decoding every value.
COUNT(key) naturally excludes NULLs, keeping count and extract
symmetric.
Addresses Copilot review feedback on #582.
* fix(safari): round-2 review — WAL replay, stable ordering, error context
- Drop immutable=1 on temp-copy SQLite opens in readLocalStorageFile /
countLocalStorageFile. Session.Acquire copies the -wal / -shm sidecars,
so mode=ro alone lets SQLite replay WAL on the ephemeral copy and
surface entries Safari committed to WAL but hasn't checkpointed yet.
Live-file reads in profiles.go keep immutable=1 as before.
- Order ItemTable query by (key, rowid) for deterministic exports across
runs and SQLite versions.
- Wrap os.ReadFile / os.ReadDir errors with the offending path so
multi-origin debug logs stay scannable.
- RFC-011 §7 rewritten to explain the live-vs-temp split.
- New regression test asserts ORDER BY surfaces rows in key order.
Addresses round-2 Copilot review on #582.
* feat: add CountEntries to skip decryption for list --detail (#549)
* test: add CountEntries and countCategory tests at browser level
* fix: address review feedback on CountRows and countLocalStorage
* test: add CountRows unit tests
* feat(darwin): add interactive terminal password prompt for keychain unlock (#556)
* test: add unit tests for keyretriever and address review feedback
- Add errStorageNotFound sentinel error for precise error matching
- Non-TTY TerminalPasswordRetriever returns nil silently (review #558)
- Add darwin tests: findStorageKey, empty password, non-TTY skip
- Add linux tests: FallbackRetriever peanuts key, DefaultRetriever chain
* fix: add nolint:unused for errStorageNotFound on Windows, clean up error message
errStorageNotFound is only used on darwin/linux; Windows lint flagged it
as unused. Also simplify error format to avoid "storage" duplication.
* fix: add nolint:unused for errStorageNotFound, simplify error message
errStorageNotFound is only referenced on darwin and linux; Windows lint
flags it as unused. Also remove redundant "storage" prefix from the
error format string.
* fix: implement proper Chromium localStorage LevelDB parsing
* feat: add IsMeta field to StorageEntry and keep META entries
* fix: add error logging for decryption and missing data fields
* fix: address PR review for localStorage parsing
* fix: use naïve instead of café in Latin-1 test to avoid typos false positive
* fix: extension enabled detection and sessionStorage decoding
* fix: session storage origin resolution and extension enabled detection
* fix: address PR review comments for storage parsing
Share a single KeyRetriever instance across all profiles of the same
browser, and add sync.Once caching to GcoredumpRetriever and
SecurityCmdRetriever. This avoids repeated keychain password prompts
(or securityd memory dumps) when extracting multiple profiles.
Closes#544
* feat: add Chromium Browser implementation with new architecture
* refactor: replace Walk with ReadDir+Stat for profile discovery
* refactor: remove queries override, use extractors for Yandex passwords
* refactor: remove dataSource wrapper, use []sourcePath directly
* fix: address Copilot review feedback on chromium_new.go
* fix: always call key retriever regardless of Local State existence
* feat: add Firefox extract methods and complete data model fields
Firefox extract methods:
- extractPasswords: JSON + ASN1PBE decryption via decryptPBE helper
- extractCookies: SQLite, plaintext (no encryption), journalOff
- extractHistories: SQLite, visit count ASC sort (matches old behavior)
- extractDownloads: SQLite, moz_annos JOIN with JSON content parsing
- extractBookmarks: SQLite, moz_bookmarks JOIN moz_places
- extractExtensions: JSON, filter by location=app-profile
- extractLocalStorage: SQLite webappsstore2, reversed originKey parsing
Complete data model fields (union of Chromium and Firefox):
- CookieEntry: add HasExpire, IsPersistent
- DownloadEntry: add MimeType
- CreditCardEntry: add NickName, Address
- ExtensionEntry: add HomepageURL, Enabled
Update Chromium extractors to populate new fields:
- extract_cookie.go: fill HasExpire, IsPersistent
- extract_download.go: SELECT and fill mime_type
- extract_creditcard.go: SELECT nickname, billing_address_id
- extract_extension.go: fill HomepageURL, Enabled (state==1)
Tests:
- Full test coverage for all 7 Firefox extract functions
- Password test uses known ASN1PBE test vectors from crypto package
- Table-driven tests for parseOriginKey
- Updated Chromium tests for new fields
* fix: add COALESCE for nullable bookmark title in Firefox query
Firefox moz_bookmarks.title can be NULL (PR #500 fixed this in old code).
Add COALESCE to handle NULL gracefully in SQL instead of relying on
driver-specific NULL→string conversion behavior.
* fix: enable journalOff for all Firefox SQLite extractors and populate cookie flags
- Set journalOff=true for extract_history, extract_download, extract_bookmark
(Firefox databases require PRAGMA journal_mode=off to avoid lock errors)
- Populate HasExpire and IsPersistent for Firefox cookies (derived from expiry>0)
- Add test assertions for HasExpire/IsPersistent in both Chromium and Firefox
* fix: strip SHA256(host_key) prefix from Chrome 130+ cookie values
Chrome 130 (Cookie DB schema v24) prepends SHA256(domain) to cookie
values before encryption to prevent cross-domain replay attacks.
After decryption, this 32-byte hash must be verified and stripped.
Changes:
- Add stripCookieHash() that verifies SHA256(host_key) and strips
the prefix only when it matches (auto-compatible with older Chrome)
- Fix edge case: cookies with empty values (exactly 32 bytes = hash only)
- Add decrypt_test.go with v10 round-trip encryption/decryption test
- Add stripCookieHash test cases for v24+, older Chrome, empty values,
short values, and host mismatch scenarios
Closes#524
* fix: strip SHA256(host_key) prefix from Chrome 130+ cookie values
Chrome 130 (Cookie DB schema v24) prepends SHA256(domain) to cookie
values before encryption to prevent cross-domain replay attacks.
After decryption, this 32-byte hash must be verified and stripped.
Changes:
- Add stripCookieHash() that verifies SHA256(host_key) and strips
the prefix only when it matches (auto-compatible with older Chrome)
- Fix edge case: cookies with empty values (exactly 32 bytes = hash only)
- Add table-driven decrypt tests for v10/v20/DPAPI per platform
- Add Windows-specific DPAPI round-trip test using CryptProtectData
- Add shared testAESKey constant in testutil_test.go
- Add stripCookieHash tests for v24+, older Chrome, empty values,
short values, and host mismatch scenarios
- Extend lint CI to run on ubuntu, windows, and macos
Closes#524
* fix: remove DPAPI test from darwin/linux (returns nil on Linux)
DecryptWithDPAPI returns nil error on Linux (silent no-op) but error
on macOS, causing the test to fail on Ubuntu CI. DPAPI round-trip
testing is properly covered in decrypt_windows_test.go.
* fix: resolve Windows CI lint errors exposed by multi-platform lint
- Add _ = before windows.CloseHandle calls to satisfy errcheck
- Add build tag to params.go (only used on macOS/Linux, not Windows)
* fix: add .gitattributes to force LF and refactor cookie tests
- Add .gitattributes with `* text=auto eol=lf` to prevent CRLF
conversion on Windows CI causing gofumpt false positives
- Add .gitattributes to .gitignore whitelist
- Refactor stripCookieHash tests into table-driven style
* fix: address Copilot review on decrypt tests
- Assert error on wrong key instead of ignoring it (AES-CBC returns
padding error, not silent empty result)
- Guard empty plaintext in encryptWithDPAPI to prevent nil pointer panic
- Convert uint32 to int for make/copy slice bounds in Windows test
* fix: assert specific error message in wrong key decrypt test
* feat: add Chromium extract methods, source mapping, and tests
Implement per-category data extraction for Chromium browsers as typed
standalone functions, preparing for Phase 8 wiring into the new
Chromium struct.
New files:
- source.go: dataSource struct, chromiumSources/yandexSources maps,
yandexQueryOverrides for Yandex action_url variant
- decrypt.go: decryptValue() wrapping platform-specific decryption
- extract_password.go: SQLite + decrypt → []LoginEntry
- extract_cookie.go: SQLite + decrypt → []CookieEntry
- extract_creditcard.go: SQLite + decrypt → []CreditCardEntry
- extract_history.go: SQLite → []HistoryEntry
- extract_download.go: SQLite → []DownloadEntry
- extract_bookmark.go: JSON recursive → []BookmarkEntry
- extract_extension.go: JSON → []ExtensionEntry
- extract_storage.go: LevelDB → []StorageEntry (local + session)
- firefox/source.go: firefoxSources map
Tests use real Chrome table schemas for SQLite fixtures, with INSERT
helpers to keep test data readable and self-documenting.
Ref #520
* fix: remove LevelDB invalid path test (Windows compatibility)
leveldb.OpenFile creates the directory on Windows instead of returning
an error, causing TestExtractLocalStorage_InvalidPath to fail in CI.
This test was verifying LevelDB behavior, not our extraction logic.
* refactor: remove unused query parameter from extract functions
Only extractPasswords needs the query override (Yandex action_url).
The other 7 SQLite extract functions always use their default query,
so remove the unnecessary query parameter from their signatures.
* refactor: use DetectVersion in decryptValue instead of blind fallback
Replace try-then-fallback pattern with explicit version detection using
crypto.DetectVersion. Routes v10 to DecryptWithChromium, DPAPI to
DecryptWithDPAPI, and adds a TODO placeholder for v20 App-Bound
Encryption.
* chore: relax gocognit and gocritic linters for test files
* revert: restore strict gocognit and gocritic linters for test files
* fix: address review feedback on extract methods
- Store DetectVersion result in local variable to avoid duplicate call
- Scan credit card expiration_month/year as int then convert to string
(matches INTEGER column type in real Chrome schema)
- Add os.Stat check before leveldb.OpenFile to prevent creating empty
directories for non-existent paths
- Rename TestExtractExtensions_InvalidJSON to
TestExtractExtensions_MissingSettingsPath (JSON is valid, path is missing)
* fix: revert creditcard scan to string type for NULL safety
modernc.org/sqlite handles INTEGER→string conversion automatically.
Scanning into string is safer for nullable columns — NULL becomes ""
instead of "0" which would be an invalid month/year.
* feat: add filemanager session and crypto version detection
* refactor: move copy logic into filemanager, remove fileutil dependency
* fix: apply review suggestions for filemanager
* feat: add Windows locked file tests, fix readFileContent with ReadFile+FileMapping fallback
* fix: remove self-PID skip in findFileHandle to fix Windows CI test
* fix: seek to file start before reading duplicated handle
* fix: use full path matching in findFileHandle to avoid cross-app handle collision
* test: enhance Windows copyLocked tests with write-then-read, large file, and normal copy scenarios
* fix: check all errors in Windows tests, use bytes.Equal for large file comparison
* fix: use stable path suffix matching to handle Windows short path names in CI
* feat: add browserdata/datautil helpers (QuerySQLite, QueryRows, DecryptChromiumValue)
Phase 2 of architecture refactoring (RFC-002 Section 3):
- datautil/sqlite.go: QuerySQLite() — shared SQLite open/query/scan helper
with optional journal_mode=off for Firefox databases
- datautil/query.go: QueryRows[T]() — generic helper (Go 1.20) that wraps
QuerySQLite and collects results into a typed slice
- datautil/decrypt.go: DecryptChromiumValue() — unified Chromium decryption
(DPAPI first, then AES-GCM/CBC fallback)
- datautil/sqlite_test.go: tests for all helpers
* refactor: move DecryptChromiumValue from datautil to browser/chromium
- Remove browserdata/datautil/decrypt.go (Chromium-specific, not a generic util)
- Will be added as browser/chromium/decrypt.go (unexported decryptValue)
in the chromium extract methods PR
- Update RFCs to reflect the change
- Remove decrypt test from datautil tests
* refactor: move datautil to utils/sqliteutil for consistency
- Rename browserdata/datautil/ → utils/sqliteutil/
- Aligns with existing utils/ convention (fileutil, typeutil, byteutil)
- QuerySQLite/QueryRows are generic SQLite helpers, not browserdata-specific
- Update package name from datautil to sqliteutil
- Update both RFCs to reflect new location
* fix: apply review suggestions for sqliteutil
- QuerySQLite: validate dbPath exists before sql.Open to prevent
silently creating empty databases
- Tests: check db.Close() errors with require.NoError
* feat: add new types.Category, data models, and browserdata.Data
Phase 1 of architecture refactoring (RFC-001/RFC-002):
- types/category.go: Category enum (9 values) replacing DataType (22 values)
with String(), IsSensitive(), AllCategories, NonSensitiveCategories()
- types/models.go: browser-agnostic data models (LoginEntry, CookieEntry,
BookmarkEntry, HistoryEntry, DownloadEntry, CreditCardEntry, StorageEntry,
ExtensionEntry) — no encrypted fields, no browser prefixes
- types/category_test.go: tests for Category methods
- browserdata/browser_data.go: new Data struct with typed slices,
coexists with old BrowserData during migration
* docs: replace Coveralls badge with Codecov in README
* fix: apply review suggestions (is_http_only tag, json tags on Data)
Roger