* Add WhatsappContacts module to extract WhatsApp disappearing messages state
WhatsApp on iOS stores the disappearing messages timer for 1:1 chats on
the contact records in ContactsV2.sqlite, not in ChatStorage.sqlite. Add
a new WhatsappContacts module which extracts contact records from this
database, including phone numbers, WhatsApp and LID identifiers, and the
per-contact disappearing messages duration, and emits a timeline event
when a disappearing messages timer was set.
The database is often missing from incremental backups, so the module
logs a clear warning and returns no results instead of failing. Columns
are selected based on the actual table schema to tolerate changes across
WhatsApp versions, and if the disappearing messages column is absent the
state is reported as unknown rather than off.
The test fixture is a synthetic ContactsV2.sqlite with fictional
contacts, stored under the backup file ID derived from the WhatsApp
shared app group domain.
* Fix InteractionC contact resolution and resolve WhatsApp LIDs to contacts
The two primary InteractionC queries contained a SQL syntax error in
their direction CASE expression (a double column alias), so they always
failed and the module silently fell back to a reduced query without the
recipient join. As a result outgoing messages were serialized with no
counterpart at all ("from None (None)"). Fix the syntax so recipient
names and identifiers are extracted again, and normalize the raw 0/1
direction values from the fallback queries to INCOMING/OUTGOING.
WhatsApp identifies chat peers in interactionC.db by LID and stores the
peer LID in the domain identifier, which InteractionC could not map to a
person. Declare a dependency on the WhatsappContacts module and resolve
sender, recipient and domain identifiers (LID, JID or phone number)
against the WhatsApp contacts database, adding resolved phone number and
name fields to WhatsApp records.
Rewrite the timeline serialization to use the resolved values, fall back
to the chat peer from the domain identifier when no recipient was
recorded, label the local user instead of printing None, and include the
message direction and group name.
* Add timeline events for all WhatsApp contact timestamps
Extract ZABOUTEXPIRATIONTIMESTAMP and emit a timeline event for each
timestamp stored on a WhatsApp contact record: disappearing messages
timer changes, "about" text changes and scheduled expiry, and contact
record updates. ContactsV2.sqlite stores no other date attributes in
any released schema version.
* Add first and last interaction timeline events for WhatsApp chats
Extract one record per ZWACHATSESSION with the first and last stored
message dates, the session's own last-message date, the group creation
date and message counts. Each chat produces chat_first_message and
chat_last_message timeline events, and groups a group_created event.
The session last-message date is preferred over the newest stored
message because it survives message deletion.
* Resolve WhatsApp LID chat identifiers via the LID pair table
Recent WhatsApp versions key 1:1 chat sessions by an opaque LID rather
than the contact's phone number. Extract the ZWAPHONENUMBERLIDPAIR
table from the dedicated LID.sqlite database (or from ChatStorage
itself in versions that store it there) and use it to populate
partner_resolved_phone_number on chat session records and in timeline
events, without requiring the often-missing ContactsV2.sqlite. Each
pair is also extracted as a record and produces a lid_pair_recorded
timeline event marking when the association was learned.
* Reduce duplicate InteractionC timeline events
The interaction record's creation date normally trails its start date
by milliseconds, so serializing both nearly doubled the timeline with
duplicate entries. Only emit the creation date when it diverges from
the start date by more than an hour, with explicit wording, since a
record created long after its event indicates backfill by sync,
restore or tampering.
Per-contact aggregate dates from ZCONTACTS repeat on every interaction
row of the same contact and carried that row's message text. Serialize
them with contact-centric data strings instead, so timeline
de-duplication collapses them into one first/last-seen event per
contact.
* Fix betterproto2 migration: update generated proto code and callers
The dependency switch from betterproto to betterproto2 was incomplete.
This updates all affected files to use the betterproto2 API:
- tombstone.py: rewrite generated code to use betterproto2.field() with
explicit TYPE_* constants, repeated/optional/group flags, and map_meta()
for map fields
- tombstone_crashes.py: update import and fix to_dict() call to use
keyword-only casing= argument required by betterproto2
- pyproject.toml: replace betterproto[compiler] dev dep with betterproto2-compiler
- Makefile: update protoc plugin flag to --python_betterproto2_out
* Fix STIX2 hash key parsing to accept spec-compliant algorithm names
The STIX2 specification requires single quotes around hash algorithm
names that contain hyphens (e.g. file:hashes.'SHA-256'). MVT only
accepted a non-standard lowercase form (file:hashes.sha256), silently
dropping any indicators using the spec-correct spelling.
Normalize hash algorithm keys in _process_indicator by stripping quotes
and hyphens from the algorithm portion before matching, so all of the
following are accepted for SHA-256, SHA-1 and MD5:
file:hashes.'SHA-256' (STIX2 spec)
file:hashes.SHA-256
file:hashes.SHA256
file:hashes.sha256 (previously the only accepted form)
The same normalization is applied to app:cert.* keys.
Update generate_stix.py to use the spec-compliant quoted forms, and add
test_parse_stix2_hash_key_variants to cover all spelling variants.
* Add initial parser for ADB dumpsys
* Add ADBState tests and support for AndroidQF and
check-adb
* Handle case where ADB is not available in device dumpsys
* Add prelimary ipv4-addr ioc matching support under collection domains
* Add IP addresses as a valid IOC type
This currently just supports IPv4 addresses which
are treated as domains internally in MVT.
---------
Co-authored-by: renini <renini@local>
* Improves STIX2 support and testing
* Adds documentation on STIX2 support in MVT
---------
Co-authored-by: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
Not every system uses 'utf-8' as a default encoding for opening files in Python.
Before you say that there must be a way to set default encoding in one line, no, there is not. At least, I didn't found a way to do this.
Donncha Ó Cearbhaill