Commit Graph

33 Commits

Author SHA1 Message Date
besendorf c782d79974 V3 (#716)
* Run bugreport and backup modules during check-androidqf

Adding support to automatically run ADB backup and bugreport modules
automatically when running the check-androidqf command. This is a first
step to deduplicate the code for Android modules.

* Deduplicate modules which are run by the sub-commands.

* Raise the proper NoAndroidQFBackup exception when a back-up isn't found

* Remove check-adb command and update docs

* Remove check-apk code and old dependencies

* Major refactor to add structured alerting and typed indicators

This commit makes a structural change to MVT by changing binary
detected/not detected logic into a structured multi-level system
of alerts. This gives far more power to extend MVT and manage
alerts.

This commit also begins the process of adding proper typing for
key objects used in MVT including Indicators, IndicatorMatches,
and ModuleResults. This will also be keep to programmatically using
the output of MVT.

* Fix up, remove ADB module base

* Rework old detections tracking into stuctured alert levels

* Quote STIX path in log line

* Fix profile events log line

* Close open archive (zip/tar) file handles

* Fix root_binaries and mounts modules to use alertstore

* Update tests to use alertstore instead of detected attribute

* Fix alertstore method calls - use high() instead of warning()

* Fix remaining test errors

- Add log_latest() call in root_binaries to log each alert
- Fix UnboundLocalError in cmd_check_androidqf by initializing bugreport variable
- Remove incorrect backup.close() call since load_backup() returns bytes
- Remove duplicate from_ab method in cmd_check_backup that was using old attributes

* Log alerts on add

* Remove slug from alertstore calls

* update alerts.py

* update alerts.py

* move indicator_match to alert object

* .

* - Remove timeline_detected and route to alertstore

* fix typing for mypy

* Remove unused type imports

* Fix check_receiver_prefix and check_android_property_name

- check_receiver_prefix() used dict syntax (ioc["value"]) on Indicator
  dataclass objects from get_iocs(). Changed to ioc.value/ioc.name.
- check_receiver_prefix() returned raw ioc instead of IndicatorMatch.
  Now returns IndicatorMatch with descriptive message.
- Fixed return type annotations on both methods to Optional[IndicatorMatch].
- Removed unused Union import.

* Fix residual self.detected usage in packages and dumpsys_receivers

These modules still used self.detected.append() which no longer exists
after the alertstore migration. Converted to alertstore calls:
- packages.py: ROOT_PACKAGES detection → alertstore.high()
- dumpsys_receivers.py: receiver IOC match → alertstore.critical()

* Fix SMS module alertstore.high() call passing slug as message

The first argument was self.get_slug() (module slug) instead of a
human-readable message. The module is already auto-detected via
AlertStore._get_calling_module(). Also removed redundant log_latest().

* Apply suggestions from code review

Fix JSON serialization in `module.save_to_json` and fix argument order in iOS alertstore calls.

Co-authored-by: tes <tesitura@users.noreply.github.com>

* Remove unsupported ADB modules

* Fail removed check-adb command

* Fix alert serialization and logging

* Close sqlite connections in iOS modules

* Fix DEBUG messages not reaching handlers, save_to_json for dictionary results and TypeError on mixed event_time types in safary_history

* add matched_indicator via alertstore instead of directly modifying json objects

* Alert on battery daily uninstall and downgrade

* Lower alert severity to medium for suspicious items

* Switch version to 2026.4.28 CalVer

---------

Co-authored-by: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
Co-authored-by: tes <tesitura@users.noreply.github.com>
Co-authored-by: Janik Besendorf <janik.besendorf@reporter-ohne-grenzen.de>
2026-04-29 14:32:29 +02:00
besendorf 6c537c624e Fix STIX2 hash key parsing to accept spec-compliant algorithm names (#767)
* Fix betterproto2 migration: update generated proto code and callers

The dependency switch from betterproto to betterproto2 was incomplete.
This updates all affected files to use the betterproto2 API:

- tombstone.py: rewrite generated code to use betterproto2.field() with
  explicit TYPE_* constants, repeated/optional/group flags, and map_meta()
  for map fields
- tombstone_crashes.py: update import and fix to_dict() call to use
  keyword-only casing= argument required by betterproto2
- pyproject.toml: replace betterproto[compiler] dev dep with betterproto2-compiler
- Makefile: update protoc plugin flag to --python_betterproto2_out

* Fix STIX2 hash key parsing to accept spec-compliant algorithm names

The STIX2 specification requires single quotes around hash algorithm
names that contain hyphens (e.g. file:hashes.'SHA-256'). MVT only
accepted a non-standard lowercase form (file:hashes.sha256), silently
dropping any indicators using the spec-correct spelling.

Normalize hash algorithm keys in _process_indicator by stripping quotes
and hyphens from the algorithm portion before matching, so all of the
following are accepted for SHA-256, SHA-1 and MD5:

  file:hashes.'SHA-256'   (STIX2 spec)
  file:hashes.SHA-256
  file:hashes.SHA256
  file:hashes.sha256      (previously the only accepted form)

The same normalization is applied to app:cert.* keys.

Update generate_stix.py to use the spec-compliant quoted forms, and add
test_parse_stix2_hash_key_variants to cover all spelling variants.
2026-04-07 20:41:40 +02:00
besendorf b795ea3129 Add root_binaries androidqf module (#676)
* Add root_binaries androidqf module

* Fix AndroidQF file count test

* fix ruff

---------

Co-authored-by: User <user@DESKTOP-3T8T346.localdomain>
2025-10-23 15:12:01 +02:00
Donncha Ó Cearbhaill f4425865c0 Add missed modules using updated settings module 2024-12-25 00:14:14 +00:00
tes 9d81b5bfa8 Add a module to parse uninstalled apps from dumpsys data, for both bugreport and AndroidQF output, and match them against package name IoCs. 2024-12-11 16:47:19 -03:00
Donncha Ó Cearbhaill 665806db98 Add initial parser for ADB state in Dumpsys (#547)
* Add initial parser for ADB dumpsys

* Add ADBState tests and support for AndroidQF and
check-adb

* Handle case where ADB is not available in device dumpsys
2024-10-18 15:31:25 +02:00
Tek a03f4e55ff Adds androidqf files module (#541)
* Adds androidqf files module

* Add new files module to module list

---------

Co-authored-by: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
2024-10-17 18:32:23 +02:00
Donncha Ó Cearbhaill 81b647beac Add basic support for IP indicators in MVT (#556)
* Add prelimary ipv4-addr ioc matching support under collection domains

* Add IP addresses as a valid IOC type

This currently just supports IPv4 addresses which
are treated as domains internally in MVT.

---------

Co-authored-by: renini <renini@local>
2024-10-17 18:20:17 +02:00
Tek 052c4e207b Improves STIX2 support and testing (#523)
* Improves STIX2 support and testing

* Adds documentation on STIX2 support in MVT

---------

Co-authored-by: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
2024-10-16 16:47:10 +02:00
Rory Flynn caeeec2816 Add packages module for androidqf (#506)
* Add Packages module for androidqf

* Update test
2024-06-24 19:00:07 +02:00
Rory Flynn 6c7ad0ac95 Convert timezone-aware datetimes automatically to UTC (#485) 2024-04-18 16:49:30 +02:00
Rory Flynn 4b4cad46ba Add CustomJSONEncoder to handle bytes types (#414)
Adds a custom JSON encoder class to fix serialisation issues where modules included bytes types containing non-utf8 bytes, which can't be serialised to JSON.

---------

Co-authored-by: Rory Flynn <rory.flynn@amnesty.org>
2023-11-15 11:40:24 +01:00
Nex 9988887d27 Updated copyright notice 2023-09-09 17:55:27 +02:00
tek e60e5fdc6e Refactors DumpsysBatteryHistory and adds related androidqf module 2023-08-04 19:20:14 +02:00
tek 7e0e071c5d Refactor DumpsysBatteryDaily module and add related artifact 2023-08-04 16:17:52 +02:00
tek 6356a4ff87 Refactor code of DumpsysDBInfo 2023-07-31 23:43:20 +02:00
Nex e1677639c4 Linted code using isort + autoflake + black, fixed wrong use of Optional[bool] 2023-06-01 23:40:26 +02:00
tek 801fe367ac Improves WebkitResourceLoadStatistics module 2023-04-07 14:43:20 +02:00
tek b5d7e528de Adds indicators for android properties 2023-03-29 12:57:41 +02:00
tek 4ed8ff51ff Improves code PEP8 compliance and adds ruff check 2023-03-01 16:43:08 -05:00
Tek 7d79844749 Improves generation of hashes (#327)
* Improves generation of hashes

* Adds generation of reference info.json hash
2023-02-21 20:16:32 +01:00
tek 704ea39569 Removes empty lines to be PEP8 compliant 2023-02-08 20:20:13 +01:00
tek 81ed0b0c19 Update copyright information 2023-02-08 20:18:16 +01:00
Nex 2781f33fb5 Added more date conversion wrappers 2022-08-13 14:04:10 +02:00
Nex 271fe5fbee Continuing enforcement of line length and simplifying date conversions 2022-08-13 02:14:24 +02:00
Nex d82c788a18 Removed AUTHORS file in favor of explicit copyright notice 2022-05-08 14:53:50 +02:00
Nex 189b1d7fc6 Fixed tests 2022-02-02 18:14:10 +01:00
Nex eb5f07a75d Updated copyright notice 2022-01-30 20:15:01 +01:00
Nex b35cd4bc73 Added support for context-aware indicators.
This way when a detection is logged, the user can know which STIX2
file was matched by the module
2022-01-21 16:26:58 +01:00
Nex b697874f56 Conforming the test files 2022-01-18 16:00:03 +01:00
Donncha Ó Cearbhaill e85c70c603 Generate stix2 for each test run 2022-01-07 17:51:21 +01:00
Donncha Ó Cearbhaill 54963b0b59 Update test PR to work with latest code, fix flake8 2022-01-07 17:03:53 +01:00
tek 513e2cc704 First test structure 2022-01-07 16:41:19 +01:00