mirror of
https://github.com/mvt-project/mvt.git
synced 2026-10-24 08:08:25 +02:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d58abbb568 | ||
|
|
d52e9fcc74 | ||
|
|
59c3f5d60c | ||
|
|
2fe7f4e261 | ||
|
|
361bc72a4e | ||
|
|
9d71661f55 |
No files matched your search
@@ -86,6 +86,11 @@ writing and installing them, and the
|
||||
[custom CLI command documentation](https://docs.mvt.re/en/latest/development/custom_commands/)
|
||||
for the entry points a package registers commands in.
|
||||
|
||||
Extraction commands run independent modules concurrently with four workers by
|
||||
default. Use `--jobs 1` for sequential execution or another positive value to
|
||||
set the worker limit. Dependencies retain their declared ordering, and parallel
|
||||
console logs are printed as grouped module blocks.
|
||||
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -62,27 +62,6 @@ AndroidQF will prompt the user to download, decrypt and collect device intrusion
|
||||
|
||||
For cases where intrusion logs were collected outside of an AndroidQF acquisition, the standalone `mvt-android check-intrusion-logs` command can analyse them directly. See [Check Android Intrusion Logs](intrusion_logs.md) for details, and the [feature announcment from Amnesty International's Security Lab](https://securitylab.amnesty.org/latest/2026/05/android-intrusion-logging-as-a-new-source-of-data-for-consensual-forensic-analysis/) for background on the data source.
|
||||
|
||||
## Browser history
|
||||
|
||||
AndroidQF can optionally collect browser `History` databases when a device
|
||||
already has working root access. When its acquisition contains the
|
||||
`browser_history/manifest.json` file, `mvt-android check-androidqf` parses every
|
||||
listed database, including any collected SQLite WAL and SHM sidecars. Browser
|
||||
visits are added to the module results, timeline, URL output, and indicator
|
||||
checks.
|
||||
|
||||
MVT can also inspect the supported database locations in a full Android
|
||||
filesystem dump, or inspect one explicitly supplied Chromium `History` file:
|
||||
|
||||
```bash
|
||||
mvt-android check-fs --output /path/to/results /path/to/filesystem-dump
|
||||
mvt-android check-fs --output /path/to/results /path/to/History
|
||||
```
|
||||
|
||||
The built-in locations cover Chrome, Brave, Microsoft Edge, and Samsung
|
||||
Internet. Other Chromium-compatible databases can be supplied explicitly as a
|
||||
file without assigning an unverified browser identity.
|
||||
|
||||
## Android Debug Bridge analysis removed
|
||||
|
||||
The ability to analyze Android devices directly over ADB has been removed from MVT. Direct extraction of data from ADB was error-prone and frequently resulted in inconsistent data collection between ADB and AndroidQF acquisitions. Use AndroidQF for device acquisition and `mvt-android check-androidqf` for analysis.
|
||||
|
||||
@@ -46,6 +46,33 @@ on it, and runs the rest of the analysis. Selecting such a module with
|
||||
A cycle in the dependency graph is a programming error rather than a
|
||||
configuration problem: the command logs a warning and runs no modules at all.
|
||||
|
||||
## Parallel module execution
|
||||
|
||||
Extraction modules run concurrently, using four worker threads by default. Use
|
||||
`--jobs INTEGER` to change the worker limit, or `--jobs 1` for sequential
|
||||
execution and live per-line logging:
|
||||
|
||||
```bash
|
||||
mvt-ios check-backup --jobs 8 --output ./out ./backup
|
||||
```
|
||||
|
||||
The scheduler starts a module only after all of its declared dependencies have
|
||||
completed. Results, alerts, and timeline entries are still aggregated in stable
|
||||
topological order. During parallel execution, console records from each module
|
||||
are buffered and printed together as a labeled block when that module finishes;
|
||||
`command.log` is written immediately and retains the complete log stream.
|
||||
|
||||
Parallel-safe modules must log through `self.log`, must not write directly to
|
||||
stdout or stderr, and must not mutate shared global state. A module that uses a
|
||||
non-thread-safe resource or direct terminal output can opt out:
|
||||
|
||||
```python
|
||||
class TerminalModule(MVTModule):
|
||||
parallel_safe = False
|
||||
```
|
||||
|
||||
Such modules run synchronously and exclusively after active workers finish.
|
||||
|
||||
## Custom modules
|
||||
|
||||
MVT's module-running `check-*` commands can run forensic modules which are not
|
||||
@@ -382,7 +409,8 @@ module came from:
|
||||
Some MVT modules extract and process significant amounts of data during the analysis process or while checking results against known indicators. Care must be
|
||||
take to avoid inefficient code paths as we add new modules.
|
||||
|
||||
MVT modules can be profiled with Python built-in `cProfile` by setting the `MVT_PROFILE` environment variable.
|
||||
MVT modules can be profiled with Python built-in `cProfile` by setting the `MVT_PROFILE` environment variable. Profiling forces sequential execution even
|
||||
when `--jobs` is greater than one.
|
||||
|
||||
```bash
|
||||
MVT_PROFILE=1 dev/mvt-ios check-backup test_backup
|
||||
|
||||
+1
-1
@@ -66,4 +66,4 @@ docker run --rm -it \
|
||||
ghcr.io/mvt-project/mvt:latest-ios lockdown info
|
||||
```
|
||||
|
||||
If you built the image from source, replace `ghcr.io/mvt-project/mvt` with `mvt`.
|
||||
If you built the image from source, replace `ghcr.io/mvt-project/mvt` with `mvt`.
|
||||
@@ -1,162 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2021-2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import sqlite3
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from mvt.common.module_types import ModuleAtomicResult, ModuleSerializedResult
|
||||
from mvt.common.utils import convert_chrometime_to_datetime, convert_datetime_to_iso
|
||||
|
||||
from .artifact import AndroidArtifact
|
||||
|
||||
|
||||
class BrowserHistoryArtifact(AndroidArtifact):
|
||||
"""Shared Chromium History database parsing and result handling."""
|
||||
|
||||
def _parse_browser_history(
|
||||
self,
|
||||
connection: sqlite3.Connection,
|
||||
*,
|
||||
browser: str,
|
||||
package: str,
|
||||
profile: str,
|
||||
source_path: str,
|
||||
) -> None:
|
||||
cursor = connection.cursor()
|
||||
try:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT
|
||||
urls.id,
|
||||
urls.url,
|
||||
urls.title,
|
||||
urls.visit_count,
|
||||
urls.typed_count,
|
||||
visits.id,
|
||||
visits.visit_time,
|
||||
visits.from_visit,
|
||||
visits.transition
|
||||
FROM urls
|
||||
JOIN visits ON visits.url = urls.id
|
||||
ORDER BY visits.visit_time;
|
||||
"""
|
||||
)
|
||||
for row in cursor:
|
||||
timestamp = int(row[6])
|
||||
self.results.append(
|
||||
{
|
||||
"id": row[0],
|
||||
"url": row[1],
|
||||
"title": row[2],
|
||||
"visit_count": row[3],
|
||||
"typed_count": row[4],
|
||||
"visit_id": row[5],
|
||||
"timestamp": timestamp,
|
||||
"isodate": convert_datetime_to_iso(
|
||||
convert_chrometime_to_datetime(timestamp)
|
||||
),
|
||||
"redirect_source": row[7],
|
||||
"transition": row[8],
|
||||
"browser": browser,
|
||||
"package": package,
|
||||
"profile": profile,
|
||||
"source_path": source_path,
|
||||
}
|
||||
)
|
||||
finally:
|
||||
cursor.close()
|
||||
|
||||
def serialize(self, record: ModuleAtomicResult) -> ModuleSerializedResult:
|
||||
return {
|
||||
"timestamp": record["isodate"],
|
||||
"module": self.__class__.__name__,
|
||||
"event": "browser_history",
|
||||
"data": (
|
||||
f"{record['browser']} visit to {record['url']} "
|
||||
f"(visit ID: {record['visit_id']}, profile: {record['profile']})"
|
||||
),
|
||||
}
|
||||
|
||||
def check_indicators(self) -> None:
|
||||
if not self.indicators:
|
||||
return
|
||||
|
||||
for result, match in zip(
|
||||
self.results,
|
||||
self.indicators.check_url_batches(
|
||||
[[result["url"]] for result in self.results]
|
||||
),
|
||||
):
|
||||
if match:
|
||||
self.alertstore.critical(
|
||||
match.message, "", result, matched_indicator=match.ioc
|
||||
)
|
||||
|
||||
def collect_url_results(self) -> None:
|
||||
for result in self.results:
|
||||
self.add_url_result(result["url"], result.get("isodate"), "browser_history")
|
||||
|
||||
|
||||
class TemporarySQLiteConnection(sqlite3.Connection):
|
||||
temporary_directory: tempfile.TemporaryDirectory | None = None
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
super().close()
|
||||
finally:
|
||||
if self.temporary_directory:
|
||||
self.temporary_directory.cleanup()
|
||||
self.temporary_directory = None
|
||||
|
||||
|
||||
def open_browser_history_database(database_path: Path) -> sqlite3.Connection:
|
||||
"""Open a staged History database without modifying forensic evidence."""
|
||||
database_uri = database_path.resolve().as_uri()
|
||||
if not Path(f"{database_path}-wal").is_file():
|
||||
return sqlite3.connect(f"{database_uri}?mode=ro&immutable=1", uri=True)
|
||||
|
||||
temporary_directory = tempfile.TemporaryDirectory(prefix="mvt_sqlite_")
|
||||
temporary_path = Path(temporary_directory.name) / database_path.name
|
||||
shutil.copy2(database_path, temporary_path)
|
||||
for suffix in ("-wal", "-shm"):
|
||||
sidecar = Path(f"{database_path}{suffix}")
|
||||
if sidecar.is_file():
|
||||
shutil.copy2(sidecar, Path(f"{temporary_path}{suffix}"))
|
||||
|
||||
try:
|
||||
connection = sqlite3.connect(
|
||||
f"{temporary_path.resolve().as_uri()}?mode=ro",
|
||||
uri=True,
|
||||
factory=TemporarySQLiteConnection,
|
||||
)
|
||||
except Exception:
|
||||
temporary_directory.cleanup()
|
||||
raise
|
||||
connection.temporary_directory = temporary_directory
|
||||
return connection
|
||||
|
||||
|
||||
def validate_manifest_database(database: Any) -> dict[str, Any]:
|
||||
if not isinstance(database, dict):
|
||||
raise ValueError("database entry is not an object")
|
||||
|
||||
required = ("browser", "package", "profile", "device_path", "archive_path")
|
||||
for field in required:
|
||||
if not isinstance(database.get(field), str) or not database[field]:
|
||||
raise ValueError(f"database entry has invalid {field}")
|
||||
|
||||
archive_path = database["archive_path"]
|
||||
path = Path(archive_path)
|
||||
if (
|
||||
"\\" in archive_path
|
||||
or path.is_absolute()
|
||||
or ".." in path.parts
|
||||
or path.parts[:1] != ("browser_history",)
|
||||
):
|
||||
raise ValueError(f"unsafe browser history archive path: {archive_path}")
|
||||
return database
|
||||
+21
-54
@@ -21,7 +21,6 @@ from mvt.common.help import (
|
||||
HELP_MSG_CHECK_ANDROID_BACKUP,
|
||||
HELP_MSG_CHECK_ANDROIDQF,
|
||||
HELP_MSG_CHECK_BUGREPORT,
|
||||
HELP_MSG_CHECK_FS,
|
||||
HELP_MSG_CHECK_IOCS,
|
||||
HELP_MSG_CHECK_INTRUSION_LOGS,
|
||||
HELP_MSG_DELAY_CHECKS,
|
||||
@@ -29,6 +28,7 @@ from mvt.common.help import (
|
||||
HELP_MSG_DISABLE_UPDATE_CHECK,
|
||||
HELP_MSG_HASHES,
|
||||
HELP_MSG_IOC,
|
||||
HELP_MSG_JOBS,
|
||||
HELP_MSG_LIST_MODULES,
|
||||
HELP_MSG_LOAD_MODULE,
|
||||
HELP_MSG_MODULE,
|
||||
@@ -162,6 +162,9 @@ def check_adb(ctx):
|
||||
),
|
||||
)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("BUGREPORT_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_bugreport(
|
||||
@@ -173,6 +176,7 @@ def check_bugreport(
|
||||
load_module,
|
||||
timezone,
|
||||
verbose,
|
||||
jobs,
|
||||
bugreport_path,
|
||||
):
|
||||
from .cmd_check_bugreport import CmdAndroidCheckBugreport
|
||||
@@ -195,6 +199,7 @@ def check_bugreport(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -237,6 +242,9 @@ def check_bugreport(
|
||||
@click.option("--non-interactive", "-n", is_flag=True, help=HELP_MSG_NONINTERACTIVE)
|
||||
@click.option("--backup-password", "-p", help=HELP_MSG_ANDROID_BACKUP_PASSWORD)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("BACKUP_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_backup(
|
||||
@@ -248,6 +256,7 @@ def check_backup(
|
||||
non_interactive,
|
||||
backup_password,
|
||||
verbose,
|
||||
jobs,
|
||||
backup_path,
|
||||
):
|
||||
from .cmd_check_backup import CmdAndroidCheckBackup
|
||||
@@ -269,6 +278,7 @@ def check_backup(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -312,6 +322,9 @@ def check_backup(
|
||||
@click.option("--non-interactive", "-n", is_flag=True, help=HELP_MSG_NONINTERACTIVE)
|
||||
@click.option("--backup-password", "-p", help=HELP_MSG_ANDROID_BACKUP_PASSWORD)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("ANDROIDQF_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_androidqf(
|
||||
@@ -327,6 +340,7 @@ def check_androidqf(
|
||||
non_interactive,
|
||||
backup_password,
|
||||
verbose,
|
||||
jobs,
|
||||
androidqf_path,
|
||||
):
|
||||
from .cmd_check_androidqf import CmdAndroidCheckAndroidQF
|
||||
@@ -350,6 +364,7 @@ def check_androidqf(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -362,59 +377,6 @@ def check_androidqf(
|
||||
cmd.show_support_message()
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Command: check-fs
|
||||
# ==============================================================================
|
||||
@cli.command("check-fs", context_settings=CONTEXT_SETTINGS, help=HELP_MSG_CHECK_FS)
|
||||
@click.option(
|
||||
"--iocs",
|
||||
"-i",
|
||||
type=click.Path(exists=True),
|
||||
multiple=True,
|
||||
default=[],
|
||||
help=HELP_MSG_IOC,
|
||||
)
|
||||
@click.option("--output", "-o", type=click.Path(exists=False), help=HELP_MSG_OUTPUT)
|
||||
@click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
|
||||
@click.option("--module", "-m", help=HELP_MSG_MODULE)
|
||||
@click.option(
|
||||
"--load-module",
|
||||
type=click.Path(exists=True),
|
||||
multiple=True,
|
||||
default=[],
|
||||
help=HELP_MSG_LOAD_MODULE,
|
||||
)
|
||||
@click.option("--hashes", "-H", is_flag=True, help=HELP_MSG_HASHES)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE)
|
||||
@click.argument("DUMP_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_fs(
|
||||
ctx, iocs, output, list_modules, module, load_module, hashes, verbose, dump_path
|
||||
):
|
||||
set_verbose_logging(verbose)
|
||||
custom_modules = _load_custom_modules(load_module)
|
||||
from .cmd_check_fs import CmdAndroidCheckFS
|
||||
|
||||
cmd = CmdAndroidCheckFS(
|
||||
target_path=dump_path,
|
||||
results_path=output,
|
||||
ioc_files=iocs,
|
||||
module_name=module,
|
||||
hashes=hashes,
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
)
|
||||
if list_modules:
|
||||
cmd.list_modules()
|
||||
return
|
||||
|
||||
log.info("Checking Android filesystem located at: %s", dump_path)
|
||||
cmd.run()
|
||||
cmd.show_alerts_brief()
|
||||
cmd.show_support_message()
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Command: check-intrusion-logs
|
||||
# ==============================================================================
|
||||
@@ -452,6 +414,9 @@ def check_fs(
|
||||
),
|
||||
)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("LOGS_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_intrusion_logs(
|
||||
@@ -463,6 +428,7 @@ def check_intrusion_logs(
|
||||
load_module,
|
||||
timezone,
|
||||
verbose,
|
||||
jobs,
|
||||
logs_path,
|
||||
):
|
||||
from .cmd_check_intrusion_logs import CmdAndroidCheckIntrusionLogs
|
||||
@@ -483,6 +449,7 @@ def check_intrusion_logs(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
|
||||
@@ -53,6 +53,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -68,6 +69,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
@@ -259,6 +261,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
hashes=self.hashes,
|
||||
sub_command=True,
|
||||
custom_modules=self.custom_modules,
|
||||
jobs=self.jobs,
|
||||
)
|
||||
cmd.from_zip(bugreport)
|
||||
cmd.run()
|
||||
@@ -290,6 +293,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
hashes=self.hashes,
|
||||
sub_command=True,
|
||||
custom_modules=self.custom_modules,
|
||||
jobs=self.jobs,
|
||||
)
|
||||
try:
|
||||
cmd.from_ab(backup)
|
||||
@@ -376,6 +380,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
hashes=self.hashes,
|
||||
sub_command=True,
|
||||
custom_modules=self.custom_modules,
|
||||
jobs=self.jobs,
|
||||
)
|
||||
cmd.run()
|
||||
|
||||
|
||||
@@ -47,6 +47,7 @@ class CmdAndroidCheckBackup(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -62,6 +63,7 @@ class CmdAndroidCheckBackup(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
|
||||
@@ -40,6 +40,7 @@ class CmdAndroidCheckBugreport(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -55,6 +56,7 @@ class CmdAndroidCheckBugreport(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2021-2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from mvt.common.command import Command
|
||||
from mvt.common.indicators import Indicators
|
||||
from mvt.common.module import MVTModule
|
||||
|
||||
from .modules.fs import FS_MODULES
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CmdAndroidCheckFS(Command):
|
||||
def __init__(
|
||||
self,
|
||||
target_path: Optional[str] = None,
|
||||
results_path: Optional[str] = None,
|
||||
ioc_files: Optional[list] = None,
|
||||
iocs: Optional[Indicators] = None,
|
||||
module_name: Optional[str] = None,
|
||||
module_options: Optional[dict] = None,
|
||||
hashes: bool = False,
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
results_path=results_path,
|
||||
ioc_files=ioc_files,
|
||||
iocs=iocs,
|
||||
module_name=module_name,
|
||||
module_options=module_options,
|
||||
hashes=hashes,
|
||||
log=log,
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
)
|
||||
self.platform = "android"
|
||||
self.name = "check-fs"
|
||||
self.modules = FS_MODULES
|
||||
@@ -37,6 +37,7 @@ class CmdAndroidCheckIntrusionLogs(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -52,6 +53,7 @@ class CmdAndroidCheckIntrusionLogs(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
|
||||
@@ -11,7 +11,6 @@ from .aqf_log_timestamps import AQFLogTimestamps
|
||||
from .aqf_packages import AQFPackages
|
||||
from .aqf_processes import AQFProcesses
|
||||
from .aqf_settings import AQFSettings
|
||||
from .browser_history import BrowserHistory
|
||||
from .mounts import Mounts
|
||||
from .root_binaries import RootBinaries
|
||||
|
||||
@@ -24,5 +23,4 @@ ANDROIDQF_MODULES: list[type[MVTModule]] = [
|
||||
AQFLogTimestamps,
|
||||
RootBinaries,
|
||||
Mounts,
|
||||
BrowserHistory,
|
||||
]
|
||||
@@ -8,8 +8,6 @@ import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from rich.progress import track
|
||||
|
||||
from mvt.android.utils import (
|
||||
BROWSER_INSTALLERS,
|
||||
PLAY_STORE_INSTALLERS,
|
||||
@@ -153,10 +151,9 @@ class AQFPackages(AndroidQFModule):
|
||||
if total_hashes == 0:
|
||||
return
|
||||
|
||||
progress_desc = f"Looking up {total_hashes} package files on VirusTotal..."
|
||||
for index, file_hash in enumerate(
|
||||
track(files_by_hash, description=progress_desc)
|
||||
):
|
||||
self.log.info("Looking up %d package files on VirusTotal...", total_hashes)
|
||||
for index, file_hash in enumerate(files_by_hash):
|
||||
self.log.debug("VirusTotal lookup %d/%d", index + 1, total_hashes)
|
||||
try:
|
||||
results = virustotal_lookup(file_hash)
|
||||
except VTNoKey as exc:
|
||||
|
||||
@@ -81,12 +81,13 @@ class AndroidQFModule(MVTModule):
|
||||
return None
|
||||
|
||||
def _get_file_content(self, file_path):
|
||||
if self.archive:
|
||||
handle = self.archive.open(file_path)
|
||||
else:
|
||||
handle = open(os.path.join(self.parent_path, file_path), "rb")
|
||||
with self.resource_lock:
|
||||
if self.archive:
|
||||
handle = self.archive.open(file_path)
|
||||
else:
|
||||
handle = open(os.path.join(self.parent_path, file_path), "rb")
|
||||
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
|
||||
return data
|
||||
return data
|
||||
@@ -1,115 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2021-2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
from mvt.android.artifacts.browser_history import (
|
||||
BrowserHistoryArtifact,
|
||||
open_browser_history_database,
|
||||
validate_manifest_database,
|
||||
)
|
||||
|
||||
from .base import AndroidQFModule
|
||||
|
||||
|
||||
class BrowserHistory(BrowserHistoryArtifact, AndroidQFModule):
|
||||
"""Extract browser visits collected by AndroidQF."""
|
||||
|
||||
supported_commands = (("android", "check-androidqf"),)
|
||||
|
||||
def _find_manifest(self) -> str | None:
|
||||
manifests = [
|
||||
file_path
|
||||
for file_path in self.files
|
||||
if file_path.replace("\\", "/").endswith("browser_history/manifest.json")
|
||||
]
|
||||
if not manifests:
|
||||
return None
|
||||
if len(manifests) > 1:
|
||||
self.log.warning(
|
||||
"Found multiple browser history manifests; using %s", manifests[0]
|
||||
)
|
||||
return manifests[0]
|
||||
|
||||
def _stage_database(
|
||||
self, archive_path: str, prefix: str, temporary_path: Path
|
||||
) -> Path:
|
||||
available_files = {
|
||||
str(PurePosixPath(file_path.replace("\\", "/"))): file_path
|
||||
for file_path in self.files
|
||||
}
|
||||
normalized_path = str(PurePosixPath(prefix, archive_path))
|
||||
source_path = available_files.get(normalized_path)
|
||||
if not source_path:
|
||||
raise FileNotFoundError(archive_path)
|
||||
|
||||
staged_path = temporary_path / "History"
|
||||
staged_path.write_bytes(self._get_file_content(source_path))
|
||||
for suffix in ("-wal", "-shm"):
|
||||
sidecar = available_files.get(normalized_path + suffix)
|
||||
if sidecar:
|
||||
Path(f"{staged_path}{suffix}").write_bytes(
|
||||
self._get_file_content(sidecar)
|
||||
)
|
||||
return staged_path
|
||||
|
||||
def run(self) -> None:
|
||||
manifest_path = self._find_manifest()
|
||||
if not manifest_path:
|
||||
self.log.info("No AndroidQF browser history manifest found")
|
||||
return
|
||||
|
||||
try:
|
||||
manifest = json.loads(self._get_file_content(manifest_path))
|
||||
except (json.JSONDecodeError, OSError, TypeError, UnicodeDecodeError) as exc:
|
||||
self.log.error("Unable to read browser history manifest: %s", exc)
|
||||
return
|
||||
|
||||
if not isinstance(manifest, dict) or manifest.get("schema_version") != 1:
|
||||
self.log.error("Unsupported AndroidQF browser history manifest")
|
||||
return
|
||||
databases = manifest.get("databases", [])
|
||||
if not isinstance(databases, list):
|
||||
self.log.error("Invalid AndroidQF browser history database list")
|
||||
return
|
||||
|
||||
normalized_manifest = manifest_path.replace("\\", "/")
|
||||
marker = "browser_history/manifest.json"
|
||||
prefix = normalized_manifest[: -len(marker)].rstrip("/")
|
||||
|
||||
for raw_database in databases:
|
||||
try:
|
||||
database = validate_manifest_database(raw_database)
|
||||
with tempfile.TemporaryDirectory(prefix="mvt_browser_history_") as temp:
|
||||
staged_path = self._stage_database(
|
||||
database["archive_path"], prefix, Path(temp)
|
||||
)
|
||||
connection = open_browser_history_database(staged_path)
|
||||
try:
|
||||
self._parse_browser_history(
|
||||
connection,
|
||||
browser=database["browser"],
|
||||
package=database["package"],
|
||||
profile=database["profile"],
|
||||
source_path=database["device_path"],
|
||||
)
|
||||
finally:
|
||||
connection.close()
|
||||
except (
|
||||
FileNotFoundError,
|
||||
OSError,
|
||||
OverflowError,
|
||||
sqlite3.Error,
|
||||
TypeError,
|
||||
ValueError,
|
||||
) as exc:
|
||||
self.log.error("Unable to parse browser history database: %s", exc)
|
||||
|
||||
self.log.info(
|
||||
"Extracted a total of %d browser history items", len(self.results)
|
||||
)
|
||||
@@ -55,21 +55,22 @@ class BackupModule(MVTModule):
|
||||
return fnmatch.filter(self.files, pattern)
|
||||
|
||||
def _get_file_content(self, file_path: str) -> bytes:
|
||||
handle = None
|
||||
if self.tar:
|
||||
try:
|
||||
member = self.tar.getmember(file_path)
|
||||
handle = self.tar.extractfile(member)
|
||||
if not handle:
|
||||
raise ValueError(f"Could not extract file: {file_path}")
|
||||
except KeyError:
|
||||
raise FileNotFoundError(f"File not found in tar: {file_path}")
|
||||
elif self.backup_path:
|
||||
handle = open(os.path.join(self.backup_path, file_path), "rb")
|
||||
else:
|
||||
raise ValueError("No backup path or tar file provided")
|
||||
with self.resource_lock:
|
||||
handle = None
|
||||
if self.tar:
|
||||
try:
|
||||
member = self.tar.getmember(file_path)
|
||||
handle = self.tar.extractfile(member)
|
||||
if not handle:
|
||||
raise ValueError(f"Could not extract file: {file_path}")
|
||||
except KeyError:
|
||||
raise FileNotFoundError(f"File not found in tar: {file_path}")
|
||||
elif self.backup_path:
|
||||
handle = open(os.path.join(self.backup_path, file_path), "rb")
|
||||
else:
|
||||
raise ValueError("No backup path or tar file provided")
|
||||
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
|
||||
return data
|
||||
return data
|
||||
@@ -4,6 +4,7 @@
|
||||
# https://github.com/mvt-project/mvt/blob/main/LICENSE
|
||||
import datetime
|
||||
import fnmatch
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
@@ -12,6 +13,7 @@ from typing import List, Optional
|
||||
from zipfile import ZipFile
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
from mvt.android.artifacts.artifact import AndroidArtifact
|
||||
from mvt.common.module import ModuleResults, MVTModule
|
||||
|
||||
# `------ 0.101s was the duration of 'SOME SECTION' ------`, printed when that
|
||||
@@ -24,6 +26,24 @@ SECTION_DURATION = re.compile(r"^-{3,}\s*[0-9.]+s was the duration of", re.IGNOR
|
||||
DUMPSTATE_ENTRY_POINTS = ("main_entry.txt", "dumpState_*.log", "*/dumpsys.txt")
|
||||
|
||||
|
||||
_DUMPSTATE_BYTES_CACHE_KEY = ("android_bugreport", "dumpstate", "bytes")
|
||||
_DUMPSTATE_SECTIONS_CACHE_KEY = ("android_bugreport", "dumpstate", "sections")
|
||||
_SYSTEM_PROPERTIES_SECTION = b"------ SYSTEM PROPERTIES"
|
||||
_DUMPSYS_SEPARATORS = {
|
||||
b"DUMP OF SERVICE accessibility:",
|
||||
b"DUMP OF SERVICE adb:",
|
||||
b"DUMP OF SERVICE appops:",
|
||||
b"DUMP OF SERVICE batterystats:",
|
||||
b"DUMP OF SERVICE dbinfo:",
|
||||
b"DUMP OF SERVICE package:",
|
||||
b"DUMP OF SERVICE platform_compat:",
|
||||
}
|
||||
_DUMPSTATE_DELIMITER = (
|
||||
b"------------------------------------------------------------------------------"
|
||||
)
|
||||
_GETPROP_LINE_RE = re.compile(rb"\[(.+?)\]: \[(.+?)\]")
|
||||
|
||||
|
||||
class BugReportModule(MVTModule):
|
||||
"""This class provides a base for all Android Bug Report modules."""
|
||||
|
||||
@@ -77,45 +97,156 @@ class BugReportModule(MVTModule):
|
||||
return []
|
||||
|
||||
def _get_file_content(self, file_path: str) -> bytes:
|
||||
if self.zip_archive:
|
||||
handle = self.zip_archive.open(file_path)
|
||||
else:
|
||||
if not self.extract_path:
|
||||
raise ValueError("extract_path is not set")
|
||||
joined = os.path.join(self.extract_path, file_path)
|
||||
if (
|
||||
not Path(joined)
|
||||
.resolve()
|
||||
.is_relative_to(Path(self.extract_path).resolve())
|
||||
):
|
||||
raise ValueError("unsafe file_path")
|
||||
handle = open(joined, "rb")
|
||||
with self.resource_lock:
|
||||
if self.zip_archive:
|
||||
handle = self.zip_archive.open(file_path)
|
||||
else:
|
||||
if not self.extract_path:
|
||||
raise ValueError("extract_path is not set")
|
||||
joined = os.path.join(self.extract_path, file_path)
|
||||
if (
|
||||
not Path(joined)
|
||||
.resolve()
|
||||
.is_relative_to(Path(self.extract_path).resolve())
|
||||
):
|
||||
raise ValueError("unsafe file_path")
|
||||
handle = open(joined, "rb")
|
||||
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
|
||||
return data
|
||||
return data
|
||||
|
||||
def _get_dumpstate_file(self) -> Optional[bytes]:
|
||||
main_entry, dumpstate_log, dumpsys_txt = DUMPSTATE_ENTRY_POINTS
|
||||
with self.resource_lock:
|
||||
if _DUMPSTATE_BYTES_CACHE_KEY in self.resource_cache:
|
||||
return self.resource_cache[_DUMPSTATE_BYTES_CACHE_KEY]
|
||||
|
||||
main = self._get_files_by_pattern(main_entry)
|
||||
if main:
|
||||
main_content = self._get_file_content(main[0])
|
||||
try:
|
||||
return self._get_file_content(main_content.decode().strip())
|
||||
except KeyError:
|
||||
return None
|
||||
content = None
|
||||
main_entry, dumpstate_log, dumpsys_txt = DUMPSTATE_ENTRY_POINTS
|
||||
main = self._get_files_by_pattern(main_entry)
|
||||
if main:
|
||||
main_content = self._get_file_content(main[0])
|
||||
try:
|
||||
content = self._get_file_content(main_content.decode().strip())
|
||||
except KeyError:
|
||||
pass
|
||||
else:
|
||||
dumpstate_logs = self._get_files_by_pattern(dumpstate_log)
|
||||
if dumpstate_logs:
|
||||
content = self._get_file_content(dumpstate_logs[0])
|
||||
else:
|
||||
dumpsys_files = self._get_files_by_pattern(dumpsys_txt)
|
||||
if dumpsys_files:
|
||||
content = self._get_file_content(dumpsys_files[0])
|
||||
|
||||
dumpstate_logs = self._get_files_by_pattern(dumpstate_log)
|
||||
if dumpstate_logs:
|
||||
return self._get_file_content(dumpstate_logs[0])
|
||||
self.resource_cache[_DUMPSTATE_BYTES_CACHE_KEY] = content
|
||||
return content
|
||||
|
||||
dumpsys_files = self._get_files_by_pattern(dumpsys_txt)
|
||||
if dumpsys_files:
|
||||
return self._get_file_content(dumpsys_files[0])
|
||||
def _get_dumpstate_sections(self) -> dict[bytes, bytes]:
|
||||
with self.resource_lock:
|
||||
if _DUMPSTATE_SECTIONS_CACHE_KEY in self.resource_cache:
|
||||
return self.resource_cache[_DUMPSTATE_SECTIONS_CACHE_KEY]
|
||||
|
||||
return None
|
||||
section_lines: dict[bytes, list[bytes]] = {
|
||||
separator: [] for separator in _DUMPSYS_SEPARATORS
|
||||
}
|
||||
section_lines[_SYSTEM_PROPERTIES_SECTION] = []
|
||||
current_section = None
|
||||
completed_sections = set()
|
||||
in_system_properties = False
|
||||
content = self._get_dumpstate_file()
|
||||
|
||||
if content:
|
||||
for raw_line in io.BytesIO(content):
|
||||
line = raw_line.rstrip(b"\r\n")
|
||||
stripped = line.strip()
|
||||
|
||||
if stripped.startswith(_SYSTEM_PROPERTIES_SECTION):
|
||||
in_system_properties = True
|
||||
elif in_system_properties:
|
||||
if stripped == b"------":
|
||||
in_system_properties = False
|
||||
elif _GETPROP_LINE_RE.search(line):
|
||||
section_lines[_SYSTEM_PROPERTIES_SECTION].append(line)
|
||||
|
||||
if stripped in _DUMPSYS_SEPARATORS:
|
||||
if current_section is not None:
|
||||
completed_sections.add(current_section)
|
||||
current_section = (
|
||||
None if stripped in completed_sections else stripped
|
||||
)
|
||||
continue
|
||||
|
||||
if stripped.startswith(_DUMPSTATE_DELIMITER):
|
||||
if current_section is not None:
|
||||
completed_sections.add(current_section)
|
||||
current_section = None
|
||||
continue
|
||||
|
||||
if current_section is not None:
|
||||
section_lines[current_section].append(line)
|
||||
|
||||
sections = {
|
||||
separator: b"\n".join(lines)
|
||||
for separator, lines in section_lines.items()
|
||||
}
|
||||
self.resource_cache[_DUMPSTATE_SECTIONS_CACHE_KEY] = sections
|
||||
return sections
|
||||
|
||||
def _get_dumpsys_section_bytes(self, separator: bytes) -> Optional[bytes]:
|
||||
sections = self._get_dumpstate_sections()
|
||||
if separator in sections:
|
||||
return sections[separator]
|
||||
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
return None
|
||||
return AndroidArtifact.extract_dumpsys_section(content, separator, binary=True)
|
||||
|
||||
def _get_dumpsys_section(
|
||||
self, separator: str, errors: str = "replace"
|
||||
) -> Optional[str]:
|
||||
cache_key = (
|
||||
"android_bugreport",
|
||||
"dumpstate",
|
||||
"section",
|
||||
separator,
|
||||
errors,
|
||||
)
|
||||
with self.resource_lock:
|
||||
if cache_key in self.resource_cache:
|
||||
return self.resource_cache[cache_key]
|
||||
|
||||
section_bytes = self._get_dumpsys_section_bytes(separator.encode("utf-8"))
|
||||
section = (
|
||||
section_bytes.decode("utf-8", errors=errors)
|
||||
if section_bytes is not None
|
||||
else None
|
||||
)
|
||||
self.resource_cache[cache_key] = section
|
||||
return section
|
||||
|
||||
def _get_system_properties_text(self, errors: str = "ignore") -> Optional[str]:
|
||||
cache_key = (
|
||||
"android_bugreport",
|
||||
"dumpstate",
|
||||
"system_properties",
|
||||
errors,
|
||||
)
|
||||
with self.resource_lock:
|
||||
if cache_key in self.resource_cache:
|
||||
return self.resource_cache[cache_key]
|
||||
|
||||
sections = self._get_dumpstate_sections()
|
||||
section_bytes = sections.get(_SYSTEM_PROPERTIES_SECTION)
|
||||
section = (
|
||||
section_bytes.decode("utf-8", errors=errors)
|
||||
if section_bytes is not None
|
||||
else None
|
||||
)
|
||||
self.resource_cache[cache_key] = section
|
||||
return section
|
||||
|
||||
@staticmethod
|
||||
def extract_command_section(content: str, heading: str) -> str:
|
||||
@@ -164,7 +295,8 @@ class BugReportModule(MVTModule):
|
||||
instant returned in UTC.
|
||||
"""
|
||||
if self.zip_archive:
|
||||
file_timetuple = self.zip_archive.getinfo(file_path).date_time
|
||||
with self.resource_lock:
|
||||
file_timetuple = self.zip_archive.getinfo(file_path).date_time
|
||||
return datetime.datetime(*file_timetuple, tzinfo=self._device_timezone())
|
||||
if not self.extract_path:
|
||||
raise ValueError("extract_path is not set")
|
||||
|
||||
@@ -34,18 +34,16 @@ class DumpsysAccessibility(DumpsysAccessibilityArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
full_dumpsys = self._get_dumpstate_file()
|
||||
if not full_dumpsys:
|
||||
content = self._get_dumpsys_section(
|
||||
"DUMP OF SERVICE accessibility:", errors="ignore"
|
||||
)
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
content = self.extract_dumpsys_section(
|
||||
full_dumpsys.decode("utf-8", errors="ignore"),
|
||||
"DUMP OF SERVICE accessibility:",
|
||||
)
|
||||
self.parse(content)
|
||||
|
||||
listed = unnamed = 0
|
||||
|
||||
@@ -38,19 +38,14 @@ class DumpsysActivities(DumpsysPackageActivitiesArtifact, BugReportModule):
|
||||
self.results = results if results else []
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
section = self._get_dumpsys_section("DUMP OF SERVICE package:", errors="ignore")
|
||||
if section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
# Extract package section
|
||||
section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="ignore"), "DUMP OF SERVICE package:"
|
||||
)
|
||||
|
||||
# Parse
|
||||
self.parse(section)
|
||||
|
||||
|
||||
@@ -38,19 +38,14 @@ class DumpsysADBState(DumpsysADBArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
full_dumpsys = self._get_dumpstate_file()
|
||||
if not full_dumpsys:
|
||||
content = self._get_dumpsys_section_bytes(b"DUMP OF SERVICE adb:")
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
content = self.extract_dumpsys_section(
|
||||
full_dumpsys,
|
||||
b"DUMP OF SERVICE adb:",
|
||||
binary=True,
|
||||
)
|
||||
self.parse(content)
|
||||
if self.results:
|
||||
self.log.info(
|
||||
|
||||
@@ -34,17 +34,14 @@ class DumpsysAppops(DumpsysAppopsArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
section = self._get_dumpsys_section("DUMP OF SERVICE appops:")
|
||||
if section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE appops:"
|
||||
)
|
||||
self.parse(section)
|
||||
|
||||
self.log.info(
|
||||
|
||||
@@ -34,17 +34,14 @@ class DumpsysBatteryDaily(DumpsysBatteryDailyArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
dumpsys_section = self._get_dumpsys_section("DUMP OF SERVICE batterystats:")
|
||||
if dumpsys_section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
dumpsys_section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE batterystats:"
|
||||
)
|
||||
self.parse(dumpsys_section)
|
||||
|
||||
self.log.info("Extracted a total of %d battery daily stats", len(self.results))
|
||||
@@ -34,17 +34,14 @@ class DumpsysBatteryHistory(DumpsysBatteryHistoryArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
dumpsys_section = self._get_dumpsys_section("DUMP OF SERVICE batterystats:")
|
||||
if dumpsys_section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
dumpsys_section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE batterystats:"
|
||||
)
|
||||
self.parse(dumpsys_section)
|
||||
|
||||
self.log.info(
|
||||
|
||||
@@ -36,18 +36,14 @@ class DumpsysDBInfo(DumpsysDBInfoArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
data = self._get_dumpstate_file()
|
||||
if not data:
|
||||
section = self._get_dumpsys_section("DUMP OF SERVICE dbinfo:", errors="ignore")
|
||||
if section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
section = self.extract_dumpsys_section(
|
||||
data.decode("utf-8", errors="ignore"), "DUMP OF SERVICE dbinfo:"
|
||||
)
|
||||
|
||||
self.parse(section)
|
||||
self.log.info(
|
||||
"Extracted a total of %d database connection pool records",
|
||||
|
||||
@@ -36,16 +36,13 @@ class DumpsysGetProp(GetPropArtifact, BugReportModule):
|
||||
self.results = [] if not results else results
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
content = self._get_system_properties_text()
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
section = self.extract_command_section(
|
||||
content.decode(errors="ignore"), "------ SYSTEM PROPERTIES"
|
||||
)
|
||||
self.parse(section)
|
||||
self.parse(content)
|
||||
self.log.info("Extracted %d Android system properties", len(self.results))
|
||||
@@ -35,17 +35,14 @@ class DumpsysPackages(DumpsysPackagesArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
data = self._get_dumpstate_file()
|
||||
if not data:
|
||||
content = self._get_dumpsys_section("DUMP OF SERVICE package:")
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
content = self.extract_dumpsys_section(
|
||||
data.decode("utf-8", errors="replace"), "DUMP OF SERVICE package:"
|
||||
)
|
||||
self.parse(content)
|
||||
|
||||
for result in self.results:
|
||||
|
||||
@@ -34,18 +34,14 @@ class DumpsysPlatformCompat(DumpsysPlatformCompatArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
data = self._get_dumpstate_file()
|
||||
if not data:
|
||||
content = self._get_dumpsys_section("DUMP OF SERVICE platform_compat:")
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
decoded_data = data.decode("utf-8", errors="replace")
|
||||
content = self.extract_dumpsys_section(
|
||||
decoded_data, "DUMP OF SERVICE platform_compat:"
|
||||
)
|
||||
self.parse(content)
|
||||
|
||||
self.log.info("Found %d package compatibility overrides", len(self.results))
|
||||
@@ -36,17 +36,13 @@ class DumpsysReceivers(DumpsysReceiversArtifact, BugReportModule):
|
||||
self.results = results if results else []
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
dumpsys_section = self._get_dumpsys_section("DUMP OF SERVICE package:")
|
||||
if dumpsys_section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
dumpsys_section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE package:"
|
||||
)
|
||||
|
||||
self.parse(dumpsys_section)
|
||||
self.log.info("Extracted %d package receivers", len(self.results))
|
||||
@@ -1,10 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2021-2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
from mvt.common.module import MVTModule
|
||||
|
||||
from .browser_history import BrowserHistory
|
||||
|
||||
FS_MODULES: list[type[MVTModule]] = [BrowserHistory]
|
||||
@@ -1,115 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2021-2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from mvt.android.artifacts.browser_history import (
|
||||
BrowserHistoryArtifact,
|
||||
open_browser_history_database,
|
||||
)
|
||||
from mvt.common.module import MVTModule
|
||||
from mvt.common.module_types import ModuleResults
|
||||
|
||||
|
||||
# These locations are deliberately limited to paths backed by public parser
|
||||
# fixtures or the historical MVT implementation.
|
||||
BROWSER_HISTORY_PATHS = {
|
||||
"data/data/com.android.chrome/app_chrome/Default/History": (
|
||||
"Chrome",
|
||||
"com.android.chrome",
|
||||
"Default",
|
||||
),
|
||||
"data/data/com.brave.browser/app_chrome/Default/History": (
|
||||
"Brave",
|
||||
"com.brave.browser",
|
||||
"Default",
|
||||
),
|
||||
"data/data/com.microsoft.emmx/app_chrome/Default/History": (
|
||||
"Microsoft Edge",
|
||||
"com.microsoft.emmx",
|
||||
"Default",
|
||||
),
|
||||
"data/data/com.sec.android.app.sbrowser/app_sbrowser/Default/History": (
|
||||
"Samsung Internet",
|
||||
"com.sec.android.app.sbrowser",
|
||||
"Default",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
class BrowserHistory(BrowserHistoryArtifact, MVTModule):
|
||||
"""Extract supported Chromium History databases from a filesystem dump."""
|
||||
|
||||
supported_commands = (("android", "check-fs"),)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
target_path: Optional[str] = None,
|
||||
results_path: Optional[str] = None,
|
||||
module_options: Optional[dict] = None,
|
||||
log: logging.Logger = logging.getLogger(__name__),
|
||||
results: Optional[ModuleResults] = None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
file_path=file_path,
|
||||
target_path=target_path,
|
||||
results_path=results_path,
|
||||
module_options=module_options,
|
||||
log=log,
|
||||
results=results,
|
||||
)
|
||||
|
||||
def _database_paths(self) -> list[tuple[Path, str, str, str]]:
|
||||
if not self.target_path:
|
||||
return []
|
||||
target = Path(self.target_path)
|
||||
if target.is_file():
|
||||
normalized_target = target.resolve().as_posix()
|
||||
for relative_path, identity in BROWSER_HISTORY_PATHS.items():
|
||||
if normalized_target.endswith(f"/{relative_path}"):
|
||||
return [(target, *identity)]
|
||||
return [(target, "Chromium", "unknown", "unknown")]
|
||||
|
||||
databases = []
|
||||
for relative_path, identity in BROWSER_HISTORY_PATHS.items():
|
||||
database_path = target / relative_path
|
||||
if database_path.is_file():
|
||||
databases.append((database_path, *identity))
|
||||
return databases
|
||||
|
||||
def run(self) -> None:
|
||||
for database_path, browser, package, profile in self._database_paths():
|
||||
try:
|
||||
connection = open_browser_history_database(database_path)
|
||||
try:
|
||||
self._parse_browser_history(
|
||||
connection,
|
||||
browser=browser,
|
||||
package=package,
|
||||
profile=profile,
|
||||
source_path=str(database_path),
|
||||
)
|
||||
finally:
|
||||
connection.close()
|
||||
except (
|
||||
OSError,
|
||||
OverflowError,
|
||||
sqlite3.Error,
|
||||
TypeError,
|
||||
ValueError,
|
||||
) as exc:
|
||||
self.log.error(
|
||||
"Unable to parse browser history database %s: %s",
|
||||
database_path,
|
||||
exc,
|
||||
)
|
||||
|
||||
self.log.info(
|
||||
"Extracted a total of %d browser history items", len(self.results)
|
||||
)
|
||||
+228
-43
@@ -6,7 +6,10 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import sys
|
||||
import threading
|
||||
from concurrent.futures import Future, ThreadPoolExecutor
|
||||
from datetime import datetime
|
||||
from functools import cached_property
|
||||
from heapq import heappop, heappush
|
||||
@@ -19,6 +22,7 @@ from rich.text import Text
|
||||
from .alerts import AlertLevel, AlertStore
|
||||
from .config import settings
|
||||
from .indicators import Indicators
|
||||
from .log import MVTLogHandler, finish_module_log_buffer, start_module_log_buffer
|
||||
from .module import EncryptedBackupError, MVTModule, run_module, save_timeline
|
||||
from .module_loader import (
|
||||
ModuleOrigin,
|
||||
@@ -52,6 +56,7 @@ class Command:
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
self.name = ""
|
||||
self.platform = ""
|
||||
@@ -67,6 +72,11 @@ class Command:
|
||||
self.sub_command = sub_command
|
||||
self.disable_version_check = disable_version_check
|
||||
self.disable_indicator_check = disable_indicator_check
|
||||
if jobs < 1:
|
||||
raise ValueError("jobs must be at least 1")
|
||||
self.jobs = jobs
|
||||
self._resource_lock = threading.RLock()
|
||||
self._resource_cache: dict[object, Any] = {}
|
||||
|
||||
# This dictionary can contain options that will be passed down from
|
||||
# the Command to all modules. This can for example be used to pass
|
||||
@@ -710,41 +720,22 @@ class Command:
|
||||
|
||||
return ordered
|
||||
|
||||
def run(self) -> None:
|
||||
# The output folder and its command.log exist for a run, so that
|
||||
# listing modules or rejecting a target leaves nothing behind.
|
||||
# Resolving the module list can warn, so the log comes first.
|
||||
self._create_storage()
|
||||
self._setup_logging()
|
||||
|
||||
ordered_modules = self._ordered_modules()
|
||||
if ordered_modules is None:
|
||||
return
|
||||
|
||||
# Read the indicators once the run is certain to happen, before
|
||||
# init() does any work on the target, so that a missing indicators
|
||||
# file is reported first and every module gets the same object.
|
||||
iocs = self.iocs
|
||||
|
||||
# Commands announce their target from init(), so it goes before the
|
||||
# module list.
|
||||
def _run_module(
|
||||
self,
|
||||
module: type[MVTModule],
|
||||
executed_by_type: dict[type[MVTModule], MVTModule],
|
||||
capture_logs: bool,
|
||||
) -> tuple[MVTModule, list[logging.LogRecord], bool]:
|
||||
"""Initialize and run one module, optionally buffering console logs."""
|
||||
token = start_module_log_buffer() if capture_logs else None
|
||||
records: list[logging.LogRecord] = []
|
||||
encrypted = False
|
||||
try:
|
||||
self.init()
|
||||
except NotImplementedError:
|
||||
pass
|
||||
|
||||
self._log_loaded_modules(ordered_modules)
|
||||
|
||||
executed_by_type: dict[type[MVTModule], MVTModule] = {}
|
||||
for module in ordered_modules:
|
||||
|
||||
module_logger = get_module_logger(module)
|
||||
|
||||
m = module(
|
||||
target_path=self.target_path,
|
||||
results_path=self.results_path,
|
||||
module_options=self.module_options,
|
||||
log=module_logger,
|
||||
log=get_module_logger(module),
|
||||
)
|
||||
# Dependencies are keyed by the module class they declare, even
|
||||
# when it was replaced, so that a module asking for the results
|
||||
@@ -753,10 +744,12 @@ class Command:
|
||||
dependency: executed_by_type[resolved]
|
||||
for dependency, resolved in self._module_dependencies(module)
|
||||
}
|
||||
m.resource_lock = self._resource_lock
|
||||
m.resource_cache = self._resource_cache
|
||||
|
||||
if iocs.total_ioc_count:
|
||||
m.indicators = iocs
|
||||
m.indicators.log = m.log
|
||||
if self.iocs.total_ioc_count:
|
||||
# IOC collections are shared read-only during module execution.
|
||||
m.indicators = self.iocs
|
||||
|
||||
if self.serial:
|
||||
m.serial = self.serial
|
||||
@@ -769,23 +762,215 @@ class Command:
|
||||
try:
|
||||
run_module(m)
|
||||
except EncryptedBackupError:
|
||||
self.log.critical(
|
||||
"The backup appears to be encrypted. "
|
||||
"Please decrypt it first using `mvt-ios decrypt-backup`."
|
||||
)
|
||||
return
|
||||
encrypted = True
|
||||
finally:
|
||||
if token is not None:
|
||||
records = finish_module_log_buffer(token)
|
||||
|
||||
self.executed.append(m)
|
||||
executed_by_type[module] = m
|
||||
self.timeline.extend(m.timeline)
|
||||
self.url_results.extend(m.url_results)
|
||||
self.alertstore.extend(m.alertstore.alerts)
|
||||
return m, records, encrypted
|
||||
|
||||
@staticmethod
|
||||
def _console_handler() -> Optional[MVTLogHandler]:
|
||||
for handler in reversed(logging.getLogger("mvt").handlers):
|
||||
if isinstance(handler, MVTLogHandler):
|
||||
return handler
|
||||
return None
|
||||
|
||||
def _publish_completed_modules(
|
||||
self,
|
||||
ordered_modules: list[type[MVTModule]],
|
||||
completed: dict[type[MVTModule], MVTModule],
|
||||
) -> None:
|
||||
"""Expose a stable snapshot of completed modules to initialization hooks."""
|
||||
self.executed = [
|
||||
completed[module] for module in ordered_modules if module in completed
|
||||
]
|
||||
|
||||
def _aggregate_modules(
|
||||
self,
|
||||
ordered_modules: list[type[MVTModule]],
|
||||
completed: dict[type[MVTModule], MVTModule],
|
||||
) -> None:
|
||||
"""Aggregate results in stable topological order."""
|
||||
for module in ordered_modules:
|
||||
if module not in completed:
|
||||
continue
|
||||
instance = completed[module]
|
||||
self.timeline.extend(instance.timeline)
|
||||
self.url_results.extend(instance.url_results)
|
||||
self.alertstore.extend(instance.alertstore.alerts)
|
||||
|
||||
def _run_sequential(
|
||||
self, ordered_modules: list[type[MVTModule]]
|
||||
) -> tuple[dict[type[MVTModule], MVTModule], bool]:
|
||||
completed: dict[type[MVTModule], MVTModule] = {}
|
||||
for module in ordered_modules:
|
||||
instance, _, encrypted = self._run_module(module, completed, False)
|
||||
if encrypted:
|
||||
return completed, True
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
return completed, False
|
||||
|
||||
def _run_parallel(
|
||||
self, ordered_modules: list[type[MVTModule]], jobs: int
|
||||
) -> tuple[dict[type[MVTModule], MVTModule], bool]:
|
||||
"""Run a stable module DAG with bounded worker threads."""
|
||||
completed: dict[type[MVTModule], MVTModule] = {}
|
||||
scheduled: set[type[MVTModule]] = set()
|
||||
running: dict[Future, type[MVTModule]] = {}
|
||||
completion_queue: queue.Queue[Future] = queue.Queue()
|
||||
handler = self._console_handler()
|
||||
status = None
|
||||
fatal = False
|
||||
|
||||
def ready_modules() -> list[type[MVTModule]]:
|
||||
return [
|
||||
module
|
||||
for module in ordered_modules
|
||||
if module not in scheduled
|
||||
and all(
|
||||
dependency in completed
|
||||
for _, dependency in self._module_dependencies(module)
|
||||
)
|
||||
]
|
||||
|
||||
def update_status() -> None:
|
||||
if status is None:
|
||||
return
|
||||
names = ", ".join(module.__name__ for module in running.values())
|
||||
status.update(
|
||||
f"Modules: {len(completed)}/{len(ordered_modules)} complete"
|
||||
+ (f"; running {names}" if names else "")
|
||||
)
|
||||
|
||||
if handler and handler.console.is_terminal:
|
||||
status = handler.console.status("")
|
||||
status.start()
|
||||
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=jobs) as executor:
|
||||
while len(completed) < len(ordered_modules):
|
||||
ready = ready_modules()
|
||||
|
||||
# An unsafe module is a scheduler barrier: it starts only
|
||||
# after active workers drain and blocks later ready work.
|
||||
if ready and not ready[0].parallel_safe:
|
||||
if running:
|
||||
future = completion_queue.get()
|
||||
module = running.pop(future)
|
||||
instance, records, encrypted = future.result()
|
||||
if handler:
|
||||
handler.emit_module_records(module.__name__, records)
|
||||
if encrypted:
|
||||
fatal = True
|
||||
break
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
update_status()
|
||||
continue
|
||||
|
||||
module = ready[0]
|
||||
scheduled.add(module)
|
||||
if status is not None:
|
||||
status.update(
|
||||
f"Modules: {len(completed)}/{len(ordered_modules)} "
|
||||
f"complete; running {module.__name__}"
|
||||
)
|
||||
instance, records, encrypted = self._run_module(
|
||||
module, completed, True
|
||||
)
|
||||
if handler:
|
||||
handler.emit_module_records(module.__name__, records)
|
||||
if encrypted:
|
||||
fatal = True
|
||||
break
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
update_status()
|
||||
continue
|
||||
|
||||
for module in ready:
|
||||
if len(running) >= jobs or not module.parallel_safe:
|
||||
break
|
||||
scheduled.add(module)
|
||||
future = executor.submit(
|
||||
self._run_module, module, dict(completed), True
|
||||
)
|
||||
running[future] = module
|
||||
future.add_done_callback(completion_queue.put)
|
||||
|
||||
update_status()
|
||||
if not running:
|
||||
break
|
||||
|
||||
future = completion_queue.get()
|
||||
module = running.pop(future)
|
||||
instance, records, encrypted = future.result()
|
||||
if handler:
|
||||
handler.emit_module_records(module.__name__, records)
|
||||
if encrypted:
|
||||
fatal = True
|
||||
break
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
update_status()
|
||||
|
||||
if fatal:
|
||||
for future in running:
|
||||
future.cancel()
|
||||
# The executor context waits for already-active work. Its
|
||||
# results are intentionally discarded after the fatal error.
|
||||
finally:
|
||||
if status is not None:
|
||||
status.stop()
|
||||
|
||||
return completed, fatal
|
||||
|
||||
def run(self) -> None:
|
||||
self._create_storage()
|
||||
self._setup_logging()
|
||||
|
||||
ordered_modules = self._ordered_modules()
|
||||
if ordered_modules is None:
|
||||
return
|
||||
|
||||
# Load shared indicators before touching the acquisition.
|
||||
self.iocs
|
||||
|
||||
try:
|
||||
self.init()
|
||||
except NotImplementedError:
|
||||
pass
|
||||
|
||||
self._log_loaded_modules(ordered_modules)
|
||||
|
||||
jobs = self.jobs
|
||||
if settings.PROFILE and jobs > 1:
|
||||
self.log.warning(
|
||||
"MVT_PROFILE is enabled; forcing sequential module execution."
|
||||
)
|
||||
jobs = 1
|
||||
|
||||
if jobs == 1:
|
||||
completed, fatal = self._run_sequential(ordered_modules)
|
||||
else:
|
||||
completed, fatal = self._run_parallel(ordered_modules, jobs)
|
||||
|
||||
self._aggregate_modules(ordered_modules, completed)
|
||||
|
||||
try:
|
||||
self.finish()
|
||||
except NotImplementedError:
|
||||
pass
|
||||
|
||||
if fatal:
|
||||
self.log.critical(
|
||||
"The backup appears to be encrypted. "
|
||||
"Please decrypt it first using `mvt-ios decrypt-backup`."
|
||||
)
|
||||
return
|
||||
|
||||
# We only store the timeline from the parent/main command
|
||||
if self.sub_command:
|
||||
return
|
||||
|
||||
@@ -17,6 +17,7 @@ HELP_MSG_LOAD_MODULE = (
|
||||
HELP_MSG_NONINTERACTIVE = "Don't ask interactive questions during processing"
|
||||
HELP_MSG_HASHES = "Generate hashes of all the files analyzed"
|
||||
HELP_MSG_VERBOSE = "Verbose mode"
|
||||
HELP_MSG_JOBS = "Number of extraction modules to run concurrently"
|
||||
HELP_MSG_VERBOSE_COMMAND = (
|
||||
"Verbose mode (kept for compatibility, pass --verbose before the command "
|
||||
"name instead)"
|
||||
|
||||
+41
-1
@@ -4,6 +4,8 @@
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from contextvars import ContextVar, Token
|
||||
from rich.console import Console
|
||||
from rich.logging import RichHandler
|
||||
from typing import Optional
|
||||
@@ -27,16 +29,33 @@ logging.addLevelName(HIGH_ALERT, "HIGH")
|
||||
logging.addLevelName(CRITICAL_ALERT, "CRITICAL")
|
||||
|
||||
|
||||
_module_log_buffer: ContextVar[Optional[list[logging.LogRecord]]] = ContextVar(
|
||||
"mvt_module_log_buffer", default=None
|
||||
)
|
||||
|
||||
|
||||
def start_module_log_buffer() -> Token:
|
||||
"""Capture console records in the current module execution context."""
|
||||
return _module_log_buffer.set([])
|
||||
|
||||
|
||||
def finish_module_log_buffer(token: Token) -> list[logging.LogRecord]:
|
||||
records = _module_log_buffer.get() or []
|
||||
_module_log_buffer.reset(token)
|
||||
return records
|
||||
|
||||
|
||||
class MVTLogHandler(RichHandler):
|
||||
def __init__(self, console: Optional[Console] = None, level: int = logging.DEBUG):
|
||||
super().__init__(console=console, level=level)
|
||||
self._console_lock = threading.RLock()
|
||||
|
||||
def __add_prefix_space(self, level: str) -> str:
|
||||
max_length = len("CRITICAL ALERT")
|
||||
space = max_length - len(level)
|
||||
return f"{level}{' ' * space}"
|
||||
|
||||
def emit(self, record: logging.LogRecord):
|
||||
def _emit_record(self, record: logging.LogRecord) -> None:
|
||||
try:
|
||||
msg = rf"[grey50]\[{record.name}][/] {self.format(record)}"
|
||||
|
||||
@@ -63,3 +82,24 @@ class MVTLogHandler(RichHandler):
|
||||
|
||||
except Exception:
|
||||
self.handleError(record)
|
||||
|
||||
def emit(self, record: logging.LogRecord):
|
||||
buffer = _module_log_buffer.get()
|
||||
if buffer is not None:
|
||||
# More than one CLI can be imported in-process (notably in tests),
|
||||
# which can install multiple console handlers for the same record.
|
||||
if not buffer or buffer[-1] is not record:
|
||||
buffer.append(record)
|
||||
return
|
||||
|
||||
with self._console_lock:
|
||||
self._emit_record(record)
|
||||
|
||||
def emit_module_records(
|
||||
self, module_name: str, records: list[logging.LogRecord]
|
||||
) -> None:
|
||||
"""Print one module's captured records without interleaving."""
|
||||
with self._console_lock:
|
||||
self.console.print(f"[bold cyan]--- {module_name} ---[/bold cyan]")
|
||||
for record in records:
|
||||
self._emit_record(record)
|
||||
@@ -8,6 +8,7 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from dataclasses import asdict, is_dataclass
|
||||
from typing import Any, Dict, Optional, Sequence
|
||||
|
||||
@@ -46,6 +47,7 @@ class MVTModule:
|
||||
slug: Optional[str] = None
|
||||
dependencies: Sequence[type["MVTModule"]] = ()
|
||||
supported_commands: Sequence[tuple[str, str]] = ()
|
||||
parallel_safe: bool = True
|
||||
# A custom module can name a module class it supersedes, usually a
|
||||
# built-in one. When both are available to a command, the named module is
|
||||
# dropped from the run and this module takes its place, including in the
|
||||
@@ -90,6 +92,12 @@ class MVTModule:
|
||||
self.timeline: ModuleTimeline = []
|
||||
self.url_results: list[URLResult] = []
|
||||
self.dependency_modules: Dict[type["MVTModule"], "MVTModule"] = {}
|
||||
# Commands replace this with a command-scoped lock for modules sharing
|
||||
# archive handles or other non-thread-safe read resources.
|
||||
self.resource_lock = threading.RLock()
|
||||
# Commands also replace this with a command-scoped cache. Cached values
|
||||
# must be immutable, because parallel modules can read them concurrently.
|
||||
self.resource_cache: Dict[object, Any] = {}
|
||||
|
||||
def get_dependency_results(
|
||||
self, module_class: type["MVTModule"]
|
||||
@@ -185,17 +193,20 @@ class MVTModule:
|
||||
|
||||
"""
|
||||
timeline_set = set()
|
||||
deduplicated = []
|
||||
for record in timeline:
|
||||
timeline_set.add(
|
||||
json.dumps(
|
||||
asdict(record)
|
||||
if is_dataclass(record) and not isinstance(record, type)
|
||||
else record,
|
||||
sort_keys=True,
|
||||
)
|
||||
serialized = json.dumps(
|
||||
asdict(record)
|
||||
if is_dataclass(record) and not isinstance(record, type)
|
||||
else record,
|
||||
sort_keys=True,
|
||||
)
|
||||
if serialized in timeline_set:
|
||||
continue
|
||||
timeline_set.add(serialized)
|
||||
deduplicated.append(json.loads(serialized))
|
||||
|
||||
return [json.loads(record) for record in timeline_set]
|
||||
return deduplicated
|
||||
|
||||
def to_timeline(self) -> None:
|
||||
"""Convert results into a timeline."""
|
||||
|
||||
@@ -36,6 +36,7 @@ from mvt.common.help import (
|
||||
HELP_MSG_LOAD_MODULE,
|
||||
HELP_MSG_MODULE,
|
||||
HELP_MSG_VERBOSE,
|
||||
HELP_MSG_JOBS,
|
||||
HELP_MSG_VERBOSE_COMMAND,
|
||||
HELP_MSG_CHECK_FS,
|
||||
HELP_MSG_CHECK_IOCS,
|
||||
@@ -279,6 +280,9 @@ def extract_key(password, key_file, backup_path):
|
||||
)
|
||||
@click.option("--hashes", "-H", is_flag=True, help=HELP_MSG_HASHES)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("BACKUP_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_backup(
|
||||
@@ -291,6 +295,7 @@ def check_backup(
|
||||
load_module,
|
||||
hashes,
|
||||
verbose,
|
||||
jobs,
|
||||
backup_path,
|
||||
):
|
||||
from .cmd_check_backup import CmdIOSCheckBackup
|
||||
@@ -309,6 +314,7 @@ def check_backup(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -348,6 +354,9 @@ def check_backup(
|
||||
)
|
||||
@click.option("--hashes", "-H", is_flag=True, help=HELP_MSG_HASHES)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("DUMP_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_fs(
|
||||
@@ -360,6 +369,7 @@ def check_fs(
|
||||
load_module,
|
||||
hashes,
|
||||
verbose,
|
||||
jobs,
|
||||
dump_path,
|
||||
):
|
||||
from .cmd_check_fs import CmdIOSCheckFS
|
||||
@@ -378,6 +388,7 @@ def check_fs(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
|
||||
@@ -38,6 +38,7 @@ class CmdIOSCheckBackup(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -53,6 +54,7 @@ class CmdIOSCheckBackup(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "ios"
|
||||
|
||||
@@ -31,6 +31,7 @@ class CmdIOSCheckFS(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -45,6 +46,7 @@ class CmdIOSCheckFS(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "ios"
|
||||
|
||||
@@ -1,261 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2021-2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from mvt.android.modules.androidqf.browser_history import (
|
||||
BrowserHistory as AndroidQFBrowserHistory,
|
||||
)
|
||||
from mvt.android.modules.fs.browser_history import BrowserHistory as FSBrowserHistory
|
||||
from mvt.android.cmd_check_fs import CmdAndroidCheckFS
|
||||
|
||||
CHROME_TIME = 13_348_540_800_000_000
|
||||
URL = "https://example.org/path"
|
||||
|
||||
|
||||
def create_history_database(path: Path, *, url: str = URL) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with sqlite3.connect(path) as connection:
|
||||
connection.executescript(
|
||||
"""
|
||||
CREATE TABLE urls (
|
||||
id INTEGER PRIMARY KEY,
|
||||
url TEXT,
|
||||
title TEXT,
|
||||
visit_count INTEGER,
|
||||
typed_count INTEGER
|
||||
);
|
||||
CREATE TABLE visits (
|
||||
id INTEGER PRIMARY KEY,
|
||||
url INTEGER,
|
||||
visit_time INTEGER,
|
||||
from_visit INTEGER,
|
||||
transition INTEGER
|
||||
);
|
||||
"""
|
||||
)
|
||||
connection.execute("INSERT INTO urls VALUES (1, ?, 'Example', 1, 0)", (url,))
|
||||
connection.execute(
|
||||
"INSERT INTO visits VALUES (7, 1, ?, 0, 805306368)",
|
||||
(CHROME_TIME,),
|
||||
)
|
||||
|
||||
|
||||
def manifest(database_paths: list[tuple[str, str]]) -> dict:
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"status": "collected",
|
||||
"databases": [
|
||||
{
|
||||
"browser": "Chrome",
|
||||
"package": package,
|
||||
"profile": "Default",
|
||||
"device_path": f"/data/data/{package}/app_chrome/Default/History",
|
||||
"archive_path": archive_path,
|
||||
"sidecars": [],
|
||||
}
|
||||
for package, archive_path in database_paths
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("use_zip", [False, True])
|
||||
def test_androidqf_browser_history_directory_and_zip(tmp_path, use_zip):
|
||||
source_database = tmp_path / "source" / "History"
|
||||
create_history_database(source_database)
|
||||
archive_path = "browser_history/com.android.chrome/Default/History"
|
||||
manifest_data = manifest([("com.android.chrome", archive_path)])
|
||||
|
||||
module = AndroidQFBrowserHistory()
|
||||
if use_zip:
|
||||
acquisition_path = tmp_path / "acquisition.zip"
|
||||
with zipfile.ZipFile(acquisition_path, "w") as archive:
|
||||
archive.write(source_database, archive_path)
|
||||
archive.writestr("browser_history/manifest.json", json.dumps(manifest_data))
|
||||
with zipfile.ZipFile(acquisition_path) as archive:
|
||||
module.from_zip(archive, archive.namelist())
|
||||
module.run()
|
||||
else:
|
||||
acquisition = tmp_path / "acquisition"
|
||||
database_path = acquisition / archive_path
|
||||
database_path.parent.mkdir(parents=True)
|
||||
database_path.write_bytes(source_database.read_bytes())
|
||||
manifest_path = acquisition / "browser_history" / "manifest.json"
|
||||
manifest_path.write_text(json.dumps(manifest_data))
|
||||
files = [
|
||||
path.relative_to(tmp_path).as_posix()
|
||||
for path in acquisition.rglob("*")
|
||||
if path.is_file()
|
||||
]
|
||||
module.from_dir(str(tmp_path), files)
|
||||
module.run()
|
||||
|
||||
assert len(module.results) == 1
|
||||
assert module.results[0]["url"] == URL
|
||||
assert module.results[0]["browser"] == "Chrome"
|
||||
assert module.results[0]["source_path"].endswith("/History")
|
||||
module.collect_url_results()
|
||||
module.to_timeline()
|
||||
assert module.url_results[0]["url"] == URL
|
||||
assert module.timeline[0]["event"] == "browser_history"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("prefix", ["", "./", "acquisition/", "./acquisition/"])
|
||||
@pytest.mark.parametrize("separator", ["/", "\\"])
|
||||
def test_androidqf_zip_paths_preserve_wal_visits(tmp_path, prefix, separator):
|
||||
database = tmp_path / "History"
|
||||
create_history_database(database)
|
||||
connection = sqlite3.connect(database)
|
||||
connection.execute("PRAGMA journal_mode=WAL")
|
||||
connection.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
||||
connection.execute(
|
||||
"INSERT INTO urls VALUES (2, ?, 'WAL', 1, 0)", (URL + "/wal",)
|
||||
)
|
||||
connection.execute(
|
||||
"INSERT INTO visits VALUES (8, 2, ?, 0, 805306368)", (CHROME_TIME,)
|
||||
)
|
||||
connection.commit()
|
||||
archive_path = "browser_history/com.android.chrome/Default/History"
|
||||
acquisition = tmp_path / "acquisition.zip"
|
||||
try:
|
||||
with zipfile.ZipFile(acquisition, "w") as archive:
|
||||
for suffix in ("", "-wal", "-shm"):
|
||||
archive.writestr(
|
||||
(prefix + archive_path + suffix).replace("/", separator),
|
||||
Path(f"{database}{suffix}").read_bytes(),
|
||||
)
|
||||
archive.writestr(
|
||||
(prefix + "browser_history/manifest.json").replace("/", separator),
|
||||
json.dumps(manifest([("com.android.chrome", archive_path)])),
|
||||
)
|
||||
module = AndroidQFBrowserHistory()
|
||||
with zipfile.ZipFile(acquisition) as archive:
|
||||
module.from_zip(archive, archive.namelist())
|
||||
module.run()
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
assert {result["url"] for result in module.results} == {URL, URL + "/wal"}
|
||||
|
||||
|
||||
def test_androidqf_browser_history_isolates_malformed_database(tmp_path, caplog):
|
||||
acquisition = tmp_path / "acquisition"
|
||||
good_archive_path = "browser_history/com.android.chrome/Default/History"
|
||||
bad_archive_path = "browser_history/com.brave.browser/Default/History"
|
||||
create_history_database(acquisition / good_archive_path)
|
||||
bad_path = acquisition / bad_archive_path
|
||||
bad_path.parent.mkdir(parents=True)
|
||||
bad_path.write_bytes(b"not sqlite")
|
||||
manifest_path = acquisition / "browser_history" / "manifest.json"
|
||||
manifest_path.write_text(
|
||||
json.dumps(
|
||||
manifest(
|
||||
[
|
||||
("com.brave.browser", bad_archive_path),
|
||||
("com.android.chrome", good_archive_path),
|
||||
]
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
module = AndroidQFBrowserHistory()
|
||||
files = [
|
||||
path.relative_to(tmp_path).as_posix()
|
||||
for path in acquisition.rglob("*")
|
||||
if path.is_file()
|
||||
]
|
||||
module.from_dir(str(tmp_path), files)
|
||||
module.run()
|
||||
|
||||
assert [result["url"] for result in module.results] == [URL]
|
||||
assert "Unable to parse browser history database" in caplog.text
|
||||
|
||||
|
||||
def test_androidqf_browser_history_rejects_unsafe_manifest_path(tmp_path, caplog):
|
||||
acquisition = tmp_path / "acquisition"
|
||||
manifest_path = acquisition / "browser_history" / "manifest.json"
|
||||
manifest_path.parent.mkdir(parents=True)
|
||||
manifest_path.write_text(
|
||||
json.dumps(manifest([("com.android.chrome", "browser_history/../secret")]))
|
||||
)
|
||||
module = AndroidQFBrowserHistory()
|
||||
module.from_dir(str(tmp_path), [manifest_path.relative_to(tmp_path).as_posix()])
|
||||
|
||||
module.run()
|
||||
|
||||
assert module.results == []
|
||||
assert "unsafe browser history archive path" in caplog.text
|
||||
|
||||
|
||||
def test_filesystem_browser_history_reads_wal_only_visit(tmp_path):
|
||||
database_path = tmp_path / "History"
|
||||
connection = sqlite3.connect(database_path)
|
||||
connection.execute("PRAGMA journal_mode=WAL")
|
||||
connection.executescript(
|
||||
"""
|
||||
CREATE TABLE urls (
|
||||
id INTEGER PRIMARY KEY,
|
||||
url TEXT,
|
||||
title TEXT,
|
||||
visit_count INTEGER,
|
||||
typed_count INTEGER
|
||||
);
|
||||
CREATE TABLE visits (
|
||||
id INTEGER PRIMARY KEY,
|
||||
url INTEGER,
|
||||
visit_time INTEGER,
|
||||
from_visit INTEGER,
|
||||
transition INTEGER
|
||||
);
|
||||
"""
|
||||
)
|
||||
connection.commit()
|
||||
connection.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
||||
connection.execute("INSERT INTO urls VALUES (1, ?, 'WAL', 1, 0)", (URL,))
|
||||
connection.execute(
|
||||
"INSERT INTO visits VALUES (8, 1, ?, 0, 805306368)", (CHROME_TIME,)
|
||||
)
|
||||
connection.commit()
|
||||
try:
|
||||
assert Path(f"{database_path}-wal").stat().st_size > 0
|
||||
module = FSBrowserHistory(target_path=str(database_path))
|
||||
module.run()
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
assert [result["url"] for result in module.results] == [URL]
|
||||
assert module.results[0]["browser"] == "Chromium"
|
||||
|
||||
|
||||
def test_android_check_fs_finds_all_supported_browser_paths(tmp_path):
|
||||
for index, relative_path in enumerate(
|
||||
(
|
||||
"data/data/com.android.chrome/app_chrome/Default/History",
|
||||
"data/data/com.brave.browser/app_chrome/Default/History",
|
||||
"data/data/com.microsoft.emmx/app_chrome/Default/History",
|
||||
"data/data/com.sec.android.app.sbrowser/app_sbrowser/Default/History",
|
||||
)
|
||||
):
|
||||
create_history_database(
|
||||
tmp_path / relative_path, url=f"https://example.org/{index}"
|
||||
)
|
||||
|
||||
command = CmdAndroidCheckFS(target_path=str(tmp_path), module_name="BrowserHistory")
|
||||
command.run()
|
||||
|
||||
assert len(command.executed) == 1
|
||||
assert {result["package"] for result in command.executed[0].results} == {
|
||||
"com.android.chrome",
|
||||
"com.brave.browser",
|
||||
"com.microsoft.emmx",
|
||||
"com.sec.android.app.sbrowser",
|
||||
}
|
||||
assert len(command.url_results) == 4
|
||||
assert len(command.timeline) == 4
|
||||
@@ -4,8 +4,13 @@
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from mvt.android.modules.bugreport.base import BugReportModule
|
||||
from mvt.android.modules.bugreport.dumpsys_appops import DumpsysAppops
|
||||
from mvt.android.modules.bugreport.dumpsys_getprop import DumpsysGetProp
|
||||
from mvt.android.modules.bugreport.dumpsys_packages import DumpsysPackages
|
||||
@@ -118,3 +123,53 @@ class TestBugreportAnalysis:
|
||||
assert len(m.results) == 2
|
||||
assert m.results[1]["pid"] == 3559
|
||||
assert m.results[0]["sources"]["text"]["parsed"] is True
|
||||
|
||||
def test_dumpstate_section_is_cached_across_parallel_modules(self):
|
||||
dumpstate = (
|
||||
b"header\n"
|
||||
b"------ SYSTEM PROPERTIES (getprop) ------\n"
|
||||
b"[persist.sys.timezone]: [Europe/Berlin]\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
b"------\n"
|
||||
b"DUMP OF SERVICE package:\n"
|
||||
b"package data\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
b"DUMP OF SERVICE package:\n"
|
||||
b"duplicate package data\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
b"DUMP OF SERVICE adb:\n"
|
||||
b"adb data\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
)
|
||||
archive = MagicMock()
|
||||
archive.open.side_effect = lambda _: BytesIO(dumpstate)
|
||||
shared_lock = threading.RLock()
|
||||
shared_cache = {}
|
||||
modules = []
|
||||
|
||||
for _ in range(4):
|
||||
module = BugReportModule()
|
||||
module.from_zip(archive, ["dumpState_test.log"])
|
||||
module.resource_lock = shared_lock
|
||||
module.resource_cache = shared_cache
|
||||
modules.append(module)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=4) as executor:
|
||||
sections = list(
|
||||
executor.map(
|
||||
lambda module: module._get_dumpsys_section(
|
||||
"DUMP OF SERVICE package:"
|
||||
),
|
||||
modules,
|
||||
)
|
||||
)
|
||||
|
||||
assert sections == ["package data"] * 4
|
||||
assert all(section is sections[0] for section in sections)
|
||||
assert modules[0]._get_dumpsys_section_bytes(b"DUMP OF SERVICE adb:") == (
|
||||
b"adb data"
|
||||
)
|
||||
assert modules[0]._get_system_properties_text() == (
|
||||
"[persist.sys.timezone]: [Europe/Berlin]"
|
||||
)
|
||||
archive.open.assert_called_once_with("dumpState_test.log")
|
||||
+304
-19
@@ -5,14 +5,22 @@
|
||||
|
||||
import json
|
||||
import logging
|
||||
from unittest.mock import patch
|
||||
import threading
|
||||
import time
|
||||
from io import StringIO
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
from rich.console import Console
|
||||
|
||||
from mvt.common.command import Command
|
||||
from mvt.common.config import settings
|
||||
from mvt.common.log import MVTLogHandler
|
||||
from mvt.common.module import EncryptedBackupError, MVTModule
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from mvt.common.indicators import Indicators
|
||||
from mvt.common.module import MVTModule
|
||||
|
||||
|
||||
class RecordingModule(MVTModule):
|
||||
@@ -298,6 +306,15 @@ class TestCommand:
|
||||
alerts = json.loads((tmp_path / "alerts.json").read_text())
|
||||
assert alerts[0]["event"]["payload"] == "\\xa8\\xa9"
|
||||
|
||||
def test_timeline_deduplication_preserves_first_seen_order(self):
|
||||
timeline = [
|
||||
{"timestamp": "same", "event": "first"},
|
||||
{"timestamp": "same", "event": "second"},
|
||||
{"timestamp": "same", "event": "first"},
|
||||
]
|
||||
|
||||
assert MVTModule._deduplicate_timeline(timeline) == timeline[:2]
|
||||
|
||||
def test_stores_collected_urls(self, tmp_path):
|
||||
cmd = RecordingCommand(results_path=str(tmp_path))
|
||||
cmd.modules = [URLRecordingModule]
|
||||
@@ -314,7 +331,7 @@ class TestCommand:
|
||||
]
|
||||
|
||||
def test_modules_run_in_stable_topological_order(self):
|
||||
cmd = RecordingCommand()
|
||||
cmd = RecordingCommand(jobs=1)
|
||||
cmd.modules = [ThirdModule, IndependentModule, SecondModule, FirstModule]
|
||||
|
||||
cmd.run()
|
||||
@@ -329,7 +346,7 @@ class TestCommand:
|
||||
assert second.results == ["first", "second"]
|
||||
|
||||
def test_selected_module_runs_transitive_dependencies(self):
|
||||
cmd = RecordingCommand(module_name="ThirdModule")
|
||||
cmd = RecordingCommand(module_name="ThirdModule", jobs=1)
|
||||
cmd.modules = [ThirdModule, SecondModule, FirstModule, IndependentModule]
|
||||
|
||||
cmd.run()
|
||||
@@ -340,6 +357,38 @@ class TestCommand:
|
||||
"ThirdModule",
|
||||
]
|
||||
|
||||
@pytest.mark.parametrize("jobs", [1, 2])
|
||||
@pytest.mark.parametrize("parallel_safe", [False, True])
|
||||
def test_initialization_hook_sees_completed_dependencies(self, jobs, parallel_safe):
|
||||
class Prerequisite(FirstModule):
|
||||
pass
|
||||
|
||||
Prerequisite.parallel_safe = parallel_safe
|
||||
|
||||
class Dependent(SecondModule):
|
||||
dependencies = (Prerequisite,)
|
||||
|
||||
def run(self):
|
||||
self.results = self.get_dependency_results(Prerequisite) + ["second"]
|
||||
|
||||
class HookCommand(RecordingCommand):
|
||||
def module_init(self, module):
|
||||
if isinstance(module, Dependent):
|
||||
self.seen = [
|
||||
executed.results
|
||||
for executed in self.executed
|
||||
if isinstance(executed, Prerequisite)
|
||||
]
|
||||
|
||||
cmd = HookCommand(jobs=jobs)
|
||||
cmd.modules = [Dependent, Prerequisite]
|
||||
|
||||
cmd.run()
|
||||
|
||||
assert cmd.seen == [["first"]]
|
||||
assert [type(module) for module in cmd.executed] == [Prerequisite, Dependent]
|
||||
assert cmd.executed[1].results == ["first", "second"]
|
||||
|
||||
def test_circular_dependency_warns_and_stops(self, caplog):
|
||||
class CircularOne(RecordingModule):
|
||||
pass
|
||||
@@ -372,7 +421,10 @@ class TestCommand:
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["IndependentModule", "FirstModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"IndependentModule",
|
||||
"FirstModule",
|
||||
]
|
||||
assert cmd.initialized
|
||||
assert "Module DependentModule will be SKIPPED" in caplog.text
|
||||
assert "depends on module UnavailableModule" in caplog.text
|
||||
@@ -476,7 +528,7 @@ class TestCommand:
|
||||
]
|
||||
|
||||
def test_selected_custom_module_runs(self):
|
||||
cmd = RecordingCommand(module_name="CustomIOSBackupModule")
|
||||
cmd = RecordingCommand(module_name="CustomIOSBackupModule", jobs=1)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.custom_modules = [CustomIOSBackupModule]
|
||||
@@ -496,7 +548,7 @@ class TestCommand:
|
||||
assert RecordingModule.run_order == []
|
||||
|
||||
def test_custom_module_dependencies_use_topological_order(self):
|
||||
cmd = RecordingCommand(module_name="CustomDependsOnBuiltin")
|
||||
cmd = RecordingCommand(module_name="CustomDependsOnBuiltin", jobs=1)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.modules = [SecondModule, FirstModule]
|
||||
@@ -506,6 +558,228 @@ class TestCommand:
|
||||
|
||||
assert RecordingModule.run_order == ["FirstModule", "CustomDependsOnBuiltin"]
|
||||
|
||||
def test_parallel_modules_overlap_and_respect_worker_limit(self):
|
||||
lock = threading.Lock()
|
||||
barrier = threading.Barrier(2)
|
||||
active = 0
|
||||
maximum = 0
|
||||
overlapped = []
|
||||
|
||||
class ParallelModule(RecordingModule):
|
||||
def run(self):
|
||||
nonlocal active, maximum
|
||||
with lock:
|
||||
active += 1
|
||||
maximum = max(maximum, active)
|
||||
try:
|
||||
barrier.wait(timeout=2)
|
||||
overlapped.append(self.__class__.__name__)
|
||||
time.sleep(0.02)
|
||||
finally:
|
||||
with lock:
|
||||
active -= 1
|
||||
|
||||
class ParallelOne(ParallelModule):
|
||||
pass
|
||||
|
||||
class ParallelTwo(ParallelModule):
|
||||
pass
|
||||
|
||||
class ParallelThree(ParallelModule):
|
||||
pass
|
||||
|
||||
cmd = RecordingCommand(jobs=2)
|
||||
cmd.modules = [ParallelOne, ParallelTwo, ParallelThree]
|
||||
cmd.run()
|
||||
|
||||
assert maximum == 2
|
||||
assert {"ParallelOne", "ParallelTwo"}.issubset(overlapped)
|
||||
|
||||
def test_dependencies_wait_and_transitive_dependencies_run_once(self):
|
||||
prerequisite_finished = threading.Event()
|
||||
runs = []
|
||||
|
||||
class Prerequisite(RecordingModule):
|
||||
def run(self):
|
||||
runs.append("prerequisite")
|
||||
self.results = ["ready"]
|
||||
prerequisite_finished.set()
|
||||
|
||||
class Dependent(RecordingModule):
|
||||
dependencies = (Prerequisite,)
|
||||
|
||||
def run(self):
|
||||
assert prerequisite_finished.is_set()
|
||||
assert self.get_dependency_results(Prerequisite) == ["ready"]
|
||||
runs.append(self.__class__.__name__)
|
||||
|
||||
class DependentOne(Dependent):
|
||||
pass
|
||||
|
||||
class DependentTwo(Dependent):
|
||||
pass
|
||||
|
||||
cmd = RecordingCommand(jobs=3)
|
||||
cmd.modules = [DependentOne, DependentTwo, Prerequisite]
|
||||
cmd.run()
|
||||
|
||||
assert runs.count("prerequisite") == 1
|
||||
assert set(runs[1:]) == {"DependentOne", "DependentTwo"}
|
||||
|
||||
def test_parallel_unsafe_module_runs_exclusively(self):
|
||||
lock = threading.Lock()
|
||||
active = 0
|
||||
unsafe_ran_exclusively = []
|
||||
|
||||
class SafeOne(RecordingModule):
|
||||
def run(self):
|
||||
nonlocal active
|
||||
with lock:
|
||||
active += 1
|
||||
time.sleep(0.03)
|
||||
with lock:
|
||||
active -= 1
|
||||
|
||||
class Unsafe(RecordingModule):
|
||||
parallel_safe = False
|
||||
|
||||
def run(self):
|
||||
nonlocal active
|
||||
with lock:
|
||||
unsafe_ran_exclusively.append(active == 0)
|
||||
active += 1
|
||||
time.sleep(0.01)
|
||||
with lock:
|
||||
active -= 1
|
||||
|
||||
class SafeTwo(SafeOne):
|
||||
pass
|
||||
|
||||
cmd = RecordingCommand(jobs=3)
|
||||
cmd.modules = [SafeOne, Unsafe, SafeTwo]
|
||||
cmd.run()
|
||||
|
||||
assert unsafe_ran_exclusively == [True]
|
||||
|
||||
def test_results_are_aggregated_in_topological_order(self):
|
||||
release_first = threading.Event()
|
||||
|
||||
class SlowFirst(RecordingModule):
|
||||
def run(self):
|
||||
release_first.wait(timeout=2)
|
||||
self.results = ["first"]
|
||||
self.timeline = [{"module": "first"}]
|
||||
self.alertstore.info("first", "", {"order": 1})
|
||||
|
||||
class FastSecond(RecordingModule):
|
||||
def run(self):
|
||||
self.results = ["second"]
|
||||
self.timeline = [{"module": "second"}]
|
||||
self.alertstore.info("second", "", {"order": 2})
|
||||
release_first.set()
|
||||
|
||||
cmd = RecordingCommand(jobs=2)
|
||||
cmd.modules = [SlowFirst, FastSecond]
|
||||
cmd.run()
|
||||
|
||||
assert [type(module) for module in cmd.executed] == [SlowFirst, FastSecond]
|
||||
assert [entry["module"] for entry in cmd.timeline] == ["first", "second"]
|
||||
assert [alert.message for alert in cmd.alertstore.alerts] == ["first", "second"]
|
||||
|
||||
def test_parallel_console_logs_are_grouped_and_file_log_is_complete(
|
||||
self, tmp_path
|
||||
):
|
||||
output = StringIO()
|
||||
handler = MVTLogHandler(
|
||||
console=Console(file=output, force_terminal=False, color_system=None)
|
||||
)
|
||||
handler.setFormatter(logging.Formatter("%(message)s"))
|
||||
logger = logging.getLogger("mvt")
|
||||
logger.setLevel(logging.DEBUG)
|
||||
logger.addHandler(handler)
|
||||
release_slow = threading.Event()
|
||||
|
||||
class SlowLog(RecordingModule):
|
||||
def run(self):
|
||||
self.log.info("slow-a")
|
||||
release_slow.wait(timeout=2)
|
||||
self.log.info("slow-b")
|
||||
|
||||
class FastLog(RecordingModule):
|
||||
def run(self):
|
||||
self.log.info("fast-a")
|
||||
self.log.info("fast-b")
|
||||
release_slow.set()
|
||||
|
||||
SlowLog.__module__ = "mvt.tests.parallel"
|
||||
FastLog.__module__ = "mvt.tests.parallel"
|
||||
try:
|
||||
cmd = RecordingCommand(results_path=str(tmp_path), jobs=2)
|
||||
cmd.modules = [SlowLog, FastLog]
|
||||
cmd.run()
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
console_output = output.getvalue()
|
||||
fast_block = (
|
||||
console_output.index("--- FastLog ---"),
|
||||
console_output.index("fast-a"),
|
||||
console_output.index("fast-b"),
|
||||
)
|
||||
slow_block = (
|
||||
console_output.index("--- SlowLog ---"),
|
||||
console_output.index("slow-a"),
|
||||
console_output.index("slow-b"),
|
||||
)
|
||||
assert fast_block == tuple(sorted(fast_block))
|
||||
assert slow_block == tuple(sorted(slow_block))
|
||||
assert fast_block[-1] < slow_block[0] or slow_block[-1] < fast_block[0]
|
||||
file_output = (tmp_path / "command.log").read_text()
|
||||
for message in ("slow-a", "slow-b", "fast-a", "fast-b"):
|
||||
assert message in file_output
|
||||
|
||||
def test_encrypted_backup_stops_scheduling_and_cleans_up(self, caplog):
|
||||
finish_called = []
|
||||
|
||||
class Encrypted(RecordingModule):
|
||||
def run(self):
|
||||
raise EncryptedBackupError
|
||||
|
||||
class NeverScheduled(RecordingModule):
|
||||
dependencies = (Encrypted,)
|
||||
|
||||
class CleanupCommand(RecordingCommand):
|
||||
def finish(self):
|
||||
finish_called.append(True)
|
||||
|
||||
cmd = CleanupCommand(jobs=2)
|
||||
cmd.modules = [Encrypted, NeverScheduled]
|
||||
with caplog.at_level(logging.CRITICAL):
|
||||
cmd.run()
|
||||
|
||||
assert finish_called == [True]
|
||||
assert not any(isinstance(module, NeverScheduled) for module in cmd.executed)
|
||||
assert "backup appears to be encrypted" in caplog.text
|
||||
|
||||
def test_profiling_forces_sequential_execution(self, monkeypatch, caplog):
|
||||
run_order = []
|
||||
|
||||
class ProfileOne(RecordingModule):
|
||||
def run(self):
|
||||
run_order.append("one")
|
||||
|
||||
class ProfileTwo(RecordingModule):
|
||||
def run(self):
|
||||
run_order.append("two")
|
||||
|
||||
monkeypatch.setattr(settings, "PROFILE", True)
|
||||
cmd = RecordingCommand(jobs=4)
|
||||
cmd.modules = [ProfileOne, ProfileTwo]
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert run_order == ["one", "two"]
|
||||
assert "forcing sequential module execution" in caplog.text
|
||||
def test_custom_module_replaces_builtin(self, caplog):
|
||||
cmd = RecordingCommand()
|
||||
cmd.platform = "ios"
|
||||
@@ -516,7 +790,10 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["IndependentModule", "ReplacementModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"IndependentModule",
|
||||
"ReplacementModule",
|
||||
]
|
||||
assert cmd.module_replacements == {FirstModule: ReplacementModule}
|
||||
assert (
|
||||
"Module ReplacementModule from" in caplog.text
|
||||
@@ -532,7 +809,10 @@ class TestCommand:
|
||||
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["FirstModule", "CustomIOSBackupModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"FirstModule",
|
||||
"CustomIOSBackupModule",
|
||||
]
|
||||
assert cmd.module_replacements == {}
|
||||
|
||||
def test_unavailable_replaced_module_is_ignored(self, caplog):
|
||||
@@ -545,7 +825,10 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["IndependentModule", "ReplacementModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"IndependentModule",
|
||||
"ReplacementModule",
|
||||
]
|
||||
assert cmd.module_replacements == {}
|
||||
assert "replaces module FirstModule" not in caplog.text
|
||||
|
||||
@@ -663,7 +946,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"ReplacementModule",
|
||||
"OtherReplacementModule",
|
||||
]
|
||||
@@ -722,8 +1005,9 @@ class TestCommand:
|
||||
assert cmd.module_replacements == {}
|
||||
assert logged_substitutions(caplog) == []
|
||||
|
||||
def test_modules_replacing_each_other_all_run(self, caplog):
|
||||
cmd = RecordingCommand()
|
||||
@pytest.mark.parametrize("jobs", [1, 4])
|
||||
def test_modules_replacing_each_other_all_run(self, caplog, jobs):
|
||||
cmd = RecordingCommand(jobs=jobs)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.custom_modules = [MutualReplacementOne, MutualReplacementTwo]
|
||||
@@ -731,7 +1015,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"MutualReplacementOne",
|
||||
"MutualReplacementTwo",
|
||||
]
|
||||
@@ -742,8 +1026,9 @@ class TestCommand:
|
||||
)
|
||||
assert logged_substitutions(caplog) == []
|
||||
|
||||
def test_replacement_cycle_of_three_modules_all_run(self, caplog):
|
||||
cmd = RecordingCommand()
|
||||
@pytest.mark.parametrize("jobs", [1, 4])
|
||||
def test_replacement_cycle_of_three_modules_all_run(self, caplog, jobs):
|
||||
cmd = RecordingCommand(jobs=jobs)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.custom_modules = [CycleOneModule, CycleTwoModule, CycleThreeModule]
|
||||
@@ -751,7 +1036,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"CycleOneModule",
|
||||
"CycleTwoModule",
|
||||
"CycleThreeModule",
|
||||
@@ -834,7 +1119,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"SharedSlugModule",
|
||||
"OtherSharedSlugModule",
|
||||
]
|
||||
|
||||
@@ -41,3 +41,11 @@ class TestCheckBackupCommand:
|
||||
result = runner.invoke(check_backup, [str(tmp_path)])
|
||||
assert result.exit_code == 1
|
||||
assert "does not appear to be an iTunes backup folder" in caplog.text
|
||||
|
||||
def test_check_rejects_non_positive_jobs(self):
|
||||
result = CliRunner().invoke(
|
||||
check_backup, ["--jobs", "0", get_ios_backup_folder()]
|
||||
)
|
||||
|
||||
assert result.exit_code == 2
|
||||
assert "Invalid value for '--jobs'" in result.output
|
||||
@@ -288,15 +288,19 @@ def test_androidqf_propagates_custom_modules_to_nested_commands(tmp_path, monkey
|
||||
cmd = CmdAndroidCheckAndroidQF(
|
||||
target_path=str(tmp_path),
|
||||
custom_modules=custom_modules,
|
||||
jobs=7,
|
||||
)
|
||||
|
||||
def record_available(name):
|
||||
def _record(command):
|
||||
records[name] = [
|
||||
module.__name__
|
||||
for module in command._available_modules()
|
||||
if module.__name__.startswith("Nested")
|
||||
]
|
||||
records[name] = {
|
||||
"jobs": command.jobs,
|
||||
"modules": [
|
||||
module.__name__
|
||||
for module in command._available_modules()
|
||||
if module.__name__.startswith("Nested")
|
||||
],
|
||||
}
|
||||
|
||||
return _record
|
||||
|
||||
@@ -339,7 +343,7 @@ def test_androidqf_propagates_custom_modules_to_nested_commands(tmp_path, monkey
|
||||
assert cmd.run_backup_cmd()
|
||||
assert cmd.run_intrusion_logs_cmd()
|
||||
assert records == {
|
||||
"bugreport": ["NestedBugreportModule"],
|
||||
"backup": ["NestedBackupModule"],
|
||||
"intrusion_logs": ["NestedIntrusionLogsModule"],
|
||||
"bugreport": {"jobs": 7, "modules": ["NestedBugreportModule"]},
|
||||
"backup": {"jobs": 7, "modules": ["NestedBackupModule"]},
|
||||
"intrusion_logs": {"jobs": 7, "modules": ["NestedIntrusionLogsModule"]},
|
||||
}
|
||||
Reference in new issue
Block a user