mirror of
https://github.com/mvt-project/mvt.git
synced 2026-10-25 23:53:42 +01:00
Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d58abbb568 | ||
|
|
d52e9fcc74 | ||
|
|
59c3f5d60c | ||
|
|
2fe7f4e261 | ||
|
|
361bc72a4e | ||
|
|
9d71661f55 |
No files matched your search
@@ -86,6 +86,11 @@ writing and installing them, and the
|
||||
[custom CLI command documentation](https://docs.mvt.re/en/latest/development/custom_commands/)
|
||||
for the entry points a package registers commands in.
|
||||
|
||||
Extraction commands run independent modules concurrently with four workers by
|
||||
default. Use `--jobs 1` for sequential execution or another positive value to
|
||||
set the worker limit. Dependencies retain their declared ordering, and parallel
|
||||
console logs are printed as grouped module blocks.
|
||||
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -46,6 +46,33 @@ on it, and runs the rest of the analysis. Selecting such a module with
|
||||
A cycle in the dependency graph is a programming error rather than a
|
||||
configuration problem: the command logs a warning and runs no modules at all.
|
||||
|
||||
## Parallel module execution
|
||||
|
||||
Extraction modules run concurrently, using four worker threads by default. Use
|
||||
`--jobs INTEGER` to change the worker limit, or `--jobs 1` for sequential
|
||||
execution and live per-line logging:
|
||||
|
||||
```bash
|
||||
mvt-ios check-backup --jobs 8 --output ./out ./backup
|
||||
```
|
||||
|
||||
The scheduler starts a module only after all of its declared dependencies have
|
||||
completed. Results, alerts, and timeline entries are still aggregated in stable
|
||||
topological order. During parallel execution, console records from each module
|
||||
are buffered and printed together as a labeled block when that module finishes;
|
||||
`command.log` is written immediately and retains the complete log stream.
|
||||
|
||||
Parallel-safe modules must log through `self.log`, must not write directly to
|
||||
stdout or stderr, and must not mutate shared global state. A module that uses a
|
||||
non-thread-safe resource or direct terminal output can opt out:
|
||||
|
||||
```python
|
||||
class TerminalModule(MVTModule):
|
||||
parallel_safe = False
|
||||
```
|
||||
|
||||
Such modules run synchronously and exclusively after active workers finish.
|
||||
|
||||
## Custom modules
|
||||
|
||||
MVT's module-running `check-*` commands can run forensic modules which are not
|
||||
@@ -382,16 +409,11 @@ module came from:
|
||||
Some MVT modules extract and process significant amounts of data during the analysis process or while checking results against known indicators. Care must be
|
||||
take to avoid inefficient code paths as we add new modules.
|
||||
|
||||
MVT modules can be profiled with Python built-in `cProfile` by setting the `MVT_PROFILE` environment variable.
|
||||
MVT modules can be profiled with Python built-in `cProfile` by setting the `MVT_PROFILE` environment variable. Profiling forces sequential execution even
|
||||
when `--jobs` is greater than one.
|
||||
|
||||
```bash
|
||||
MVT_PROFILE=1 dev/mvt-ios check-backup test_backup
|
||||
```
|
||||
|
||||
Open an issue or PR if you are encountering significant performance issues when analyzing a device with MVT.
|
||||
|
||||
## Output schemas
|
||||
|
||||
Custom modules should declare an `output_model` using a Pydantic `RootModel`.
|
||||
This validates saved results and lets consumers obtain the module's JSON Schema.
|
||||
See [Output schemas](../output-schemas.md#custom-modules) for a complete example.
|
||||
+1
-1
@@ -66,4 +66,4 @@ docker run --rm -it \
|
||||
ghcr.io/mvt-project/mvt:latest-ios lockdown info
|
||||
```
|
||||
|
||||
If you built the image from source, replace `ghcr.io/mvt-project/mvt` with `mvt`.
|
||||
If you built the image from source, replace `ghcr.io/mvt-project/mvt` with `mvt`.
|
||||
@@ -1,77 +0,0 @@
|
||||
# Output schemas
|
||||
|
||||
MVT validates its JSON output with versioned Pydantic models. The schema version
|
||||
used for a run is recorded as `output_schema_version` in `info.json`.
|
||||
|
||||
MVT preserves the established on-disk format: most module files contain an array
|
||||
of result objects, while modules whose results are naturally grouped contain an
|
||||
object keyed by source or namespace. Detection files and `alerts.json` contain
|
||||
arrays of alert objects. Timestamps remain strings because their timezone and
|
||||
precision depend on the source artifact.
|
||||
|
||||
## Exporting JSON Schema
|
||||
|
||||
Both platform commands can print a versioned JSON Schema bundle:
|
||||
|
||||
```bash
|
||||
mvt-ios schemas
|
||||
mvt-android schemas
|
||||
```
|
||||
|
||||
Use `--output` to write one Draft 2020-12 JSON Schema file for each output:
|
||||
|
||||
```bash
|
||||
mvt-ios schemas --output ./mvt-ios-schemas
|
||||
mvt-android schemas --output ./mvt-android-schemas
|
||||
```
|
||||
|
||||
## Python API
|
||||
|
||||
Models and schema discovery functions are available from `mvt.schemas`:
|
||||
|
||||
```python
|
||||
from mvt.schemas import get_output_model
|
||||
|
||||
SafariHistoryOutput = get_output_model("safari_history", platform="ios")
|
||||
validated = SafariHistoryOutput.model_validate(records)
|
||||
json_schema = SafariHistoryOutput.model_json_schema()
|
||||
```
|
||||
|
||||
Common outputs have dedicated field-level models. Built-in module outputs have a
|
||||
declared root shape, and modules with an established dedicated record model expose
|
||||
its complete field schema.
|
||||
|
||||
## Custom modules
|
||||
|
||||
Custom modules can publish a precise contract by assigning a Pydantic root model
|
||||
to `output_model`:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, RootModel
|
||||
|
||||
from mvt.common.module import MVTModule
|
||||
|
||||
|
||||
class ExampleRecord(BaseModel):
|
||||
message: str
|
||||
timestamp: str | None = None
|
||||
|
||||
|
||||
class ExampleOutput(RootModel[list[ExampleRecord]]):
|
||||
pass
|
||||
|
||||
|
||||
class ExampleModule(MVTModule):
|
||||
output_model = ExampleOutput
|
||||
```
|
||||
|
||||
For compatibility, a custom module without `output_model` can still write an
|
||||
array or object containing JSON values. MVT logs a warning when it uses this
|
||||
generic contract. A future major release may require custom modules to declare
|
||||
their output models.
|
||||
|
||||
## Compatibility policy
|
||||
|
||||
Within one output schema major version, required fields are not removed and field
|
||||
types are not narrowed. New optional fields may be added. A breaking output
|
||||
change requires a new schema major version and a migration note.
|
||||
@@ -32,7 +32,6 @@ nav:
|
||||
- Introduction: "introduction.md"
|
||||
- Installation: "install.md"
|
||||
- Command Completion: "command_completion.md"
|
||||
- Output Schemas: "output-schemas.md"
|
||||
- Using Docker: "docker.md"
|
||||
- MVT for iOS:
|
||||
- iOS Forensic Methodology: "ios/methodology.md"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import datetime
|
||||
from typing import ClassVar, List, Optional
|
||||
from typing import List, Optional
|
||||
|
||||
import pydantic
|
||||
import betterproto2
|
||||
@@ -13,7 +13,6 @@ from dateutil import parser
|
||||
from mvt.android.parsers.proto.tombstone import Tombstone
|
||||
from mvt.common.module_types import ModuleAtomicResult, ModuleSerializedResult
|
||||
from mvt.common.utils import convert_datetime_to_iso
|
||||
from mvt.schemas import OutputModel
|
||||
|
||||
from .artifact import AndroidArtifact
|
||||
|
||||
@@ -74,10 +73,6 @@ class TombstoneCrashResult(pydantic.BaseModel):
|
||||
extra: Optional[str] = None
|
||||
|
||||
|
||||
class TombstoneCrashOutput(pydantic.RootModel[List[TombstoneCrashResult]]):
|
||||
"""Complete JSON document written by tombstone crash modules."""
|
||||
|
||||
|
||||
class TombstoneCrashArtifact(AndroidArtifact):
|
||||
"""
|
||||
Parser for Android tombstone crash files.
|
||||
@@ -85,8 +80,6 @@ class TombstoneCrashArtifact(AndroidArtifact):
|
||||
This parser can parse both text and protobuf tombstone crash files.
|
||||
"""
|
||||
|
||||
output_model: ClassVar[OutputModel] = TombstoneCrashOutput
|
||||
|
||||
def serialize(self, record: ModuleAtomicResult) -> ModuleSerializedResult:
|
||||
return {
|
||||
"timestamp": record["timestamp"],
|
||||
|
||||
+26
-21
@@ -28,6 +28,7 @@ from mvt.common.help import (
|
||||
HELP_MSG_DISABLE_UPDATE_CHECK,
|
||||
HELP_MSG_HASHES,
|
||||
HELP_MSG_IOC,
|
||||
HELP_MSG_JOBS,
|
||||
HELP_MSG_LIST_MODULES,
|
||||
HELP_MSG_LOAD_MODULE,
|
||||
HELP_MSG_MODULE,
|
||||
@@ -98,13 +99,12 @@ def cli(ctx, disable_update_check, disable_indicator_update_check, verbose):
|
||||
ctx.obj["verbose"] = verbose
|
||||
set_verbose_logging(verbose)
|
||||
|
||||
if ctx.invoked_subcommand not in ("completion", "schemas"):
|
||||
from mvt.common.logo import logo
|
||||
from mvt.common.logo import logo
|
||||
|
||||
logo(
|
||||
disable_version_check=disable_update_check,
|
||||
disable_indicator_check=disable_indicator_update_check,
|
||||
)
|
||||
logo(
|
||||
disable_version_check=disable_update_check,
|
||||
disable_indicator_check=disable_indicator_update_check,
|
||||
)
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
@@ -116,21 +116,6 @@ def version():
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Command: schemas
|
||||
# ==============================================================================
|
||||
@cli.command("schemas", help="Print or export the JSON Schemas for MVT outputs.")
|
||||
@click.option(
|
||||
"--output",
|
||||
"-o",
|
||||
type=click.Path(file_okay=False, dir_okay=True),
|
||||
help="Write one JSON Schema file per output to this directory.",
|
||||
)
|
||||
def schemas(output):
|
||||
from mvt.common.cmd_schemas import emit_schemas
|
||||
|
||||
emit_schemas("android", output)
|
||||
|
||||
|
||||
# Command: check-adb (removed)
|
||||
# ==============================================================================
|
||||
@cli.command(
|
||||
@@ -177,6 +162,9 @@ def check_adb(ctx):
|
||||
),
|
||||
)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("BUGREPORT_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_bugreport(
|
||||
@@ -188,6 +176,7 @@ def check_bugreport(
|
||||
load_module,
|
||||
timezone,
|
||||
verbose,
|
||||
jobs,
|
||||
bugreport_path,
|
||||
):
|
||||
from .cmd_check_bugreport import CmdAndroidCheckBugreport
|
||||
@@ -210,6 +199,7 @@ def check_bugreport(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -252,6 +242,9 @@ def check_bugreport(
|
||||
@click.option("--non-interactive", "-n", is_flag=True, help=HELP_MSG_NONINTERACTIVE)
|
||||
@click.option("--backup-password", "-p", help=HELP_MSG_ANDROID_BACKUP_PASSWORD)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("BACKUP_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_backup(
|
||||
@@ -263,6 +256,7 @@ def check_backup(
|
||||
non_interactive,
|
||||
backup_password,
|
||||
verbose,
|
||||
jobs,
|
||||
backup_path,
|
||||
):
|
||||
from .cmd_check_backup import CmdAndroidCheckBackup
|
||||
@@ -284,6 +278,7 @@ def check_backup(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -327,6 +322,9 @@ def check_backup(
|
||||
@click.option("--non-interactive", "-n", is_flag=True, help=HELP_MSG_NONINTERACTIVE)
|
||||
@click.option("--backup-password", "-p", help=HELP_MSG_ANDROID_BACKUP_PASSWORD)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("ANDROIDQF_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_androidqf(
|
||||
@@ -342,6 +340,7 @@ def check_androidqf(
|
||||
non_interactive,
|
||||
backup_password,
|
||||
verbose,
|
||||
jobs,
|
||||
androidqf_path,
|
||||
):
|
||||
from .cmd_check_androidqf import CmdAndroidCheckAndroidQF
|
||||
@@ -365,6 +364,7 @@ def check_androidqf(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -414,6 +414,9 @@ def check_androidqf(
|
||||
),
|
||||
)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("LOGS_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_intrusion_logs(
|
||||
@@ -425,6 +428,7 @@ def check_intrusion_logs(
|
||||
load_module,
|
||||
timezone,
|
||||
verbose,
|
||||
jobs,
|
||||
logs_path,
|
||||
):
|
||||
from .cmd_check_intrusion_logs import CmdAndroidCheckIntrusionLogs
|
||||
@@ -445,6 +449,7 @@ def check_intrusion_logs(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
|
||||
@@ -53,6 +53,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -68,6 +69,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
@@ -259,6 +261,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
hashes=self.hashes,
|
||||
sub_command=True,
|
||||
custom_modules=self.custom_modules,
|
||||
jobs=self.jobs,
|
||||
)
|
||||
cmd.from_zip(bugreport)
|
||||
cmd.run()
|
||||
@@ -290,6 +293,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
hashes=self.hashes,
|
||||
sub_command=True,
|
||||
custom_modules=self.custom_modules,
|
||||
jobs=self.jobs,
|
||||
)
|
||||
try:
|
||||
cmd.from_ab(backup)
|
||||
@@ -376,6 +380,7 @@ class CmdAndroidCheckAndroidQF(Command):
|
||||
hashes=self.hashes,
|
||||
sub_command=True,
|
||||
custom_modules=self.custom_modules,
|
||||
jobs=self.jobs,
|
||||
)
|
||||
cmd.run()
|
||||
|
||||
|
||||
@@ -47,6 +47,7 @@ class CmdAndroidCheckBackup(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -62,6 +63,7 @@ class CmdAndroidCheckBackup(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
|
||||
@@ -40,6 +40,7 @@ class CmdAndroidCheckBugreport(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -55,6 +56,7 @@ class CmdAndroidCheckBugreport(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
|
||||
@@ -37,6 +37,7 @@ class CmdAndroidCheckIntrusionLogs(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -52,6 +53,7 @@ class CmdAndroidCheckIntrusionLogs(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "android"
|
||||
|
||||
@@ -8,8 +8,6 @@ import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from rich.progress import track
|
||||
|
||||
from mvt.android.utils import (
|
||||
BROWSER_INSTALLERS,
|
||||
PLAY_STORE_INSTALLERS,
|
||||
@@ -153,10 +151,9 @@ class AQFPackages(AndroidQFModule):
|
||||
if total_hashes == 0:
|
||||
return
|
||||
|
||||
progress_desc = f"Looking up {total_hashes} package files on VirusTotal..."
|
||||
for index, file_hash in enumerate(
|
||||
track(files_by_hash, description=progress_desc)
|
||||
):
|
||||
self.log.info("Looking up %d package files on VirusTotal...", total_hashes)
|
||||
for index, file_hash in enumerate(files_by_hash):
|
||||
self.log.debug("VirusTotal lookup %d/%d", index + 1, total_hashes)
|
||||
try:
|
||||
results = virustotal_lookup(file_hash)
|
||||
except VTNoKey as exc:
|
||||
|
||||
@@ -4,7 +4,6 @@
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
from mvt.android.artifacts.settings import Settings as SettingsArtifact
|
||||
from mvt.schemas import RecordListOutput
|
||||
|
||||
from .base import AndroidQFModule
|
||||
|
||||
@@ -12,8 +11,6 @@ from .base import AndroidQFModule
|
||||
class AQFSettings(SettingsArtifact, AndroidQFModule):
|
||||
"""This module analyse setting files"""
|
||||
|
||||
output_model = RecordListOutput
|
||||
|
||||
def run(self) -> None:
|
||||
for setting_file in self._get_files_by_pattern("*/settings_*.txt"):
|
||||
namespace = setting_file[setting_file.rfind("_") + 1 : -4]
|
||||
|
||||
@@ -7,18 +7,15 @@ import fnmatch
|
||||
import logging
|
||||
import os
|
||||
import zipfile
|
||||
from typing import ClassVar, List, Optional
|
||||
from typing import List, Optional
|
||||
|
||||
from mvt.common.module import MVTModule
|
||||
from mvt.common.module_types import ModuleResults
|
||||
from mvt.schemas import OutputModel, RecordListOutput
|
||||
|
||||
|
||||
class AndroidQFModule(MVTModule):
|
||||
"""This class provides a base for all Android Data analysis modules."""
|
||||
|
||||
output_model: ClassVar[OutputModel] = RecordListOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
@@ -84,12 +81,13 @@ class AndroidQFModule(MVTModule):
|
||||
return None
|
||||
|
||||
def _get_file_content(self, file_path):
|
||||
if self.archive:
|
||||
handle = self.archive.open(file_path)
|
||||
else:
|
||||
handle = open(os.path.join(self.parent_path, file_path), "rb")
|
||||
with self.resource_lock:
|
||||
if self.archive:
|
||||
handle = self.archive.open(file_path)
|
||||
else:
|
||||
handle = open(os.path.join(self.parent_path, file_path), "rb")
|
||||
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
|
||||
return data
|
||||
return data
|
||||
@@ -7,17 +7,14 @@ import fnmatch
|
||||
import logging
|
||||
import os
|
||||
from tarfile import TarFile
|
||||
from typing import ClassVar, List, Optional
|
||||
from typing import List, Optional
|
||||
|
||||
from mvt.common.module import ModuleResults, MVTModule
|
||||
from mvt.schemas import OutputModel, RecordListOutput
|
||||
|
||||
|
||||
class BackupModule(MVTModule):
|
||||
"""This class provides a base for all backup extractios modules"""
|
||||
|
||||
output_model: ClassVar[OutputModel] = RecordListOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
@@ -58,21 +55,22 @@ class BackupModule(MVTModule):
|
||||
return fnmatch.filter(self.files, pattern)
|
||||
|
||||
def _get_file_content(self, file_path: str) -> bytes:
|
||||
handle = None
|
||||
if self.tar:
|
||||
try:
|
||||
member = self.tar.getmember(file_path)
|
||||
handle = self.tar.extractfile(member)
|
||||
if not handle:
|
||||
raise ValueError(f"Could not extract file: {file_path}")
|
||||
except KeyError:
|
||||
raise FileNotFoundError(f"File not found in tar: {file_path}")
|
||||
elif self.backup_path:
|
||||
handle = open(os.path.join(self.backup_path, file_path), "rb")
|
||||
else:
|
||||
raise ValueError("No backup path or tar file provided")
|
||||
with self.resource_lock:
|
||||
handle = None
|
||||
if self.tar:
|
||||
try:
|
||||
member = self.tar.getmember(file_path)
|
||||
handle = self.tar.extractfile(member)
|
||||
if not handle:
|
||||
raise ValueError(f"Could not extract file: {file_path}")
|
||||
except KeyError:
|
||||
raise FileNotFoundError(f"File not found in tar: {file_path}")
|
||||
elif self.backup_path:
|
||||
handle = open(os.path.join(self.backup_path, file_path), "rb")
|
||||
else:
|
||||
raise ValueError("No backup path or tar file provided")
|
||||
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
|
||||
return data
|
||||
return data
|
||||
@@ -4,16 +4,17 @@
|
||||
# https://github.com/mvt-project/mvt/blob/main/LICENSE
|
||||
import datetime
|
||||
import fnmatch
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import ClassVar, List, Optional
|
||||
from typing import List, Optional
|
||||
from zipfile import ZipFile
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
from mvt.android.artifacts.artifact import AndroidArtifact
|
||||
from mvt.common.module import ModuleResults, MVTModule
|
||||
from mvt.schemas import OutputModel, RecordListOutput
|
||||
|
||||
# `------ 0.101s was the duration of 'SOME SECTION' ------`, printed when that
|
||||
# section finishes and not necessarily between two sections.
|
||||
@@ -25,11 +26,27 @@ SECTION_DURATION = re.compile(r"^-{3,}\s*[0-9.]+s was the duration of", re.IGNOR
|
||||
DUMPSTATE_ENTRY_POINTS = ("main_entry.txt", "dumpState_*.log", "*/dumpsys.txt")
|
||||
|
||||
|
||||
_DUMPSTATE_BYTES_CACHE_KEY = ("android_bugreport", "dumpstate", "bytes")
|
||||
_DUMPSTATE_SECTIONS_CACHE_KEY = ("android_bugreport", "dumpstate", "sections")
|
||||
_SYSTEM_PROPERTIES_SECTION = b"------ SYSTEM PROPERTIES"
|
||||
_DUMPSYS_SEPARATORS = {
|
||||
b"DUMP OF SERVICE accessibility:",
|
||||
b"DUMP OF SERVICE adb:",
|
||||
b"DUMP OF SERVICE appops:",
|
||||
b"DUMP OF SERVICE batterystats:",
|
||||
b"DUMP OF SERVICE dbinfo:",
|
||||
b"DUMP OF SERVICE package:",
|
||||
b"DUMP OF SERVICE platform_compat:",
|
||||
}
|
||||
_DUMPSTATE_DELIMITER = (
|
||||
b"------------------------------------------------------------------------------"
|
||||
)
|
||||
_GETPROP_LINE_RE = re.compile(rb"\[(.+?)\]: \[(.+?)\]")
|
||||
|
||||
|
||||
class BugReportModule(MVTModule):
|
||||
"""This class provides a base for all Android Bug Report modules."""
|
||||
|
||||
output_model: ClassVar[OutputModel] = RecordListOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
@@ -80,45 +97,156 @@ class BugReportModule(MVTModule):
|
||||
return []
|
||||
|
||||
def _get_file_content(self, file_path: str) -> bytes:
|
||||
if self.zip_archive:
|
||||
handle = self.zip_archive.open(file_path)
|
||||
else:
|
||||
if not self.extract_path:
|
||||
raise ValueError("extract_path is not set")
|
||||
joined = os.path.join(self.extract_path, file_path)
|
||||
if (
|
||||
not Path(joined)
|
||||
.resolve()
|
||||
.is_relative_to(Path(self.extract_path).resolve())
|
||||
):
|
||||
raise ValueError("unsafe file_path")
|
||||
handle = open(joined, "rb")
|
||||
with self.resource_lock:
|
||||
if self.zip_archive:
|
||||
handle = self.zip_archive.open(file_path)
|
||||
else:
|
||||
if not self.extract_path:
|
||||
raise ValueError("extract_path is not set")
|
||||
joined = os.path.join(self.extract_path, file_path)
|
||||
if (
|
||||
not Path(joined)
|
||||
.resolve()
|
||||
.is_relative_to(Path(self.extract_path).resolve())
|
||||
):
|
||||
raise ValueError("unsafe file_path")
|
||||
handle = open(joined, "rb")
|
||||
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
data = handle.read()
|
||||
handle.close()
|
||||
|
||||
return data
|
||||
return data
|
||||
|
||||
def _get_dumpstate_file(self) -> Optional[bytes]:
|
||||
main_entry, dumpstate_log, dumpsys_txt = DUMPSTATE_ENTRY_POINTS
|
||||
with self.resource_lock:
|
||||
if _DUMPSTATE_BYTES_CACHE_KEY in self.resource_cache:
|
||||
return self.resource_cache[_DUMPSTATE_BYTES_CACHE_KEY]
|
||||
|
||||
main = self._get_files_by_pattern(main_entry)
|
||||
if main:
|
||||
main_content = self._get_file_content(main[0])
|
||||
try:
|
||||
return self._get_file_content(main_content.decode().strip())
|
||||
except KeyError:
|
||||
return None
|
||||
content = None
|
||||
main_entry, dumpstate_log, dumpsys_txt = DUMPSTATE_ENTRY_POINTS
|
||||
main = self._get_files_by_pattern(main_entry)
|
||||
if main:
|
||||
main_content = self._get_file_content(main[0])
|
||||
try:
|
||||
content = self._get_file_content(main_content.decode().strip())
|
||||
except KeyError:
|
||||
pass
|
||||
else:
|
||||
dumpstate_logs = self._get_files_by_pattern(dumpstate_log)
|
||||
if dumpstate_logs:
|
||||
content = self._get_file_content(dumpstate_logs[0])
|
||||
else:
|
||||
dumpsys_files = self._get_files_by_pattern(dumpsys_txt)
|
||||
if dumpsys_files:
|
||||
content = self._get_file_content(dumpsys_files[0])
|
||||
|
||||
dumpstate_logs = self._get_files_by_pattern(dumpstate_log)
|
||||
if dumpstate_logs:
|
||||
return self._get_file_content(dumpstate_logs[0])
|
||||
self.resource_cache[_DUMPSTATE_BYTES_CACHE_KEY] = content
|
||||
return content
|
||||
|
||||
dumpsys_files = self._get_files_by_pattern(dumpsys_txt)
|
||||
if dumpsys_files:
|
||||
return self._get_file_content(dumpsys_files[0])
|
||||
def _get_dumpstate_sections(self) -> dict[bytes, bytes]:
|
||||
with self.resource_lock:
|
||||
if _DUMPSTATE_SECTIONS_CACHE_KEY in self.resource_cache:
|
||||
return self.resource_cache[_DUMPSTATE_SECTIONS_CACHE_KEY]
|
||||
|
||||
return None
|
||||
section_lines: dict[bytes, list[bytes]] = {
|
||||
separator: [] for separator in _DUMPSYS_SEPARATORS
|
||||
}
|
||||
section_lines[_SYSTEM_PROPERTIES_SECTION] = []
|
||||
current_section = None
|
||||
completed_sections = set()
|
||||
in_system_properties = False
|
||||
content = self._get_dumpstate_file()
|
||||
|
||||
if content:
|
||||
for raw_line in io.BytesIO(content):
|
||||
line = raw_line.rstrip(b"\r\n")
|
||||
stripped = line.strip()
|
||||
|
||||
if stripped.startswith(_SYSTEM_PROPERTIES_SECTION):
|
||||
in_system_properties = True
|
||||
elif in_system_properties:
|
||||
if stripped == b"------":
|
||||
in_system_properties = False
|
||||
elif _GETPROP_LINE_RE.search(line):
|
||||
section_lines[_SYSTEM_PROPERTIES_SECTION].append(line)
|
||||
|
||||
if stripped in _DUMPSYS_SEPARATORS:
|
||||
if current_section is not None:
|
||||
completed_sections.add(current_section)
|
||||
current_section = (
|
||||
None if stripped in completed_sections else stripped
|
||||
)
|
||||
continue
|
||||
|
||||
if stripped.startswith(_DUMPSTATE_DELIMITER):
|
||||
if current_section is not None:
|
||||
completed_sections.add(current_section)
|
||||
current_section = None
|
||||
continue
|
||||
|
||||
if current_section is not None:
|
||||
section_lines[current_section].append(line)
|
||||
|
||||
sections = {
|
||||
separator: b"\n".join(lines)
|
||||
for separator, lines in section_lines.items()
|
||||
}
|
||||
self.resource_cache[_DUMPSTATE_SECTIONS_CACHE_KEY] = sections
|
||||
return sections
|
||||
|
||||
def _get_dumpsys_section_bytes(self, separator: bytes) -> Optional[bytes]:
|
||||
sections = self._get_dumpstate_sections()
|
||||
if separator in sections:
|
||||
return sections[separator]
|
||||
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
return None
|
||||
return AndroidArtifact.extract_dumpsys_section(content, separator, binary=True)
|
||||
|
||||
def _get_dumpsys_section(
|
||||
self, separator: str, errors: str = "replace"
|
||||
) -> Optional[str]:
|
||||
cache_key = (
|
||||
"android_bugreport",
|
||||
"dumpstate",
|
||||
"section",
|
||||
separator,
|
||||
errors,
|
||||
)
|
||||
with self.resource_lock:
|
||||
if cache_key in self.resource_cache:
|
||||
return self.resource_cache[cache_key]
|
||||
|
||||
section_bytes = self._get_dumpsys_section_bytes(separator.encode("utf-8"))
|
||||
section = (
|
||||
section_bytes.decode("utf-8", errors=errors)
|
||||
if section_bytes is not None
|
||||
else None
|
||||
)
|
||||
self.resource_cache[cache_key] = section
|
||||
return section
|
||||
|
||||
def _get_system_properties_text(self, errors: str = "ignore") -> Optional[str]:
|
||||
cache_key = (
|
||||
"android_bugreport",
|
||||
"dumpstate",
|
||||
"system_properties",
|
||||
errors,
|
||||
)
|
||||
with self.resource_lock:
|
||||
if cache_key in self.resource_cache:
|
||||
return self.resource_cache[cache_key]
|
||||
|
||||
sections = self._get_dumpstate_sections()
|
||||
section_bytes = sections.get(_SYSTEM_PROPERTIES_SECTION)
|
||||
section = (
|
||||
section_bytes.decode("utf-8", errors=errors)
|
||||
if section_bytes is not None
|
||||
else None
|
||||
)
|
||||
self.resource_cache[cache_key] = section
|
||||
return section
|
||||
|
||||
@staticmethod
|
||||
def extract_command_section(content: str, heading: str) -> str:
|
||||
@@ -167,7 +295,8 @@ class BugReportModule(MVTModule):
|
||||
instant returned in UTC.
|
||||
"""
|
||||
if self.zip_archive:
|
||||
file_timetuple = self.zip_archive.getinfo(file_path).date_time
|
||||
with self.resource_lock:
|
||||
file_timetuple = self.zip_archive.getinfo(file_path).date_time
|
||||
return datetime.datetime(*file_timetuple, tzinfo=self._device_timezone())
|
||||
if not self.extract_path:
|
||||
raise ValueError("extract_path is not set")
|
||||
|
||||
@@ -34,18 +34,16 @@ class DumpsysAccessibility(DumpsysAccessibilityArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
full_dumpsys = self._get_dumpstate_file()
|
||||
if not full_dumpsys:
|
||||
content = self._get_dumpsys_section(
|
||||
"DUMP OF SERVICE accessibility:", errors="ignore"
|
||||
)
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
content = self.extract_dumpsys_section(
|
||||
full_dumpsys.decode("utf-8", errors="ignore"),
|
||||
"DUMP OF SERVICE accessibility:",
|
||||
)
|
||||
self.parse(content)
|
||||
|
||||
listed = unnamed = 0
|
||||
|
||||
@@ -38,19 +38,14 @@ class DumpsysActivities(DumpsysPackageActivitiesArtifact, BugReportModule):
|
||||
self.results = results if results else []
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
section = self._get_dumpsys_section("DUMP OF SERVICE package:", errors="ignore")
|
||||
if section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
# Extract package section
|
||||
section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="ignore"), "DUMP OF SERVICE package:"
|
||||
)
|
||||
|
||||
# Parse
|
||||
self.parse(section)
|
||||
|
||||
|
||||
@@ -38,19 +38,14 @@ class DumpsysADBState(DumpsysADBArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
full_dumpsys = self._get_dumpstate_file()
|
||||
if not full_dumpsys:
|
||||
content = self._get_dumpsys_section_bytes(b"DUMP OF SERVICE adb:")
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
content = self.extract_dumpsys_section(
|
||||
full_dumpsys,
|
||||
b"DUMP OF SERVICE adb:",
|
||||
binary=True,
|
||||
)
|
||||
self.parse(content)
|
||||
if self.results:
|
||||
self.log.info(
|
||||
|
||||
@@ -34,17 +34,14 @@ class DumpsysAppops(DumpsysAppopsArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
section = self._get_dumpsys_section("DUMP OF SERVICE appops:")
|
||||
if section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE appops:"
|
||||
)
|
||||
self.parse(section)
|
||||
|
||||
self.log.info(
|
||||
|
||||
@@ -34,17 +34,14 @@ class DumpsysBatteryDaily(DumpsysBatteryDailyArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
dumpsys_section = self._get_dumpsys_section("DUMP OF SERVICE batterystats:")
|
||||
if dumpsys_section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
dumpsys_section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE batterystats:"
|
||||
)
|
||||
self.parse(dumpsys_section)
|
||||
|
||||
self.log.info("Extracted a total of %d battery daily stats", len(self.results))
|
||||
@@ -34,17 +34,14 @@ class DumpsysBatteryHistory(DumpsysBatteryHistoryArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
dumpsys_section = self._get_dumpsys_section("DUMP OF SERVICE batterystats:")
|
||||
if dumpsys_section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
dumpsys_section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE batterystats:"
|
||||
)
|
||||
self.parse(dumpsys_section)
|
||||
|
||||
self.log.info(
|
||||
|
||||
@@ -36,18 +36,14 @@ class DumpsysDBInfo(DumpsysDBInfoArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
data = self._get_dumpstate_file()
|
||||
if not data:
|
||||
section = self._get_dumpsys_section("DUMP OF SERVICE dbinfo:", errors="ignore")
|
||||
if section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
section = self.extract_dumpsys_section(
|
||||
data.decode("utf-8", errors="ignore"), "DUMP OF SERVICE dbinfo:"
|
||||
)
|
||||
|
||||
self.parse(section)
|
||||
self.log.info(
|
||||
"Extracted a total of %d database connection pool records",
|
||||
|
||||
@@ -36,16 +36,13 @@ class DumpsysGetProp(GetPropArtifact, BugReportModule):
|
||||
self.results = [] if not results else results
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
content = self._get_system_properties_text()
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
section = self.extract_command_section(
|
||||
content.decode(errors="ignore"), "------ SYSTEM PROPERTIES"
|
||||
)
|
||||
self.parse(section)
|
||||
self.parse(content)
|
||||
self.log.info("Extracted %d Android system properties", len(self.results))
|
||||
@@ -35,17 +35,14 @@ class DumpsysPackages(DumpsysPackagesArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
data = self._get_dumpstate_file()
|
||||
if not data:
|
||||
content = self._get_dumpsys_section("DUMP OF SERVICE package:")
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
content = self.extract_dumpsys_section(
|
||||
data.decode("utf-8", errors="replace"), "DUMP OF SERVICE package:"
|
||||
)
|
||||
self.parse(content)
|
||||
|
||||
for result in self.results:
|
||||
|
||||
@@ -34,18 +34,14 @@ class DumpsysPlatformCompat(DumpsysPlatformCompatArtifact, BugReportModule):
|
||||
)
|
||||
|
||||
def run(self) -> None:
|
||||
data = self._get_dumpstate_file()
|
||||
if not data:
|
||||
content = self._get_dumpsys_section("DUMP OF SERVICE platform_compat:")
|
||||
if content is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
decoded_data = data.decode("utf-8", errors="replace")
|
||||
content = self.extract_dumpsys_section(
|
||||
decoded_data, "DUMP OF SERVICE platform_compat:"
|
||||
)
|
||||
self.parse(content)
|
||||
|
||||
self.log.info("Found %d package compatibility overrides", len(self.results))
|
||||
@@ -8,7 +8,6 @@ from typing import Optional
|
||||
|
||||
from mvt.android.artifacts.dumpsys_receivers import DumpsysReceiversArtifact
|
||||
from mvt.common.module_types import ModuleResults
|
||||
from mvt.schemas import MappingOutput
|
||||
|
||||
from .base import BugReportModule
|
||||
|
||||
@@ -16,8 +15,6 @@ from .base import BugReportModule
|
||||
class DumpsysReceivers(DumpsysReceiversArtifact, BugReportModule):
|
||||
"""This module extracts details on receivers for risky activities."""
|
||||
|
||||
output_model = MappingOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
@@ -39,17 +36,13 @@ class DumpsysReceivers(DumpsysReceiversArtifact, BugReportModule):
|
||||
self.results = results if results else []
|
||||
|
||||
def run(self) -> None:
|
||||
content = self._get_dumpstate_file()
|
||||
if not content:
|
||||
dumpsys_section = self._get_dumpsys_section("DUMP OF SERVICE package:")
|
||||
if dumpsys_section is None:
|
||||
self.log.error(
|
||||
"Unable to find dumpstate file. "
|
||||
"Did you provide a valid bug report archive?"
|
||||
)
|
||||
return
|
||||
|
||||
dumpsys_section = self.extract_dumpsys_section(
|
||||
content.decode("utf-8", errors="replace"), "DUMP OF SERVICE package:"
|
||||
)
|
||||
|
||||
self.parse(dumpsys_section)
|
||||
self.log.info("Extracted %d package receivers", len(self.results))
|
||||
@@ -9,7 +9,7 @@ import json
|
||||
import logging
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import ClassVar, Optional, Union
|
||||
from typing import Optional, Union
|
||||
|
||||
try:
|
||||
import zoneinfo
|
||||
@@ -18,7 +18,6 @@ except ImportError:
|
||||
|
||||
from mvt.common.module import MVTModule
|
||||
from mvt.common.utils import convert_datetime_to_iso, convert_unix_to_iso
|
||||
from mvt.schemas import OutputModel, RecordListOutput
|
||||
|
||||
|
||||
class IntrusionLogsModule(MVTModule):
|
||||
@@ -42,8 +41,6 @@ class IntrusionLogsModule(MVTModule):
|
||||
original file-loading code path.
|
||||
"""
|
||||
|
||||
output_model: ClassVar[OutputModel] = RecordListOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
"""Shared implementation for the platform schema-export commands."""
|
||||
|
||||
import json
|
||||
from typing import Optional
|
||||
|
||||
import click
|
||||
|
||||
from mvt.schemas import export_json_schemas, schema_bundle
|
||||
|
||||
|
||||
def emit_schemas(platform: str, output: Optional[str]) -> None:
|
||||
if output:
|
||||
paths = export_json_schemas(output, platform)
|
||||
click.echo(f"Exported {len(paths)} {platform} output schemas to {output}")
|
||||
return
|
||||
|
||||
click.echo(json.dumps(schema_bundle(platform), indent=2))
|
||||
+229
-56
@@ -6,7 +6,10 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import sys
|
||||
import threading
|
||||
from concurrent.futures import Future, ThreadPoolExecutor
|
||||
from datetime import datetime
|
||||
from functools import cached_property
|
||||
from heapq import heappop, heappush
|
||||
@@ -19,6 +22,7 @@ from rich.text import Text
|
||||
from .alerts import AlertLevel, AlertStore
|
||||
from .config import settings
|
||||
from .indicators import Indicators
|
||||
from .log import MVTLogHandler, finish_module_log_buffer, start_module_log_buffer
|
||||
from .module import EncryptedBackupError, MVTModule, run_module, save_timeline
|
||||
from .module_loader import (
|
||||
ModuleOrigin,
|
||||
@@ -34,13 +38,6 @@ from .utils import (
|
||||
get_sha256_from_file_path,
|
||||
)
|
||||
from .version import MVT_VERSION
|
||||
from mvt.schemas import (
|
||||
OUTPUT_SCHEMA_VERSION,
|
||||
AlertResults,
|
||||
RunInfo,
|
||||
URLResults,
|
||||
validate_output,
|
||||
)
|
||||
|
||||
|
||||
class Command:
|
||||
@@ -59,6 +56,7 @@ class Command:
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
self.name = ""
|
||||
self.platform = ""
|
||||
@@ -74,6 +72,11 @@ class Command:
|
||||
self.sub_command = sub_command
|
||||
self.disable_version_check = disable_version_check
|
||||
self.disable_indicator_check = disable_indicator_check
|
||||
if jobs < 1:
|
||||
raise ValueError("jobs must be at least 1")
|
||||
self.jobs = jobs
|
||||
self._resource_lock = threading.RLock()
|
||||
self._resource_cache: dict[object, Any] = {}
|
||||
|
||||
# This dictionary can contain options that will be passed down from
|
||||
# the Command to all modules. This can for example be used to pass
|
||||
@@ -164,8 +167,6 @@ class Command:
|
||||
if not alerts:
|
||||
return
|
||||
|
||||
alerts = validate_output(AlertResults, alerts)
|
||||
|
||||
alerts_path = os.path.join(self.results_path, "alerts.json")
|
||||
with open(alerts_path, "w+", encoding="utf-8") as handle:
|
||||
json.dump(alerts, handle, indent=4, cls=CustomJSONEncoder)
|
||||
@@ -174,10 +175,9 @@ class Command:
|
||||
if not self.results_path or not self.url_results:
|
||||
return
|
||||
|
||||
urls = validate_output(URLResults, self.url_results)
|
||||
urls_path = os.path.join(self.results_path, "urls.json")
|
||||
with open(urls_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(urls, handle, indent=4, cls=CustomJSONEncoder)
|
||||
json.dump(self.url_results, handle, indent=4, cls=CustomJSONEncoder)
|
||||
|
||||
def _store_alerts_timeline(self) -> None:
|
||||
if not self.results_path:
|
||||
@@ -200,7 +200,6 @@ class Command:
|
||||
"date": convert_datetime_to_iso(datetime.now()),
|
||||
"ioc_files": [],
|
||||
"hashes": [],
|
||||
"output_schema_version": OUTPUT_SCHEMA_VERSION,
|
||||
}
|
||||
|
||||
for coll in self.iocs.ioc_collections:
|
||||
@@ -213,7 +212,6 @@ class Command:
|
||||
|
||||
info["hashes"] = self.hash_values
|
||||
|
||||
info = validate_output(RunInfo, info)
|
||||
info_path = os.path.join(self.results_path, "info.json")
|
||||
with open(info_path, "w+", encoding="utf-8") as handle:
|
||||
json.dump(info, handle, indent=4)
|
||||
@@ -722,41 +720,22 @@ class Command:
|
||||
|
||||
return ordered
|
||||
|
||||
def run(self) -> None:
|
||||
# The output folder and its command.log exist for a run, so that
|
||||
# listing modules or rejecting a target leaves nothing behind.
|
||||
# Resolving the module list can warn, so the log comes first.
|
||||
self._create_storage()
|
||||
self._setup_logging()
|
||||
|
||||
ordered_modules = self._ordered_modules()
|
||||
if ordered_modules is None:
|
||||
return
|
||||
|
||||
# Read the indicators once the run is certain to happen, before
|
||||
# init() does any work on the target, so that a missing indicators
|
||||
# file is reported first and every module gets the same object.
|
||||
iocs = self.iocs
|
||||
|
||||
# Commands announce their target from init(), so it goes before the
|
||||
# module list.
|
||||
def _run_module(
|
||||
self,
|
||||
module: type[MVTModule],
|
||||
executed_by_type: dict[type[MVTModule], MVTModule],
|
||||
capture_logs: bool,
|
||||
) -> tuple[MVTModule, list[logging.LogRecord], bool]:
|
||||
"""Initialize and run one module, optionally buffering console logs."""
|
||||
token = start_module_log_buffer() if capture_logs else None
|
||||
records: list[logging.LogRecord] = []
|
||||
encrypted = False
|
||||
try:
|
||||
self.init()
|
||||
except NotImplementedError:
|
||||
pass
|
||||
|
||||
self._log_loaded_modules(ordered_modules)
|
||||
|
||||
executed_by_type: dict[type[MVTModule], MVTModule] = {}
|
||||
for module in ordered_modules:
|
||||
|
||||
module_logger = get_module_logger(module)
|
||||
|
||||
m = module(
|
||||
target_path=self.target_path,
|
||||
results_path=self.results_path,
|
||||
module_options=self.module_options,
|
||||
log=module_logger,
|
||||
log=get_module_logger(module),
|
||||
)
|
||||
# Dependencies are keyed by the module class they declare, even
|
||||
# when it was replaced, so that a module asking for the results
|
||||
@@ -765,10 +744,12 @@ class Command:
|
||||
dependency: executed_by_type[resolved]
|
||||
for dependency, resolved in self._module_dependencies(module)
|
||||
}
|
||||
m.resource_lock = self._resource_lock
|
||||
m.resource_cache = self._resource_cache
|
||||
|
||||
if iocs.total_ioc_count:
|
||||
m.indicators = iocs
|
||||
m.indicators.log = m.log
|
||||
if self.iocs.total_ioc_count:
|
||||
# IOC collections are shared read-only during module execution.
|
||||
m.indicators = self.iocs
|
||||
|
||||
if self.serial:
|
||||
m.serial = self.serial
|
||||
@@ -781,23 +762,215 @@ class Command:
|
||||
try:
|
||||
run_module(m)
|
||||
except EncryptedBackupError:
|
||||
self.log.critical(
|
||||
"The backup appears to be encrypted. "
|
||||
"Please decrypt it first using `mvt-ios decrypt-backup`."
|
||||
)
|
||||
return
|
||||
encrypted = True
|
||||
finally:
|
||||
if token is not None:
|
||||
records = finish_module_log_buffer(token)
|
||||
|
||||
self.executed.append(m)
|
||||
executed_by_type[module] = m
|
||||
self.timeline.extend(m.timeline)
|
||||
self.url_results.extend(m.url_results)
|
||||
self.alertstore.extend(m.alertstore.alerts)
|
||||
return m, records, encrypted
|
||||
|
||||
@staticmethod
|
||||
def _console_handler() -> Optional[MVTLogHandler]:
|
||||
for handler in reversed(logging.getLogger("mvt").handlers):
|
||||
if isinstance(handler, MVTLogHandler):
|
||||
return handler
|
||||
return None
|
||||
|
||||
def _publish_completed_modules(
|
||||
self,
|
||||
ordered_modules: list[type[MVTModule]],
|
||||
completed: dict[type[MVTModule], MVTModule],
|
||||
) -> None:
|
||||
"""Expose a stable snapshot of completed modules to initialization hooks."""
|
||||
self.executed = [
|
||||
completed[module] for module in ordered_modules if module in completed
|
||||
]
|
||||
|
||||
def _aggregate_modules(
|
||||
self,
|
||||
ordered_modules: list[type[MVTModule]],
|
||||
completed: dict[type[MVTModule], MVTModule],
|
||||
) -> None:
|
||||
"""Aggregate results in stable topological order."""
|
||||
for module in ordered_modules:
|
||||
if module not in completed:
|
||||
continue
|
||||
instance = completed[module]
|
||||
self.timeline.extend(instance.timeline)
|
||||
self.url_results.extend(instance.url_results)
|
||||
self.alertstore.extend(instance.alertstore.alerts)
|
||||
|
||||
def _run_sequential(
|
||||
self, ordered_modules: list[type[MVTModule]]
|
||||
) -> tuple[dict[type[MVTModule], MVTModule], bool]:
|
||||
completed: dict[type[MVTModule], MVTModule] = {}
|
||||
for module in ordered_modules:
|
||||
instance, _, encrypted = self._run_module(module, completed, False)
|
||||
if encrypted:
|
||||
return completed, True
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
return completed, False
|
||||
|
||||
def _run_parallel(
|
||||
self, ordered_modules: list[type[MVTModule]], jobs: int
|
||||
) -> tuple[dict[type[MVTModule], MVTModule], bool]:
|
||||
"""Run a stable module DAG with bounded worker threads."""
|
||||
completed: dict[type[MVTModule], MVTModule] = {}
|
||||
scheduled: set[type[MVTModule]] = set()
|
||||
running: dict[Future, type[MVTModule]] = {}
|
||||
completion_queue: queue.Queue[Future] = queue.Queue()
|
||||
handler = self._console_handler()
|
||||
status = None
|
||||
fatal = False
|
||||
|
||||
def ready_modules() -> list[type[MVTModule]]:
|
||||
return [
|
||||
module
|
||||
for module in ordered_modules
|
||||
if module not in scheduled
|
||||
and all(
|
||||
dependency in completed
|
||||
for _, dependency in self._module_dependencies(module)
|
||||
)
|
||||
]
|
||||
|
||||
def update_status() -> None:
|
||||
if status is None:
|
||||
return
|
||||
names = ", ".join(module.__name__ for module in running.values())
|
||||
status.update(
|
||||
f"Modules: {len(completed)}/{len(ordered_modules)} complete"
|
||||
+ (f"; running {names}" if names else "")
|
||||
)
|
||||
|
||||
if handler and handler.console.is_terminal:
|
||||
status = handler.console.status("")
|
||||
status.start()
|
||||
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=jobs) as executor:
|
||||
while len(completed) < len(ordered_modules):
|
||||
ready = ready_modules()
|
||||
|
||||
# An unsafe module is a scheduler barrier: it starts only
|
||||
# after active workers drain and blocks later ready work.
|
||||
if ready and not ready[0].parallel_safe:
|
||||
if running:
|
||||
future = completion_queue.get()
|
||||
module = running.pop(future)
|
||||
instance, records, encrypted = future.result()
|
||||
if handler:
|
||||
handler.emit_module_records(module.__name__, records)
|
||||
if encrypted:
|
||||
fatal = True
|
||||
break
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
update_status()
|
||||
continue
|
||||
|
||||
module = ready[0]
|
||||
scheduled.add(module)
|
||||
if status is not None:
|
||||
status.update(
|
||||
f"Modules: {len(completed)}/{len(ordered_modules)} "
|
||||
f"complete; running {module.__name__}"
|
||||
)
|
||||
instance, records, encrypted = self._run_module(
|
||||
module, completed, True
|
||||
)
|
||||
if handler:
|
||||
handler.emit_module_records(module.__name__, records)
|
||||
if encrypted:
|
||||
fatal = True
|
||||
break
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
update_status()
|
||||
continue
|
||||
|
||||
for module in ready:
|
||||
if len(running) >= jobs or not module.parallel_safe:
|
||||
break
|
||||
scheduled.add(module)
|
||||
future = executor.submit(
|
||||
self._run_module, module, dict(completed), True
|
||||
)
|
||||
running[future] = module
|
||||
future.add_done_callback(completion_queue.put)
|
||||
|
||||
update_status()
|
||||
if not running:
|
||||
break
|
||||
|
||||
future = completion_queue.get()
|
||||
module = running.pop(future)
|
||||
instance, records, encrypted = future.result()
|
||||
if handler:
|
||||
handler.emit_module_records(module.__name__, records)
|
||||
if encrypted:
|
||||
fatal = True
|
||||
break
|
||||
completed[module] = instance
|
||||
self._publish_completed_modules(ordered_modules, completed)
|
||||
update_status()
|
||||
|
||||
if fatal:
|
||||
for future in running:
|
||||
future.cancel()
|
||||
# The executor context waits for already-active work. Its
|
||||
# results are intentionally discarded after the fatal error.
|
||||
finally:
|
||||
if status is not None:
|
||||
status.stop()
|
||||
|
||||
return completed, fatal
|
||||
|
||||
def run(self) -> None:
|
||||
self._create_storage()
|
||||
self._setup_logging()
|
||||
|
||||
ordered_modules = self._ordered_modules()
|
||||
if ordered_modules is None:
|
||||
return
|
||||
|
||||
# Load shared indicators before touching the acquisition.
|
||||
self.iocs
|
||||
|
||||
try:
|
||||
self.init()
|
||||
except NotImplementedError:
|
||||
pass
|
||||
|
||||
self._log_loaded_modules(ordered_modules)
|
||||
|
||||
jobs = self.jobs
|
||||
if settings.PROFILE and jobs > 1:
|
||||
self.log.warning(
|
||||
"MVT_PROFILE is enabled; forcing sequential module execution."
|
||||
)
|
||||
jobs = 1
|
||||
|
||||
if jobs == 1:
|
||||
completed, fatal = self._run_sequential(ordered_modules)
|
||||
else:
|
||||
completed, fatal = self._run_parallel(ordered_modules, jobs)
|
||||
|
||||
self._aggregate_modules(ordered_modules, completed)
|
||||
|
||||
try:
|
||||
self.finish()
|
||||
except NotImplementedError:
|
||||
pass
|
||||
|
||||
if fatal:
|
||||
self.log.critical(
|
||||
"The backup appears to be encrypted. "
|
||||
"Please decrypt it first using `mvt-ios decrypt-backup`."
|
||||
)
|
||||
return
|
||||
|
||||
# We only store the timeline from the parent/main command
|
||||
if self.sub_command:
|
||||
return
|
||||
|
||||
@@ -17,6 +17,7 @@ HELP_MSG_LOAD_MODULE = (
|
||||
HELP_MSG_NONINTERACTIVE = "Don't ask interactive questions during processing"
|
||||
HELP_MSG_HASHES = "Generate hashes of all the files analyzed"
|
||||
HELP_MSG_VERBOSE = "Verbose mode"
|
||||
HELP_MSG_JOBS = "Number of extraction modules to run concurrently"
|
||||
HELP_MSG_VERBOSE_COMMAND = (
|
||||
"Verbose mode (kept for compatibility, pass --verbose before the command "
|
||||
"name instead)"
|
||||
|
||||
+41
-1
@@ -4,6 +4,8 @@
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from contextvars import ContextVar, Token
|
||||
from rich.console import Console
|
||||
from rich.logging import RichHandler
|
||||
from typing import Optional
|
||||
@@ -27,16 +29,33 @@ logging.addLevelName(HIGH_ALERT, "HIGH")
|
||||
logging.addLevelName(CRITICAL_ALERT, "CRITICAL")
|
||||
|
||||
|
||||
_module_log_buffer: ContextVar[Optional[list[logging.LogRecord]]] = ContextVar(
|
||||
"mvt_module_log_buffer", default=None
|
||||
)
|
||||
|
||||
|
||||
def start_module_log_buffer() -> Token:
|
||||
"""Capture console records in the current module execution context."""
|
||||
return _module_log_buffer.set([])
|
||||
|
||||
|
||||
def finish_module_log_buffer(token: Token) -> list[logging.LogRecord]:
|
||||
records = _module_log_buffer.get() or []
|
||||
_module_log_buffer.reset(token)
|
||||
return records
|
||||
|
||||
|
||||
class MVTLogHandler(RichHandler):
|
||||
def __init__(self, console: Optional[Console] = None, level: int = logging.DEBUG):
|
||||
super().__init__(console=console, level=level)
|
||||
self._console_lock = threading.RLock()
|
||||
|
||||
def __add_prefix_space(self, level: str) -> str:
|
||||
max_length = len("CRITICAL ALERT")
|
||||
space = max_length - len(level)
|
||||
return f"{level}{' ' * space}"
|
||||
|
||||
def emit(self, record: logging.LogRecord):
|
||||
def _emit_record(self, record: logging.LogRecord) -> None:
|
||||
try:
|
||||
msg = rf"[grey50]\[{record.name}][/] {self.format(record)}"
|
||||
|
||||
@@ -63,3 +82,24 @@ class MVTLogHandler(RichHandler):
|
||||
|
||||
except Exception:
|
||||
self.handleError(record)
|
||||
|
||||
def emit(self, record: logging.LogRecord):
|
||||
buffer = _module_log_buffer.get()
|
||||
if buffer is not None:
|
||||
# More than one CLI can be imported in-process (notably in tests),
|
||||
# which can install multiple console handlers for the same record.
|
||||
if not buffer or buffer[-1] is not record:
|
||||
buffer.append(record)
|
||||
return
|
||||
|
||||
with self._console_lock:
|
||||
self._emit_record(record)
|
||||
|
||||
def emit_module_records(
|
||||
self, module_name: str, records: list[logging.LogRecord]
|
||||
) -> None:
|
||||
"""Print one module's captured records without interleaving."""
|
||||
with self._console_lock:
|
||||
self.console.print(f"[bold cyan]--- {module_name} ---[/bold cyan]")
|
||||
for record in records:
|
||||
self._emit_record(record)
|
||||
+43
-36
@@ -8,13 +8,9 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from dataclasses import asdict, is_dataclass
|
||||
from typing import Any, ClassVar, Dict, Optional, Sequence
|
||||
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from mvt.schemas.models import AlertResults, GenericModuleOutput, TimelineResults
|
||||
from mvt.schemas.serialization import validate_output
|
||||
from typing import Any, Dict, Optional, Sequence
|
||||
|
||||
from .alerts import AlertStore
|
||||
from .indicators import Indicators
|
||||
@@ -51,7 +47,7 @@ class MVTModule:
|
||||
slug: Optional[str] = None
|
||||
dependencies: Sequence[type["MVTModule"]] = ()
|
||||
supported_commands: Sequence[tuple[str, str]] = ()
|
||||
output_model: ClassVar[type[BaseModel]] = GenericModuleOutput
|
||||
parallel_safe: bool = True
|
||||
# A custom module can name a module class it supersedes, usually a
|
||||
# built-in one. When both are available to a command, the named module is
|
||||
# dropped from the run and this module takes its place, including in the
|
||||
@@ -96,6 +92,12 @@ class MVTModule:
|
||||
self.timeline: ModuleTimeline = []
|
||||
self.url_results: list[URLResult] = []
|
||||
self.dependency_modules: Dict[type["MVTModule"], "MVTModule"] = {}
|
||||
# Commands replace this with a command-scoped lock for modules sharing
|
||||
# archive handles or other non-thread-safe read resources.
|
||||
self.resource_lock = threading.RLock()
|
||||
# Commands also replace this with a command-scoped cache. Cached values
|
||||
# must be immutable, because parallel modules can read them concurrently.
|
||||
self.resource_cache: Dict[object, Any] = {}
|
||||
|
||||
def get_dependency_results(
|
||||
self, module_class: type["MVTModule"]
|
||||
@@ -106,7 +108,7 @@ class MVTModule:
|
||||
@classmethod
|
||||
def from_json(cls, json_path: str, log: logging.Logger):
|
||||
with open(json_path, "r", encoding="utf-8") as handle:
|
||||
results = validate_output(cls.output_model, json.load(handle))
|
||||
results = json.load(handle)
|
||||
if log:
|
||||
log.info('Loaded %d results from "%s"', len(results), json_path)
|
||||
|
||||
@@ -147,35 +149,38 @@ class MVTModule:
|
||||
name = self.get_slug()
|
||||
|
||||
if self.results:
|
||||
converted_results: Any
|
||||
if isinstance(self.results, dict):
|
||||
converted_results = self.results
|
||||
else:
|
||||
converted_results = [
|
||||
asdict(result)
|
||||
if is_dataclass(result) and not isinstance(result, type)
|
||||
else result
|
||||
for result in self.results
|
||||
]
|
||||
results_file_name = f"{name}.json"
|
||||
results_json_path = os.path.join(self.results_path, results_file_name)
|
||||
if self.output_model is GenericModuleOutput:
|
||||
self.log.warning(
|
||||
"Module %s does not declare an output_model; using the "
|
||||
"generic compatibility schema",
|
||||
self.__class__.__name__,
|
||||
)
|
||||
try:
|
||||
converted_results = validate_output(self.output_model, self.results)
|
||||
except (TypeError, ValueError, ValidationError) as exc:
|
||||
self.log.error(
|
||||
"Output from module %s does not match schema %s: %s",
|
||||
self.__class__.__name__,
|
||||
self.output_model.__name__,
|
||||
exc,
|
||||
)
|
||||
else:
|
||||
with open(results_json_path, "w", encoding="utf-8") as handle:
|
||||
with open(results_json_path, "w", encoding="utf-8") as handle:
|
||||
try:
|
||||
json.dump(
|
||||
converted_results, handle, indent=4, cls=CustomJSONEncoder
|
||||
)
|
||||
except Exception as exc:
|
||||
self.log.error(
|
||||
"Unable to store results of module %s to file %s: %s",
|
||||
self.__class__.__name__,
|
||||
results_file_name,
|
||||
exc,
|
||||
)
|
||||
|
||||
if self.alertstore.alerts:
|
||||
detected_file_name = f"{name}_detected.json"
|
||||
detected_json_path = os.path.join(self.results_path, detected_file_name)
|
||||
alerts = validate_output(AlertResults, self.alertstore.as_json())
|
||||
with open(detected_json_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(alerts, handle, indent=4, cls=CustomJSONEncoder)
|
||||
json.dump(
|
||||
self.alertstore.as_json(), handle, indent=4, cls=CustomJSONEncoder
|
||||
)
|
||||
|
||||
def serialize(self, result: ModuleAtomicResult) -> ModuleSerializedResult:
|
||||
raise NotImplementedError
|
||||
@@ -188,17 +193,20 @@ class MVTModule:
|
||||
|
||||
"""
|
||||
timeline_set = set()
|
||||
deduplicated = []
|
||||
for record in timeline:
|
||||
timeline_set.add(
|
||||
json.dumps(
|
||||
asdict(record)
|
||||
if is_dataclass(record) and not isinstance(record, type)
|
||||
else record,
|
||||
sort_keys=True,
|
||||
)
|
||||
serialized = json.dumps(
|
||||
asdict(record)
|
||||
if is_dataclass(record) and not isinstance(record, type)
|
||||
else record,
|
||||
sort_keys=True,
|
||||
)
|
||||
if serialized in timeline_set:
|
||||
continue
|
||||
timeline_set.add(serialized)
|
||||
deduplicated.append(json.loads(serialized))
|
||||
|
||||
return [json.loads(record) for record in timeline_set]
|
||||
return deduplicated
|
||||
|
||||
def to_timeline(self) -> None:
|
||||
"""Convert results into a timeline."""
|
||||
@@ -306,7 +314,6 @@ def save_timeline(timeline: list, timeline_path: str, is_utc: bool = True) -> No
|
||||
:param timeline_path: Path to the csv file to store the timeline to
|
||||
|
||||
"""
|
||||
timeline = validate_output(TimelineResults, timeline)
|
||||
with open(timeline_path, "w", encoding="utf-8") as handle:
|
||||
csvoutput = csv.writer(
|
||||
handle, delimiter=",", quotechar='"', quoting=csv.QUOTE_ALL, escapechar="\\"
|
||||
|
||||
+16
-21
@@ -36,6 +36,7 @@ from mvt.common.help import (
|
||||
HELP_MSG_LOAD_MODULE,
|
||||
HELP_MSG_MODULE,
|
||||
HELP_MSG_VERBOSE,
|
||||
HELP_MSG_JOBS,
|
||||
HELP_MSG_VERBOSE_COMMAND,
|
||||
HELP_MSG_CHECK_FS,
|
||||
HELP_MSG_CHECK_IOCS,
|
||||
@@ -110,13 +111,12 @@ def cli(ctx, disable_update_check, disable_indicator_update_check, verbose):
|
||||
ctx.obj["verbose"] = verbose
|
||||
set_verbose_logging(verbose)
|
||||
|
||||
if ctx.invoked_subcommand not in ("completion", "schemas"):
|
||||
from mvt.common.logo import logo
|
||||
from mvt.common.logo import logo
|
||||
|
||||
logo(
|
||||
disable_version_check=disable_update_check,
|
||||
disable_indicator_check=disable_indicator_update_check,
|
||||
)
|
||||
logo(
|
||||
disable_version_check=disable_update_check,
|
||||
disable_indicator_check=disable_indicator_update_check,
|
||||
)
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
@@ -128,21 +128,6 @@ def version():
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Command: schemas
|
||||
# ==============================================================================
|
||||
@cli.command("schemas", help="Print or export the JSON Schemas for MVT outputs.")
|
||||
@click.option(
|
||||
"--output",
|
||||
"-o",
|
||||
type=click.Path(file_okay=False, dir_okay=True),
|
||||
help="Write one JSON Schema file per output to this directory.",
|
||||
)
|
||||
def schemas(output):
|
||||
from mvt.common.cmd_schemas import emit_schemas
|
||||
|
||||
emit_schemas("ios", output)
|
||||
|
||||
|
||||
# Command: decrypt-backup
|
||||
# ==============================================================================
|
||||
@cli.command(
|
||||
@@ -295,6 +280,9 @@ def extract_key(password, key_file, backup_path):
|
||||
)
|
||||
@click.option("--hashes", "-H", is_flag=True, help=HELP_MSG_HASHES)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("BACKUP_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_backup(
|
||||
@@ -307,6 +295,7 @@ def check_backup(
|
||||
load_module,
|
||||
hashes,
|
||||
verbose,
|
||||
jobs,
|
||||
backup_path,
|
||||
):
|
||||
from .cmd_check_backup import CmdIOSCheckBackup
|
||||
@@ -325,6 +314,7 @@ def check_backup(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
@@ -364,6 +354,9 @@ def check_backup(
|
||||
)
|
||||
@click.option("--hashes", "-H", is_flag=True, help=HELP_MSG_HASHES)
|
||||
@click.option("--verbose", "-v", is_flag=True, help=HELP_MSG_VERBOSE_COMMAND)
|
||||
@click.option(
|
||||
"--jobs", type=click.IntRange(min=1), default=4, show_default=True, help=HELP_MSG_JOBS
|
||||
)
|
||||
@click.argument("DUMP_PATH", type=click.Path(exists=True))
|
||||
@click.pass_context
|
||||
def check_fs(
|
||||
@@ -376,6 +369,7 @@ def check_fs(
|
||||
load_module,
|
||||
hashes,
|
||||
verbose,
|
||||
jobs,
|
||||
dump_path,
|
||||
):
|
||||
from .cmd_check_fs import CmdIOSCheckFS
|
||||
@@ -394,6 +388,7 @@ def check_fs(
|
||||
disable_version_check=_get_disable_flags(ctx)[0],
|
||||
disable_indicator_check=_get_disable_flags(ctx)[1],
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
if list_modules:
|
||||
|
||||
@@ -38,6 +38,7 @@ class CmdIOSCheckBackup(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -53,6 +54,7 @@ class CmdIOSCheckBackup(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "ios"
|
||||
|
||||
@@ -31,6 +31,7 @@ class CmdIOSCheckFS(Command):
|
||||
disable_version_check: bool = False,
|
||||
disable_indicator_check: bool = False,
|
||||
custom_modules: Optional[list[type[MVTModule]]] = None,
|
||||
jobs: int = 4,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
target_path=target_path,
|
||||
@@ -45,6 +46,7 @@ class CmdIOSCheckFS(Command):
|
||||
disable_version_check=disable_version_check,
|
||||
disable_indicator_check=disable_indicator_check,
|
||||
custom_modules=custom_modules,
|
||||
jobs=jobs,
|
||||
)
|
||||
|
||||
self.platform = "ios"
|
||||
|
||||
@@ -10,7 +10,6 @@ from typing import Optional
|
||||
|
||||
from mvt.common.module import DatabaseNotFoundError
|
||||
from mvt.common.module_types import ModuleResults
|
||||
from mvt.schemas import MappingOutput
|
||||
from mvt.ios.versions import get_device_desc_from_id, is_ios_version_outdated
|
||||
|
||||
from ..base import IOSExtraction
|
||||
@@ -19,8 +18,6 @@ from ..base import IOSExtraction
|
||||
class BackupInfo(IOSExtraction):
|
||||
"""This module extracts information about the device and the backup."""
|
||||
|
||||
output_model = MappingOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
|
||||
@@ -11,7 +11,7 @@ import sqlite3
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import ClassVar, Iterator, Optional, Union
|
||||
from typing import Iterator, Optional, Union
|
||||
|
||||
from mvt.common.module import (
|
||||
DatabaseCorruptedError,
|
||||
@@ -19,7 +19,6 @@ from mvt.common.module import (
|
||||
ModuleResults,
|
||||
MVTModule,
|
||||
)
|
||||
from mvt.schemas import OutputModel, RecordListOutput
|
||||
|
||||
|
||||
class TemporarySQLiteConnection(sqlite3.Connection):
|
||||
@@ -40,8 +39,6 @@ class IOSExtraction(MVTModule):
|
||||
"""This class provides a base for all iOS filesystem/backup extraction
|
||||
modules."""
|
||||
|
||||
output_model: ClassVar[OutputModel] = RecordListOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
|
||||
@@ -13,14 +13,11 @@ from mvt.common.module_types import (
|
||||
ModuleResults,
|
||||
ModuleSerializedResult,
|
||||
)
|
||||
from mvt.schemas import MappingOutput
|
||||
|
||||
from ..base import IOSExtraction
|
||||
|
||||
|
||||
class CacheFiles(IOSExtraction):
|
||||
output_model = MappingOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
|
||||
@@ -10,7 +10,6 @@ from typing import Optional
|
||||
|
||||
from mvt.common.module_types import ModuleResults
|
||||
from mvt.common.utils import convert_datetime_to_iso
|
||||
from mvt.schemas import MappingOutput
|
||||
|
||||
from ..base import IOSExtraction
|
||||
|
||||
@@ -33,8 +32,6 @@ class WebkitSessionResourceLog(IOSExtraction):
|
||||
|
||||
"""
|
||||
|
||||
output_model = MappingOutput
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Optional[str] = None,
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
"""Stable public API for MVT output models and JSON Schemas."""
|
||||
|
||||
from .models import (
|
||||
OUTPUT_SCHEMA_VERSION,
|
||||
AlertResult,
|
||||
AlertResults,
|
||||
FileHash,
|
||||
GenericModuleOutput,
|
||||
MappingOutput,
|
||||
ModuleRecord,
|
||||
OutputModel,
|
||||
RecordListOutput,
|
||||
RunInfo,
|
||||
TimelineEvent,
|
||||
TimelineResults,
|
||||
URLResult,
|
||||
URLResults,
|
||||
)
|
||||
from .registry import (
|
||||
export_json_schemas,
|
||||
get_output_model,
|
||||
get_output_models,
|
||||
schema_bundle,
|
||||
)
|
||||
from .serialization import json_compatible, validate_output, write_output
|
||||
|
||||
__all__ = [
|
||||
"OUTPUT_SCHEMA_VERSION",
|
||||
"AlertResult",
|
||||
"AlertResults",
|
||||
"FileHash",
|
||||
"GenericModuleOutput",
|
||||
"MappingOutput",
|
||||
"ModuleRecord",
|
||||
"OutputModel",
|
||||
"RecordListOutput",
|
||||
"RunInfo",
|
||||
"TimelineEvent",
|
||||
"TimelineResults",
|
||||
"URLResult",
|
||||
"URLResults",
|
||||
"export_json_schemas",
|
||||
"get_output_model",
|
||||
"get_output_models",
|
||||
"json_compatible",
|
||||
"schema_bundle",
|
||||
"validate_output",
|
||||
"write_output",
|
||||
]
|
||||
@@ -1,101 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
"""Public Pydantic models for files written by MVT.
|
||||
|
||||
Module records intentionally remain extensible: each extraction module has its own
|
||||
record fields, while the root models make the top-level output shape predictable.
|
||||
More specific modules can replace :class:`ModuleRecord` with a dedicated model.
|
||||
"""
|
||||
|
||||
from typing import Literal, Optional
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, JsonValue, RootModel
|
||||
|
||||
OUTPUT_SCHEMA_VERSION: Literal["1.0"] = "1.0"
|
||||
OutputModel = type[BaseModel]
|
||||
|
||||
|
||||
class MVTOutputModel(BaseModel):
|
||||
"""Base for stable output objects with no undocumented fields."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
|
||||
class ModuleRecord(BaseModel):
|
||||
"""A JSON object produced by an extraction module.
|
||||
|
||||
Module-specific keys are retained and represented in JSON Schema through
|
||||
``additionalProperties``. Dedicated module models can provide stronger field
|
||||
contracts without changing the serialization machinery.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="allow", strict=True)
|
||||
|
||||
|
||||
class RecordListOutput(RootModel[list[ModuleRecord]]):
|
||||
"""The standard output shape for built-in extraction modules."""
|
||||
|
||||
|
||||
class MappingOutput(RootModel[dict[str, JsonValue]]):
|
||||
"""Output shape for modules grouped by a dynamic string key."""
|
||||
|
||||
|
||||
class GenericModuleOutput(RootModel[list[JsonValue] | dict[str, JsonValue]]):
|
||||
"""Compatibility output shape for third-party modules without a model."""
|
||||
|
||||
|
||||
class FileHash(MVTOutputModel):
|
||||
file_path: str
|
||||
sha256: str
|
||||
|
||||
|
||||
class RunInfo(MVTOutputModel):
|
||||
target_path: Optional[str]
|
||||
mvt_version: str
|
||||
date: str
|
||||
ioc_files: list[str]
|
||||
hashes: list[FileHash]
|
||||
output_schema_version: Literal["1.0"]
|
||||
|
||||
|
||||
class URLResult(MVTOutputModel):
|
||||
url: str
|
||||
expanded_url: Optional[str]
|
||||
timestamp: Optional[str]
|
||||
source: str
|
||||
|
||||
|
||||
class URLResults(RootModel[list[URLResult]]):
|
||||
pass
|
||||
|
||||
|
||||
AlertLevelName = Literal["INFORMATIONAL", "LOW", "MEDIUM", "HIGH", "CRITICAL"]
|
||||
|
||||
|
||||
class AlertResult(MVTOutputModel):
|
||||
level: AlertLevelName
|
||||
module: str
|
||||
message: str
|
||||
event_time: str
|
||||
event: dict[str, JsonValue]
|
||||
matched_indicator: Optional[JsonValue] = None
|
||||
|
||||
|
||||
class AlertResults(RootModel[list[AlertResult]]):
|
||||
pass
|
||||
|
||||
|
||||
class TimelineEvent(MVTOutputModel):
|
||||
"""Public contract for the records used to create ``timeline.csv``."""
|
||||
|
||||
timestamp: Optional[str]
|
||||
module: str
|
||||
event: str
|
||||
data: str
|
||||
|
||||
|
||||
class TimelineResults(RootModel[list[TimelineEvent]]):
|
||||
pass
|
||||
@@ -1,133 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
"""Discovery and JSON Schema export for MVT output models."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Iterable
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from .models import (
|
||||
OUTPUT_SCHEMA_VERSION,
|
||||
AlertResults,
|
||||
RunInfo,
|
||||
TimelineResults,
|
||||
URLResults,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from mvt.common.module import MVTModule
|
||||
|
||||
COMMON_OUTPUT_MODELS: dict[str, type[BaseModel]] = {
|
||||
"alerts": AlertResults,
|
||||
"info": RunInfo,
|
||||
"timeline": TimelineResults,
|
||||
"urls": URLResults,
|
||||
}
|
||||
|
||||
JSON_SCHEMA_DIALECT = "https://json-schema.org/draft/2020-12/schema"
|
||||
|
||||
|
||||
def _module_classes(platform: str) -> Iterable[type["MVTModule"]]:
|
||||
if platform == "ios":
|
||||
from mvt.ios.modules.backup import BACKUP_MODULES as IOS_BACKUP_MODULES
|
||||
from mvt.ios.modules.fs import FS_MODULES
|
||||
from mvt.ios.modules.mixed import MIXED_MODULES
|
||||
|
||||
return IOS_BACKUP_MODULES + FS_MODULES + MIXED_MODULES
|
||||
if platform == "android":
|
||||
from mvt.android.modules.androidqf import ANDROIDQF_MODULES
|
||||
from mvt.android.modules.backup import BACKUP_MODULES as ANDROID_BACKUP_MODULES
|
||||
from mvt.android.modules.bugreport import BUGREPORT_MODULES
|
||||
from mvt.android.modules.intrusion_logs import INTRUSION_LOGS_MODULES
|
||||
|
||||
return (
|
||||
ANDROID_BACKUP_MODULES
|
||||
+ BUGREPORT_MODULES
|
||||
+ ANDROIDQF_MODULES
|
||||
+ INTRUSION_LOGS_MODULES
|
||||
)
|
||||
raise ValueError(f"Unsupported MVT platform: {platform}")
|
||||
|
||||
|
||||
def get_output_models(platform: str) -> dict[str, type[BaseModel]]:
|
||||
"""Return common and module output models available for a platform."""
|
||||
|
||||
models = dict(COMMON_OUTPUT_MODELS)
|
||||
for module in _module_classes(platform):
|
||||
models[module.get_slug()] = module.output_model
|
||||
return dict(sorted(models.items()))
|
||||
|
||||
|
||||
def get_output_model(name: str, platform: str | None = None) -> type[BaseModel]:
|
||||
"""Look up an output model by file stem/module slug.
|
||||
|
||||
``platform`` is optional for common outputs. It is required when an output
|
||||
slug exists on both platforms and resolves to different models.
|
||||
"""
|
||||
|
||||
normalized = name.removesuffix(".json")
|
||||
if normalized.endswith("_detected"):
|
||||
return AlertResults
|
||||
if normalized in COMMON_OUTPUT_MODELS:
|
||||
return COMMON_OUTPUT_MODELS[normalized]
|
||||
if platform:
|
||||
try:
|
||||
return get_output_models(platform)[normalized]
|
||||
except KeyError as exc:
|
||||
raise KeyError(f"Unknown {platform} output schema: {name}") from exc
|
||||
|
||||
matches = {
|
||||
models[normalized]
|
||||
for candidate in ("ios", "android")
|
||||
if normalized in (models := get_output_models(candidate))
|
||||
}
|
||||
if len(matches) == 1:
|
||||
return matches.pop()
|
||||
if len(matches) > 1:
|
||||
raise KeyError(f"Output schema {name!r} is ambiguous; specify a platform")
|
||||
raise KeyError(f"Unknown MVT output schema: {name}")
|
||||
|
||||
|
||||
def schema_bundle(platform: str) -> dict[str, Any]:
|
||||
"""Build a versioned bundle of Draft 2020-12 JSON Schemas."""
|
||||
|
||||
return {
|
||||
"output_schema_version": OUTPUT_SCHEMA_VERSION,
|
||||
"platform": platform,
|
||||
"schemas": {
|
||||
name: _output_json_schema(name, model, platform)
|
||||
for name, model in get_output_models(platform).items()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _output_json_schema(
|
||||
name: str, model: type[BaseModel], platform: str
|
||||
) -> dict[str, Any]:
|
||||
schema = model.model_json_schema()
|
||||
return {
|
||||
"$schema": JSON_SCHEMA_DIALECT,
|
||||
"$id": f"urn:mvt:output-schema:{OUTPUT_SCHEMA_VERSION}:{platform}:{name}",
|
||||
**schema,
|
||||
}
|
||||
|
||||
|
||||
def export_json_schemas(destination: str | os.PathLike[str], platform: str) -> list[Path]:
|
||||
"""Write one JSON Schema per output and return the paths created."""
|
||||
|
||||
destination_path = Path(destination)
|
||||
destination_path.mkdir(parents=True, exist_ok=True)
|
||||
written = []
|
||||
for name, model in get_output_models(platform).items():
|
||||
output_path = destination_path / f"{name}.schema.json"
|
||||
with output_path.open("w", encoding="utf-8") as handle:
|
||||
json.dump(_output_json_schema(name, model, platform), handle, indent=2)
|
||||
handle.write("\n")
|
||||
written.append(output_path)
|
||||
return written
|
||||
@@ -1,35 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
"""Validation and serialization helpers shared by all MVT output writers."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from mvt.common.utils import CustomJSONEncoder
|
||||
|
||||
|
||||
def json_compatible(value: Any) -> Any:
|
||||
"""Apply MVT's legacy conversions before validating JSON-native values."""
|
||||
|
||||
return json.loads(json.dumps(value, cls=CustomJSONEncoder))
|
||||
|
||||
|
||||
def validate_output(model: type[BaseModel], value: Any) -> Any:
|
||||
"""Validate an output document and return JSON-serializable Python values."""
|
||||
|
||||
compatible = json_compatible(value)
|
||||
return model.model_validate(compatible).model_dump(mode="json")
|
||||
|
||||
|
||||
def write_output(path: str | Path, model: type[BaseModel], value: Any) -> None:
|
||||
"""Validate and write an MVT JSON document."""
|
||||
|
||||
validated = validate_output(model, value)
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
json.dump(validated, handle, indent=4)
|
||||
@@ -4,8 +4,13 @@
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from mvt.android.modules.bugreport.base import BugReportModule
|
||||
from mvt.android.modules.bugreport.dumpsys_appops import DumpsysAppops
|
||||
from mvt.android.modules.bugreport.dumpsys_getprop import DumpsysGetProp
|
||||
from mvt.android.modules.bugreport.dumpsys_packages import DumpsysPackages
|
||||
@@ -118,3 +123,53 @@ class TestBugreportAnalysis:
|
||||
assert len(m.results) == 2
|
||||
assert m.results[1]["pid"] == 3559
|
||||
assert m.results[0]["sources"]["text"]["parsed"] is True
|
||||
|
||||
def test_dumpstate_section_is_cached_across_parallel_modules(self):
|
||||
dumpstate = (
|
||||
b"header\n"
|
||||
b"------ SYSTEM PROPERTIES (getprop) ------\n"
|
||||
b"[persist.sys.timezone]: [Europe/Berlin]\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
b"------\n"
|
||||
b"DUMP OF SERVICE package:\n"
|
||||
b"package data\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
b"DUMP OF SERVICE package:\n"
|
||||
b"duplicate package data\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
b"DUMP OF SERVICE adb:\n"
|
||||
b"adb data\n"
|
||||
b"------------------------------------------------------------------------------\n"
|
||||
)
|
||||
archive = MagicMock()
|
||||
archive.open.side_effect = lambda _: BytesIO(dumpstate)
|
||||
shared_lock = threading.RLock()
|
||||
shared_cache = {}
|
||||
modules = []
|
||||
|
||||
for _ in range(4):
|
||||
module = BugReportModule()
|
||||
module.from_zip(archive, ["dumpState_test.log"])
|
||||
module.resource_lock = shared_lock
|
||||
module.resource_cache = shared_cache
|
||||
modules.append(module)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=4) as executor:
|
||||
sections = list(
|
||||
executor.map(
|
||||
lambda module: module._get_dumpsys_section(
|
||||
"DUMP OF SERVICE package:"
|
||||
),
|
||||
modules,
|
||||
)
|
||||
)
|
||||
|
||||
assert sections == ["package data"] * 4
|
||||
assert all(section is sections[0] for section in sections)
|
||||
assert modules[0]._get_dumpsys_section_bytes(b"DUMP OF SERVICE adb:") == (
|
||||
b"adb data"
|
||||
)
|
||||
assert modules[0]._get_system_properties_text() == (
|
||||
"[persist.sys.timezone]: [Europe/Berlin]"
|
||||
)
|
||||
archive.open.assert_called_once_with("dumpState_test.log")
|
||||
+304
-19
@@ -5,14 +5,22 @@
|
||||
|
||||
import json
|
||||
import logging
|
||||
from unittest.mock import patch
|
||||
import threading
|
||||
import time
|
||||
from io import StringIO
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
from rich.console import Console
|
||||
|
||||
from mvt.common.command import Command
|
||||
from mvt.common.config import settings
|
||||
from mvt.common.log import MVTLogHandler
|
||||
from mvt.common.module import EncryptedBackupError, MVTModule
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from mvt.common.indicators import Indicators
|
||||
from mvt.common.module import MVTModule
|
||||
|
||||
|
||||
class RecordingModule(MVTModule):
|
||||
@@ -298,6 +306,15 @@ class TestCommand:
|
||||
alerts = json.loads((tmp_path / "alerts.json").read_text())
|
||||
assert alerts[0]["event"]["payload"] == "\\xa8\\xa9"
|
||||
|
||||
def test_timeline_deduplication_preserves_first_seen_order(self):
|
||||
timeline = [
|
||||
{"timestamp": "same", "event": "first"},
|
||||
{"timestamp": "same", "event": "second"},
|
||||
{"timestamp": "same", "event": "first"},
|
||||
]
|
||||
|
||||
assert MVTModule._deduplicate_timeline(timeline) == timeline[:2]
|
||||
|
||||
def test_stores_collected_urls(self, tmp_path):
|
||||
cmd = RecordingCommand(results_path=str(tmp_path))
|
||||
cmd.modules = [URLRecordingModule]
|
||||
@@ -314,7 +331,7 @@ class TestCommand:
|
||||
]
|
||||
|
||||
def test_modules_run_in_stable_topological_order(self):
|
||||
cmd = RecordingCommand()
|
||||
cmd = RecordingCommand(jobs=1)
|
||||
cmd.modules = [ThirdModule, IndependentModule, SecondModule, FirstModule]
|
||||
|
||||
cmd.run()
|
||||
@@ -329,7 +346,7 @@ class TestCommand:
|
||||
assert second.results == ["first", "second"]
|
||||
|
||||
def test_selected_module_runs_transitive_dependencies(self):
|
||||
cmd = RecordingCommand(module_name="ThirdModule")
|
||||
cmd = RecordingCommand(module_name="ThirdModule", jobs=1)
|
||||
cmd.modules = [ThirdModule, SecondModule, FirstModule, IndependentModule]
|
||||
|
||||
cmd.run()
|
||||
@@ -340,6 +357,38 @@ class TestCommand:
|
||||
"ThirdModule",
|
||||
]
|
||||
|
||||
@pytest.mark.parametrize("jobs", [1, 2])
|
||||
@pytest.mark.parametrize("parallel_safe", [False, True])
|
||||
def test_initialization_hook_sees_completed_dependencies(self, jobs, parallel_safe):
|
||||
class Prerequisite(FirstModule):
|
||||
pass
|
||||
|
||||
Prerequisite.parallel_safe = parallel_safe
|
||||
|
||||
class Dependent(SecondModule):
|
||||
dependencies = (Prerequisite,)
|
||||
|
||||
def run(self):
|
||||
self.results = self.get_dependency_results(Prerequisite) + ["second"]
|
||||
|
||||
class HookCommand(RecordingCommand):
|
||||
def module_init(self, module):
|
||||
if isinstance(module, Dependent):
|
||||
self.seen = [
|
||||
executed.results
|
||||
for executed in self.executed
|
||||
if isinstance(executed, Prerequisite)
|
||||
]
|
||||
|
||||
cmd = HookCommand(jobs=jobs)
|
||||
cmd.modules = [Dependent, Prerequisite]
|
||||
|
||||
cmd.run()
|
||||
|
||||
assert cmd.seen == [["first"]]
|
||||
assert [type(module) for module in cmd.executed] == [Prerequisite, Dependent]
|
||||
assert cmd.executed[1].results == ["first", "second"]
|
||||
|
||||
def test_circular_dependency_warns_and_stops(self, caplog):
|
||||
class CircularOne(RecordingModule):
|
||||
pass
|
||||
@@ -372,7 +421,10 @@ class TestCommand:
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["IndependentModule", "FirstModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"IndependentModule",
|
||||
"FirstModule",
|
||||
]
|
||||
assert cmd.initialized
|
||||
assert "Module DependentModule will be SKIPPED" in caplog.text
|
||||
assert "depends on module UnavailableModule" in caplog.text
|
||||
@@ -476,7 +528,7 @@ class TestCommand:
|
||||
]
|
||||
|
||||
def test_selected_custom_module_runs(self):
|
||||
cmd = RecordingCommand(module_name="CustomIOSBackupModule")
|
||||
cmd = RecordingCommand(module_name="CustomIOSBackupModule", jobs=1)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.custom_modules = [CustomIOSBackupModule]
|
||||
@@ -496,7 +548,7 @@ class TestCommand:
|
||||
assert RecordingModule.run_order == []
|
||||
|
||||
def test_custom_module_dependencies_use_topological_order(self):
|
||||
cmd = RecordingCommand(module_name="CustomDependsOnBuiltin")
|
||||
cmd = RecordingCommand(module_name="CustomDependsOnBuiltin", jobs=1)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.modules = [SecondModule, FirstModule]
|
||||
@@ -506,6 +558,228 @@ class TestCommand:
|
||||
|
||||
assert RecordingModule.run_order == ["FirstModule", "CustomDependsOnBuiltin"]
|
||||
|
||||
def test_parallel_modules_overlap_and_respect_worker_limit(self):
|
||||
lock = threading.Lock()
|
||||
barrier = threading.Barrier(2)
|
||||
active = 0
|
||||
maximum = 0
|
||||
overlapped = []
|
||||
|
||||
class ParallelModule(RecordingModule):
|
||||
def run(self):
|
||||
nonlocal active, maximum
|
||||
with lock:
|
||||
active += 1
|
||||
maximum = max(maximum, active)
|
||||
try:
|
||||
barrier.wait(timeout=2)
|
||||
overlapped.append(self.__class__.__name__)
|
||||
time.sleep(0.02)
|
||||
finally:
|
||||
with lock:
|
||||
active -= 1
|
||||
|
||||
class ParallelOne(ParallelModule):
|
||||
pass
|
||||
|
||||
class ParallelTwo(ParallelModule):
|
||||
pass
|
||||
|
||||
class ParallelThree(ParallelModule):
|
||||
pass
|
||||
|
||||
cmd = RecordingCommand(jobs=2)
|
||||
cmd.modules = [ParallelOne, ParallelTwo, ParallelThree]
|
||||
cmd.run()
|
||||
|
||||
assert maximum == 2
|
||||
assert {"ParallelOne", "ParallelTwo"}.issubset(overlapped)
|
||||
|
||||
def test_dependencies_wait_and_transitive_dependencies_run_once(self):
|
||||
prerequisite_finished = threading.Event()
|
||||
runs = []
|
||||
|
||||
class Prerequisite(RecordingModule):
|
||||
def run(self):
|
||||
runs.append("prerequisite")
|
||||
self.results = ["ready"]
|
||||
prerequisite_finished.set()
|
||||
|
||||
class Dependent(RecordingModule):
|
||||
dependencies = (Prerequisite,)
|
||||
|
||||
def run(self):
|
||||
assert prerequisite_finished.is_set()
|
||||
assert self.get_dependency_results(Prerequisite) == ["ready"]
|
||||
runs.append(self.__class__.__name__)
|
||||
|
||||
class DependentOne(Dependent):
|
||||
pass
|
||||
|
||||
class DependentTwo(Dependent):
|
||||
pass
|
||||
|
||||
cmd = RecordingCommand(jobs=3)
|
||||
cmd.modules = [DependentOne, DependentTwo, Prerequisite]
|
||||
cmd.run()
|
||||
|
||||
assert runs.count("prerequisite") == 1
|
||||
assert set(runs[1:]) == {"DependentOne", "DependentTwo"}
|
||||
|
||||
def test_parallel_unsafe_module_runs_exclusively(self):
|
||||
lock = threading.Lock()
|
||||
active = 0
|
||||
unsafe_ran_exclusively = []
|
||||
|
||||
class SafeOne(RecordingModule):
|
||||
def run(self):
|
||||
nonlocal active
|
||||
with lock:
|
||||
active += 1
|
||||
time.sleep(0.03)
|
||||
with lock:
|
||||
active -= 1
|
||||
|
||||
class Unsafe(RecordingModule):
|
||||
parallel_safe = False
|
||||
|
||||
def run(self):
|
||||
nonlocal active
|
||||
with lock:
|
||||
unsafe_ran_exclusively.append(active == 0)
|
||||
active += 1
|
||||
time.sleep(0.01)
|
||||
with lock:
|
||||
active -= 1
|
||||
|
||||
class SafeTwo(SafeOne):
|
||||
pass
|
||||
|
||||
cmd = RecordingCommand(jobs=3)
|
||||
cmd.modules = [SafeOne, Unsafe, SafeTwo]
|
||||
cmd.run()
|
||||
|
||||
assert unsafe_ran_exclusively == [True]
|
||||
|
||||
def test_results_are_aggregated_in_topological_order(self):
|
||||
release_first = threading.Event()
|
||||
|
||||
class SlowFirst(RecordingModule):
|
||||
def run(self):
|
||||
release_first.wait(timeout=2)
|
||||
self.results = ["first"]
|
||||
self.timeline = [{"module": "first"}]
|
||||
self.alertstore.info("first", "", {"order": 1})
|
||||
|
||||
class FastSecond(RecordingModule):
|
||||
def run(self):
|
||||
self.results = ["second"]
|
||||
self.timeline = [{"module": "second"}]
|
||||
self.alertstore.info("second", "", {"order": 2})
|
||||
release_first.set()
|
||||
|
||||
cmd = RecordingCommand(jobs=2)
|
||||
cmd.modules = [SlowFirst, FastSecond]
|
||||
cmd.run()
|
||||
|
||||
assert [type(module) for module in cmd.executed] == [SlowFirst, FastSecond]
|
||||
assert [entry["module"] for entry in cmd.timeline] == ["first", "second"]
|
||||
assert [alert.message for alert in cmd.alertstore.alerts] == ["first", "second"]
|
||||
|
||||
def test_parallel_console_logs_are_grouped_and_file_log_is_complete(
|
||||
self, tmp_path
|
||||
):
|
||||
output = StringIO()
|
||||
handler = MVTLogHandler(
|
||||
console=Console(file=output, force_terminal=False, color_system=None)
|
||||
)
|
||||
handler.setFormatter(logging.Formatter("%(message)s"))
|
||||
logger = logging.getLogger("mvt")
|
||||
logger.setLevel(logging.DEBUG)
|
||||
logger.addHandler(handler)
|
||||
release_slow = threading.Event()
|
||||
|
||||
class SlowLog(RecordingModule):
|
||||
def run(self):
|
||||
self.log.info("slow-a")
|
||||
release_slow.wait(timeout=2)
|
||||
self.log.info("slow-b")
|
||||
|
||||
class FastLog(RecordingModule):
|
||||
def run(self):
|
||||
self.log.info("fast-a")
|
||||
self.log.info("fast-b")
|
||||
release_slow.set()
|
||||
|
||||
SlowLog.__module__ = "mvt.tests.parallel"
|
||||
FastLog.__module__ = "mvt.tests.parallel"
|
||||
try:
|
||||
cmd = RecordingCommand(results_path=str(tmp_path), jobs=2)
|
||||
cmd.modules = [SlowLog, FastLog]
|
||||
cmd.run()
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
console_output = output.getvalue()
|
||||
fast_block = (
|
||||
console_output.index("--- FastLog ---"),
|
||||
console_output.index("fast-a"),
|
||||
console_output.index("fast-b"),
|
||||
)
|
||||
slow_block = (
|
||||
console_output.index("--- SlowLog ---"),
|
||||
console_output.index("slow-a"),
|
||||
console_output.index("slow-b"),
|
||||
)
|
||||
assert fast_block == tuple(sorted(fast_block))
|
||||
assert slow_block == tuple(sorted(slow_block))
|
||||
assert fast_block[-1] < slow_block[0] or slow_block[-1] < fast_block[0]
|
||||
file_output = (tmp_path / "command.log").read_text()
|
||||
for message in ("slow-a", "slow-b", "fast-a", "fast-b"):
|
||||
assert message in file_output
|
||||
|
||||
def test_encrypted_backup_stops_scheduling_and_cleans_up(self, caplog):
|
||||
finish_called = []
|
||||
|
||||
class Encrypted(RecordingModule):
|
||||
def run(self):
|
||||
raise EncryptedBackupError
|
||||
|
||||
class NeverScheduled(RecordingModule):
|
||||
dependencies = (Encrypted,)
|
||||
|
||||
class CleanupCommand(RecordingCommand):
|
||||
def finish(self):
|
||||
finish_called.append(True)
|
||||
|
||||
cmd = CleanupCommand(jobs=2)
|
||||
cmd.modules = [Encrypted, NeverScheduled]
|
||||
with caplog.at_level(logging.CRITICAL):
|
||||
cmd.run()
|
||||
|
||||
assert finish_called == [True]
|
||||
assert not any(isinstance(module, NeverScheduled) for module in cmd.executed)
|
||||
assert "backup appears to be encrypted" in caplog.text
|
||||
|
||||
def test_profiling_forces_sequential_execution(self, monkeypatch, caplog):
|
||||
run_order = []
|
||||
|
||||
class ProfileOne(RecordingModule):
|
||||
def run(self):
|
||||
run_order.append("one")
|
||||
|
||||
class ProfileTwo(RecordingModule):
|
||||
def run(self):
|
||||
run_order.append("two")
|
||||
|
||||
monkeypatch.setattr(settings, "PROFILE", True)
|
||||
cmd = RecordingCommand(jobs=4)
|
||||
cmd.modules = [ProfileOne, ProfileTwo]
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert run_order == ["one", "two"]
|
||||
assert "forcing sequential module execution" in caplog.text
|
||||
def test_custom_module_replaces_builtin(self, caplog):
|
||||
cmd = RecordingCommand()
|
||||
cmd.platform = "ios"
|
||||
@@ -516,7 +790,10 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["IndependentModule", "ReplacementModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"IndependentModule",
|
||||
"ReplacementModule",
|
||||
]
|
||||
assert cmd.module_replacements == {FirstModule: ReplacementModule}
|
||||
assert (
|
||||
"Module ReplacementModule from" in caplog.text
|
||||
@@ -532,7 +809,10 @@ class TestCommand:
|
||||
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["FirstModule", "CustomIOSBackupModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"FirstModule",
|
||||
"CustomIOSBackupModule",
|
||||
]
|
||||
assert cmd.module_replacements == {}
|
||||
|
||||
def test_unavailable_replaced_module_is_ignored(self, caplog):
|
||||
@@ -545,7 +825,10 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == ["IndependentModule", "ReplacementModule"]
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"IndependentModule",
|
||||
"ReplacementModule",
|
||||
]
|
||||
assert cmd.module_replacements == {}
|
||||
assert "replaces module FirstModule" not in caplog.text
|
||||
|
||||
@@ -663,7 +946,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"ReplacementModule",
|
||||
"OtherReplacementModule",
|
||||
]
|
||||
@@ -722,8 +1005,9 @@ class TestCommand:
|
||||
assert cmd.module_replacements == {}
|
||||
assert logged_substitutions(caplog) == []
|
||||
|
||||
def test_modules_replacing_each_other_all_run(self, caplog):
|
||||
cmd = RecordingCommand()
|
||||
@pytest.mark.parametrize("jobs", [1, 4])
|
||||
def test_modules_replacing_each_other_all_run(self, caplog, jobs):
|
||||
cmd = RecordingCommand(jobs=jobs)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.custom_modules = [MutualReplacementOne, MutualReplacementTwo]
|
||||
@@ -731,7 +1015,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"MutualReplacementOne",
|
||||
"MutualReplacementTwo",
|
||||
]
|
||||
@@ -742,8 +1026,9 @@ class TestCommand:
|
||||
)
|
||||
assert logged_substitutions(caplog) == []
|
||||
|
||||
def test_replacement_cycle_of_three_modules_all_run(self, caplog):
|
||||
cmd = RecordingCommand()
|
||||
@pytest.mark.parametrize("jobs", [1, 4])
|
||||
def test_replacement_cycle_of_three_modules_all_run(self, caplog, jobs):
|
||||
cmd = RecordingCommand(jobs=jobs)
|
||||
cmd.platform = "ios"
|
||||
cmd.name = "check-backup"
|
||||
cmd.custom_modules = [CycleOneModule, CycleTwoModule, CycleThreeModule]
|
||||
@@ -751,7 +1036,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.INFO):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"CycleOneModule",
|
||||
"CycleTwoModule",
|
||||
"CycleThreeModule",
|
||||
@@ -834,7 +1119,7 @@ class TestCommand:
|
||||
with caplog.at_level(logging.WARNING):
|
||||
cmd.run()
|
||||
|
||||
assert RecordingModule.run_order == [
|
||||
assert [type(module).__name__ for module in cmd.executed] == [
|
||||
"SharedSlugModule",
|
||||
"OtherSharedSlugModule",
|
||||
]
|
||||
|
||||
@@ -1,131 +0,0 @@
|
||||
# Mobile Verification Toolkit (MVT)
|
||||
# Copyright (c) 2026 The MVT Authors.
|
||||
# Use of this software is governed by the MVT License 1.1 that can be found at
|
||||
# https://license.mvt.re/1.1/
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
from pydantic import BaseModel, ConfigDict, RootModel, ValidationError
|
||||
|
||||
from mvt.android.cli import cli as android_cli
|
||||
from mvt.common.module import MVTModule
|
||||
from mvt.ios.cli import cli as ios_cli
|
||||
from mvt.schemas import (
|
||||
OUTPUT_SCHEMA_VERSION,
|
||||
GenericModuleOutput,
|
||||
MappingOutput,
|
||||
RecordListOutput,
|
||||
export_json_schemas,
|
||||
get_output_model,
|
||||
get_output_models,
|
||||
schema_bundle,
|
||||
validate_output,
|
||||
)
|
||||
|
||||
|
||||
class CountRecord(BaseModel):
|
||||
model_config = ConfigDict(strict=True)
|
||||
count: int
|
||||
|
||||
|
||||
class CountOutput(RootModel[list[CountRecord]]):
|
||||
pass
|
||||
|
||||
|
||||
class CountModule(MVTModule):
|
||||
output_model = CountOutput
|
||||
|
||||
|
||||
def test_common_output_model_validates_without_changing_root_shape():
|
||||
assert validate_output(RecordListOutput, [{"name": "example", "size": 2}]) == [
|
||||
{"name": "example", "size": 2}
|
||||
]
|
||||
assert validate_output(MappingOutput, {"global": {"enabled": "1"}}) == {
|
||||
"global": {"enabled": "1"}
|
||||
}
|
||||
|
||||
|
||||
def test_module_model_is_used_when_loading_existing_output(tmp_path):
|
||||
path = tmp_path / "count.json"
|
||||
path.write_text('[{"count": 4}]', encoding="utf-8")
|
||||
|
||||
module = CountModule.from_json(str(path), logging.getLogger(__name__))
|
||||
|
||||
assert module.results == [{"count": 4}]
|
||||
|
||||
|
||||
def test_invalid_module_output_is_not_written(tmp_path, caplog):
|
||||
module = CountModule(results_path=str(tmp_path), results=[{"count": "four"}])
|
||||
|
||||
with caplog.at_level(logging.ERROR):
|
||||
module.save_to_json()
|
||||
|
||||
assert not (tmp_path / "count_module.json").exists()
|
||||
assert "does not match schema CountOutput" in caplog.text
|
||||
|
||||
|
||||
def test_invalid_existing_output_raises_validation_error(tmp_path):
|
||||
path = tmp_path / "count.json"
|
||||
path.write_text('[{"count": "four"}]', encoding="utf-8")
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
CountModule.from_json(str(path), logging.getLogger(__name__))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("platform", ["ios", "android"])
|
||||
def test_every_builtin_module_has_a_non_generic_output_model(platform):
|
||||
module_models = {
|
||||
name: model
|
||||
for name, model in get_output_models(platform).items()
|
||||
if name not in {"alerts", "info", "timeline", "urls"}
|
||||
}
|
||||
|
||||
assert module_models
|
||||
assert GenericModuleOutput not in module_models.values()
|
||||
|
||||
|
||||
def test_known_mapping_and_specific_models_are_registered():
|
||||
assert get_output_model("backup_info.json", "ios") is MappingOutput
|
||||
assert get_output_model("dumpsys_receivers", "android") is MappingOutput
|
||||
assert get_output_model("tombstones", "android").__name__ == "TombstoneCrashOutput"
|
||||
assert get_output_model("sms_detected.json", "ios").__name__ == "AlertResults"
|
||||
|
||||
|
||||
def test_schema_bundle_has_stable_versioned_shape():
|
||||
bundle = schema_bundle("android")
|
||||
|
||||
assert bundle["output_schema_version"] == OUTPUT_SCHEMA_VERSION
|
||||
assert bundle["platform"] == "android"
|
||||
assert bundle["schemas"]["urls"]["$schema"].endswith("2020-12/schema")
|
||||
assert bundle["schemas"]["urls"]["type"] == "array"
|
||||
assert bundle["schemas"]["info"]["additionalProperties"] is False
|
||||
assert bundle["schemas"]["dumpsys_receivers"]["type"] == "object"
|
||||
|
||||
|
||||
def test_export_json_schemas_writes_one_file_per_registered_output(tmp_path):
|
||||
paths = export_json_schemas(tmp_path, "android")
|
||||
|
||||
assert len(paths) == len(get_output_models("android"))
|
||||
assert json.loads((tmp_path / "info.schema.json").read_text())["title"] == (
|
||||
"RunInfo"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("cli", "platform"), [(ios_cli, "ios"), (android_cli, "android")]
|
||||
)
|
||||
def test_schema_cli_prints_machine_readable_bundle(cli, platform):
|
||||
result = CliRunner().invoke(cli, ["schemas"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert json.loads(result.output)["platform"] == platform
|
||||
|
||||
|
||||
def test_schema_cli_exports_schema_files(tmp_path):
|
||||
result = CliRunner().invoke(android_cli, ["schemas", "--output", str(tmp_path)])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert (tmp_path / "urls.schema.json").exists()
|
||||
@@ -41,3 +41,11 @@ class TestCheckBackupCommand:
|
||||
result = runner.invoke(check_backup, [str(tmp_path)])
|
||||
assert result.exit_code == 1
|
||||
assert "does not appear to be an iTunes backup folder" in caplog.text
|
||||
|
||||
def test_check_rejects_non_positive_jobs(self):
|
||||
result = CliRunner().invoke(
|
||||
check_backup, ["--jobs", "0", get_ios_backup_folder()]
|
||||
)
|
||||
|
||||
assert result.exit_code == 2
|
||||
assert "Invalid value for '--jobs'" in result.output
|
||||
@@ -288,15 +288,19 @@ def test_androidqf_propagates_custom_modules_to_nested_commands(tmp_path, monkey
|
||||
cmd = CmdAndroidCheckAndroidQF(
|
||||
target_path=str(tmp_path),
|
||||
custom_modules=custom_modules,
|
||||
jobs=7,
|
||||
)
|
||||
|
||||
def record_available(name):
|
||||
def _record(command):
|
||||
records[name] = [
|
||||
module.__name__
|
||||
for module in command._available_modules()
|
||||
if module.__name__.startswith("Nested")
|
||||
]
|
||||
records[name] = {
|
||||
"jobs": command.jobs,
|
||||
"modules": [
|
||||
module.__name__
|
||||
for module in command._available_modules()
|
||||
if module.__name__.startswith("Nested")
|
||||
],
|
||||
}
|
||||
|
||||
return _record
|
||||
|
||||
@@ -339,7 +343,7 @@ def test_androidqf_propagates_custom_modules_to_nested_commands(tmp_path, monkey
|
||||
assert cmd.run_backup_cmd()
|
||||
assert cmd.run_intrusion_logs_cmd()
|
||||
assert records == {
|
||||
"bugreport": ["NestedBugreportModule"],
|
||||
"backup": ["NestedBackupModule"],
|
||||
"intrusion_logs": ["NestedIntrusionLogsModule"],
|
||||
"bugreport": {"jobs": 7, "modules": ["NestedBugreportModule"]},
|
||||
"backup": {"jobs": 7, "modules": ["NestedBackupModule"]},
|
||||
"intrusion_logs": {"jobs": 7, "modules": ["NestedIntrusionLogsModule"]},
|
||||
}
|
||||
Reference in new issue
Block a user