From 1cf82f44e91cdd9b861366a8ea3e61e53efb2b71 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Donncha=20=C3=93=20Cearbhaill?= Date: Wed, 26 Aug 2026 21:24:25 +0200 Subject: [PATCH] Document how to write MVT plugins The custom modules page now leads with plugin packages. Loading module files with --load-module and MVT_CUSTOM_MODULES moves to a section on developing a module locally. A new "Writing a module" section shows a module which subclasses IOSExtraction. It lists each base class, the command pair it serves and the helpers it provides. "Depending on a built-in module" says to import a built-in class from its family package. "Importing from MVT" says what mvt.plugin exports and what importing from it means. The custom commands page shows a Command subclass which lists its own modules. The sysdiagnose and plugin configuration pages import from mvt.plugin. --- docs/development/custom_commands.md | 62 +++++++ docs/development/index.md | 214 +++++++++++++++++------ docs/development/plugin_configuration.md | 22 ++- docs/ios/sysdiagnose.md | 32 ++-- 4 files changed, 253 insertions(+), 77 deletions(-) diff --git a/docs/development/custom_commands.md b/docs/development/custom_commands.md index 71ac623..3161cef 100644 --- a/docs/development/custom_commands.md +++ b/docs/development/custom_commands.md @@ -29,6 +29,10 @@ def summarize(path): click.echo(f"Summarizing {path}") ``` +Log through `get_plugin_logger(__name__)` from `mvt.plugin`. Records logged +through `logging.getLogger(__name__)` do not reach `command.log`, and MVT's +console handler does not show them. + Register the object in the package's `pyproject.toml`. The entry-point name is the command users invoke: @@ -125,6 +129,64 @@ export MVT_IOS_CUSTOM_COMMANDS=./ios_commands export MVT_ANDROID_CUSTOM_COMMANDS=./android_commands ``` +## Building a Module-Running Command + +A command which runs forensic modules over an acquisition subclasses `Command`. +`Command` creates the output folder and writes `command.log`. It orders the +modules, resolves their dependencies and runs them. It writes the result files, +`alerts.json` and `info.json`. The subclass sets `platform`, `name` and +`modules`: + +```python +from mvt.plugin import Command, MVTModule, convert_unix_to_iso, get_plugin_logger + +log = get_plugin_logger(__name__) + + +class APKManifest(MVTModule): + supported_commands = (("android", "check-apks"),) + + def run(self): + self.results = [{"checked_at": convert_unix_to_iso(0)}] + + +class CmdCheckAPKs(Command): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.platform = "android" + self.name = "check-apks" + self.modules = [APKManifest] +``` + +`platform` and `name` are the pair the modules declare in +`supported_commands`. `modules` lists the module classes the command runs, +imported directly. A command which runs modules of another plugin depends on +that package in `pyproject.toml` and imports them the same way. `init()`, +`module_init(module)` and `finish()` are optional hooks. `run()` calls them +before the run, before each module and after the run. + +Wrap the command in a Click command with an `--output` option. Run it, then +print the alert summary: + +```python +import click + + +@click.command("check-apks") +@click.option("--output", "-o", type=click.Path(exists=False)) +@click.argument("TARGET_PATH", type=click.Path(exists=True)) +def cli(output, target_path): + cmd = CmdCheckAPKs(target_path=target_path, results_path=output) + log.info("Checking APK files at path: %s", target_path) + cmd.run() + cmd.show_alerts_brief() +``` + +The `--verbose` option of `mvt`, `mvt-ios` and `mvt-android` applies to the +command. The command defines no `--verbose` option of its own. The pair a +plugin command adds is not listed anywhere in MVT. Name it in the plugin's +README. + ## Naming and Errors Built-in MVT commands cannot be replaced. External command names must also be diff --git a/docs/development/index.md b/docs/development/index.md index 692a2ab..d1c167f 100644 --- a/docs/development/index.md +++ b/docs/development/index.md @@ -48,51 +48,21 @@ configuration problem: the command logs a warning and runs no modules at all. ## Custom modules -Module-running `check-*` commands can load custom modules from Python files that -are not installed as part of MVT. Load one file with: +MVT's module-running `check-*` commands can run forensic modules which are not +part of MVT. Custom modules are distributed as plugin packages. A Python +package installed next to MVT registers its modules through an entry point. +The modules then load automatically in every command they support, see +[Installed module packages](#installed-module-packages). `mvt plugins list` +shows the installed packages and where each one was installed from. -```bash -mvt-ios check-backup --load-module ./example_module.py --output ./out ./backup -``` +MVT can also load module files by path, with `--load-module` and +`MVT_CUSTOM_MODULES`, see +[Developing modules locally](#developing-modules-locally). This can be used +while writing a module. Use a Python package to distribute one. -You can also load a folder. MVT loads non-hidden top-level `*.py` files in -sorted order and skips `__init__.py`: - -```bash -mvt-ios check-fs --load-module ./custom_modules ./filesystem-dump -``` - -Set `MVT_CUSTOM_MODULES` to load a folder for every module-running command. This -folder is loaded before any `--load-module` path: - -```bash -MVT_CUSTOM_MODULES=./custom_modules mvt-android check-bugreport ./bugreport.zip -``` - -Custom modules are normal `MVTModule` subclasses: - -```python -from mvt.common.module import MVTModule - - -class ExampleCustomModule(MVTModule): - supported_commands = (("ios", "check-backup"), ("ios", "check-fs")) - slug = "example_custom_module" - - def run(self): - self.results = [{"message": "custom module ran"}] - - def check_indicators(self): - pass - - def serialize(self, result): - return None -``` - -Use `supported_commands` to declare the platform/command pairs a module -supports. Empty `supported_commands` means the module will not run and MVT logs -a warning. This explicit declaration is required for every command. Supported -pairs are: +A custom module declares in `supported_commands` the platform and command +pairs it runs in. A module with empty `supported_commands` does not run and +MVT logs a warning. The nine pairs are: ```python ("ios", "check-backup") @@ -106,13 +76,86 @@ pairs are: ("android", "check-iocs") ``` -Custom modules can depend on existing MVT module classes. Dependencies are -resolved with the same ordering logic as built-in modules, and custom modules -are appended after built-ins before ordering: +`check-iocs` re-checks stored results rather than an acquisition. It matches +every `.json` file in the results folder to the module with that slug. +It then runs that module's `check_indicators()` again. + +### Writing a module + +A module subclasses `MVTModule` or one of the base classes below and +implements `run()`. `check_indicators()` and `serialize()` are optional. The +first matches results against IOCs or detections. The second returns timeline +records. ```python -from mvt.common.module import MVTModule -from mvt.ios.modules.backup.manifest import Manifest +from mvt.plugin import IOSExtraction, convert_unix_to_iso + + +class ExampleCustomModule(IOSExtraction): + supported_commands = ( + ("ios", "check-backup"), + ("ios", "check-fs"), + ) + slug = "example_custom_module" + + def run(self): + self.results = [{"checked_at": convert_unix_to_iso(0)}] + + def check_indicators(self): + pass + + def serialize(self, result): + return None +``` + +The base classes are: + +- `MVTModule`: the base of every module. It provides `self.results`, + `self.alertstore`, `self.log`, `self.indicators` and + `get_dependency_results()`. Subclass it directly for a module which reads + only the results of other modules. +- `IOSExtraction`: `("ios", "check-backup")` and `("ios", "check-fs")`. Adds + `_find_ios_database()`, which locates a module's database in a backup or in + a filesystem dump and repairs it if it is malformed. Adds + `_get_backup_files_from_manifest()`, `_get_backup_file_from_id()` and + `_get_fs_files_from_patterns()`. Adds `_open_sqlite_db()`, which opens a + database read-only. +- `SysdiagnoseExtraction`: `("ios", "check-sysdiagnose")`. MVT extracts the + archive and calls `from_sysdiagnose_folder()` before `run()`. The module + reads files with `_get_files_by_pattern()` and `_get_file_content()`. + `ips_files` lists the crash reports. See + [Check an iOS Sysdiagnose](../ios/sysdiagnose.md). +- `AndroidQFModule`: `("android", "check-androidqf")`. MVT calls `from_dir()` + or `from_zip()` with the file list of the acquisition. The module reads + files with `_get_files_by_pattern()` and `_get_file_content()`. + `_get_device_timezone()` returns the device timezone. +- `AndroidBackupModule`: `("android", "check-backup")`. MVT calls `from_dir()` + or `from_ab()`. The module reads files with `_get_files_by_pattern()` and + `_get_file_content()`. +- `BugReportModule`: `("android", "check-bugreport")`. MVT calls `from_dir()` + or `from_zip()`. The module reads files with `_get_files_by_pattern()`, + `_get_files_by_patterns()` and `_get_file_content()`. + `_get_dumpstate_file()` returns the dumpstate file, and + `_get_file_modification_time()` the modification time of a file. + +The underscore-named helpers are internal to the base classes. Plugin modules +can call them. Their names and signatures can change between releases. Read +the base class in `src/mvt/ios/modules` or `src/mvt/android/modules` before +relying on one. + +### Depending on a built-in module + +A module which post-processes records generated by one or more built-in MVT +modules must declare the source modules in `dependencies`. It reads their +results with `get_dependency_results()`. Import the class from its family +package: `mvt.ios.modules.backup`, `mvt.ios.modules.fs`, +`mvt.ios.modules.mixed`, `mvt.android.modules.androidqf`, +`mvt.android.modules.backup`, `mvt.android.modules.bugreport` or +`mvt.android.modules.intrusion_logs`. + +```python +from mvt.ios.modules.backup import Manifest +from mvt.plugin import MVTModule class DependentCustomModule(MVTModule): @@ -124,6 +167,30 @@ class DependentCustomModule(MVTModule): self.results = [{"manifest_entries": len(manifest_results)}] ``` +Dependencies are ordered as for the built-in modules, with custom modules +appended after the built-ins, see [Module dependencies](#module-dependencies). +A dependency has to run in every command the module supports. Where it does +not, MVT skips the module with a warning. + +`get_dependency_results()` returns the plain dictionaries the module produced. +They are the same records it writes to `.json`. Typed results per module +are planned. + +### Importing from MVT + +Import from `mvt.plugin` if it has what you need. The names it exports are +kept working on a best-effort basis. A change to one of them is announced in +the release notes. Anything else in `mvt` can be imported too, but may change +between releases without notice. The plugin interface is best effort. + +`mvt.plugin` exports the base classes above and `Command`, `Alert` and +`AlertLevel`, the result types, `DatabaseNotFoundError` and +`DatabaseCorruptedError`, the timestamp converters, the settings API of +[Plugin Configuration](plugin_configuration.md), MVT's own `settings`, +`get_plugin_logger()` and `MVT_VERSION`. `src/mvt/plugin.py` holds the list. +Read MVT's `settings` for values such as `NETWORK_ACCESS_ALLOWED` and +`NETWORK_TIMEOUT`. Plugin values go in the plugin's own settings file. + ## Installed module packages Python packages can register modules so they load automatically in every @@ -133,14 +200,14 @@ the package's `pyproject.toml`: ```toml [project.entry-points."mvt.modules"] -mvt-plugin-amnesty-custom = "mvt_plugin_amnesty_custom:get_modules" +mvt-plugin-example-org = "mvt_plugin_example_org:get_modules" ``` The entry point must resolve to an iterable of `MVTModule` subclasses, or to a callable returning one: ```python -from mvt.common.module import MVTModule +from mvt.plugin import MVTModule class PackagedModule(MVTModule): @@ -154,6 +221,10 @@ def get_modules() -> list[type[MVTModule]]: return [PackagedModule] ``` +`get_modules()` is the package's module list, written by hand. A package which +keeps its modules in separate files imports each class there and lists it. A +module missing from the list does not load. + Installed modules follow the same rules as other custom modules: each module must declare `supported_commands`, and dependencies are resolved with the standard ordering logic. A broken entry point is skipped with a warning and @@ -164,7 +235,7 @@ from sources you trust. For a `pipx` installation of MVT, inject the package into MVT's environment: ```bash -pipx inject mvt mvt-plugin-amnesty-custom +pipx inject mvt mvt-plugin-example-org ``` Module packages that need their own settings, such as an API key, should store @@ -175,9 +246,9 @@ rather than in MVT's own `config.yaml`. Name module packages `mvt-plugin-` (import package `mvt_plugin_`), and include the name of the publishing organization or author so packages from -different groups do not collide: for example, Amnesty International's custom -modules would be distributed as `mvt-plugin-amnesty-custom` with the import -package `mvt_plugin_amnesty_custom`. +different groups do not collide: for example, an organisation's custom modules +would be distributed as `mvt-plugin-example-org` with the import package +`mvt_plugin_example_org`. The prefix makes module packages easy to find on PyPI and keeps their import names from clashing with unrelated Python packages. It is a convention, not a @@ -196,11 +267,42 @@ came from. MVT's own modules log under their dotted path (for example MVT's internal logger tree: - Installed packages log under `mvt.ext.`, with the `mvt_plugin_` - prefix stripped: modules in `mvt_plugin_amnesty_custom` log as - `mvt.ext.amnesty_custom.*`. + prefix stripped: modules in `mvt_plugin_example_org` log as + `mvt.ext.example_org.*`. - Files loaded with `--load-module` or `MVT_CUSTOM_MODULES` log as `mvt.ext.`. +Outside a module class, for example in a custom command line handler, log +through `get_plugin_logger(__name__)`. It returns a logger in the same +namespace. + +## Developing modules locally + +While a module is being written, load it from its file. `--load-module` takes a +Python file, or a folder of them, on every module-running command, and can be +repeated: + +```bash +mvt-ios check-backup --load-module ./example_module.py --output ./out ./backup +``` + +For a folder, MVT loads its non-hidden top-level `*.py` files in sorted order +and skips `__init__.py`. `MVT_CUSTOM_MODULES` names a folder to load on every +module-running command, before any `--load-module` path: + +```bash +MVT_CUSTOM_MODULES=./custom_modules mvt-android check-bugreport ./bugreport.zip +``` + +Files loaded this way follow the same rules as packaged modules. +`--list-modules` reports them with the SHA-256 hash of the file in place of a +version. An editable install of the package (`pip install -e .`) also works: +the modules load through the entry point, and `mvt plugins list` shows the +package with the `local` origin. + +Loading by path is for development. Move a module into a package once it +works. + ## Auditing loaded modules Because installed module packages load automatically, MVT records where every diff --git a/docs/development/plugin_configuration.md b/docs/development/plugin_configuration.md index e41f4e4..62d35a7 100644 --- a/docs/development/plugin_configuration.md +++ b/docs/development/plugin_configuration.md @@ -24,8 +24,7 @@ configuration: The exact parent folder follows the platform convention used for MVT's `config.yaml` (for example `~/Library/Application Support/mvt` on macOS). Use -`mvt.common.plugin_config.plugin_config_path()` instead of building the path by -hand. +`plugin_config_path()` from `mvt.plugin` instead of building the path by hand. Plugin names must be lowercase and may only contain letters, digits and dashes, matching the `mvt-plugin-` package naming convention. MVT creates the @@ -38,9 +37,8 @@ leaves a partially written settings file behind. Everything else a plugin keeps on disk, such as a cache, a downloaded artifact or synchronization state, belongs in the folder returned by the `data_folder()` -class method of the plugin's settings class, or by -`mvt.common.plugin_config.plugin_data_folder()` called with the plugin name if -the plugin has no settings class: +class method of the plugin's settings class, or by `plugin_data_folder()` from +`mvt.plugin`, called with the plugin name if the plugin has no settings class: ``` ~/.local/share/mvt/plugin-data// # Linux @@ -76,7 +74,7 @@ defaults: ```python from typing import Optional -from mvt.common.plugin_config import MVTPluginSettings +from mvt.plugin import MVTPluginSettings class ExamplePluginSettings(MVTPluginSettings): @@ -94,12 +92,15 @@ from datetime import datetime, timezone import click +from mvt.plugin import plugin_env_prefix + def sync(): settings = ExamplePluginSettings.load() if not settings.API_KEY: + prefix = plugin_env_prefix(settings.plugin_name) raise click.ClickException( - "No API key configured. Set MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY or " + f"No API key configured. Set {prefix}API_KEY or " "run 'example-plugin configure'." ) @@ -107,6 +108,9 @@ def sync(): settings.save() ``` +The message builds the variable name with `plugin_env_prefix()`, see +[Environment Variables](#environment-variables). + A missing settings file is not an error: the plugin then runs on the field defaults and on whatever the environment provides. `save()` only persists the values that differ from the defaults, and it never touches MVT's `config.yaml`. @@ -129,6 +133,10 @@ export MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY=... export MVT_PLUGIN_EXAMPLE_PLUGIN_MAX_RESULTS=50 ``` +Do not repeat the plugin name in a field name: a plugin named `example-scanner` +with a `SCANNER_API_KEY` field asks the user for +`MVT_PLUGIN_EXAMPLE_SCANNER_SCANNER_API_KEY`. Name the field `API_KEY`. + Settings resolve in this order, from highest to lowest priority: 1. Arguments passed to the settings class directly, such as diff --git a/docs/ios/sysdiagnose.md b/docs/ios/sysdiagnose.md index 6488bc2..d0d84fb 100644 --- a/docs/ios/sysdiagnose.md +++ b/docs/ios/sysdiagnose.md @@ -1,30 +1,32 @@ # Check an iOS Sysdiagnose `mvt-ios check-sysdiagnose` prepares an iOS sysdiagnose archive for analysis by -custom MVT modules. MVT does not include built-in sysdiagnose modules. You must -load at least one custom module that explicitly supports this command. +custom MVT modules. MVT does not include built-in sysdiagnose modules. The +command runs the modules of the installed +[plugin packages](../development/index.md#installed-module-packages) which +declare support for it. Install at least one such package first. The command accepts either an extracted sysdiagnose directory or the original gzip-compressed tar archive. ```bash -mvt-ios check-sysdiagnose \ - --load-module ./sysdiagnose_modules.py \ - --output ./results \ +mvt-ios check-sysdiagnose --output ./results \ ./sysdiagnose_2024.01.02_03-04-05+0200.tar.gz ``` Use `--hashes` to include hashes for analyzed files in `info.json`, and -`--list-modules` to display the eligible custom modules without running them. +`--list-modules` to display the eligible modules without running them. ## Writing a custom module -Extend `SysdiagnoseExtraction` to access the archive contents consistently for -both directory and tar inputs. Each module must declare the command explicitly -in `supported_commands`. +Extend `SysdiagnoseExtraction` from `mvt.plugin`, see +[Writing a module](../development/index.md#writing-a-module). The module reads +the archive the same way whether MVT was given a folder or a tar archive. It +declares the command in `supported_commands`. While writing one, +[load it from its file](../development/index.md#developing-modules-locally). ```python -from mvt.ios.modules.sysdiagnose import SysdiagnoseExtraction +from mvt.plugin import SysdiagnoseExtraction class ExampleSysdiagnoseModule(SysdiagnoseExtraction): @@ -44,7 +46,9 @@ class ExampleSysdiagnoseModule(SysdiagnoseExtraction): return None ``` -The base class provides `from_sysdiagnose_folder()` and -`from_sysdiagnose_tar()` setup hooks, as well as protected file lookup, file -reading, and timezone extraction helpers. IPS crash-report metadata is exposed -on `ips_files`. +MVT extracts a tar archive first. It calls `from_sysdiagnose_folder()` on each +module before `run()`. `ips_files` lists the IPS crash reports. + +`_get_files_by_pattern()` and `_get_file_content()` are internal helpers of the +base class. Use them to read the archive. Their names and signatures can change +between releases. See `src/mvt/ios/modules/sysdiagnose/base.py`.