Document how to write MVT plugins

The custom modules page now leads with plugin packages. Loading module
files with --load-module and MVT_CUSTOM_MODULES moves to a section on
developing a module locally.

A new "Writing a module" section shows a module which subclasses
IOSExtraction. It lists each base class, the command pair it serves and the
helpers it provides. "Depending on a built-in module" says to import a
built-in class from its family package.

"Importing from MVT" says what mvt.plugin exports and what importing from
it means. The custom commands page shows a Command subclass which lists its
own modules. The sysdiagnose and plugin configuration pages import from
mvt.plugin.

docs/development/custom_commands.md

@@ -29,6 +29,10 @@ def summarize(path):
     click.echo(f"Summarizing {path}")
 ```
 
+Log through `get_plugin_logger(__name__)` from `mvt.plugin`. Records logged
+through `logging.getLogger(__name__)` do not reach `command.log`, and MVT's
+console handler does not show them.
+
 Register the object in the package's `pyproject.toml`. The entry-point name is
 the command users invoke:
 
@@ -125,6 +129,64 @@ export MVT_IOS_CUSTOM_COMMANDS=./ios_commands
 export MVT_ANDROID_CUSTOM_COMMANDS=./android_commands
 ```
 
+## Building a Module-Running Command
+
+A command which runs forensic modules over an acquisition subclasses `Command`.
+`Command` creates the output folder and writes `command.log`. It orders the
+modules, resolves their dependencies and runs them. It writes the result files,
+`alerts.json` and `info.json`. The subclass sets `platform`, `name` and
+`modules`:
+
+```python
+from mvt.plugin import Command, MVTModule, convert_unix_to_iso, get_plugin_logger
+
+log = get_plugin_logger(__name__)
+
+
+class APKManifest(MVTModule):
+    supported_commands = (("android", "check-apks"),)
+
+    def run(self):
+        self.results = [{"checked_at": convert_unix_to_iso(0)}]
+
+
+class CmdCheckAPKs(Command):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.platform = "android"
+        self.name = "check-apks"
+        self.modules = [APKManifest]
+```
+
+`platform` and `name` are the pair the modules declare in
+`supported_commands`. `modules` lists the module classes the command runs,
+imported directly. A command which runs modules of another plugin depends on
+that package in `pyproject.toml` and imports them the same way. `init()`,
+`module_init(module)` and `finish()` are optional hooks. `run()` calls them
+before the run, before each module and after the run.
+
+Wrap the command in a Click command with an `--output` option. Run it, then
+print the alert summary:
+
+```python
+import click
+
+
+@click.command("check-apks")
+@click.option("--output", "-o", type=click.Path(exists=False))
+@click.argument("TARGET_PATH", type=click.Path(exists=True))
+def cli(output, target_path):
+    cmd = CmdCheckAPKs(target_path=target_path, results_path=output)
+    log.info("Checking APK files at path: %s", target_path)
+    cmd.run()
+    cmd.show_alerts_brief()
+```
+
+The `--verbose` option of `mvt`, `mvt-ios` and `mvt-android` applies to the
+command. The command defines no `--verbose` option of its own. The pair a
+plugin command adds is not listed anywhere in MVT. Name it in the plugin's
+README.
+
 ## Naming and Errors
 
 Built-in MVT commands cannot be replaced. External command names must also be

docs/development/index.md

@@ -48,51 +48,21 @@ configuration problem: the command logs a warning and runs no modules at all.
 
 ## Custom modules
 
-Module-running `check-*` commands can load custom modules from Python files that
+MVT's module-running `check-*` commands can run forensic modules which are not
-are not installed as part of MVT. Load one file with:
+part of MVT. Custom modules are distributed as plugin packages. A Python
+package installed next to MVT registers its modules through an entry point.
+The modules then load automatically in every command they support, see
+[Installed module packages](#installed-module-packages). `mvt plugins list`
+shows the installed packages and where each one was installed from.
 
-```bash
+MVT can also load module files by path, with `--load-module` and
-mvt-ios check-backup --load-module ./example_module.py --output ./out ./backup
+`MVT_CUSTOM_MODULES`, see
-```
+[Developing modules locally](#developing-modules-locally). This can be used
+while writing a module. Use a Python package to distribute one.
 
-You can also load a folder. MVT loads non-hidden top-level `*.py` files in
+A custom module declares in `supported_commands` the platform and command
-sorted order and skips `__init__.py`:
+pairs it runs in. A module with empty `supported_commands` does not run and
-
+MVT logs a warning. The nine pairs are:
-```bash
-mvt-ios check-fs --load-module ./custom_modules ./filesystem-dump
-```
-
-Set `MVT_CUSTOM_MODULES` to load a folder for every module-running command. This
-folder is loaded before any `--load-module` path:
-
-```bash
-MVT_CUSTOM_MODULES=./custom_modules mvt-android check-bugreport ./bugreport.zip
-```
-
-Custom modules are normal `MVTModule` subclasses:
-
-```python
-from mvt.common.module import MVTModule
-
-
-class ExampleCustomModule(MVTModule):
-    supported_commands = (("ios", "check-backup"), ("ios", "check-fs"))
-    slug = "example_custom_module"
-
-    def run(self):
-        self.results = [{"message": "custom module ran"}]
-
-    def check_indicators(self):
-        pass
-
-    def serialize(self, result):
-        return None
-```
-
-Use `supported_commands` to declare the platform/command pairs a module
-supports. Empty `supported_commands` means the module will not run and MVT logs
-a warning. This explicit declaration is required for every command. Supported
-pairs are:
 
 ```python
 ("ios", "check-backup")
@@ -106,13 +76,86 @@ pairs are:
 ("android", "check-iocs")
 ```
 
-Custom modules can depend on existing MVT module classes. Dependencies are
+`check-iocs` re-checks stored results rather than an acquisition. It matches
-resolved with the same ordering logic as built-in modules, and custom modules
+every `<slug>.json` file in the results folder to the module with that slug.
-are appended after built-ins before ordering:
+It then runs that module's `check_indicators()` again.
+
+### Writing a module
+
+A module subclasses `MVTModule` or one of the base classes below and
+implements `run()`. `check_indicators()` and `serialize()` are optional. The
+first matches results against IOCs or detections. The second returns timeline
+records.
 
 ```python
-from mvt.common.module import MVTModule
+from mvt.plugin import IOSExtraction, convert_unix_to_iso
-from mvt.ios.modules.backup.manifest import Manifest
+
+
+class ExampleCustomModule(IOSExtraction):
+    supported_commands = (
+        ("ios", "check-backup"),
+        ("ios", "check-fs"),
+    )
+    slug = "example_custom_module"
+
+    def run(self):
+        self.results = [{"checked_at": convert_unix_to_iso(0)}]
+
+    def check_indicators(self):
+        pass
+
+    def serialize(self, result):
+        return None
+```
+
+The base classes are:
+
+- `MVTModule`: the base of every module. It provides `self.results`,
+  `self.alertstore`, `self.log`, `self.indicators` and
+  `get_dependency_results()`. Subclass it directly for a module which reads
+  only the results of other modules.
+- `IOSExtraction`: `("ios", "check-backup")` and `("ios", "check-fs")`. Adds
+  `_find_ios_database()`, which locates a module's database in a backup or in
+  a filesystem dump and repairs it if it is malformed. Adds
+  `_get_backup_files_from_manifest()`, `_get_backup_file_from_id()` and
+  `_get_fs_files_from_patterns()`. Adds `_open_sqlite_db()`, which opens a
+  database read-only.
+- `SysdiagnoseExtraction`: `("ios", "check-sysdiagnose")`. MVT extracts the
+  archive and calls `from_sysdiagnose_folder()` before `run()`. The module
+  reads files with `_get_files_by_pattern()` and `_get_file_content()`.
+  `ips_files` lists the crash reports. See
+  [Check an iOS Sysdiagnose](../ios/sysdiagnose.md).
+- `AndroidQFModule`: `("android", "check-androidqf")`. MVT calls `from_dir()`
+  or `from_zip()` with the file list of the acquisition. The module reads
+  files with `_get_files_by_pattern()` and `_get_file_content()`.
+  `_get_device_timezone()` returns the device timezone.
+- `AndroidBackupModule`: `("android", "check-backup")`. MVT calls `from_dir()`
+  or `from_ab()`. The module reads files with `_get_files_by_pattern()` and
+  `_get_file_content()`.
+- `BugReportModule`: `("android", "check-bugreport")`. MVT calls `from_dir()`
+  or `from_zip()`. The module reads files with `_get_files_by_pattern()`,
+  `_get_files_by_patterns()` and `_get_file_content()`.
+  `_get_dumpstate_file()` returns the dumpstate file, and
+  `_get_file_modification_time()` the modification time of a file.
+
+The underscore-named helpers are internal to the base classes. Plugin modules
+can call them. Their names and signatures can change between releases. Read
+the base class in `src/mvt/ios/modules` or `src/mvt/android/modules` before
+relying on one.
+
+### Depending on a built-in module
+
+A module which post-processes records generated by one or more built-in MVT
+modules must declare the source modules in `dependencies`. It reads their
+results with `get_dependency_results()`. Import the class from its family
+package: `mvt.ios.modules.backup`, `mvt.ios.modules.fs`,
+`mvt.ios.modules.mixed`, `mvt.android.modules.androidqf`,
+`mvt.android.modules.backup`, `mvt.android.modules.bugreport` or
+`mvt.android.modules.intrusion_logs`.
+
+```python
+from mvt.ios.modules.backup import Manifest
+from mvt.plugin import MVTModule
 
 
 class DependentCustomModule(MVTModule):
@@ -124,6 +167,30 @@ class DependentCustomModule(MVTModule):
         self.results = [{"manifest_entries": len(manifest_results)}]
 ```
 
+Dependencies are ordered as for the built-in modules, with custom modules
+appended after the built-ins, see [Module dependencies](#module-dependencies).
+A dependency has to run in every command the module supports. Where it does
+not, MVT skips the module with a warning.
+
+`get_dependency_results()` returns the plain dictionaries the module produced.
+They are the same records it writes to `<slug>.json`. Typed results per module
+are planned.
+
+### Importing from MVT
+
+Import from `mvt.plugin` if it has what you need. The names it exports are
+kept working on a best-effort basis. A change to one of them is announced in
+the release notes. Anything else in `mvt` can be imported too, but may change
+between releases without notice. The plugin interface is best effort.
+
+`mvt.plugin` exports the base classes above and `Command`, `Alert` and
+`AlertLevel`, the result types, `DatabaseNotFoundError` and
+`DatabaseCorruptedError`, the timestamp converters, the settings API of
+[Plugin Configuration](plugin_configuration.md), MVT's own `settings`,
+`get_plugin_logger()` and `MVT_VERSION`. `src/mvt/plugin.py` holds the list.
+Read MVT's `settings` for values such as `NETWORK_ACCESS_ALLOWED` and
+`NETWORK_TIMEOUT`. Plugin values go in the plugin's own settings file.
+
 ## Installed module packages
 
 Python packages can register modules so they load automatically in every
@@ -133,14 +200,14 @@ the package's `pyproject.toml`:
 
 ```toml
 [project.entry-points."mvt.modules"]
-mvt-plugin-amnesty-custom = "mvt_plugin_amnesty_custom:get_modules"
+mvt-plugin-example-org = "mvt_plugin_example_org:get_modules"
 ```
 
 The entry point must resolve to an iterable of `MVTModule` subclasses, or to
 a callable returning one:
 
 ```python
-from mvt.common.module import MVTModule
+from mvt.plugin import MVTModule
 
 
 class PackagedModule(MVTModule):
@@ -154,6 +221,10 @@ def get_modules() -> list[type[MVTModule]]:
     return [PackagedModule]
 ```
 
+`get_modules()` is the package's module list, written by hand. A package which
+keeps its modules in separate files imports each class there and lists it. A
+module missing from the list does not load.
+
 Installed modules follow the same rules as other custom modules: each module
 must declare `supported_commands`, and dependencies are resolved with the
 standard ordering logic. A broken entry point is skipped with a warning and
@@ -164,7 +235,7 @@ from sources you trust.
 For a `pipx` installation of MVT, inject the package into MVT's environment:
 
 ```bash
-pipx inject mvt mvt-plugin-amnesty-custom
+pipx inject mvt mvt-plugin-example-org
 ```
 
 Module packages that need their own settings, such as an API key, should store
@@ -175,9 +246,9 @@ rather than in MVT's own `config.yaml`.
 
 Name module packages `mvt-plugin-<name>` (import package `mvt_plugin_<name>`),
 and include the name of the publishing organization or author so packages from
-different groups do not collide: for example, Amnesty International's custom
+different groups do not collide: for example, an organisation's custom modules
-modules would be distributed as `mvt-plugin-amnesty-custom` with the import
+would be distributed as `mvt-plugin-example-org` with the import package
-package `mvt_plugin_amnesty_custom`.
+`mvt_plugin_example_org`.
 
 The prefix makes module packages easy to find on PyPI and keeps their import
 names from clashing with unrelated Python packages. It is a convention, not a
@@ -196,11 +267,42 @@ came from. MVT's own modules log under their dotted path (for example
 MVT's internal logger tree:
 
 - Installed packages log under `mvt.ext.<package>`, with the `mvt_plugin_`
-  prefix stripped: modules in `mvt_plugin_amnesty_custom` log as
+  prefix stripped: modules in `mvt_plugin_example_org` log as
-  `mvt.ext.amnesty_custom.*`.
+  `mvt.ext.example_org.*`.
 - Files loaded with `--load-module` or `MVT_CUSTOM_MODULES` log as
   `mvt.ext.<file name>`.
 
+Outside a module class, for example in a custom command line handler, log
+through `get_plugin_logger(__name__)`. It returns a logger in the same
+namespace.
+
+## Developing modules locally
+
+While a module is being written, load it from its file. `--load-module` takes a
+Python file, or a folder of them, on every module-running command, and can be
+repeated:
+
+```bash
+mvt-ios check-backup --load-module ./example_module.py --output ./out ./backup
+```
+
+For a folder, MVT loads its non-hidden top-level `*.py` files in sorted order
+and skips `__init__.py`. `MVT_CUSTOM_MODULES` names a folder to load on every
+module-running command, before any `--load-module` path:
+
+```bash
+MVT_CUSTOM_MODULES=./custom_modules mvt-android check-bugreport ./bugreport.zip
+```
+
+Files loaded this way follow the same rules as packaged modules.
+`--list-modules` reports them with the SHA-256 hash of the file in place of a
+version. An editable install of the package (`pip install -e .`) also works:
+the modules load through the entry point, and `mvt plugins list` shows the
+package with the `local` origin.
+
+Loading by path is for development. Move a module into a package once it
+works.
+
 ## Auditing loaded modules
 
 Because installed module packages load automatically, MVT records where every

docs/development/plugin_configuration.md

@@ -24,8 +24,7 @@ configuration:
 
 The exact parent folder follows the platform convention used for MVT's
 `config.yaml` (for example `~/Library/Application Support/mvt` on macOS). Use
-`mvt.common.plugin_config.plugin_config_path()` instead of building the path by
+`plugin_config_path()` from `mvt.plugin` instead of building the path by hand.
-hand.
 
 Plugin names must be lowercase and may only contain letters, digits and dashes,
 matching the `mvt-plugin-<name>` package naming convention. MVT creates the
@@ -38,9 +37,8 @@ leaves a partially written settings file behind.
 
 Everything else a plugin keeps on disk, such as a cache, a downloaded artifact
 or synchronization state, belongs in the folder returned by the `data_folder()`
-class method of the plugin's settings class, or by
+class method of the plugin's settings class, or by `plugin_data_folder()` from
-`mvt.common.plugin_config.plugin_data_folder()` called with the plugin name if
+`mvt.plugin`, called with the plugin name if the plugin has no settings class:
-the plugin has no settings class:
 
 ```
 ~/.local/share/mvt/plugin-data/<plugin name>/                # Linux
@@ -76,7 +74,7 @@ defaults:
 ```python
 from typing import Optional
 
-from mvt.common.plugin_config import MVTPluginSettings
+from mvt.plugin import MVTPluginSettings
 
 
 class ExamplePluginSettings(MVTPluginSettings):
@@ -94,12 +92,15 @@ from datetime import datetime, timezone
 
 import click
 
+from mvt.plugin import plugin_env_prefix
+
 
 def sync():
     settings = ExamplePluginSettings.load()
     if not settings.API_KEY:
+        prefix = plugin_env_prefix(settings.plugin_name)
         raise click.ClickException(
-            "No API key configured. Set MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY or "
+            f"No API key configured. Set {prefix}API_KEY or "
             "run 'example-plugin configure'."
         )
 
@@ -107,6 +108,9 @@ def sync():
     settings.save()
 ```
 
+The message builds the variable name with `plugin_env_prefix()`, see
+[Environment Variables](#environment-variables).
+
 A missing settings file is not an error: the plugin then runs on the field
 defaults and on whatever the environment provides. `save()` only persists the
 values that differ from the defaults, and it never touches MVT's `config.yaml`.
@@ -129,6 +133,10 @@ export MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY=...
 export MVT_PLUGIN_EXAMPLE_PLUGIN_MAX_RESULTS=50
 ```
 
+Do not repeat the plugin name in a field name: a plugin named `example-scanner`
+with a `SCANNER_API_KEY` field asks the user for
+`MVT_PLUGIN_EXAMPLE_SCANNER_SCANNER_API_KEY`. Name the field `API_KEY`.
+
 Settings resolve in this order, from highest to lowest priority:
 
 1. Arguments passed to the settings class directly, such as

docs/ios/sysdiagnose.md

@@ -1,30 +1,32 @@
 # Check an iOS Sysdiagnose
 
 `mvt-ios check-sysdiagnose` prepares an iOS sysdiagnose archive for analysis by
-custom MVT modules. MVT does not include built-in sysdiagnose modules. You must
+custom MVT modules. MVT does not include built-in sysdiagnose modules. The
-load at least one custom module that explicitly supports this command.
+command runs the modules of the installed
+[plugin packages](../development/index.md#installed-module-packages) which
+declare support for it. Install at least one such package first.
 
 The command accepts either an extracted sysdiagnose directory or the original
 gzip-compressed tar archive.
 
 ```bash
-mvt-ios check-sysdiagnose \
+mvt-ios check-sysdiagnose --output ./results \
-    --load-module ./sysdiagnose_modules.py \
-    --output ./results \
     ./sysdiagnose_2024.01.02_03-04-05+0200.tar.gz
 ```
 
 Use `--hashes` to include hashes for analyzed files in `info.json`, and
-`--list-modules` to display the eligible custom modules without running them.
+`--list-modules` to display the eligible modules without running them.
 
 ## Writing a custom module
 
-Extend `SysdiagnoseExtraction` to access the archive contents consistently for
+Extend `SysdiagnoseExtraction` from `mvt.plugin`, see
-both directory and tar inputs. Each module must declare the command explicitly
+[Writing a module](../development/index.md#writing-a-module). The module reads
-in `supported_commands`.
+the archive the same way whether MVT was given a folder or a tar archive. It
+declares the command in `supported_commands`. While writing one,
+[load it from its file](../development/index.md#developing-modules-locally).
 
 ```python
-from mvt.ios.modules.sysdiagnose import SysdiagnoseExtraction
+from mvt.plugin import SysdiagnoseExtraction
 
 
 class ExampleSysdiagnoseModule(SysdiagnoseExtraction):
@@ -44,7 +46,9 @@ class ExampleSysdiagnoseModule(SysdiagnoseExtraction):
         return None
 ```
 
-The base class provides `from_sysdiagnose_folder()` and
+MVT extracts a tar archive first. It calls `from_sysdiagnose_folder()` on each
-`from_sysdiagnose_tar()` setup hooks, as well as protected file lookup, file
+module before `run()`. `ips_files` lists the IPS crash reports.
-reading, and timezone extraction helpers. IPS crash-report metadata is exposed
+
-on `ips_files`.
+`_get_files_by_pattern()` and `_get_file_content()` are internal helpers of the
+base class. Use them to read the archive. Their names and signatures can change
+between releases. See `src/mvt/ios/modules/sysdiagnose/base.py`.