Run extraction modules in parallel

This commit is contained in:
Janik Besendorf
2026-07-15 09:40:18 +02:00
parent afcfda4720
commit 9d71661f55
21 changed files with 643 additions and 90 deletions
+33 -1
View File
@@ -39,6 +39,33 @@ Selecting a single module also runs its transitive dependencies. If a dependency
is unavailable or the dependency graph contains a cycle, the command logs a
warning and does not run any modules.
## Parallel module execution
Extraction modules run concurrently, using four worker threads by default. Use
`--jobs INTEGER` to change the worker limit, or `--jobs 1` for sequential
execution and live per-line logging:
```bash
mvt-ios check-backup --jobs 8 --output ./out ./backup
```
The scheduler starts a module only after all of its declared dependencies have
completed. Results, alerts, and timeline entries are still aggregated in stable
topological order. During parallel execution, console records from each module
are buffered and printed together as a labeled block when that module finishes;
`command.log` is written immediately and retains the complete log stream.
Parallel-safe modules must log through `self.log`, must not write directly to
stdout or stderr, and must not mutate shared global state. A module that uses a
non-thread-safe resource or direct terminal output can opt out:
```python
class TerminalModule(MVTModule):
parallel_safe = False
```
Such modules run synchronously and exclusively after active workers finish.
## Custom modules
Module-running `check-*` commands can load custom modules from Python files that
@@ -82,6 +109,9 @@ class ExampleCustomModule(MVTModule):
return None
```
Custom modules are considered parallel-safe by default. Follow the parallel
module requirements above or set `parallel_safe = False`.
Use `supported_commands` to restrict a module to specific platform/command
pairs. Missing or empty `supported_commands` means the module is available to
all commands, which keeps older modules compatible. Supported pairs are:
@@ -120,7 +150,9 @@ class DependentCustomModule(MVTModule):
Some MVT modules extract and process significant amounts of data during the analysis process or while checking results against known indicators. Care must be
take to avoid inefficient code paths as we add new modules.
MVT modules can be profiled with Python built-in `cProfile` by setting the `MVT_PROFILE` environment variable.
MVT modules can be profiled with Python built-in `cProfile` by setting the
`MVT_PROFILE` environment variable. Profiling forces sequential execution even
when `--jobs` is greater than one.
```bash
MVT_PROFILE=1 dev/mvt-ios check-backup test_backup