mirror of
https://github.com/mvt-project/mvt.git
synced 2026-07-29 23:58:52 +02:00
Run extraction modules in parallel
This commit is contained in:
+33
-1
@@ -39,6 +39,33 @@ Selecting a single module also runs its transitive dependencies. If a dependency
|
||||
is unavailable or the dependency graph contains a cycle, the command logs a
|
||||
warning and does not run any modules.
|
||||
|
||||
## Parallel module execution
|
||||
|
||||
Extraction modules run concurrently, using four worker threads by default. Use
|
||||
`--jobs INTEGER` to change the worker limit, or `--jobs 1` for sequential
|
||||
execution and live per-line logging:
|
||||
|
||||
```bash
|
||||
mvt-ios check-backup --jobs 8 --output ./out ./backup
|
||||
```
|
||||
|
||||
The scheduler starts a module only after all of its declared dependencies have
|
||||
completed. Results, alerts, and timeline entries are still aggregated in stable
|
||||
topological order. During parallel execution, console records from each module
|
||||
are buffered and printed together as a labeled block when that module finishes;
|
||||
`command.log` is written immediately and retains the complete log stream.
|
||||
|
||||
Parallel-safe modules must log through `self.log`, must not write directly to
|
||||
stdout or stderr, and must not mutate shared global state. A module that uses a
|
||||
non-thread-safe resource or direct terminal output can opt out:
|
||||
|
||||
```python
|
||||
class TerminalModule(MVTModule):
|
||||
parallel_safe = False
|
||||
```
|
||||
|
||||
Such modules run synchronously and exclusively after active workers finish.
|
||||
|
||||
## Custom modules
|
||||
|
||||
Module-running `check-*` commands can load custom modules from Python files that
|
||||
@@ -82,6 +109,9 @@ class ExampleCustomModule(MVTModule):
|
||||
return None
|
||||
```
|
||||
|
||||
Custom modules are considered parallel-safe by default. Follow the parallel
|
||||
module requirements above or set `parallel_safe = False`.
|
||||
|
||||
Use `supported_commands` to restrict a module to specific platform/command
|
||||
pairs. Missing or empty `supported_commands` means the module is available to
|
||||
all commands, which keeps older modules compatible. Supported pairs are:
|
||||
@@ -120,7 +150,9 @@ class DependentCustomModule(MVTModule):
|
||||
Some MVT modules extract and process significant amounts of data during the analysis process or while checking results against known indicators. Care must be
|
||||
take to avoid inefficient code paths as we add new modules.
|
||||
|
||||
MVT modules can be profiled with Python built-in `cProfile` by setting the `MVT_PROFILE` environment variable.
|
||||
MVT modules can be profiled with Python built-in `cProfile` by setting the
|
||||
`MVT_PROFILE` environment variable. Profiling forces sequential execution even
|
||||
when `--jobs` is greater than one.
|
||||
|
||||
```bash
|
||||
MVT_PROFILE=1 dev/mvt-ios check-backup test_backup
|
||||
|
||||
Reference in New Issue
Block a user