Add namespaced plugin configuration support

Plugin packages need somewhere to keep their own settings, but MVT
rewrites its config.yaml with only the fields it knows about, so any
foreign section is dropped. Add MVTPluginSettings, a pydantic-settings
base class that gives each plugin its own file under the MVT config
folder and its own MVT_PLUGIN_<NAME>_ environment variable namespace.

Settings resolve from constructor arguments, then the environment, then
the plugin file, then the field defaults. Saving skips the values the
environment currently supplies, so credentials passed as environment
variables are not copied to disk, and writes through a private temporary
file so a settings file is never partially written or briefly readable
by other users.
This commit is contained in:
Donncha Ó Cearbhaill
2026-08-26 11:53:47 +02:00
committed by Donncha Ó Cearbhaill
parent 4e031cd43f
commit dab82e38d0
6 changed files with 682 additions and 0 deletions
+4
View File
@@ -56,6 +56,10 @@ pipx inject mvt my-mvt-plugin
When MVT is installed in an active virtual environment, install the plugin with
`pip` in that environment.
Command packages that need their own settings, such as an API key, should store
them in a namespaced [plugin configuration file](plugin_configuration.md)
rather than in MVT's own `config.yaml`.
## Load a Command File
For local commands that are not packaged, create a Python file that exports one
+4
View File
@@ -160,6 +160,10 @@ For a `pipx` installation of MVT, inject the package into MVT's environment:
pipx inject mvt mvt-plugin-amnesty-custom
```
Module packages that need their own settings, such as an API key, should store
them in a namespaced [plugin configuration file](plugin_configuration.md)
rather than in MVT's own `config.yaml`.
### Naming module packages
Name module packages `mvt-plugin-<name>` (import package `mvt_plugin_<name>`),
+114
View File
@@ -0,0 +1,114 @@
# Plugin Configuration
Plugin packages that add [custom CLI commands](custom_commands.md) or
[modules](index.md#installed-module-packages) often need to store their
own settings, such as an API key, a cache folder or the timestamp of the last
synchronization. MVT provides a namespaced settings base class so each plugin
keeps its configuration in its own file.
!!! warning
Do not write plugin settings to MVT's own `config.yaml`. MVT rewrites that
file with the settings it knows about every time it starts, so any other
section is deleted.
## Where Settings Are Stored
Each plugin gets one YAML file in a `plugins` folder next to MVT's own
configuration:
```
~/.config/mvt/plugins/<plugin name>.yaml
```
The exact parent folder follows the platform convention used for MVT's
`config.yaml` (for example `~/Library/Application Support/mvt` on macOS). Use
`mvt.common.plugin_config.plugin_config_path()` instead of building the path by
hand.
Plugin names must be lowercase and may only contain letters, digits and dashes,
matching the `mvt-plugin-<name>` package naming convention. MVT creates the
`plugins` folder with `0700` permissions and writes the settings files with
`0600` permissions, because they commonly hold credentials. Files are written
through a temporary file and moved into place, so an interrupted save never
leaves a partially written settings file behind.
## Defining Plugin Settings
Subclass `MVTPluginSettings`, set `plugin_name` and declare typed fields with
defaults:
```python
from typing import Optional
from mvt.common.plugin_config import MVTPluginSettings
class ExamplePluginSettings(MVTPluginSettings):
plugin_name = "example-plugin"
API_KEY: Optional[str] = None
CACHE_FOLDER: str = "~/.cache/example-plugin"
LAST_SYNC: Optional[str] = None
```
`load()` returns the current settings and `save()` writes them back:
```python
from datetime import datetime, timezone
import click
def sync():
settings = ExamplePluginSettings.load()
if not settings.API_KEY:
raise click.ClickException(
"No API key configured. Set MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY or "
"run 'example-plugin configure'."
)
settings.LAST_SYNC = datetime.now(timezone.utc).isoformat()
settings.save()
```
A missing settings file is not an error: the plugin then runs on the field
defaults and on whatever the environment provides. `save()` only persists the
values that differ from the defaults, and it never touches MVT's `config.yaml`.
A settings file that cannot be parsed, or that does not hold a mapping of
setting names to values, raises a `PluginConfigLoadError` naming the file.
Every subclass that sets its own `plugin_name` gets its own file and its own
environment namespace. A subclass that does not redefine `plugin_name` inherits
it, and therefore shares the file and the environment variables of its parent
class.
## Environment Variables
Every field can also be set with an environment variable. The prefix is
`MVT_PLUGIN_`, followed by the plugin name upper-cased with dashes replaced by
underscores, followed by the field name. For the example above:
```bash
export MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY=...
export MVT_PLUGIN_EXAMPLE_PLUGIN_CACHE_FOLDER=/tmp/example-cache
```
Settings resolve in this order, from highest to lowest priority:
1. Arguments passed to the settings class directly, such as
`ExamplePluginSettings(API_KEY="...")`
2. Environment variables
3. The plugin's YAML file
4. The field defaults declared on the settings class
!!! tip
On shared or multi-user machines, prefer passing API keys through
environment variables rather than saving them to the plugin file. `save()`
skips every value that the environment currently supplies, so a credential
provided that way is not copied into the settings file when a plugin saves
an unrelated setting.
Unknown keys in a plugin's YAML file are ignored, so a settings file written by
a newer version of a plugin does not break an older one.
+1
View File
@@ -52,4 +52,5 @@ nav:
- Development:
- Development Instructions: "development/index.md"
- Custom CLI Commands: "development/custom_commands.md"
- Plugin Configuration: "development/plugin_configuration.md"
- License: "license.md"
+256
View File
@@ -0,0 +1,256 @@
# Mobile Verification Toolkit (MVT)
# Copyright (c) 2021-2026 The MVT Authors.
# Use of this software is governed by the MVT License 1.1 that can be found at
# https://license.mvt.re/1.1/
import contextlib
import json
import os
import re
import tempfile
from typing import Any, ClassVar, Dict, List, Tuple, Type, TypeVar
import yaml
from appdirs import user_config_dir
from pydantic import ValidationError
from pydantic_settings import (
BaseSettings,
EnvSettingsSource,
PydanticBaseSettingsSource,
SettingsConfigDict,
YamlConfigSettingsSource,
)
PLUGIN_CONFIG_FOLDER_NAME = "plugins"
PLUGIN_ENV_PREFIX = "MVT_PLUGIN_"
PLUGIN_NAME_PATTERN = re.compile(r"[a-z0-9][a-z0-9-]*")
PluginSettingsType = TypeVar("PluginSettingsType", bound="MVTPluginSettings")
class PluginConfigLoadError(Exception):
pass
def validate_plugin_name(plugin_name: Any) -> str:
"""
Check that a plugin name is safe to use in a file name and an env variable.
:param plugin_name: Name to validate.
:returns: The validated plugin name.
"""
if not isinstance(plugin_name, str):
raise TypeError(
f"Plugin name must be a string, not {type(plugin_name).__name__}"
)
if not PLUGIN_NAME_PATTERN.fullmatch(plugin_name):
raise ValueError(
f"Invalid plugin name {plugin_name!r}: plugin names must start with a "
"lowercase letter or a digit and may only contain lowercase letters, "
"digits and dashes"
)
return plugin_name
def plugin_config_folder() -> str:
"""
Return the folder where plugins store their configuration files.
The path is resolved on every call so it always reflects the current
environment.
"""
return os.path.join(user_config_dir("mvt"), PLUGIN_CONFIG_FOLDER_NAME)
def plugin_config_path(plugin_name: str) -> str:
"""
Return the path of the configuration file of a given plugin.
:param plugin_name: Name of the plugin.
"""
return os.path.join(
plugin_config_folder(), f"{validate_plugin_name(plugin_name)}.yaml"
)
def plugin_env_prefix(plugin_name: str) -> str:
"""
Return the environment variable prefix used by a given plugin.
Dashes are replaced by underscores. Plugin names cannot contain underscores,
so two different plugin names never share an environment namespace.
:param plugin_name: Name of the plugin.
"""
name = validate_plugin_name(plugin_name).upper().replace("-", "_")
return f"{PLUGIN_ENV_PREFIX}{name}_"
def _settings_plugin_name(settings_cls: Type[BaseSettings]) -> str:
plugin_name = getattr(settings_cls, "plugin_name", None)
if plugin_name is None:
raise TypeError(
f"{settings_cls.__name__} must set a 'plugin_name' class attribute to "
"namespace its configuration file and environment variables"
)
return validate_plugin_name(plugin_name)
def _plugin_yaml_source(
settings_cls: Type[BaseSettings], config_path: str
) -> YamlConfigSettingsSource:
"""
Build the YAML settings source of a plugin, reporting unusable files.
A missing file is not an error, but a file which cannot be parsed or which
does not hold a mapping of setting names is reported with its path.
"""
try:
return YamlConfigSettingsSource(settings_cls, config_path)
except yaml.YAMLError as exc:
raise PluginConfigLoadError(
f"Invalid plugin configuration file {config_path}: {exc}"
) from exc
except (TypeError, ValueError) as exc:
raise PluginConfigLoadError(
f"Invalid plugin configuration file {config_path}: the file must "
"contain a mapping of setting names to values"
) from exc
class MVTPluginSettings(BaseSettings):
"""
Base class for plugin-namespaced settings.
Subclass with typed fields and set `plugin_name`. Values resolve from
constructor arguments, then environment variables (MVT_PLUGIN_<NAME>_*),
then the plugin's YAML file (~/.config/mvt/plugins/<name>.yaml), then field
defaults.
Plugins must not store their settings in MVT's own configuration file: MVT
rewrites it with the fields it knows about, dropping anything else.
"""
model_config = SettingsConfigDict(extra="ignore")
plugin_name: ClassVar[str]
def __init_subclass__(cls, **kwargs: Any) -> None:
super().__init_subclass__(**kwargs)
plugin_name = _settings_plugin_name(cls)
# Namespace the environment variables of this plugin. Each pydantic
# model gets its own configuration dictionary, so this does not leak
# into other plugins.
cls.model_config["env_prefix"] = plugin_env_prefix(plugin_name)
@classmethod
def settings_customise_sources(
cls,
settings_cls: Type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> Tuple[PydanticBaseSettingsSource, ...]:
config_path = plugin_config_path(_settings_plugin_name(settings_cls))
yaml_source = _plugin_yaml_source(settings_cls, config_path)
# Explicit arguments take precedence over environment variables, which
# in turn take precedence over the configuration file.
return (init_settings, env_settings, yaml_source)
@classmethod
def load(cls: Type[PluginSettingsType]) -> PluginSettingsType:
"""
Load the settings of the plugin.
A missing configuration file is not an error: the settings then come
from the environment and from the field defaults.
"""
return cls()
def _environment_values(self) -> Dict[str, Any]:
"""
Return the settings values currently supplied by the environment.
Values are validated by the model, so they can be compared with the
values held by this instance.
"""
settings_cls = type(self)
raw_values = EnvSettingsSource(settings_cls)()
names = [name for name in raw_values if name in settings_cls.model_fields]
if not names:
return {}
# Fall back on the current values for the fields the environment does
# not set, so that required fields do not fail validation here.
current_values = json.loads(self.model_dump_json())
try:
from_environment = settings_cls.model_validate(
{**current_values, **raw_values}
)
except ValidationError:
# An environment variable which the model cannot validate must not
# stop the other environment values from being recognised, or a
# credential would be written to the configuration file.
return self._environment_values_by_field(current_values, raw_values, names)
return {name: getattr(from_environment, name) for name in names}
def _environment_values_by_field(
self,
current_values: Dict[str, Any],
raw_values: Dict[str, Any],
names: List[str],
) -> Dict[str, Any]:
"""
Validate each environment value on its own, skipping unusable ones.
:param current_values: Serialized values held by this instance.
:param raw_values: Values supplied by the environment.
:param names: Names of the fields set by the environment.
"""
settings_cls = type(self)
values = {}
for name in names:
try:
from_environment = settings_cls.model_validate(
{**current_values, name: raw_values[name]}
)
except ValidationError:
continue
values[name] = getattr(from_environment, name)
return values
def save(self) -> None:
"""
Save the current settings to the configuration file of the plugin.
Only values which differ from the field defaults are persisted. Values
which come from the environment are not written to disk, so credentials
passed as environment variables stay out of the configuration file.
MVT's own configuration file is never modified.
"""
config_folder = plugin_config_folder()
if not os.path.isdir(config_folder):
os.makedirs(config_folder, mode=0o700, exist_ok=True)
values = json.loads(self.model_dump_json(exclude_defaults=True))
for name, environment_value in self._environment_values().items():
if name in values and getattr(self, name, None) == environment_value:
del values[name]
# Settings files can hold credentials, so write them through a private
# temporary file and move it in place. The file is then never partially
# written and never briefly readable by other users.
config_path = plugin_config_path(self.plugin_name)
descriptor, temporary_path = tempfile.mkstemp(
dir=config_folder, prefix=f".{self.plugin_name}-", suffix=".yaml"
)
try:
with os.fdopen(descriptor, "w") as config_file:
config_file.write(yaml.dump(values, default_flow_style=False))
os.replace(temporary_path, config_path)
except BaseException:
with contextlib.suppress(OSError):
os.unlink(temporary_path)
raise
+303
View File
@@ -0,0 +1,303 @@
# Mobile Verification Toolkit (MVT)
# Copyright (c) 2021-2026 The MVT Authors.
# Use of this software is governed by the MVT License 1.1 that can be found at
# https://license.mvt.re/1.1/
import os
import stat
import sys
from typing import Optional
import pytest
import yaml
from mvt.common.plugin_config import (
MVTPluginSettings,
PluginConfigLoadError,
plugin_config_folder,
plugin_config_path,
plugin_env_prefix,
)
class ExamplePluginSettings(MVTPluginSettings):
plugin_name = "example-plugin"
API_KEY: Optional[str] = None
CACHE_FOLDER: str = "cache"
MAX_RESULTS: int = 25
class OtherPluginSettings(MVTPluginSettings):
plugin_name = "other-plugin"
API_KEY: Optional[str] = None
@pytest.fixture
def config_folder(tmp_path, monkeypatch):
monkeypatch.setattr(
"mvt.common.plugin_config.user_config_dir",
lambda *args, **kwargs: str(tmp_path),
)
return tmp_path
def _write_plugin_file(plugin_name, values):
config_path = plugin_config_path(plugin_name)
os.makedirs(os.path.dirname(config_path), exist_ok=True)
content = values if isinstance(values, str) else yaml.dump(values)
with open(config_path, "w") as config_file:
config_file.write(content)
return config_path
def test_plugin_paths_and_prefixes_are_namespaced(config_folder):
assert plugin_config_folder() == str(config_folder / "plugins")
assert plugin_config_path("example-plugin") == str(
config_folder / "plugins" / "example-plugin.yaml"
)
assert plugin_env_prefix("example-plugin") == "MVT_PLUGIN_EXAMPLE_PLUGIN_"
assert plugin_env_prefix("other-plugin") == "MVT_PLUGIN_OTHER_PLUGIN_"
def test_defaults_are_used_without_file_or_environment(config_folder):
settings = ExamplePluginSettings.load()
assert settings.API_KEY is None
assert settings.CACHE_FOLDER == "cache"
assert settings.MAX_RESULTS == 25
assert not os.path.exists(plugin_config_path("example-plugin"))
def test_values_are_loaded_from_the_plugin_file(config_folder):
_write_plugin_file("example-plugin", {"API_KEY": "from-file", "MAX_RESULTS": 5})
settings = ExamplePluginSettings.load()
assert settings.API_KEY == "from-file"
assert settings.MAX_RESULTS == 5
assert settings.CACHE_FOLDER == "cache"
def test_environment_overrides_the_plugin_file(config_folder, monkeypatch):
_write_plugin_file("example-plugin", {"API_KEY": "from-file", "MAX_RESULTS": 5})
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY", "from-environment")
settings = ExamplePluginSettings.load()
assert settings.API_KEY == "from-environment"
assert settings.MAX_RESULTS == 5
def test_arguments_override_the_environment_and_the_plugin_file(
config_folder, monkeypatch
):
_write_plugin_file("example-plugin", {"API_KEY": "from-file", "MAX_RESULTS": 5})
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY", "from-environment")
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_MAX_RESULTS", "10")
settings = ExamplePluginSettings(API_KEY="from-argument")
assert settings.API_KEY == "from-argument"
assert settings.MAX_RESULTS == 10
def test_save_and_load_round_trip(config_folder):
settings = ExamplePluginSettings.load()
settings.API_KEY = "saved-key"
settings.MAX_RESULTS = 100
settings.save()
config_path = plugin_config_path("example-plugin")
assert os.path.isfile(config_path)
with open(config_path) as config_file:
assert yaml.safe_load(config_file) == {
"API_KEY": "saved-key",
"MAX_RESULTS": 100,
}
reloaded = ExamplePluginSettings.load()
assert reloaded.API_KEY == "saved-key"
assert reloaded.MAX_RESULTS == 100
assert reloaded.CACHE_FOLDER == "cache"
@pytest.mark.skipif(
sys.platform == "win32", reason="POSIX file permissions are not available"
)
def test_saved_file_is_only_readable_by_the_user(config_folder):
settings = ExamplePluginSettings.load()
settings.API_KEY = "saved-key"
settings.save()
config_path = plugin_config_path("example-plugin")
assert stat.S_IMODE(os.stat(config_path).st_mode) == 0o600
folder_mode = stat.S_IMODE(os.stat(plugin_config_folder()).st_mode)
assert folder_mode & 0o077 == 0
@pytest.mark.skipif(
sys.platform == "win32", reason="POSIX file permissions are not available"
)
def test_save_restricts_the_permissions_of_an_existing_file(config_folder):
config_path = _write_plugin_file("example-plugin", {"API_KEY": "from-file"})
os.chmod(config_path, 0o644)
settings = ExamplePluginSettings.load()
settings.MAX_RESULTS = 100
settings.save()
assert stat.S_IMODE(os.stat(config_path).st_mode) == 0o600
assert os.listdir(plugin_config_folder()) == ["example-plugin.yaml"]
def test_save_only_persists_non_default_values(config_folder):
settings = ExamplePluginSettings.load()
settings.CACHE_FOLDER = "another-cache"
settings.save()
with open(plugin_config_path("example-plugin")) as config_file:
assert yaml.safe_load(config_file) == {"CACHE_FOLDER": "another-cache"}
def test_save_does_not_persist_values_coming_from_the_environment(
config_folder, monkeypatch
):
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY", "environment-secret")
settings = ExamplePluginSettings.load()
assert settings.API_KEY == "environment-secret"
settings.MAX_RESULTS = 100
settings.save()
with open(plugin_config_path("example-plugin")) as config_file:
assert yaml.safe_load(config_file) == {"MAX_RESULTS": 100}
def test_an_invalid_environment_variable_still_protects_the_other_values(
config_folder, monkeypatch
):
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY", "environment-secret")
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_MAX_RESULTS", "not-an-int")
settings = ExamplePluginSettings(MAX_RESULTS=100)
assert settings.API_KEY == "environment-secret"
settings.save()
with open(plugin_config_path("example-plugin")) as config_file:
saved_values = yaml.safe_load(config_file)
assert saved_values == {"MAX_RESULTS": 100}
assert "API_KEY" not in saved_values
def test_save_persists_values_which_differ_from_the_environment(
config_folder, monkeypatch
):
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY", "environment-secret")
settings = ExamplePluginSettings.load()
settings.API_KEY = "chosen-key"
settings.save()
with open(plugin_config_path("example-plugin")) as config_file:
assert yaml.safe_load(config_file) == {"API_KEY": "chosen-key"}
def test_save_does_not_write_the_mvt_configuration_file(config_folder):
settings = ExamplePluginSettings.load()
settings.API_KEY = "saved-key"
settings.save()
assert os.listdir(config_folder) == ["plugins"]
def test_unknown_keys_in_the_plugin_file_are_ignored(config_folder):
_write_plugin_file(
"example-plugin",
{"API_KEY": "from-file", "UNKNOWN_SETTING": "ignored"},
)
settings = ExamplePluginSettings.load()
assert settings.API_KEY == "from-file"
assert not hasattr(settings, "UNKNOWN_SETTING")
def test_unparsable_plugin_file_is_reported_with_its_path(config_folder):
config_path = _write_plugin_file("example-plugin", "API_KEY: [unclosed\n")
with pytest.raises(PluginConfigLoadError) as raised:
ExamplePluginSettings.load()
assert config_path in str(raised.value)
def test_plugin_file_which_is_not_a_mapping_is_reported_with_its_path(config_folder):
config_path = _write_plugin_file("example-plugin", "- one\n- two\n")
with pytest.raises(PluginConfigLoadError) as raised:
ExamplePluginSettings.load()
assert config_path in str(raised.value)
assert "mapping of setting names" in str(raised.value)
def test_plugins_do_not_interfere_with_each_other(config_folder, monkeypatch):
_write_plugin_file("other-plugin", {"API_KEY": "other-file-key"})
monkeypatch.setenv("MVT_PLUGIN_EXAMPLE_PLUGIN_API_KEY", "example-environment-key")
example_settings = ExamplePluginSettings.load()
other_settings = OtherPluginSettings.load()
assert example_settings.API_KEY == "example-environment-key"
assert other_settings.API_KEY == "other-file-key"
example_settings.MAX_RESULTS = 100
example_settings.save()
assert sorted(os.listdir(plugin_config_folder())) == [
"example-plugin.yaml",
"other-plugin.yaml",
]
with open(plugin_config_path("other-plugin")) as config_file:
assert yaml.safe_load(config_file) == {"API_KEY": "other-file-key"}
def test_subclass_without_plugin_name_is_rejected():
with pytest.raises(TypeError, match="plugin_name"):
class MissingNameSettings(MVTPluginSettings):
API_KEY: Optional[str] = None
def test_subclass_with_invalid_plugin_name_is_rejected():
with pytest.raises(ValueError, match="Invalid plugin name"):
class InvalidNameSettings(MVTPluginSettings):
plugin_name = "Bad/Name"
def test_underscores_are_not_allowed_in_plugin_names():
# Underscores are replaced by dashes in the environment prefix, so allowing
# both would let two plugin names share one environment namespace.
with pytest.raises(ValueError, match="Invalid plugin name"):
class UnderscoreNameSettings(MVTPluginSettings):
plugin_name = "under_score"
with pytest.raises(ValueError, match="Invalid plugin name"):
plugin_config_path("under_score")
with pytest.raises(ValueError, match="Invalid plugin name"):
plugin_env_prefix("under_score")
@pytest.mark.parametrize(
"plugin_name", ["../escape", "folder/name", "UPPER", "-dash", ""]
)
def test_unsafe_plugin_names_have_no_configuration_path(plugin_name):
with pytest.raises(ValueError, match="Invalid plugin name"):
plugin_config_path(plugin_name)