Sorting imports and removing unused ones

Added process to automatically check for indicators updates

.github/workflows/python-package.yml

@@ -28,7 +28,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install flake8 pytest safety stix2
+        python -m pip install flake8 pytest safety stix2 pytest-mock
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         python -m pip install .
     - name: Lint with flake8

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-     <img src="./docs/mvt.png" width="200" />
+     <img src="https://docs.mvt.re/en/latest/mvt.png" width="200" />
 </p>
 
 # Mobile Verification Toolkit

docs/android/download_apks.md

@@ -13,22 +13,16 @@ It might take several minutes to complete.
 !!! info
     MVT will likely warn you it was unable to download certain installed packages. There is no reason to be alarmed: this is typically expected behavior when MVT attempts to download a system package it has no privileges to access.
 
-Optionally, you can decide to enable lookups of the SHA256 hash of all the extracted APKs on [VirusTotal](https://www.virustotal.com) and/or [Koodous](https://koodous.com). While these lookups do not provide any conclusive assessment on all of the extracted APKs, they might highlight any known malicious ones:
+Optionally, you can decide to enable lookups of the SHA256 hash of all the extracted APKs on [VirusTotal](https://www.virustotal.com). While these lookups do not provide any conclusive assessment on all of the extracted APKs, they might highlight any known malicious ones:
 
 ```bash
-mvt-android download-apks --output /path/to/folder --virustotal
-mvt-android download-apks --output /path/to/folder --koodous
+MVT_VT_API_KEY=<key> mvt-android download-apks --output /path/to/folder --virustotal
 ```
 
-Or, to launch all available lookups:
+Please note that in order to use VirusTotal lookups you are required to provide your own API key through the `MVT_VT_API_KEY` environment variable. You should also note that VirusTotal enforces strict API usage. Be mindful that MVT might consume your hourly search quota.
+
+In case you have a previous extraction of APKs you want to later check against VirusTotal, you can do so with the following arguments:
 
 ```bash
-mvt-android download-apks --output /path/to/folder --all-checks
+MVT_VT_API_KEY=<key> mvt-android download-apks --from-file /path/to/folder/apks.json --virustotal
 ```
-
-In case you have a previous extraction of APKs you want to later check against VirusTotal and Koodous, you can do so with the following arguments:
-
-```bash
-mvt-android download-apks --from-file /path/to/folder/apks.json --all-checks
-```
-

docs/android/methodology.md

@@ -8,8 +8,10 @@ However, not all is lost.
 
 Because malware attacks over Android typically take the form of malicious or backdoored apps, the very first thing you might want to do is to extract and verify all installed Android packages and triage quickly if there are any which stand out as malicious or which might be atypical.
 
-While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly lookup services such as [VirusTotal](https://www.virustotal.com) or [Koodous](https://koodous.com) which might quickly indicate known bad apps.
+While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly look them up on services such as [VirusTotal](https://www.virustotal.com).
 
+!!! info "Using VirusTotal"
+	Please note that in order to use VirusTotal lookups you are required to provide your own API key through the `MVT_VT_API_KEY` environment variable. You should also note that VirusTotal enforces strict API usage. Be mindful that MVT might consume your hourly search quota.
 
 ## Check the device over Android Debug Bridge
 

docs/iocs.md

@@ -41,6 +41,6 @@ export MVT_STIX2="/home/user/IOC1.stix2:/home/user/IOC2.stix2"
     - [Predator from Cytrox](https://citizenlab.ca/2021/12/pegasus-vs-predator-dissidents-doubly-infected-iphone-reveals-cytrox-mercenary-spyware/) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-12-16_cytrox/cytrox.stix2))
 - [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/generated/stalkerware.stix2).
 
-You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators listed [here](https://github.com/mvt-project/mvt/blob/main/public_indicators.json) and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by mvt.
+You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators listed [here](https://github.com/mvt-project/mvt/blob/main/public_indicators.json) and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by MVT.
 
 Please [open an issue](https://github.com/mvt-project/mvt/issues/) to suggest new sources of STIX-formatted IOCs.

docs/ios/records.md

@@ -18,7 +18,7 @@ If indicators are provided through the command-line, processes and domains are c
 
 ### `backup_info.json`
 
-!!! info "Availabiliy"
+!!! info "Availability"
     Backup: :material-check:
     Full filesystem dump: :material-close:
 

mkdocs.yml

@@ -1,7 +1,7 @@
 site_name: Mobile Verification Toolkit
 repo_url: https://github.com/mvt-project/mvt
 edit_uri: edit/main/docs/
-copyright: Copyright © 2021 MVT Project Developers
+copyright: Copyright © 2021-2022 MVT Project Developers
 site_description: Mobile Verification Toolkit Documentation
 markdown_extensions:
     - attr_list

mvt/android/cli.py

@@ -3,31 +3,25 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
-import getpass
-import io
 import logging
 import os
-import tarfile
-from pathlib import Path
-from zipfile import ZipFile
 
 import click
 from rich.logging import RichHandler
 
-from mvt.android.parsers.backup import (AndroidBackupParsingError,
-                                        InvalidBackupPassword, parse_ab_header,
-                                        parse_backup_file)
+from mvt.common.cmd_check_iocs import CmdCheckIOCS
 from mvt.common.help import (HELP_MSG_FAST, HELP_MSG_IOC,
                              HELP_MSG_LIST_MODULES, HELP_MSG_MODULE,
                              HELP_MSG_OUTPUT, HELP_MSG_SERIAL)
-from mvt.common.indicators import Indicators, download_indicators_files
 from mvt.common.logo import logo
-from mvt.common.module import run_module, save_timeline
+from mvt.common.updates import IndicatorsUpdates
 
-from .download_apks import DownloadAPKs
-from .lookups.koodous import koodous_lookup
-from .lookups.virustotal import virustotal_lookup
+from .cmd_check_adb import CmdAndroidCheckADB
+from .cmd_check_backup import CmdAndroidCheckBackup
+from .cmd_check_bugreport import CmdAndroidCheckBugreport
+from .cmd_download_apks import DownloadAPKs
 from .modules.adb import ADB_MODULES
+from .modules.adb.packages import Packages
 from .modules.backup import BACKUP_MODULES
 from .modules.bugreport import BUGREPORT_MODULES
 
@@ -62,14 +56,12 @@ def version():
 @click.option("--all-apks", "-a", is_flag=True,
               help="Extract all packages installed on the phone, including system packages")
 @click.option("--virustotal", "-v", is_flag=True, help="Check packages on VirusTotal")
-@click.option("--koodous", "-k", is_flag=True, help="Check packages on Koodous")
-@click.option("--all-checks", "-A", is_flag=True, help="Run all available checks")
 @click.option("--output", "-o", type=click.Path(exists=False),
               help="Specify a path to a folder where you want to store the APKs")
 @click.option("--from-file", "-f", type=click.Path(exists=True),
               help="Instead of acquiring from phone, load an existing packages.json file for lookups (mainly for debug purposes)")
 @click.pass_context
-def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_file, serial):
+def download_apks(ctx, all_apks, virustotal, output, from_file, serial):
     try:
         if from_file:
             download = DownloadAPKs.from_json(from_file)
@@ -92,16 +84,20 @@ def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_f
                 download.serial = serial
             download.run()
 
-        packages = download.packages
+        packages_to_lookup = []
+        if all_apks:
+            packages_to_lookup = download.packages
+        else:
+            for package in download.packages:
+                if not package.get("system", False):
+                    packages_to_lookup.append(package)
 
-        if len(packages) == 0:
-            return
+            if len(packages_to_lookup) == 0:
+                return
 
-        if virustotal or all_checks:
-            virustotal_lookup(packages)
-
-        if koodous or all_checks:
-            koodous_lookup(packages)
+        if virustotal:
+            m = Packages()
+            m.check_virustotal(packages_to_lookup)
     except KeyboardInterrupt:
         print("")
         ctx.exit(1)
@@ -120,49 +116,21 @@ def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_f
 @click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
 @click.option("--module", "-m", help=HELP_MSG_MODULE)
 @click.pass_context
-def check_adb(ctx, iocs, output, fast, list_modules, module, serial):
-    if list_modules:
-        log.info("Following is the list of available check-adb modules:")
-        for adb_module in ADB_MODULES:
-            log.info(" - %s", adb_module.__name__)
+def check_adb(ctx, serial, iocs, output, fast, list_modules, module):
+    cmd = CmdAndroidCheckADB(results_path=output, ioc_files=iocs,
+                             module_name=module, serial=serial, fast_mode=fast)
 
+    if list_modules:
+        cmd.list_modules()
         return
 
-    log.info("Checking Android through adb bridge")
+    log.info("Checking Android device over debug bridge")
 
-    if output and not os.path.exists(output):
-        try:
-            os.makedirs(output)
-        except Exception as e:
-            log.critical("Unable to create output folder %s: %s", output, e)
-            ctx.exit(1)
+    cmd.run()
 
-    indicators = Indicators(log=log)
-    indicators.load_indicators_files(iocs)
-
-    timeline = []
-    timeline_detected = []
-    for adb_module in ADB_MODULES:
-        if module and adb_module.__name__ != module:
-            continue
-
-        m = adb_module(output_folder=output, fast_mode=fast,
-                       log=logging.getLogger(adb_module.__module__))
-        if indicators.total_ioc_count:
-            m.indicators = indicators
-            m.indicators.log = m.log
-        if serial:
-            m.serial = serial
-
-        run_module(m)
-        timeline.extend(m.timeline)
-        timeline_detected.extend(m.timeline_detected)
-
-    if output:
-        if len(timeline) > 0:
-            save_timeline(timeline, os.path.join(output, "timeline.csv"))
-        if len(timeline_detected) > 0:
-            save_timeline(timeline_detected, os.path.join(output, "timeline_detected.csv"))
+    if len(cmd.timeline_detected) > 0:
+        log.warning("The analysis of the Android device produced %d detections!",
+                    len(cmd.timeline_detected))
 
 
 #==============================================================================
@@ -177,66 +145,20 @@ def check_adb(ctx, iocs, output, fast, list_modules, module, serial):
 @click.argument("BUGREPORT_PATH", type=click.Path(exists=True))
 @click.pass_context
 def check_bugreport(ctx, iocs, output, list_modules, module, bugreport_path):
-    if list_modules:
-        log.info("Following is the list of available check-bugreport modules:")
-        for adb_module in BUGREPORT_MODULES:
-            log.info(" - %s", adb_module.__name__)
+    cmd = CmdAndroidCheckBugreport(target_path=bugreport_path, results_path=output,
+                                   ioc_files=iocs, module_name=module)
 
+    if list_modules:
+        cmd.list_modules()
         return
 
-    log.info("Checking an Android Bug Report located at: %s", bugreport_path)
+    log.info("Checking Android bug report at path: %s", bugreport_path)
 
-    if output and not os.path.exists(output):
-        try:
-            os.makedirs(output)
-        except Exception as e:
-            log.critical("Unable to create output folder %s: %s", output, e)
-            ctx.exit(1)
+    cmd.run()
 
-    indicators = Indicators(log=log)
-    indicators.load_indicators_files(iocs)
-
-    if os.path.isfile(bugreport_path):
-        bugreport_format = "zip"
-        zip_archive = ZipFile(bugreport_path)
-        zip_files = []
-        for file_name in zip_archive.namelist():
-            zip_files.append(file_name)
-    elif os.path.isdir(bugreport_path):
-        bugreport_format = "dir"
-        folder_files = []
-        parent_path = Path(bugreport_path).absolute().as_posix()
-        for root, subdirs, subfiles in os.walk(os.path.abspath(bugreport_path)):
-            for file_name in subfiles:
-                folder_files.append(os.path.relpath(os.path.join(root, file_name), parent_path))
-
-    timeline = []
-    timeline_detected = []
-    for bugreport_module in BUGREPORT_MODULES:
-        if module and bugreport_module.__name__ != module:
-            continue
-
-        m = bugreport_module(base_folder=bugreport_path, output_folder=output,
-                             log=logging.getLogger(bugreport_module.__module__))
-
-        if bugreport_format == "zip":
-            m.from_zip(zip_archive, zip_files)
-        else:
-            m.from_folder(bugreport_path, folder_files)
-
-        if indicators.total_ioc_count:
-            m.indicators = indicators
-            m.indicators.log = m.log
-
-        run_module(m)
-        timeline.extend(m.timeline)
-        timeline_detected.extend(m.timeline_detected)
-
-    if output:
-        if len(timeline) > 0:
-            save_timeline(timeline, os.path.join(output, "timeline.csv"))
-        if len(timeline_detected) > 0:
-            save_timeline(timeline_detected, os.path.join(output, "timeline_detected.csv"))
+    if len(cmd.timeline_detected) > 0:
+        log.warning("The analysis of the Android bug report produced %d detections!",
+                    len(cmd.timeline_detected))
 
 
 #==============================================================================
@@ -247,74 +169,24 @@ def check_bugreport(ctx, iocs, output, list_modules, module, bugreport_path):
 @click.option("--iocs", "-i", type=click.Path(exists=True), multiple=True,
               default=[], help=HELP_MSG_IOC)
 @click.option("--output", "-o", type=click.Path(exists=False), help=HELP_MSG_OUTPUT)
+@click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
 @click.argument("BACKUP_PATH", type=click.Path(exists=True))
 @click.pass_context
-def check_backup(ctx, iocs, output, backup_path, serial):
-    log.info("Checking ADB backup located at: %s", backup_path)
+def check_backup(ctx, serial, iocs, output, list_modules, backup_path):
+    cmd = CmdAndroidCheckBackup(target_path=backup_path, results_path=output,
+                                ioc_files=iocs)
 
-    if os.path.isfile(backup_path):
-        # AB File
-        backup_type = "ab"
-        with open(backup_path, "rb") as handle:
-            data = handle.read()
-        header = parse_ab_header(data)
-        if not header["backup"]:
-            log.critical("Invalid backup format, file should be in .ab format")
-            ctx.exit(1)
-        password = None
-        if header["encryption"] != "none":
-            password = getpass.getpass(prompt="Backup Password: ", stream=None)
-        try:
-            tardata = parse_backup_file(data, password=password)
-        except InvalidBackupPassword:
-            log.critical("Invalid backup password")
-            ctx.exit(1)
-        except AndroidBackupParsingError:
-            log.critical("Impossible to parse this backup file, please use Android Backup Extractor instead")
-            ctx.exit(1)
+    if list_modules:
+        cmd.list_modules()
+        return
 
-        dbytes = io.BytesIO(tardata)
-        tar = tarfile.open(fileobj=dbytes)
-        files = []
-        for member in tar:
-            files.append(member.name)
+    log.info("Checking Android backup at path: %s", backup_path)
 
-    elif os.path.isdir(backup_path):
-        backup_type = "folder"
-        backup_path = Path(backup_path).absolute().as_posix()
-        files = []
-        for root, subdirs, subfiles in os.walk(os.path.abspath(backup_path)):
-            for fname in subfiles:
-                files.append(os.path.relpath(os.path.join(root, fname), backup_path))
-    else:
-        log.critical("Invalid backup path, path should be a folder or an Android Backup (.ab) file")
-        ctx.exit(1)
+    cmd.run()
 
-    if output and not os.path.exists(output):
-        try:
-            os.makedirs(output)
-        except Exception as e:
-            log.critical("Unable to create output folder %s: %s", output, e)
-            ctx.exit(1)
-
-    indicators = Indicators(log=log)
-    indicators.load_indicators_files(iocs)
-
-    for module in BACKUP_MODULES:
-        m = module(base_folder=backup_path, output_folder=output,
-                   log=logging.getLogger(module.__module__))
-        if indicators.total_ioc_count:
-            m.indicators = indicators
-            m.indicators.log = m.log
-        if serial:
-            m.serial = serial
-
-        if backup_type == "folder":
-            m.from_folder(backup_path, files)
-        else:
-            m.from_ab(backup_path, tar, files)
-
-        run_module(m)
+    if len(cmd.timeline_detected) > 0:
+        log.warning("The analysis of the Android backup produced %d detections!",
+                    len(cmd.timeline_detected))
 
 
 #==============================================================================
@@ -328,59 +200,14 @@ def check_backup(ctx, iocs, output, backup_path, serial):
 @click.argument("FOLDER", type=click.Path(exists=True))
 @click.pass_context
 def check_iocs(ctx, iocs, list_modules, module, folder):
-    all_modules = []
-    for entry in BACKUP_MODULES + ADB_MODULES:
-        if entry not in all_modules:
-            all_modules.append(entry)
+    cmd = CmdCheckIOCS(target_path=folder, ioc_files=iocs, module_name=module)
+    cmd.modules = BACKUP_MODULES + ADB_MODULES + BUGREPORT_MODULES
 
     if list_modules:
-        log.info("Following is the list of available check-iocs modules:")
-        for iocs_module in all_modules:
-            log.info(" - %s", iocs_module.__name__)
-
+        cmd.list_modules()
         return
 
-    log.info("Checking stored results against provided indicators...")
-
-    indicators = Indicators(log=log)
-    indicators.load_indicators_files(iocs)
-
-    total_detections = 0
-    for file_name in os.listdir(folder):
-        name_only, ext = os.path.splitext(file_name)
-        file_path = os.path.join(folder, file_name)
-
-        # TODO: Skipping processing of result files that are not json.
-        #       We might want to revisit this eventually.
-        if ext != ".json":
-            continue
-
-        for iocs_module in all_modules:
-            if module and iocs_module.__name__ != module:
-                continue
-
-            if iocs_module().get_slug() != name_only:
-                continue
-
-            log.info("Loading results from \"%s\" with module %s", file_name,
-                     iocs_module.__name__)
-
-            m = iocs_module.from_json(file_path,
-                                      log=logging.getLogger(iocs_module.__module__))
-            if indicators.total_ioc_count > 0:
-                m.indicators = indicators
-                m.indicators.log = m.log
-
-            try:
-                m.check_indicators()
-            except NotImplementedError:
-                continue
-            else:
-                total_detections += len(m.detected)
-
-    if total_detections > 0:
-        log.warning("The check of the results produced %d detections!",
-                    total_detections)
+    cmd.run()
 
 
 #==============================================================================
@@ -388,4 +215,5 @@ def check_iocs(ctx, iocs, list_modules, module, folder):
 #==============================================================================
 @cli.command("download-iocs", help="Download public STIX2 indicators")
 def download_indicators():
-    download_indicators_files(log)
+    ioc_updates = IndicatorsUpdates()
+    ioc_updates.update()

mvt/android/cmd_check_adb.py

@@ -0,0 +1,25 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+
+from mvt.common.command import Command
+
+from .modules.adb import ADB_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdAndroidCheckADB(Command):
+
+    name = "check-adb"
+    modules = ADB_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)

mvt/android/cmd_check_backup.py

@@ -0,0 +1,83 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import getpass
+import io
+import logging
+import os
+import sys
+import tarfile
+from pathlib import Path
+
+from mvt.android.parsers.backup import (AndroidBackupParsingError,
+                                        InvalidBackupPassword, parse_ab_header,
+                                        parse_backup_file)
+from mvt.common.command import Command
+
+from .modules.backup import BACKUP_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdAndroidCheckBackup(Command):
+
+    name = "check-backup"
+    modules = BACKUP_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+
+        self.backup_type = None
+        self.backup_archive = None
+        self.backup_files = []
+
+    def init(self):
+        if os.path.isfile(self.target_path):
+            self.backup_type = "ab"
+            with open(self.target_path, "rb") as handle:
+                data = handle.read()
+
+            header = parse_ab_header(data)
+            if not header["backup"]:
+                log.critical("Invalid backup format, file should be in .ab format")
+                sys.exit(1)
+
+            password = None
+            if header["encryption"] != "none":
+                password = getpass.getpass(prompt="Backup Password: ", stream=None)
+            try:
+                tardata = parse_backup_file(data, password=password)
+            except InvalidBackupPassword:
+                log.critical("Invalid backup password")
+                sys.exit(1)
+            except AndroidBackupParsingError as e:
+                log.critical("Impossible to parse this backup file: %s", e)
+                log.critical("Please use Android Backup Extractor (ABE) instead")
+                sys.exit(1)
+
+            dbytes = io.BytesIO(tardata)
+            self.backup_archive = tarfile.open(fileobj=dbytes)
+            for member in self.backup_archive:
+                self.backup_files.append(member.name)
+
+        elif os.path.isdir(self.target_path):
+            self.backup_type = "folder"
+            self.target_path = Path(self.target_path).absolute().as_posix()
+            for root, subdirs, subfiles in os.walk(os.path.abspath(self.target_path)):
+                for fname in subfiles:
+                    self.backup_files.append(os.path.relpath(os.path.join(root, fname), self.target_path))
+        else:
+            log.critical("Invalid backup path, path should be a folder or an Android Backup (.ab) file")
+            sys.exit(1)
+
+    def module_init(self, module):
+        if self.backup_type == "folder":
+            module.from_folder(self.target_path, self.backup_files)
+        else:
+            module.from_ab(self.target_path, self.backup_archive, self.backup_files)

mvt/android/cmd_check_bugreport.py

@@ -0,0 +1,51 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+from pathlib import Path
+from zipfile import ZipFile
+
+from mvt.common.command import Command
+
+from .modules.bugreport import BUGREPORT_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdAndroidCheckBugreport(Command):
+
+    name = "check-bugreport"
+    modules = BUGREPORT_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+
+        self.bugreport_format = None
+        self.bugreport_archive = None
+        self.bugreport_files = []
+
+    def init(self):
+        if os.path.isfile(self.target_path):
+            self.bugreport_format = "zip"
+            self.bugreport_archive = ZipFile(self.target_path)
+            for file_name in self.bugreport_archive.namelist():
+                self.bugreport_files.append(file_name)
+        elif os.path.isdir(self.target_path):
+            self.bugreport_format = "dir"
+            parent_path = Path(self.target_path).absolute().as_posix()
+            for root, subdirs, subfiles in os.walk(os.path.abspath(self.target_path)):
+                for file_name in subfiles:
+                    self.bugreport_files.append(os.path.relpath(os.path.join(root, file_name), parent_path))
+
+    def module_init(self, module):
+        if self.bugreport_format == "zip":
+            module.from_zip(self.bugreport_archive, self.bugreport_files)
+        else:
+            module.from_folder(self.target_path, self.bugreport_files)

mvt/android/cmd_download_apks.py

@@ -176,7 +176,7 @@ class DownloadAPKs(AndroidExtraction):
         with open(json_path, "w", encoding="utf-8") as handle:
             json.dump(self.packages, handle, indent=4)
 
-    def run(self):
+    def run(self) -> None:
         """Run all steps of fetch-apk."""
         self.get_packages()
         self._adb_connect()

mvt/android/lookups/koodous.py

@@ -1,58 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 Claudio Guarnieri.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/
-
-import logging
-
-import requests
-from rich.console import Console
-from rich.progress import track
-from rich.table import Table
-from rich.text import Text
-
-log = logging.getLogger(__name__)
-
-
-def koodous_lookup(packages):
-    log.info("Looking up all extracted files on Koodous (www.koodous.com)")
-    log.info("This might take a while...")
-
-    table = Table(title="Koodous Packages Detections")
-    table.add_column("Package name")
-    table.add_column("File name")
-    table.add_column("Trusted")
-    table.add_column("Detected")
-    table.add_column("Rating")
-
-    total_packages = len(packages)
-    for i in track(range(total_packages), description=f"Looking up {total_packages} packages..."):
-        package = packages[i]
-        for file in package.get("files", []):
-            url = f"https://api.koodous.com/apks/{file['sha256']}"
-            res = requests.get(url)
-            report = res.json()
-
-            row = [package["package_name"], file["path"]]
-
-            if "package_name" in report:
-                trusted = "no"
-                if report["trusted"]:
-                    trusted = Text("yes", "green bold")
-
-                detected = "no"
-                if report["detected"]:
-                    detected = Text("yes", "red bold")
-
-                rating = "0"
-                if int(report["rating"]) < 0:
-                    rating = Text(str(report["rating"]), "red bold")
-
-                row.extend([trusted, detected, rating])
-            else:
-                row.extend(["n/a", "n/a", "n/a"])
-
-            table.add_row(*row)
-
-    console = Console()
-    console.print(table)

mvt/android/lookups/virustotal.py

@@ -1,100 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 Claudio Guarnieri.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/
-
-import logging
-
-import requests
-from rich.console import Console
-from rich.progress import track
-from rich.table import Table
-from rich.text import Text
-
-log = logging.getLogger(__name__)
-
-
-def get_virustotal_report(hashes):
-    apikey = "233f22e200ca5822bd91103043ccac138b910db79f29af5616a9afe8b6f215ad"
-    url = f"https://www.virustotal.com/partners/sysinternals/file-reports?apikey={apikey}"
-
-    items = []
-    for sha256 in hashes:
-        items.append({
-            "autostart_location": "",
-            "autostart_entry": "",
-            "hash": sha256,
-            "local_name": "",
-            "creation_datetime": "",
-        })
-    headers = {"User-Agent": "VirusTotal", "Content-Type": "application/json"}
-    res = requests.post(url, headers=headers, json=items)
-
-    if res.status_code == 200:
-        report = res.json()
-        return report["data"]
-    else:
-        log.error("Unexpected response from VirusTotal: %s", res.status_code)
-        return None
-
-
-def virustotal_lookup(packages):
-    # NOTE: This is temporary, until we resolved the issue.
-    log.error("Unfortunately VirusTotal lookup is disabled until further notice, due to unresolved issues with the API service.")
-    return
-
-    log.info("Looking up all extracted files on VirusTotal (www.virustotal.com)")
-
-    unique_hashes = []
-    for package in packages:
-        for file in package.get("files", []):
-            if file["sha256"] not in unique_hashes:
-                unique_hashes.append(file["sha256"])
-
-    total_unique_hashes = len(unique_hashes)
-
-    detections = {}
-
-    def virustotal_query(batch):
-        report = get_virustotal_report(batch)
-        if not report:
-            return
-
-        for entry in report:
-            if entry["hash"] not in detections and entry["found"] is True:
-                detections[entry["hash"]] = entry["detection_ratio"]
-
-    batch = []
-    for i in track(range(total_unique_hashes), description=f"Looking up {total_unique_hashes} files..."):
-        file_hash = unique_hashes[i]
-        batch.append(file_hash)
-        if len(batch) == 25:
-            virustotal_query(batch)
-            batch = []
-
-    if batch:
-        virustotal_query(batch)
-
-    table = Table(title="VirusTotal Packages Detections")
-    table.add_column("Package name")
-    table.add_column("File path")
-    table.add_column("Detections")
-
-    for package in packages:
-        for file in package.get("files", []):
-            row = [package["package_name"], file["path"]]
-
-            if file["sha256"] in detections:
-                detection = detections[file["sha256"]]
-                positives = detection.split("/")[0]
-                if int(positives) > 0:
-                    row.append(Text(detection, "red bold"))
-                else:
-                    row.append(detection)
-            else:
-                row.append("not found")
-
-            table.add_row(*row)
-
-    console = Console()
-    console.print(table)

mvt/android/modules/adb/base.py

@@ -12,6 +12,7 @@ import string
 import sys
 import tempfile
 import time
+from typing import Callable
 
 from adb_shell.adb_device import AdbDeviceTcp, AdbDeviceUsb
 from adb_shell.auth.keygen import keygen, write_public_keyfile
@@ -33,17 +34,18 @@ ADB_PUB_KEY_PATH = os.path.expanduser("~/.android/adbkey.pub")
 class AndroidExtraction(MVTModule):
     """This class provides a base for all Android extraction modules."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.device = None
         self.serial = None
 
     @staticmethod
-    def _adb_check_keys():
+    def _adb_check_keys() -> None:
         """Make sure Android adb keys exist."""
         if not os.path.isdir(os.path.dirname(ADB_KEY_PATH)):
             os.makedirs(os.path.dirname(ADB_KEY_PATH))
@@ -54,7 +56,7 @@ class AndroidExtraction(MVTModule):
         if not os.path.exists(ADB_PUB_KEY_PATH):
             write_public_keyfile(ADB_KEY_PATH, ADB_PUB_KEY_PATH)
 
-    def _adb_connect(self):
+    def _adb_connect(self) -> None:
         """Connect to the device over adb."""
         self._adb_check_keys()
 
@@ -103,17 +105,17 @@ class AndroidExtraction(MVTModule):
             else:
                 break
 
-    def _adb_disconnect(self):
+    def _adb_disconnect(self) -> None:
         """Close adb connection to the device."""
         self.device.close()
 
-    def _adb_reconnect(self):
+    def _adb_reconnect(self) -> None:
         """Reconnect to device using adb."""
         log.info("Reconnecting ...")
         self._adb_disconnect()
         self._adb_connect()
 
-    def _adb_command(self, command):
+    def _adb_command(self, command: str) -> str:
         """Execute an adb shell command.
 
         :param command: Shell command to execute
@@ -122,7 +124,7 @@ class AndroidExtraction(MVTModule):
         """
         return self.device.shell(command, read_timeout_s=200.0)
 
-    def _adb_check_if_root(self):
+    def _adb_check_if_root(self) -> bool:
         """Check if we have a `su` binary on the Android device.
 
 
@@ -131,7 +133,7 @@ class AndroidExtraction(MVTModule):
         """
         return bool(self._adb_command("command -v su"))
 
-    def _adb_root_or_die(self):
+    def _adb_root_or_die(self) -> None:
         """Check if we have a `su` binary, otherwise raise an Exception."""
         if not self._adb_check_if_root():
             raise InsufficientPrivileges("This module is optionally available in case the device is already rooted. Do NOT root your own device!")
@@ -145,7 +147,7 @@ class AndroidExtraction(MVTModule):
         """
         return self._adb_command(f"su -c {command}")
 
-    def _adb_check_file_exists(self, file):
+    def _adb_check_file_exists(self, file: str) -> bool:
         """Verify that a file exists.
 
         :param file: Path of the file
@@ -162,7 +164,9 @@ class AndroidExtraction(MVTModule):
 
         return bool(self._adb_command_as_root(f"[ ! -f {file} ] || echo 1"))
 
-    def _adb_download(self, remote_path, local_path, progress_callback=None, retry_root=True):
+    def _adb_download(self, remote_path: str, local_path: str,
+                      progress_callback: Callable = None,
+                      retry_root: bool = True) -> None:
         """Download a file form the device.
 
         :param remote_path: Path to download from the device
@@ -179,7 +183,8 @@ class AndroidExtraction(MVTModule):
             else:
                 raise Exception(f"Unable to download file {remote_path}: {e}")
 
-    def _adb_download_root(self, remote_path, local_path, progress_callback=None):
+    def _adb_download_root(self, remote_path: str, local_path: str,
+                           progress_callback: Callable = None) -> None:
         try:
             # Check if we have root, if not raise an Exception.
             self._adb_root_or_die()
@@ -207,7 +212,8 @@ class AndroidExtraction(MVTModule):
         except AdbCommandFailureException as e:
             raise Exception(f"Unable to download file {remote_path}: {e}")
 
-    def _adb_process_file(self, remote_path, process_routine):
+    def _adb_process_file(self, remote_path: str,
+                          process_routine: Callable) -> None:
         """Download a local copy of a file which is only accessible as root.
         This is a wrapper around process_routine.
 
@@ -247,7 +253,7 @@ class AndroidExtraction(MVTModule):
         # Disconnect from the device.
         self._adb_disconnect()
 
-    def _generate_backup(self, package_name):
+    def _generate_backup(self, package_name: str) -> bytes:
         self.log.warning("Please check phone and accept Android backup prompt. You may need to set a backup password. \a")
 
         # TODO: Base64 encoding as temporary fix to avoid byte-mangling over the shell transport...
@@ -273,6 +279,6 @@ class AndroidExtraction(MVTModule):
 
         self.log.warn("All attempts to decrypt backup with password failed!")
 
-    def run(self):
+    def run(self) -> None:
         """Run the main procedure."""
         raise NotImplementedError

mvt/android/modules/adb/chrome_history.py

@@ -20,13 +20,14 @@ CHROME_HISTORY_PATH = "data/data/com.android.chrome/app_chrome/Default/History"
 class ChromeHistory(AndroidExtraction):
     """This module extracts records from Android's Chrome browsing history."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -34,7 +35,7 @@ class ChromeHistory(AndroidExtraction):
             "data": f"{record['id']} - {record['url']} (visit ID: {record['visit_id']}, redirect source: {record['redirect_source']})"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -42,7 +43,7 @@ class ChromeHistory(AndroidExtraction):
             if self.indicators.check_domain(result["url"]):
                 self.detected.append(result)
 
-    def _parse_db(self, db_path):
+    def _parse_db(self, db_path: str) -> None:
         """Parse a Chrome History database file.
 
         :param db_path: Path to the History database to process.
@@ -77,7 +78,7 @@ class ChromeHistory(AndroidExtraction):
 
         log.info("Extracted a total of %d history items", len(self.results))
 
-    def run(self):
+    def run(self) -> None:
         try:
             self._adb_process_file(os.path.join("/", CHROME_HISTORY_PATH),
                                    self._parse_db)

mvt/android/modules/adb/dumpsys_accessibility.py

@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class DumpsysAccessibility(AndroidExtraction):
     """This module extracts stats on accessibility."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -32,7 +33,7 @@ class DumpsysAccessibility(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys accessibility")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_activities.py

@@ -15,15 +15,16 @@ log = logging.getLogger(__name__)
 class DumpsysActivities(AndroidExtraction):
     """This module extracts details on receivers for risky activities."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = results if results else {}
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -35,7 +36,7 @@ class DumpsysActivities(AndroidExtraction):
                     self.detected.append({intent: activity})
                     continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys package")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_appops.py

@@ -4,7 +4,6 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import re
 
 from mvt.android.parsers.dumpsys import parse_dumpsys_appops
 
@@ -18,13 +17,14 @@ class DumpsysAppOps(AndroidExtraction):
 
     slug = "dumpsys_appops"
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
         for perm in record["permissions"]:
             if "entries" not in perm:
@@ -36,12 +36,12 @@ class DumpsysAppOps(AndroidExtraction):
                         "timestamp": entry["timestamp"],
                         "module": self.__class__.__name__,
                         "event": entry["access"],
-                        "data": f"{record['package_name']} access to {perm['name']} : {entry['access']}",
+                        "data": f"{record['package_name']} access to {perm['name']}: {entry['access']}",
                     })
 
         return records
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         for result in self.results:
             if self.indicators:
                 ioc = self.indicators.check_app_id(result.get("package_name"))
@@ -55,7 +55,7 @@ class DumpsysAppOps(AndroidExtraction):
                     self.log.info("Package %s with REQUEST_INSTALL_PACKAGES permission",
                                   result["package_name"])
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys appops")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_battery_daily.py

@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class DumpsysBatteryDaily(AndroidExtraction):
     """This module extracts records from battery daily updates."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["from"],
             "module": self.__class__.__name__,
@@ -29,7 +30,7 @@ class DumpsysBatteryDaily(AndroidExtraction):
             "data": f"Recorded update of package {record['package_name']} with vers {record['vers']}"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -40,7 +41,7 @@ class DumpsysBatteryDaily(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys batterystats --daily")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_battery_history.py

@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class DumpsysBatteryHistory(AndroidExtraction):
     """This module extracts records from battery history events."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -32,7 +33,7 @@ class DumpsysBatteryHistory(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys batterystats --history")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_dbinfo.py

@@ -4,7 +4,6 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import re
 
 from mvt.android.parsers import parse_dumpsys_dbinfo
 
@@ -18,13 +17,14 @@ class DumpsysDBInfo(AndroidExtraction):
 
     slug = "dumpsys_dbinfo"
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -37,7 +37,7 @@ class DumpsysDBInfo(AndroidExtraction):
                     self.detected.append(result)
                     continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys dbinfo")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_full.py

@@ -14,18 +14,19 @@ log = logging.getLogger(__name__)
 class DumpsysFull(AndroidExtraction):
     """This module extracts stats on battery consumption by processes."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
 
         output = self._adb_command("dumpsys")
-        if self.output_folder:
-            output_path = os.path.join(self.output_folder, "dumpsys.txt")
+        if self.results_path:
+            output_path = os.path.join(self.results_path, "dumpsys.txt")
             with open(output_path, "w", encoding="utf-8") as handle:
                 handle.write(output)
 

mvt/android/modules/adb/dumpsys_receivers.py

@@ -21,15 +21,16 @@ INTENT_NEW_OUTGOING_CALL = "android.intent.action.NEW_OUTGOING_CALL"
 class DumpsysReceivers(AndroidExtraction):
     """This module extracts details on receivers for risky activities."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = results if results else {}
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -57,7 +58,7 @@ class DumpsysReceivers(AndroidExtraction):
                 self.detected.append({intent: receiver})
                 continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
 
         output = self._adb_command("dumpsys package")

mvt/android/modules/adb/files.py

@@ -17,14 +17,15 @@ log = logging.getLogger(__name__)
 class Files(AndroidExtraction):
     """This module extracts the list of files on the device."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
         self.full_find = False
 
-    def find_files(self, folder):
+    def find_files(self, folder: str) -> None:
         if self.full_find:
             output = self._adb_command(f"find '{folder}' -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")
 
@@ -46,7 +47,7 @@ class Files(AndroidExtraction):
             for file_line in output.splitlines():
                 self.results.append({"path": file_line.rstrip()})
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         if "modified_time" in record:
             return {
                 "timestamp": record["modified_time"],
@@ -55,7 +56,7 @@ class Files(AndroidExtraction):
                 "data": record["path"],
             }
 
-    def check_suspicious(self):
+    def check_suspicious(self) -> None:
         """Check for files with suspicious permissions"""
         for result in sorted(self.results, key=lambda item: item["path"]):
             if result.get("is_suid"):
@@ -63,7 +64,7 @@ class Files(AndroidExtraction):
                                  result["path"])
                 self.detected.append(result)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         """Check file list for known suspicious files or suspicious properties"""
         self.check_suspicious()
 
@@ -75,7 +76,7 @@ class Files(AndroidExtraction):
                 self.log.warning("Found a known suspicous file at path: \"%s\"", result["path"])
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
 
         output = self._adb_command("find '/' -maxdepth 1 -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")

mvt/android/modules/adb/getprop.py

@@ -4,7 +4,6 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import re
 from datetime import datetime, timedelta
 
 from mvt.android.parsers import parse_getprop
@@ -17,15 +16,16 @@ log = logging.getLogger(__name__)
 class Getprop(AndroidExtraction):
     """This module extracts device properties from getprop command."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = {} if not results else results
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("getprop")
         self._adb_disconnect()

mvt/android/modules/adb/logcat.py

@@ -14,13 +14,14 @@ log = logging.getLogger(__name__)
 class Logcat(AndroidExtraction):
     """This module extracts details on installed packages."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
 
         # Get the current logcat.
@@ -28,8 +29,8 @@ class Logcat(AndroidExtraction):
         # Get the locat prior to last reboot.
         last_output = self._adb_command("logcat -L")
 
-        if self.output_folder:
-            logcat_path = os.path.join(self.output_folder,
+        if self.results_path:
+            logcat_path = os.path.join(self.results_path,
                                        "logcat.txt")
             with open(logcat_path, "w", encoding="utf-8") as handle:
                 handle.write(output)
@@ -37,7 +38,7 @@ class Logcat(AndroidExtraction):
             log.info("Current logcat logs stored at %s",
                      logcat_path)
 
-            logcat_last_path = os.path.join(self.output_folder,
+            logcat_last_path = os.path.join(self.results_path,
                                             "logcat_last.txt")
             with open(logcat_last_path, "w", encoding="utf-8") as handle:
                 handle.write(last_output)

mvt/android/modules/adb/packages.py

@@ -4,12 +4,13 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import os
 
-import pkg_resources
+from rich.console import Console
+from rich.progress import track
+from rich.table import Table
+from rich.text import Text
 
-from mvt.android.lookups.koodous import koodous_lookup
-from mvt.android.lookups.virustotal import virustotal_lookup
+from mvt.common.virustotal import VTNoKey, VTQuotaExceeded, virustotal_lookup
 
 from .base import AndroidExtraction
 
@@ -71,13 +72,14 @@ ROOT_PACKAGES = [
 class Packages(AndroidExtraction):
     """This module extracts the list of installed packages."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
 
         timestamps = [
@@ -96,7 +98,7 @@ class Packages(AndroidExtraction):
 
         return records
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         for result in self.results:
             if result["package_name"] in ROOT_PACKAGES:
                 self.log.warning("Found an installed package related to rooting/jailbreaking: \"%s\"",
@@ -113,14 +115,67 @@ class Packages(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-            for package_file in result["files"]:
+            for package_file in result.get("files", []):
                 ioc = self.indicators.check_file_hash(package_file["sha256"])
                 if ioc:
                     result["matched_indicator"] = ioc
                     self.detected.append(result)
 
     @staticmethod
-    def parse_package_for_details(output):
+    def check_virustotal(packages: list) -> None:
+        hashes = []
+        for package in packages:
+            for file in package.get("files", []):
+                if file["sha256"] not in hashes:
+                    hashes.append(file["sha256"])
+
+        total_hashes = len(hashes)
+        detections = {}
+
+        for i in track(range(total_hashes), description=f"Looking up {total_hashes} files..."):
+            try:
+                results = virustotal_lookup(hashes[i])
+            except VTNoKey as e:
+                log.info(e)
+                return
+            except VTQuotaExceeded as e:
+                log.error("Unable to continue: %s", e)
+                break
+
+            if not results:
+                continue
+
+            positives = results["attributes"]["last_analysis_stats"]["malicious"]
+            total = len(results["attributes"]["last_analysis_results"])
+
+            detections[hashes[i]] = f"{positives}/{total}"
+
+        table = Table(title="VirusTotal Packages Detections")
+        table.add_column("Package name")
+        table.add_column("File path")
+        table.add_column("Detections")
+
+        for package in packages:
+            for file in package.get("files", []):
+                row = [package["package_name"], file["path"]]
+
+                if file["sha256"] in detections:
+                    detection = detections[file["sha256"]]
+                    positives = detection.split("/")[0]
+                    if int(positives) > 0:
+                        row.append(Text(detection, "red bold"))
+                    else:
+                        row.append(detection)
+                else:
+                    row.append("not found")
+
+                table.add_row(*row)
+
+        console = Console()
+        console.print(table)
+
+    @staticmethod
+    def parse_package_for_details(output: str) -> dict:
         details = {
             "uid": "",
             "version_name": "",
@@ -159,7 +214,7 @@ class Packages(AndroidExtraction):
 
         return details
 
-    def _get_files_for_package(self, package_name):
+    def _get_files_for_package(self, package_name: str) -> list:
         output = self._adb_command(f"pm path {package_name}")
         output = output.strip().replace("package:", "")
         if not output:
@@ -184,7 +239,7 @@ class Packages(AndroidExtraction):
 
         return package_files
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
 
         packages = self._adb_command("pm list packages -u -i -f")
@@ -262,8 +317,7 @@ class Packages(AndroidExtraction):
                           result["package_name"], result["installer"], result["timestamp"])
 
         if not self.fast_mode:
-            virustotal_lookup(packages_to_lookup)
-            koodous_lookup(packages_to_lookup)
+            self.check_virustotal(packages_to_lookup)
 
         self.log.info("Extracted at total of %d installed package names",
                       len(self.results))

mvt/android/modules/adb/processes.py

@@ -13,13 +13,14 @@ log = logging.getLogger(__name__)
 class Processes(AndroidExtraction):
     """This module extracts details on running processes."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -29,7 +30,7 @@ class Processes(AndroidExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
 
         output = self._adb_command("ps -e")

mvt/android/modules/adb/root_binaries.py

@@ -13,13 +13,14 @@ log = logging.getLogger(__name__)
 class RootBinaries(AndroidExtraction):
     """This module extracts the list of installed packages."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def run(self):
+    def run(self) -> None:
         root_binaries = [
             "su",
             "busybox",

mvt/android/modules/adb/selinux_status.py

@@ -4,9 +4,6 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import os
-
-import pkg_resources
 
 from .base import AndroidExtraction
 
@@ -18,15 +15,16 @@ class SELinuxStatus(AndroidExtraction):
 
     slug = "selinux_status"
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = {} if not results else results
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("getenforce")
         self._adb_disconnect()

mvt/android/modules/adb/settings.py

@@ -57,15 +57,16 @@ ANDROID_DANGEROUS_SETTINGS = [
 class Settings(AndroidExtraction):
     """This module extracts Android system settings."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = {} if not results else results
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         for namespace, settings in self.results.items():
             for key, value in settings.items():
                 for danger in ANDROID_DANGEROUS_SETTINGS:
@@ -76,7 +77,7 @@ class Settings(AndroidExtraction):
                                          key, value, danger["description"])
                         break
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
 
         for namespace in ["system", "secure", "global"]:

mvt/android/modules/adb/sms.py

@@ -3,8 +3,6 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
-import base64
-import getpass
 import logging
 import os
 import sqlite3
@@ -48,13 +46,14 @@ FROM sms;
 class SMS(AndroidExtraction):
     """This module extracts all SMS messages containing links."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         body = record["body"].replace("\n", "\\n")
         return {
             "timestamp": record["isodate"],
@@ -63,7 +62,7 @@ class SMS(AndroidExtraction):
             "data": f"{record['address']}: \"{body}\""
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -76,7 +75,7 @@ class SMS(AndroidExtraction):
             if self.indicators.check_domains(message_links):
                 self.detected.append(message)
 
-    def _parse_db(self, db_path):
+    def _parse_db(self, db_path: str) -> None:
         """Parse an Android bugle_db SMS database file.
 
         :param db_path: Path to the Android SMS database file to process
@@ -110,7 +109,7 @@ class SMS(AndroidExtraction):
 
         log.info("Extracted a total of %d SMS messages containing links", len(self.results))
 
-    def _extract_sms_adb(self):
+    def _extract_sms_adb(self) -> None:
         """Use the Android backup command to extract SMS data from the native SMS app
 
         It is crucial to use the under-documented "-nocompress" flag to disable the non-standard Java compression
@@ -128,7 +127,7 @@ class SMS(AndroidExtraction):
 
         log.info("Extracted a total of %d SMS messages containing links", len(self.results))
 
-    def run(self):
+    def run(self) -> None:
         try:
             if (self._adb_check_file_exists(os.path.join("/", SMS_BUGLE_PATH))):
                 self.SMS_DB_TYPE = 1

mvt/android/modules/adb/whatsapp.py

@@ -20,13 +20,14 @@ WHATSAPP_PATH = "data/data/com.whatsapp/databases/msgstore.db"
 class Whatsapp(AndroidExtraction):
     """This module extracts all WhatsApp messages containing links."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         text = record["data"].replace("\n", "\\n")
         return {
             "timestamp": record["isodate"],
@@ -35,7 +36,7 @@ class Whatsapp(AndroidExtraction):
             "data": f"\"{text}\""
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -47,7 +48,7 @@ class Whatsapp(AndroidExtraction):
             if self.indicators.check_domains(message_links):
                 self.detected.append(message)
 
-    def _parse_db(self, db_path):
+    def _parse_db(self, db_path: str) -> None:
         """Parse an Android msgstore.db WhatsApp database file.
 
         :param db_path: Path to the Android WhatsApp database file to process
@@ -84,7 +85,7 @@ class Whatsapp(AndroidExtraction):
         log.info("Extracted a total of %d WhatsApp messages containing links", len(messages))
         self.results = messages
 
-    def run(self):
+    def run(self) -> None:
         try:
             self._adb_process_file(os.path.join("/", WHATSAPP_PATH), self._parse_db)
         except Exception as e:

mvt/android/modules/backup/base.py

@@ -5,6 +5,7 @@
 
 import fnmatch
 import os
+from tarfile import TarFile
 
 from mvt.common.module import MVTModule
 
@@ -13,14 +14,14 @@ class BackupExtraction(MVTModule):
     """This class provides a base for all backup extractios modules"""
     ab = None
 
-    def from_folder(self, backup_path, files):
+    def from_folder(self, backup_path: str, files: list) -> None:
         """
         Get all the files and list them
         """
         self.backup_path = backup_path
         self.files = files
 
-    def from_ab(self, file_path, tar, files):
+    def from_ab(self, file_path: str, tar: TarFile, files: list) -> None:
         """
         Extract the files
         """
@@ -28,10 +29,10 @@ class BackupExtraction(MVTModule):
         self.tar = tar
         self.files = files
 
-    def _get_files_by_pattern(self, pattern):
+    def _get_files_by_pattern(self, pattern: str) -> list:
         return fnmatch.filter(self.files, pattern)
 
-    def _get_file_content(self, file_path):
+    def _get_file_content(self, file_path: str) -> bytes:
         if self.ab:
             try:
                 member = self.tar.getmember(file_path)
@@ -43,4 +44,5 @@ class BackupExtraction(MVTModule):
 
         data = handle.read()
         handle.close()
+
         return data

mvt/android/modules/backup/sms.py

@@ -3,20 +3,22 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
+
 from mvt.android.modules.backup.base import BackupExtraction
 from mvt.android.parsers.backup import parse_sms_file
-from mvt.common.utils import check_for_links
 
 
 class SMS(BackupExtraction):
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
         self.results = []
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -27,10 +29,16 @@ class SMS(BackupExtraction):
             if self.indicators.check_domains(message["links"]):
                 self.detected.append(message)
 
-    def run(self):
+    def run(self) -> None:
         for file in self._get_files_by_pattern("apps/com.android.providers.telephony/d_f/*_sms_backup"):
             self.log.info("Processing SMS backup file at %s", file)
             data = self._get_file_content(file)
             self.results.extend(parse_sms_file(data))
-        self.log.info("Extracted a total of %d SMS messages containing links",
+
+        for file in self._get_files_by_pattern("apps/com.android.providers.telephony/d_f/*_mms_backup"):
+            self.log.info("Processing MMS backup file at %s", file)
+            data = self._get_file_content(file)
+            self.results.extend(parse_sms_file(data))
+
+        self.log.info("Extracted a total of %d SMS & MMS messages containing links",
                       len(self.results))

mvt/android/modules/bugreport/accessibility.py

@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class Accessibility(BugReportModule):
     """This module extracts stats on accessibility."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -32,7 +33,7 @@ class Accessibility(BugReportModule):
                 self.detected.append(result)
                 continue
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/activities.py

@@ -15,15 +15,16 @@ log = logging.getLogger(__name__)
 class Activities(BugReportModule):
     """This module extracts details on receivers for risky activities."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = results if results else {}
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -35,7 +36,7 @@ class Activities(BugReportModule):
                     self.detected.append({intent: activity})
                     continue
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/appops.py

@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class Appops(BugReportModule):
     """This module extracts information on package from App-Ops Manager."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
         for perm in record["permissions"]:
             if "entries" not in perm:
@@ -33,12 +34,12 @@ class Appops(BugReportModule):
                         "timestamp": entry["timestamp"],
                         "module": self.__class__.__name__,
                         "event": entry["access"],
-                        "data": f"{record['package_name']} access to {perm['name']} : {entry['access']}",
+                        "data": f"{record['package_name']} access to {perm['name']}: {entry['access']}",
                     })
 
         return records
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         for result in self.results:
             if self.indicators:
                 ioc = self.indicators.check_app_id(result.get("package_name"))
@@ -51,7 +52,7 @@ class Appops(BugReportModule):
                 if perm["name"] == "REQUEST_INSTALL_PACKAGES" and perm["access"] == "allow":
                     self.log.info("Package %s with REQUEST_INSTALL_PACKAGES permission", result["package_name"])
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/base.py

@@ -6,6 +6,7 @@
 import fnmatch
 import logging
 import os
+from zipfile import ZipFile
 
 from mvt.common.module import MVTModule
 
@@ -17,15 +18,15 @@ class BugReportModule(MVTModule):
 
     zip_archive = None
 
-    def from_folder(self, extract_path, extract_files):
+    def from_folder(self, extract_path: str, extract_files: str) -> None:
         self.extract_path = extract_path
         self.extract_files = extract_files
 
-    def from_zip(self, zip_archive, zip_files):
+    def from_zip(self, zip_archive: ZipFile, zip_files: list) -> None:
         self.zip_archive = zip_archive
         self.zip_files = zip_files
 
-    def _get_files_by_pattern(self, pattern):
+    def _get_files_by_pattern(self, pattern: str) -> list:
         file_names = []
         if self.zip_archive:
             for zip_file in self.zip_files:
@@ -35,13 +36,13 @@ class BugReportModule(MVTModule):
 
         return fnmatch.filter(file_names, pattern)
 
-    def _get_files_by_patterns(self, patterns):
+    def _get_files_by_patterns(self, patterns: list) -> list:
         for pattern in patterns:
             matches = self._get_files_by_pattern(pattern)
             if matches:
                 return matches
 
-    def _get_file_content(self, file_path):
+    def _get_file_content(self, file_path: str) -> bytes:
         if self.zip_archive:
             handle = self.zip_archive.open(file_path)
         else:
@@ -52,7 +53,7 @@ class BugReportModule(MVTModule):
 
         return data
 
-    def _get_dumpstate_file(self):
+    def _get_dumpstate_file(self) -> bytes:
         main = self._get_files_by_pattern("main_entry.txt")
         if main:
             main_content = self._get_file_content(main[0])

mvt/android/modules/bugreport/battery_daily.py

@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class BatteryDaily(BugReportModule):
     """This module extracts records from battery daily updates."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["from"],
             "module": self.__class__.__name__,
@@ -29,7 +30,7 @@ class BatteryDaily(BugReportModule):
             "data": f"Recorded update of package {record['package_name']} with vers {record['vers']}"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -40,7 +41,7 @@ class BatteryDaily(BugReportModule):
                 self.detected.append(result)
                 continue
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/battery_history.py

@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class BatteryHistory(BugReportModule):
     """This module extracts records from battery daily updates."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -32,7 +33,7 @@ class BatteryHistory(BugReportModule):
                 self.detected.append(result)
                 continue
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/dbinfo.py

@@ -17,13 +17,14 @@ class DBInfo(BugReportModule):
 
     slug = "dbinfo"
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -36,7 +37,7 @@ class DBInfo(BugReportModule):
                     self.detected.append(result)
                     continue
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/getprop.py

@@ -4,7 +4,6 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import re
 from datetime import datetime, timedelta
 
 from mvt.android.parsers import parse_getprop
@@ -17,15 +16,16 @@ log = logging.getLogger(__name__)
 class Getprop(BugReportModule):
     """This module extracts device properties from getprop command."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = {} if not results else results
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/packages.py

@@ -14,13 +14,14 @@ log = logging.getLogger(__name__)
 class Packages(BugReportModule):
     """This module extracts details on receivers for risky activities."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
 
         timestamps = [
@@ -39,7 +40,7 @@ class Packages(BugReportModule):
 
         return records
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -51,7 +52,7 @@ class Packages(BugReportModule):
                 continue
 
     @staticmethod
-    def parse_package_for_details(output):
+    def parse_package_for_details(output: str) -> dict:
         details = {
             "uid": "",
             "version_name": "",
@@ -102,7 +103,7 @@ class Packages(BugReportModule):
 
         return details
 
-    def parse_packages_list(self, output):
+    def parse_packages_list(self, output: str) -> list:
         pkg_rxp = re.compile(r"  Package \[(.+?)\].*")
 
         results = []
@@ -133,7 +134,7 @@ class Packages(BugReportModule):
 
         return results
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/modules/bugreport/receivers.py

@@ -21,15 +21,16 @@ INTENT_NEW_OUTGOING_CALL = "android.intent.action.NEW_OUTGOING_CALL"
 class Receivers(BugReportModule):
     """This module extracts details on receivers for risky activities."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = results if results else {}
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -57,7 +58,7 @@ class Receivers(BugReportModule):
                 self.detected.append({intent: receiver})
                 continue
 
-    def run(self):
+    def run(self) -> None:
         content = self._get_dumpstate_file()
         if not content:
             self.log.error("Unable to find dumpstate file. Did you provide a valid bug report archive?")

mvt/android/parsers/backup.py

@@ -70,11 +70,11 @@ def decrypt_master_key(password, user_salt, user_iv, pbkdf2_rounds, master_key_b
 
     The backup master key is extracted from the master key blog after decryption.
     """
-    # Derive key from password using PBKDF2
+    # Derive key from password using PBKDF2.
     kdf = PBKDF2HMAC(algorithm=hashes.SHA1(), length=32, salt=user_salt, iterations=pbkdf2_rounds)
     key = kdf.derive(password.encode("utf-8"))
 
-    # Decrypt master key blob
+    # Decrypt master key blob.
     cipher = Cipher(algorithms.AES(key), modes.CBC(user_iv))
     decryptor = cipher.decryptor()
     try:
@@ -93,7 +93,7 @@ def decrypt_master_key(password, user_salt, user_iv, pbkdf2_rounds, master_key_b
     except TypeError:
         raise InvalidBackupPassword()
 
-    # Handle quirky encoding of master key bytes in Android original Java crypto code
+    # Handle quirky encoding of master key bytes in Android original Java crypto code.
     if format_version > 1:
         hmac_mk = to_utf8_bytes(master_key)
     else:
@@ -112,6 +112,7 @@ def decrypt_master_key(password, user_salt, user_iv, pbkdf2_rounds, master_key_b
 def decrypt_backup_data(encrypted_backup, password, encryption_algo, format_version):
     """
     Generate encryption keyffrom password and do decryption
+
     """
     if encryption_algo != b"AES-256":
         raise AndroidBackupNotImplemented("Encryption Algorithm not implemented")
@@ -126,12 +127,12 @@ def decrypt_backup_data(encrypted_backup, password, encryption_algo, format_vers
     user_iv = bytes.fromhex(user_iv.decode("utf-8"))
     master_key_blob = bytes.fromhex(master_key_blob.decode("utf-8"))
 
-    # Derive decryption master key from password
+    # Derive decryption master key from password.
     master_key, master_iv = decrypt_master_key(password=password, user_salt=user_salt, user_iv=user_iv,
                                                pbkdf2_rounds=pbkdf2_rounds, master_key_blob=master_key_blob,
                                                format_version=format_version, checksum_salt=checksum_salt)
 
-    # Decrypt and unpad backup data using derivied key
+    # Decrypt and unpad backup data using derivied key.
     cipher = Cipher(algorithms.AES(master_key), modes.CBC(master_iv))
     decryptor = cipher.decryptor()
     decrypted_tar = decryptor.update(encrypted_data) + decryptor.finalize()
@@ -143,6 +144,7 @@ def decrypt_backup_data(encrypted_backup, password, encryption_algo, format_vers
 def parse_backup_file(data, password=None):
     """
     Parse an ab file, returns a tar file
+
     """
     if not data.startswith(b"ANDROID BACKUP"):
         raise AndroidBackupParsingError("Invalid file header")
@@ -170,26 +172,33 @@ def parse_tar_for_sms(data):
     """
     dbytes = io.BytesIO(data)
     tar = tarfile.open(fileobj=dbytes)
-    try:
-        member = tar.getmember("apps/com.android.providers.telephony/d_f/000000_sms_backup")
-    except KeyError:
-        return []
+    res = []
+    for member in tar.getmembers():
+        if member.name.startswith("apps/com.android.providers.telephony/d_f/") and \
+                (member.name.endswith("_sms_backup") or member.name.endswith("_mms_backup")):
+            dhandler = tar.extractfile(member)
+            res.extend(parse_sms_file(dhandler.read()))
 
-    dhandler = tar.extractfile(member)
-    return parse_sms_file(dhandler.read())
+    return res
 
 
 def parse_sms_file(data):
     """
-    Parse an SMS file extracted from a folder
-    Returns a list of SMS entries
+    Parse an SMS or MMS file extracted from a backup
+    Returns a list of SMS or MMS entries
     """
     res = []
     data = zlib.decompress(data)
     json_data = json.loads(data)
 
     for entry in json_data:
+        # Adapt MMS format to SMS format
+        if "mms_body" in entry:
+            entry["body"] = entry["mms_body"]
+            entry.pop("mms_body")
+
         message_links = check_for_links(entry["body"])
+
         utc_timestamp = datetime.datetime.utcfromtimestamp(int(entry["date"]) / 1000)
         entry["isodate"] = convert_timestamp_to_iso(utc_timestamp)
         entry["direction"] = ("sent" if int(entry["date_sent"]) else "received")

mvt/android/parsers/dumpsys.py

@@ -9,7 +9,7 @@ from datetime import datetime
 from mvt.common.utils import convert_timestamp_to_iso
 
 
-def parse_dumpsys_accessibility(output):
+def parse_dumpsys_accessibility(output: str) -> list:
     results = []
 
     in_services = False
@@ -34,7 +34,7 @@ def parse_dumpsys_accessibility(output):
     return results
 
 
-def parse_dumpsys_activity_resolver_table(output):
+def parse_dumpsys_activity_resolver_table(output: str) -> dict:
     results = {}
 
     in_activity_resolver_table = False
@@ -90,7 +90,7 @@ def parse_dumpsys_activity_resolver_table(output):
     return results
 
 
-def parse_dumpsys_battery_daily(output):
+def parse_dumpsys_battery_daily(output: str) -> list:
     results = []
     daily = None
     daily_updates = []
@@ -136,7 +136,7 @@ def parse_dumpsys_battery_daily(output):
     return results
 
 
-def parse_dumpsys_battery_history(output):
+def parse_dumpsys_battery_history(output: str) -> list:
     results = []
 
     for line in output.splitlines():
@@ -181,7 +181,7 @@ def parse_dumpsys_battery_history(output):
     return results
 
 
-def parse_dumpsys_dbinfo(output):
+def parse_dumpsys_dbinfo(output: str) -> list:
     results = []
 
     rxp = re.compile(r'.*\[([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3})\].*\[Pid:\((\d+)\)\](\w+).*sql\=\"(.+?)\"')
@@ -234,7 +234,7 @@ def parse_dumpsys_dbinfo(output):
     return results
 
 
-def parse_dumpsys_receiver_resolver_table(output):
+def parse_dumpsys_receiver_resolver_table(output: str) -> dict:
     results = {}
 
     in_receiver_resolver_table = False
@@ -290,7 +290,7 @@ def parse_dumpsys_receiver_resolver_table(output):
     return results
 
 
-def parse_dumpsys_appops(output):
+def parse_dumpsys_appops(output: str) -> list:
     results = []
     perm = {}
     package = {}

mvt/android/parsers/getprop.py

@@ -6,7 +6,7 @@
 import re
 
 
-def parse_getprop(output):
+def parse_getprop(output: str) -> dict:
     results = {}
     rxp = re.compile(r"\[(.+?)\]: \[(.+?)\]")
 

mvt/common/cmd_check_iocs.py

@@ -0,0 +1,64 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+
+from mvt.common.command import Command
+
+log = logging.getLogger(__name__)
+
+
+class CmdCheckIOCS(Command):
+
+    name = "check-iocs"
+    modules = []
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+
+    def run(self) -> None:
+        all_modules = []
+        for entry in self.modules:
+            if entry not in all_modules:
+                all_modules.append(entry)
+
+        log.info("Checking stored results against provided indicators...")
+
+        total_detections = 0
+        for file_name in os.listdir(self.target_path):
+            name_only, ext = os.path.splitext(file_name)
+            file_path = os.path.join(self.target_path, file_name)
+
+            for iocs_module in all_modules:
+                if self.module_name and iocs_module.__name__ != self.module_name:
+                    continue
+
+                if iocs_module().get_slug() != name_only:
+                    continue
+
+                log.info("Loading results from \"%s\" with module %s", file_name,
+                         iocs_module.__name__)
+
+                m = iocs_module.from_json(file_path,
+                                          log=logging.getLogger(iocs_module.__module__))
+                if self.iocs.total_ioc_count > 0:
+                    m.indicators = self.iocs
+                    m.indicators.log = m.log
+
+                try:
+                    m.check_indicators()
+                except NotImplementedError:
+                    continue
+                else:
+                    total_detections += len(m.detected)
+
+        if total_detections > 0:
+            log.warning("The check of the results produced %d detections!",
+                        total_detections)

mvt/common/command.py

@@ -0,0 +1,178 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import hashlib
+import json
+import logging
+import os
+import sys
+from datetime import datetime
+from typing import Callable
+
+from mvt.common.indicators import Indicators
+from mvt.common.module import run_module, save_timeline
+from mvt.common.utils import convert_timestamp_to_iso
+from mvt.common.version import MVT_VERSION
+
+
+class Command(object):
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False,
+                 log: logging.Logger = logging.getLogger(__name__)):
+        self.name = ""
+
+        self.target_path = target_path
+        self.results_path = results_path
+        self.ioc_files = ioc_files
+        self.module_name = module_name
+        self.serial = serial
+        self.fast_mode = fast_mode
+        self.log = log
+
+        self.iocs = Indicators(log=log)
+        self.iocs.load_indicators_files(ioc_files)
+
+        self.timeline = []
+        self.timeline_detected = []
+
+    def _create_storage(self) -> None:
+        if self.results_path and not os.path.exists(self.results_path):
+            try:
+                os.makedirs(self.results_path)
+            except Exception as e:
+                self.log.critical("Unable to create output folder %s: %s",
+                                  self.results_path, e)
+                sys.exit(1)
+
+    def _add_log_file_handler(self, logger: logging.Logger) -> None:
+        if not self.results_path:
+            return
+
+        fh = logging.FileHandler(os.path.join(self.results_path, "command.log"))
+        formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+        fh.setLevel(logging.DEBUG)
+        fh.setFormatter(formatter)
+        logger.addHandler(fh)
+
+    def _store_timeline(self) -> None:
+        if not self.results_path:
+            return
+
+        if len(self.timeline) > 0:
+            save_timeline(self.timeline,
+                          os.path.join(self.results_path, "timeline.csv"))
+
+        if len(self.timeline_detected) > 0:
+            save_timeline(self.timeline_detected,
+                          os.path.join(self.results_path, "timeline_detected.csv"))
+
+    def _store_info(self) -> None:
+        if not self.results_path:
+            return
+
+        target_path = None
+        if self.target_path:
+            target_path = os.path.abspath(self.target_path)
+
+        info = {
+            "target_path": target_path,
+            "mvt_version": MVT_VERSION,
+            "date": convert_timestamp_to_iso(datetime.now()),
+            "ioc_files": [],
+            "hashes": [],
+        }
+
+        for coll in self.iocs.ioc_collections:
+            info["ioc_files"].append(coll.get("stix2_file_path", ""))
+
+        # TODO: Revisit if setting this from environment variable is good
+        #       enough.
+        if self.target_path and os.environ.get("MVT_HASH_FILES"):
+            if os.path.isfile(self.target_path):
+                h = hashlib.sha256()
+                with open(self.target_path, "rb") as handle:
+                    h.update(handle.read())
+
+                info["hashes"].append({
+                    "file_path": self.target_path,
+                    "sha256": h.hexdigest(),
+                })
+            elif os.path.isdir(self.target_path):
+                for (root, dirs, files) in os.walk(self.target_path):
+                    for file in files:
+                        file_path = os.path.join(root, file)
+                        h = hashlib.sha256()
+
+                        try:
+                            with open(file_path, "rb") as handle:
+                                h.update(handle.read())
+                        except FileNotFoundError:
+                            self.log.error("Failed to hash the file %s: might be a symlink", file_path)
+                            continue
+                        except PermissionError:
+                            self.log.error("Failed to hash the file %s: permission denied", file_path)
+                            continue
+
+                        info["hashes"].append({
+                            "file_path": file_path,
+                            "sha256": h.hexdigest(),
+                        })
+
+        with open(os.path.join(self.results_path, "info.json"), "w+") as handle:
+            json.dump(info, handle, indent=4)
+
+    def list_modules(self) -> None:
+        self.log.info("Following is the list of available %s modules:", self.name)
+        for module in self.modules:
+            self.log.info(" - %s", module.__name__)
+
+    def init(self) -> None:
+        raise NotImplementedError
+
+    def module_init(self, module: Callable) -> None:
+        raise NotImplementedError
+
+    def run(self) -> None:
+        self._create_storage()
+        self._add_log_file_handler(self.log)
+
+        try:
+            self.init()
+        except NotImplementedError:
+            pass
+
+        for module in self.modules:
+            if self.module_name and module.__name__ != self.module_name:
+                continue
+
+            module_logger = logging.getLogger(module.__module__)
+            self._add_log_file_handler(module_logger)
+
+            m = module(target_path=self.target_path,
+                       results_path=self.results_path,
+                       fast_mode=self.fast_mode,
+                       log=module_logger)
+
+            if self.iocs.total_ioc_count:
+                m.indicators = self.iocs
+                m.indicators.log = m.log
+
+            if self.serial:
+                m.serial = self.serial
+
+            try:
+                self.module_init(m)
+            except NotImplementedError:
+                pass
+
+            run_module(m)
+
+            self.timeline.extend(m.timeline)
+            self.timeline_detected.extend(m.timeline_detected)
+
+        self._store_timeline()
+        self._store_info()

mvt/common/indicators.py

@@ -4,34 +4,36 @@
 #   https://license.mvt.re/1.1/
 
 import json
+import logging
 import os
 
-import requests
 from appdirs import user_data_dir
 
 from .url import URL
 
+MVT_DATA_FOLDER = user_data_dir("mvt")
+MVT_INDICATORS_FOLDER = os.path.join(MVT_DATA_FOLDER, "indicators")
+
 
 class Indicators:
     """This class is used to parse indicators from a STIX2 file and provide
     functions to compare extracted artifacts to the indicators.
     """
 
-    def __init__(self, log=None):
-        self.data_dir = user_data_dir("mvt")
+    def __init__(self, log=logging.Logger) -> None:
         self.log = log
         self.ioc_collections = []
         self.total_ioc_count = 0
 
-    def _load_downloaded_indicators(self):
-        if not os.path.isdir(self.data_dir):
+    def _load_downloaded_indicators(self) -> None:
+        if not os.path.isdir(MVT_INDICATORS_FOLDER):
             return
 
-        for f in os.listdir(self.data_dir):
-            if f.lower().endswith(".stix2"):
-                self.parse_stix2(os.path.join(self.data_dir, f))
+        for ioc_file_name in os.listdir(MVT_INDICATORS_FOLDER):
+            if ioc_file_name.lower().endswith(".stix2"):
+                self.parse_stix2(os.path.join(MVT_INDICATORS_FOLDER, ioc_file_name))
 
-    def _check_stix2_env_variable(self):
+    def _check_stix2_env_variable(self) -> None:
         """
         Checks if a variable MVT_STIX2 contains path to a STIX files.
         """
@@ -46,8 +48,8 @@ class Indicators:
                 self.log.error("Path specified with env MVT_STIX2 is not a valid file: %s",
                                path)
 
-    def _new_collection(self, cid="", name="", description="", file_name="",
-                        file_path=""):
+    def _new_collection(self, cid: str = "", name: str = "", description: str = "",
+                        file_name: str = "", file_path: str = "") -> dict:
         return {
             "id": cid,
             "name": name,
@@ -65,14 +67,14 @@ class Indicators:
             "count": 0,
         }
 
-    def _add_indicator(self, ioc, ioc_coll, ioc_coll_list):
+    def _add_indicator(self, ioc: str, ioc_coll: dict, ioc_coll_list: list) -> None:
         ioc = ioc.strip("'")
         if ioc not in ioc_coll_list:
             ioc_coll_list.append(ioc)
             ioc_coll["count"] += 1
             self.total_ioc_count += 1
 
-    def parse_stix2(self, file_path):
+    def parse_stix2(self, file_path: str) -> None:
         """Extract indicators from a STIX2 file.
 
         :param file_path: Path to the STIX2 file to parse
@@ -97,7 +99,7 @@ class Indicators:
             if entry_type == "malware":
                 malware[entry["id"]] = {
                     "name": entry["name"],
-                    "description": entry["description"],
+                    "description": entry.get("description", ""),
                 }
             elif entry_type == "indicator":
                 indicators.append(entry)
@@ -178,7 +180,7 @@ class Indicators:
 
         self.ioc_collections.extend(collections)
 
-    def load_indicators_files(self, files, load_default=True):
+    def load_indicators_files(self, files: list, load_default: bool = True) -> None:
         """
         Load a list of indicators files.
         """
@@ -196,7 +198,7 @@ class Indicators:
         self._check_stix2_env_variable()
         self.log.info("Loaded a total of %d unique indicators", self.total_ioc_count)
 
-    def get_iocs(self, ioc_type):
+    def get_iocs(self, ioc_type: str) -> dict:
         for ioc_collection in self.ioc_collections:
             for ioc in ioc_collection.get(ioc_type, []):
                 yield {
@@ -206,7 +208,7 @@ class Indicators:
                     "stix2_file_name": ioc_collection["stix2_file_name"],
                 }
 
-    def check_domain(self, url):
+    def check_domain(self, url: str) -> dict:
         """Check if a given URL matches any of the provided domain indicators.
 
         :param url: URL to match against domain indicators
@@ -278,7 +280,7 @@ class Indicators:
 
                 return ioc
 
-    def check_domains(self, urls):
+    def check_domains(self, urls: list) -> dict:
         """Check a list of URLs against the provided list of domain indicators.
 
         :param urls: List of URLs to check against domain indicators
@@ -294,7 +296,7 @@ class Indicators:
             if check:
                 return check
 
-    def check_process(self, process):
+    def check_process(self, process: str) -> dict:
         """Check the provided process name against the list of process
         indicators.
 
@@ -319,7 +321,7 @@ class Indicators:
                                      process, ioc["name"])
                     return ioc
 
-    def check_processes(self, processes):
+    def check_processes(self, processes: list) -> dict:
         """Check the provided list of processes against the list of
         process indicators.
 
@@ -336,7 +338,7 @@ class Indicators:
             if check:
                 return check
 
-    def check_email(self, email):
+    def check_email(self, email: str) -> dict:
         """Check the provided email against the list of email indicators.
 
         :param email: Email address to check against email indicators
@@ -353,7 +355,7 @@ class Indicators:
                                  email, ioc["name"])
                 return ioc
 
-    def check_file_name(self, file_name):
+    def check_file_name(self, file_name: str) -> dict:
         """Check the provided file name against the list of file indicators.
 
         :param file_name: File name to check against file
@@ -371,7 +373,7 @@ class Indicators:
                                  file_name, ioc["name"])
                 return ioc
 
-    def check_file_path(self, file_path):
+    def check_file_path(self, file_path: str) -> dict:
         """Check the provided file path against the list of file indicators (both path and name).
 
         :param file_path: File path or file name to check against file
@@ -394,7 +396,7 @@ class Indicators:
                                  file_path, ioc["name"])
                 return ioc
 
-    def check_profile(self, profile_uuid):
+    def check_profile(self, profile_uuid: str) -> dict:
         """Check the provided configuration profile UUID against the list of indicators.
 
         :param profile_uuid: Profile UUID to check against configuration profile indicators
@@ -411,7 +413,7 @@ class Indicators:
                                  profile_uuid, ioc["name"])
                 return ioc
 
-    def check_file_hash(self, file_hash):
+    def check_file_hash(self, file_hash: str) -> dict:
         """Check the provided SHA256 file hash against the list of indicators.
 
         :param file_hash: SHA256 hash to check
@@ -428,7 +430,7 @@ class Indicators:
                                  file_hash, ioc["name"])
                 return ioc
 
-    def check_app_id(self, app_id):
+    def check_app_id(self, app_id: str) -> dict:
         """Check the provided app identifier (typically an Android package name)
         against the list of indicators.
 
@@ -445,36 +447,3 @@ class Indicators:
                 self.log.warning("Found a known suspicious app with ID \"%s\" matching indicators from \"%s\"",
                                  app_id, ioc["name"])
                 return ioc
-
-
-def download_indicators_files(log):
-    """
-    Download indicators from repo into MVT app data directory.
-    """
-    data_dir = user_data_dir("mvt")
-    if not os.path.isdir(data_dir):
-        os.makedirs(data_dir, exist_ok=True)
-
-    # Download latest list of indicators from the MVT repo.
-    res = requests.get("https://github.com/mvt-project/mvt/raw/main/public_indicators.json")
-    if res.status_code != 200:
-        log.warning("Unable to find retrieve list of indicators from the MVT repository.")
-        return
-
-    for ioc_entry in res.json():
-        ioc_url = ioc_entry["stix2_url"]
-        log.info("Downloading indicator file %s from %s", ioc_entry["name"], ioc_url)
-
-        res = requests.get(ioc_url)
-        if res.status_code != 200:
-            log.warning("Could not find indicator file %s", ioc_url)
-            continue
-
-        clean_file_name = ioc_url.lstrip("https://").replace("/", "_")
-        ioc_path = os.path.join(data_dir, clean_file_name)
-
-        # Write file to disk. This will overwrite any older version of the STIX2 file.
-        with open(ioc_path, "w", encoding="utf-8") as handle:
-            handle.write(res.text)
-
-        log.info("Saved indicator file to %s", os.path.basename(ioc_path))

mvt/common/logo.py

@@ -5,22 +5,55 @@
 
 from rich import print
 
-from .updates import check_for_updates
+from .updates import IndicatorsUpdates, MVTUpdates
 from .version import MVT_VERSION
 
 
-def logo():
-    print("\n")
-    print("\t[bold]MVT[/bold] - Mobile Verification Toolkit")
-    print("\t\thttps://mvt.re")
-    print(f"\t\tVersion: {MVT_VERSION}")
-
+def check_updates() -> None:
+    # First we check for MVT version udpates.
+    mvt_updates = MVTUpdates()
     try:
-        latest_version = check_for_updates()
+        latest_version = mvt_updates.check()
     except Exception:
         pass
     else:
         if latest_version:
             print(f"\t\t[bold]Version {latest_version} is available! Upgrade mvt![/bold]")
 
+    # Then we check for indicators files updates.
+    ioc_updates = IndicatorsUpdates()
+
+    # Before proceeding, we check if we have downloaded an indicators index.
+    # If not, there's no point in proceeding with the updates check.
+    if ioc_updates.get_latest_update() == 0:
+        print("\t\t[bold]You have not yet downloaded any indicators, check the `download-iocs` command![/bold]")
+        return
+
+    # We only perform this check at a fixed frequency, in order to not
+    # overburden the user with too many lookups if the command is being run
+    # multiple times.
+    should_check, hours = ioc_updates.should_check()
+    if not should_check:
+        print(f"\t\tIndicators updates checked recently, next automatic check in {int(hours)} hours")
+        return
+
+    try:
+        ioc_to_update = ioc_updates.check()
+    except Exception:
+        pass
+    else:
+        if ioc_to_update:
+            print("\t\t[bold]There are updates to your indicators files! Run the `download-iocs` command to update![/bold]")
+        else:
+            print("\t\tYour indicators files seem to be up to date.")
+
+
+def logo() -> None:
+    print("\n")
+    print("\t[bold]MVT[/bold] - Mobile Verification Toolkit")
+    print("\t\thttps://mvt.re")
+    print(f"\t\tVersion: {MVT_VERSION}")
+
+    check_updates()
+
     print("\n")

mvt/common/module.py

@@ -4,8 +4,10 @@
 #   https://license.mvt.re/1.1/
 
 import csv
+import logging
 import os
 import re
+from typing import Callable
 
 import simplejson as json
 
@@ -28,16 +30,17 @@ class MVTModule(object):
     enabled = True
     slug = None
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=None):
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = None):
         """Initialize module.
 
         :param file_path: Path to the module's database file, if there is any
         :type file_path: str
-        :param base_folder: Path to the base folder (backup or filesystem dump)
+        :param target_path: Path to the target folder (backup or filesystem dump)
         :type file_path: str
-        :param output_folder: Folder where results will be stored
-        :type output_folder: str
+        :param results_path: Folder where results will be stored
+        :type results_path: str
         :param fast_mode: Flag to enable or disable slow modules
         :type fast_mode: bool
         :param log: Handle to logger
@@ -45,8 +48,8 @@ class MVTModule(object):
         :type results: list
         """
         self.file_path = file_path
-        self.base_folder = base_folder
-        self.output_folder = output_folder
+        self.target_path = target_path
+        self.results_path = results_path
         self.fast_mode = fast_mode
         self.log = log
         self.indicators = None
@@ -56,7 +59,7 @@ class MVTModule(object):
         self.timeline_detected = []
 
     @classmethod
-    def from_json(cls, json_path, log=None):
+    def from_json(cls, json_path: str, log: logging.Logger = None):
         with open(json_path, "r", encoding="utf-8") as handle:
             results = json.load(handle)
             if log:
@@ -64,7 +67,7 @@ class MVTModule(object):
                          len(results), json_path)
             return cls(results=results, log=log)
 
-    def get_slug(self):
+    def get_slug(self) -> str:
         """Use the module's class name to retrieve a slug"""
         if self.slug:
             return self.slug
@@ -72,7 +75,7 @@ class MVTModule(object):
         sub = re.sub("(.)([A-Z][a-z]+)", r"\1_\2", self.__class__.__name__)
         return re.sub("([a-z0-9])([A-Z])", r"\1_\2", sub).lower()
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         """Check the results of this module against a provided list of
         indicators.
 
@@ -80,16 +83,16 @@ class MVTModule(object):
         """
         raise NotImplementedError
 
-    def save_to_json(self):
+    def save_to_json(self) -> None:
         """Save the collected results to a json file."""
-        if not self.output_folder:
+        if not self.results_path:
             return
 
         name = self.get_slug()
 
         if self.results:
             results_file_name = f"{name}.json"
-            results_json_path = os.path.join(self.output_folder, results_file_name)
+            results_json_path = os.path.join(self.results_path, results_file_name)
             with open(results_json_path, "w", encoding="utf-8") as handle:
                 try:
                     json.dump(self.results, handle, indent=4, default=str)
@@ -99,15 +102,15 @@ class MVTModule(object):
 
         if self.detected:
             detected_file_name = f"{name}_detected.json"
-            detected_json_path = os.path.join(self.output_folder, detected_file_name)
+            detected_json_path = os.path.join(self.results_path, detected_file_name)
             with open(detected_json_path, "w", encoding="utf-8") as handle:
                 json.dump(self.detected, handle, indent=4, default=str)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         raise NotImplementedError
 
     @staticmethod
-    def _deduplicate_timeline(timeline):
+    def _deduplicate_timeline(timeline: list) -> list:
         """Serialize entry as JSON to deduplicate repeated entries
 
         :param timeline: List of entries from timeline to deduplicate
@@ -118,7 +121,7 @@ class MVTModule(object):
             timeline_set.add(json.dumps(record, sort_keys=True))
         return [json.loads(record) for record in timeline_set]
 
-    def to_timeline(self):
+    def to_timeline(self) -> None:
         """Convert results into a timeline."""
         for result in self.results:
             record = self.serialize(result)
@@ -140,12 +143,12 @@ class MVTModule(object):
         self.timeline = self._deduplicate_timeline(self.timeline)
         self.timeline_detected = self._deduplicate_timeline(self.timeline_detected)
 
-    def run(self):
+    def run(self) -> None:
         """Run the main module procedure."""
         raise NotImplementedError
 
 
-def run_module(module):
+def run_module(module: Callable) -> None:
     module.log.info("Running module %s...", module.__class__.__name__)
 
     try:
@@ -184,7 +187,7 @@ def run_module(module):
         module.save_to_json()
 
 
-def save_timeline(timeline, timeline_path):
+def save_timeline(timeline: list, timeline_path: str) -> None:
     """Save the timeline in a csv file.
 
     :param timeline: List of records to order and store

mvt/common/updates.py

@@ -3,18 +3,209 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
+import os
+from datetime import datetime
+
 import requests
+import yaml
 from packaging import version
 
+from .indicators import MVT_DATA_FOLDER, MVT_INDICATORS_FOLDER
 from .version import MVT_VERSION
 
+log = logging.getLogger(__name__)
 
-def check_for_updates():
-    res = requests.get("https://pypi.org/pypi/mvt/json")
-    data = res.json()
-    latest_version = data.get("info", {}).get("version", "")
+# In hours.
+INDICATORS_CHECK_FREQUENCY = 12
 
-    if version.parse(latest_version) > version.parse(MVT_VERSION):
-        return latest_version
 
-    return None
+class MVTUpdates:
+
+    def check(self) -> str:
+        res = requests.get("https://pypi.org/pypi/mvt/json")
+        data = res.json()
+        latest_version = data.get("info", {}).get("version", "")
+
+        if version.parse(latest_version) > version.parse(MVT_VERSION):
+            return latest_version
+
+        return ""
+
+
+class IndicatorsUpdates:
+
+    def __init__(self) -> None:
+        self.github_raw_url = "https://raw.githubusercontent.com/{}/{}/{}/{}"
+
+        self.index_owner = "mvt-project"
+        self.index_repo = "mvt-indicators"
+        self.index_branch = "main"
+        self.index_path = "indicators.yaml"
+
+        self.latest_update_path = os.path.join(MVT_DATA_FOLDER,
+                                               "latest_indicators_update")
+        self.latest_check_path = os.path.join(MVT_DATA_FOLDER,
+                                              "latest_indicators_check")
+
+    def get_latest_check(self) -> int:
+        if not os.path.exists(self.latest_check_path):
+            return 0
+
+        with open(self.latest_check_path, "r") as handle:
+            data = handle.read().strip()
+            if data:
+                return int(data)
+
+        return 0
+
+    def set_latest_check(self) -> None:
+        timestamp = int(datetime.utcnow().timestamp())
+        with open(self.latest_check_path, "w") as handle:
+            handle.write(str(timestamp))
+
+    def get_latest_update(self) -> int:
+        if not os.path.exists(self.latest_update_path):
+            return 0
+
+        with open(self.latest_update_path, "r") as handle:
+            data = handle.read().strip()
+            if data:
+                return int(data)
+
+        return 0
+
+    def set_latest_update(self) -> None:
+        timestamp = int(datetime.utcnow().timestamp())
+        with open(self.latest_update_path, "w") as handle:
+            handle.write(str(timestamp))
+
+    def get_remote_index(self) -> dict:
+        url = self.github_raw_url.format(self.index_owner, self.index_repo,
+                                         self.index_branch, self.index_path)
+        res = requests.get(url)
+        if res.status_code != 200:
+            log.error("Failed to retrieve indicators index located at %s (error %d)",
+                      url, res.status_code)
+            return None
+
+        return yaml.safe_load(res.content)
+
+    def download_remote_ioc(self, ioc_url: str) -> str:
+        res = requests.get(ioc_url)
+        if res.status_code != 200:
+            log.error("Failed to download indicators file from %s (error %d)",
+                      ioc_url, res.status_code)
+            return None
+
+        clean_file_name = ioc_url.lstrip("https://").replace("/", "_")
+        ioc_path = os.path.join(MVT_INDICATORS_FOLDER, clean_file_name)
+
+        with open(ioc_path, "w", encoding="utf-8") as handle:
+            handle.write(res.text)
+
+        return ioc_path
+
+    def update(self) -> None:
+        self.set_latest_check()
+
+        if not os.path.exists(MVT_INDICATORS_FOLDER):
+            os.makedirs(MVT_INDICATORS_FOLDER)
+
+        index = self.get_remote_index()
+        for ioc in index.get("indicators", []):
+            ioc_type = ioc.get("type", "")
+
+            if ioc_type == "github":
+                github = ioc.get("github", {})
+                owner = github.get("owner", "")
+                repo = github.get("repo", "")
+                branch = github.get("branch", "main")
+                path = github.get("path", "")
+
+                ioc_url = self.github_raw_url.format(owner, repo, branch, path)
+            else:
+                ioc_url = ioc.get("download_url", "")
+
+            if not ioc_url:
+                log.error("Could not find a way to download indicator file for %s",
+                          ioc.get("name"))
+                continue
+
+            ioc_local_path = self.download_remote_ioc(ioc_url)
+            if not ioc_local_path:
+                continue
+
+            log.info("Downloaded indicators \"%s\" to %s",
+                     ioc.get("name"), ioc_local_path)
+
+        self.set_latest_update()
+
+    def _get_remote_file_latest_commit(self, owner: str, repo: str,
+                                       branch: str, path: str) -> bool:
+        file_commit_url = f"https://api.github.com/repos/{self.index_owner}/{self.index_repo}/commits?path={self.index_path}"
+        res = requests.get(file_commit_url)
+        if res.status_code != 200:
+            log.error("Failed to get details about file %s (error %d)",
+                      file_commit_url, res.status_code)
+            return False
+
+        details = res.json()
+        if len(details) == 0:
+            return False
+
+        latest_commit = details[0]
+        latest_commit_date = latest_commit.get("commit", {}).get("author", {}).get("date", None)
+        if not latest_commit_date:
+            log.error("Failed to retrieve date of latest update to indicators index file")
+            return False
+
+        latest_commit_dt = datetime.strptime(latest_commit_date, '%Y-%m-%dT%H:%M:%SZ')
+        latest_commit_ts = int(latest_commit_dt.timestamp())
+
+        return latest_commit_ts
+
+    def should_check(self) -> (bool, int):
+        now = datetime.utcnow()
+        latest_check_ts = self.get_latest_check()
+        latest_check_dt = datetime.fromtimestamp(latest_check_ts)
+
+        diff = now - latest_check_dt
+        diff_hours = divmod(diff.total_seconds(), 3600)[0]
+
+        if diff_hours >= INDICATORS_CHECK_FREQUENCY:
+            return True, 0
+
+        return False, INDICATORS_CHECK_FREQUENCY - diff_hours
+
+    def check(self) -> bool:
+        self.set_latest_check()
+
+        latest_update = self.get_latest_update()
+        latest_commit_ts = self._get_remote_file_latest_commit(self.index_owner,
+                                                               self.index_repo,
+                                                               self.index_branch,
+                                                               self.index_path)
+
+        if latest_update < latest_commit_ts:
+            return True
+
+        index = self.get_remote_index()
+        for ioc in index.get("indicators", []):
+            if ioc.get("type", "") != "github":
+                continue
+
+            github = ioc.get("github", {})
+            owner = github.get("owner", "")
+            repo = github.get("repo", "")
+            branch = github.get("branch", "main")
+            path = github.get("path", "")
+
+            file_latest_commit_ts = self._get_remote_file_latest_commit(owner,
+                                                                        repo,
+                                                                        branch,
+                                                                        path)
+            if latest_update < file_latest_commit_ts:
+                return True
+
+        return False

mvt/common/url.py

@@ -253,7 +253,7 @@ SHORTENER_DOMAINS = [
 
 class URL:
 
-    def __init__(self, url):
+    def __init__(self, url: str) -> None:
         if type(url) == bytes:
             url = url.decode()
 
@@ -262,7 +262,7 @@ class URL:
         self.top_level = self.get_top_level()
         self.is_shortened = False
 
-    def get_domain(self):
+    def get_domain(self) -> None:
         """Get the domain from a URL.
 
         :param url: URL to parse
@@ -273,11 +273,13 @@ class URL:
         """
         # TODO: Properly handle exception.
         try:
-            return get_tld(self.url, as_object=True, fix_protocol=True).parsed_url.netloc.lower().lstrip("www.")
+            return get_tld(self.url,
+                           as_object=True,
+                           fix_protocol=True).parsed_url.netloc.lower().lstrip("www.")
         except Exception:
             return None
 
-    def get_top_level(self):
+    def get_top_level(self) -> None:
         """Get only the top-level domain from a URL.
 
         :param url: URL to parse
@@ -306,7 +308,7 @@ class URL:
 
         return self.is_shortened
 
-    def unshorten(self):
+    def unshorten(self) -> None:
         """Unshorten the URL by requesting an HTTP HEAD response."""
         res = requests.head(self.url)
         if str(res.status_code).startswith("30"):

mvt/common/utils.py

@@ -8,7 +8,7 @@ import hashlib
 import re
 
 
-def convert_mactime_to_unix(timestamp, from_2001=True):
+def convert_mactime_to_unix(timestamp, from_2001: bool = True):
     """Converts Mac Standard Time to a Unix timestamp.
 
     :param timestamp: MacTime timestamp (either int or float).
@@ -37,7 +37,7 @@ def convert_mactime_to_unix(timestamp, from_2001=True):
         return None
 
 
-def convert_chrometime_to_unix(timestamp):
+def convert_chrometime_to_unix(timestamp: int) -> int:
     """Converts Chrome timestamp to a Unix timestamp.
 
     :param timestamp: Chrome timestamp as int.
@@ -50,7 +50,7 @@ def convert_chrometime_to_unix(timestamp):
     return epoch_start + delta
 
 
-def convert_timestamp_to_iso(timestamp):
+def convert_timestamp_to_iso(timestamp: str) -> str:
     """Converts Unix timestamp to ISO string.
 
     :param timestamp: Unix timestamp.
@@ -65,7 +65,7 @@ def convert_timestamp_to_iso(timestamp):
         return None
 
 
-def check_for_links(text):
+def check_for_links(text: str) -> list:
     """Checks if a given text contains HTTP links.
 
     :param text: Any provided text.
@@ -76,7 +76,7 @@ def check_for_links(text):
     return re.findall(r"(?P<url>https?://[^\s]+)", text, re.IGNORECASE)
 
 
-def get_sha256_from_file_path(file_path):
+def get_sha256_from_file_path(file_path: str) -> str:
     """Calculate the SHA256 hash of a file from a file path.
 
     :param file_path: Path to the file to hash
@@ -93,7 +93,7 @@ def get_sha256_from_file_path(file_path):
 
 # Note: taken from here:
 # https://stackoverflow.com/questions/57014259/json-dumps-on-dictionary-with-bytes-for-keys
-def keys_bytes_to_string(obj):
+def keys_bytes_to_string(obj) -> str:
     """Convert object keys from bytes to string.
 
     :param obj: Object to convert from bytes to string.

mvt/common/version.py

@@ -3,4 +3,4 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
-MVT_VERSION = "1.5.5"
+MVT_VERSION = "1.6"

mvt/common/virustotal.py

@@ -0,0 +1,45 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+
+import requests
+
+log = logging.getLogger(__name__)
+
+MVT_VT_API_KEY = "MVT_VT_API_KEY"
+
+
+class VTNoKey(Exception):
+    pass
+
+
+class VTQuotaExceeded(Exception):
+    pass
+
+
+def virustotal_lookup(file_hash: str):
+    if MVT_VT_API_KEY not in os.environ:
+        raise VTNoKey("No VirusTotal API key provided: to use VirusTotal lookups please provide your API key with `export MVT_VT_API_KEY=<key>`")
+
+    headers = {
+        "User-Agent": "VirusTotal",
+        "Content-Type": "application/json",
+        "x-apikey": os.environ[MVT_VT_API_KEY],
+    }
+    res = requests.get(f"https://www.virustotal.com/api/v3/files/{file_hash}", headers=headers)
+
+    if res.status_code == 200:
+        report = res.json()
+        return report["data"]
+    elif res.status_code == 404:
+        log.info("Could not find results for file with hash %s", file_hash)
+    elif res.status_code == 429:
+        raise VTQuotaExceeded("You have exceeded the quota for your VirusTotal API key")
+    else:
+        raise Exception("Unexpected response from VirusTotal: %s", res.status_code)
+
+    return None

mvt/ios/cli.py

@@ -10,14 +10,17 @@ import click
 from rich.logging import RichHandler
 from rich.prompt import Prompt
 
+from mvt.common.cmd_check_iocs import CmdCheckIOCS
 from mvt.common.help import (HELP_MSG_FAST, HELP_MSG_IOC,
                              HELP_MSG_LIST_MODULES, HELP_MSG_MODULE,
-                             HELP_MSG_OUTPUT)
-from mvt.common.indicators import Indicators, download_indicators_files
+                             HELP_MSG_OUTPUT, HELP_MSG_SERIAL)
 from mvt.common.logo import logo
-from mvt.common.module import run_module, save_timeline
 from mvt.common.options import MutuallyExclusiveOption
+from mvt.common.updates import IndicatorsUpdates
 
+from .cmd_check_backup import CmdIOSCheckBackup
+from .cmd_check_fs import CmdIOSCheckFS
+from .cmd_check_usb import CmdIOSCheckUSB
 from .decrypt import DecryptBackup
 from .modules.backup import BACKUP_MODULES
 from .modules.fs import FS_MODULES
@@ -30,7 +33,7 @@ logging.basicConfig(level="INFO", format=LOG_FORMAT, handlers=[
 log = logging.getLogger(__name__)
 
 # Set this environment variable to a password if needed.
-PASSWD_ENV = "MVT_IOS_BACKUP_PASSWORD"
+MVT_IOS_BACKUP_PASSWORD = "MVT_IOS_BACKUP_PASSWORD"
 
 
 #==============================================================================
@@ -56,7 +59,7 @@ def version():
 @click.option("--destination", "-d", required=True,
               help="Path to the folder where to store the decrypted backup")
 @click.option("--password", "-p", cls=MutuallyExclusiveOption,
-              help=f"Password to use to decrypt the backup (or, set {PASSWD_ENV} environment variable)",
+              help=f"Password to use to decrypt the backup (or, set {MVT_IOS_BACKUP_PASSWORD} environment variable)",
               mutually_exclusive=["key_file"])
 @click.option("--key-file", "-k", cls=MutuallyExclusiveOption,
               type=click.Path(exists=True),
@@ -68,22 +71,22 @@ def decrypt_backup(ctx, destination, password, key_file, backup_path):
     backup = DecryptBackup(backup_path, destination)
 
     if key_file:
-        if PASSWD_ENV in os.environ:
+        if MVT_IOS_BACKUP_PASSWORD in os.environ:
             log.info("Ignoring environment variable, using --key-file '%s' instead",
-                     PASSWD_ENV, key_file)
+                     MVT_IOS_BACKUP_PASSWORD, key_file)
 
         backup.decrypt_with_key_file(key_file)
     elif password:
         log.info("Your password may be visible in the process table because it was supplied on the command line!")
 
-        if PASSWD_ENV in os.environ:
+        if MVT_IOS_BACKUP_PASSWORD in os.environ:
             log.info("Ignoring %s environment variable, using --password argument instead",
-                     PASSWD_ENV)
+                     MVT_IOS_BACKUP_PASSWORD)
 
         backup.decrypt_with_password(password)
-    elif PASSWD_ENV in os.environ:
-        log.info("Using password from %s environment variable", PASSWD_ENV)
-        backup.decrypt_with_password(os.environ[PASSWD_ENV])
+    elif MVT_IOS_BACKUP_PASSWORD in os.environ:
+        log.info("Using password from %s environment variable", MVT_IOS_BACKUP_PASSWORD)
+        backup.decrypt_with_password(os.environ[MVT_IOS_BACKUP_PASSWORD])
     else:
         sekrit = Prompt.ask("Enter backup password", password=True)
         backup.decrypt_with_password(sekrit)
@@ -99,24 +102,24 @@ def decrypt_backup(ctx, destination, password, key_file, backup_path):
 #==============================================================================
 @cli.command("extract-key", help="Extract decryption key from an iTunes backup")
 @click.option("--password", "-p",
-              help=f"Password to use to decrypt the backup (or, set {PASSWD_ENV} environment variable)")
+              help=f"Password to use to decrypt the backup (or, set {MVT_IOS_BACKUP_PASSWORD} environment variable)")
 @click.option("--key-file", "-k",
               help="Key file to be written (if unset, will print to STDOUT)",
               required=False,
               type=click.Path(exists=False, file_okay=True, dir_okay=False, writable=True))
 @click.argument("BACKUP_PATH", type=click.Path(exists=True))
-def extract_key(password, backup_path, key_file):
+def extract_key(password, key_file, backup_path):
     backup = DecryptBackup(backup_path)
 
     if password:
         log.info("Your password may be visible in the process table because it was supplied on the command line!")
 
-        if PASSWD_ENV in os.environ:
+        if MVT_IOS_BACKUP_PASSWORD in os.environ:
             log.info("Ignoring %s environment variable, using --password argument instead",
-                     PASSWD_ENV)
-    elif PASSWD_ENV in os.environ:
-        log.info("Using password from %s environment variable", PASSWD_ENV)
-        password = os.environ[PASSWD_ENV]
+                     MVT_IOS_BACKUP_PASSWORD)
+    elif MVT_IOS_BACKUP_PASSWORD in os.environ:
+        log.info("Using password from %s environment variable", MVT_IOS_BACKUP_PASSWORD)
+        password = os.environ[MVT_IOS_BACKUP_PASSWORD]
     else:
         password = Prompt.ask("Enter backup password", password=True)
 
@@ -139,52 +142,21 @@ def extract_key(password, backup_path, key_file):
 @click.option("--module", "-m", help=HELP_MSG_MODULE)
 @click.argument("BACKUP_PATH", type=click.Path(exists=True))
 @click.pass_context
-def check_backup(ctx, iocs, output, fast, backup_path, list_modules, module):
-    if list_modules:
-        log.info("Following is the list of available check-backup modules:")
-        for backup_module in BACKUP_MODULES + MIXED_MODULES:
-            log.info(" - %s", backup_module.__name__)
+def check_backup(ctx, iocs, output, fast, list_modules, module, backup_path):
+    cmd = CmdIOSCheckBackup(target_path=backup_path, results_path=output,
+                            ioc_files=iocs, module_name=module, fast_mode=fast)
 
+    if list_modules:
+        cmd.list_modules()
         return
 
     log.info("Checking iTunes backup located at: %s", backup_path)
 
-    if output and not os.path.exists(output):
-        try:
-            os.makedirs(output)
-        except Exception as e:
-            log.critical("Unable to create output folder %s: %s", output, e)
-            ctx.exit(1)
+    cmd.run()
 
-    indicators = Indicators(log=log)
-    indicators.load_indicators_files(iocs)
-
-    timeline = []
-    timeline_detected = []
-    for backup_module in BACKUP_MODULES + MIXED_MODULES:
-        if module and backup_module.__name__ != module:
-            continue
-
-        m = backup_module(base_folder=backup_path, output_folder=output, fast_mode=fast,
-                          log=logging.getLogger(backup_module.__module__))
-        m.is_backup = True
-        if indicators.total_ioc_count > 0:
-            m.indicators = indicators
-            m.indicators.log = m.log
-
-        run_module(m)
-        timeline.extend(m.timeline)
-        timeline_detected.extend(m.timeline_detected)
-
-    if output:
-        if len(timeline) > 0:
-            save_timeline(timeline, os.path.join(output, "timeline.csv"))
-        if len(timeline_detected) > 0:
-            save_timeline(timeline_detected, os.path.join(output, "timeline_detected.csv"))
-
-    if len(timeline_detected) > 0:
+    if len(cmd.timeline_detected) > 0:
         log.warning("The analysis of the backup produced %d detections!",
-                    len(timeline_detected))
+                    len(cmd.timeline_detected))
 
 
 #==============================================================================
@@ -199,53 +171,21 @@ def check_backup(ctx, iocs, output, fast, backup_path, list_modules, module):
 @click.option("--module", "-m", help=HELP_MSG_MODULE)
 @click.argument("DUMP_PATH", type=click.Path(exists=True))
 @click.pass_context
-def check_fs(ctx, iocs, output, fast, dump_path, list_modules, module):
-    if list_modules:
-        log.info("Following is the list of available check-fs modules:")
-        for fs_module in FS_MODULES + MIXED_MODULES:
-            log.info(" - %s", fs_module.__name__)
+def check_fs(ctx, iocs, output, fast, list_modules, module, dump_path):
+    cmd = CmdIOSCheckFS(target_path=dump_path, results_path=output,
+                        ioc_files=iocs, module_name=module, fast_mode=fast)
 
+    if list_modules:
+        cmd.list_modules()
         return
 
-    log.info("Checking filesystem dump located at: %s", dump_path)
+    log.info("Checking iOS filesystem located at: %s", dump_path)
 
-    if output and not os.path.exists(output):
-        try:
-            os.makedirs(output)
-        except Exception as e:
-            log.critical("Unable to create output folder %s: %s", output, e)
-            ctx.exit(1)
+    cmd.run()
 
-    indicators = Indicators(log=log)
-    indicators.load_indicators_files(iocs)
-
-    timeline = []
-    timeline_detected = []
-    for fs_module in FS_MODULES + MIXED_MODULES:
-        if module and fs_module.__name__ != module:
-            continue
-
-        m = fs_module(base_folder=dump_path, output_folder=output, fast_mode=fast,
-                      log=logging.getLogger(fs_module.__module__))
-
-        m.is_fs_dump = True
-        if indicators.total_ioc_count > 0:
-            m.indicators = indicators
-            m.indicators.log = m.log
-
-        run_module(m)
-        timeline.extend(m.timeline)
-        timeline_detected.extend(m.timeline_detected)
-
-    if output:
-        if len(timeline) > 0:
-            save_timeline(timeline, os.path.join(output, "timeline.csv"))
-        if len(timeline_detected) > 0:
-            save_timeline(timeline_detected, os.path.join(output, "timeline_detected.csv"))
-
-    if len(timeline_detected) > 0:
-        log.warning("The analysis of the filesystem produced %d detections!",
-                    len(timeline_detected))
+    if len(cmd.timeline_detected) > 0:
+        log.warning("The analysis of the iOS filesystem produced %d detections!",
+                    len(cmd.timeline_detected))
 
 
 #==============================================================================
@@ -259,54 +199,14 @@ def check_fs(ctx, iocs, output, fast, dump_path, list_modules, module):
 @click.argument("FOLDER", type=click.Path(exists=True))
 @click.pass_context
 def check_iocs(ctx, iocs, list_modules, module, folder):
-    all_modules = []
-    for entry in BACKUP_MODULES + FS_MODULES + MIXED_MODULES:
-        if entry not in all_modules:
-            all_modules.append(entry)
+    cmd = CmdCheckIOCS(target_path=folder, ioc_files=iocs, module_name=module)
+    cmd.modules = BACKUP_MODULES + FS_MODULES + MIXED_MODULES
 
     if list_modules:
-        log.info("Following is the list of available check-iocs modules:")
-        for iocs_module in all_modules:
-            log.info(" - %s", iocs_module.__name__)
-
+        cmd.list_modules()
         return
 
-    log.info("Checking stored results against provided indicators...")
-
-    indicators = Indicators(log=log)
-    indicators.load_indicators_files(iocs)
-
-    total_detections = 0
-    for file_name in os.listdir(folder):
-        name_only, ext = os.path.splitext(file_name)
-        file_path = os.path.join(folder, file_name)
-
-        for iocs_module in all_modules:
-            if module and iocs_module.__name__ != module:
-                continue
-
-            if iocs_module().get_slug() != name_only:
-                continue
-
-            log.info("Loading results from \"%s\" with module %s", file_name,
-                     iocs_module.__name__)
-
-            m = iocs_module.from_json(file_path,
-                                      log=logging.getLogger(iocs_module.__module__))
-            if indicators.total_ioc_count > 0:
-                m.indicators = indicators
-                m.indicators.log = m.log
-
-            try:
-                m.check_indicators()
-            except NotImplementedError:
-                continue
-            else:
-                total_detections += len(m.detected)
-
-    if total_detections > 0:
-        log.warning("The check of the results produced %d detections!",
-                    total_detections)
+    cmd.run()
 
 
 #==============================================================================
@@ -314,4 +214,36 @@ def check_iocs(ctx, iocs, list_modules, module, folder):
 #==============================================================================
 @cli.command("download-iocs", help="Download public STIX2 indicators")
 def download_iocs():
-    download_indicators_files(log)
+    ioc_updates = IndicatorsUpdates()
+    ioc_updates.update()
+
+
+#==============================================================================
+# Command: check-usb
+#==============================================================================
+@cli.command("check-usb", help="Extract artifacts from a live iPhone through USB / lockdown")
+@click.option("--serial", "-s", type=str, help=HELP_MSG_SERIAL)
+@click.option("--iocs", "-i", type=click.Path(exists=True), multiple=True,
+              default=[], help=HELP_MSG_IOC)
+@click.option("--output", "-o", type=click.Path(exists=False), help=HELP_MSG_OUTPUT)
+@click.option("--fast", "-f", is_flag=True, help=HELP_MSG_FAST)
+@click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
+@click.option("--module", "-m", help=HELP_MSG_MODULE)
+# TODO: serial
+# @click.argument("BACKUP_PATH", type=click.Path(exists=True))
+@click.pass_context
+def check_usb(ctx, serial, iocs, output, fast, list_modules, module):
+    cmd = CmdIOSCheckUSB(results_path=output, ioc_files=iocs,
+                         module_name=module, fast_mode=fast,
+                         serial=serial)
+
+    if list_modules:
+        cmd.list_modules()
+        return
+
+    log.info("Checking iPhone through USB, this may take a while")
+    cmd.run()
+
+    if len(cmd.timeline_detected) > 0:
+        log.warning("The analysis of the data produced %d detections!",
+                    len(cmd.timeline_detected))

mvt/ios/cmd_check_backup.py

@@ -0,0 +1,29 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+
+from mvt.common.command import Command
+
+from .modules.backup import BACKUP_MODULES
+from .modules.mixed import MIXED_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdIOSCheckBackup(Command):
+
+    name = "check-backup"
+    modules = BACKUP_MODULES + MIXED_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+
+    def module_init(self, module):
+        module.is_backup = True

mvt/ios/cmd_check_fs.py

@@ -0,0 +1,29 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+
+from mvt.common.command import Command
+
+from .modules.fs import FS_MODULES
+from .modules.mixed import MIXED_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdIOSCheckFS(Command):
+
+    name = "check-fs"
+    modules = FS_MODULES + MIXED_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+
+    def module_init(self, module):
+        module.is_fs_dump = True

mvt/ios/cmd_check_usb.py

@@ -0,0 +1,46 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import sys
+
+from pymobiledevice3.exceptions import ConnectionFailedError
+from pymobiledevice3.lockdown import LockdownClient
+
+from mvt.common.command import Command
+
+from .modules.usb import USB_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdIOSCheckUSB(Command):
+
+    name = "check-usb"
+    modules = USB_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+        self.lockdown = None
+
+    def init(self):
+        try:
+            if self.serial:
+                self.lockdown = LockdownClient(udid=self.serial)
+            else:
+                self.lockdown = LockdownClient()
+        except ConnectionRefusedError:
+            log.error("Unable to connect to the device over USB. Try to unplug, plug the device and start again.")
+            sys.exit(-1)
+        except ConnectionFailedError:
+            log.error("Unable to connect to the device %s", self.serial)
+            sys.exit(-1)
+
+    def module_init(self, module):
+        module.lockdown = self.lockdown

mvt/ios/decrypt.py

@@ -24,7 +24,7 @@ class DecryptBackup:
 
     """
 
-    def __init__(self, backup_path, dest_path=None):
+    def __init__(self, backup_path: str, dest_path: str = None) -> None:
         """Decrypts an encrypted iOS backup.
         :param backup_path: Path to the encrypted backup folder
         :param dest_path: Path to the folder where to store the decrypted backup
@@ -38,7 +38,7 @@ class DecryptBackup:
         return self._backup is not None
 
     @staticmethod
-    def is_encrypted(backup_path) -> bool:
+    def is_encrypted(backup_path: str) -> bool:
         """Query Manifest.db file to see if it's encrypted or not.
 
         :param backup_path: Path to the backup to decrypt
@@ -54,13 +54,14 @@ class DecryptBackup:
             log.critical("The backup does not seem encrypted!")
             return False
 
-    def _process_file(self, relative_path, domain, item, file_id, item_folder):
+    def _process_file(self, relative_path: str, domain: str, item,
+                      file_id: str, item_folder: str) -> None:
         self._backup.getFileDecryptedCopy(manifestEntry=item,
                                           targetName=file_id,
                                           targetFolder=item_folder)
         log.info("Decrypted file %s [%s] to %s/%s", relative_path, domain, item_folder, file_id)
 
-    def process_backup(self):
+    def process_backup(self) -> None:
         if not os.path.exists(self.dest_path):
             os.makedirs(self.dest_path)
 
@@ -111,7 +112,7 @@ class DecryptBackup:
                 shutil.copy(os.path.join(self.backup_path, file_name),
                             self.dest_path)
 
-    def decrypt_with_password(self, password):
+    def decrypt_with_password(self, password: str) -> None:
         """Decrypts an encrypted iOS backup.
 
         :param password: Password to use to decrypt the original backup
@@ -149,7 +150,7 @@ class DecryptBackup:
                 log.exception(e)
                 log.critical("Failed to decrypt backup. Did you provide the correct password? Did you point to the right backup path?")
 
-    def decrypt_with_key_file(self, key_file):
+    def decrypt_with_key_file(self, key_file: str) -> None:
         """Decrypts an encrypted iOS backup using a key file.
 
         :param key_file: File to read the key bytes to decrypt the backup
@@ -179,7 +180,7 @@ class DecryptBackup:
             log.exception(e)
             log.critical("Failed to decrypt backup. Did you provide the correct key file?")
 
-    def get_key(self):
+    def get_key(self) -> None:
         """Retrieve and prints the encryption key."""
         if not self._backup:
             return
@@ -188,7 +189,7 @@ class DecryptBackup:
         log.info("Derived decryption key for backup at path %s is: \"%s\"",
                  self.backup_path, self._decryption_key)
 
-    def write_key(self, key_path):
+    def write_key(self, key_path: str) -> None:
         """Save extracted key to file.
 
         :param key_path: Path to the file where to write the derived decryption key.

mvt/ios/modules/backup/backup_info.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import os
 import plistlib
 
@@ -15,16 +16,17 @@ from ..base import IOSExtraction
 class BackupInfo(IOSExtraction):
     """This module extracts information about the device and the backup."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = {}
 
-    def run(self):
-        info_path = os.path.join(self.base_folder, "Info.plist")
+    def run(self) -> None:
+        info_path = os.path.join(self.target_path, "Info.plist")
         if not os.path.exists(info_path):
             raise DatabaseNotFoundError("No Info.plist at backup path, unable to extract device information")
 

mvt/ios/modules/backup/configuration_profiles.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import os
 import plistlib
 from base64 import b64encode
@@ -17,13 +18,14 @@ CONF_PROFILES_DOMAIN = "SysSharedContainerDomain-systemgroup.com.apple.configura
 class ConfigurationProfiles(IOSExtraction):
     """This module extracts the full plist data from configuration profiles."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         if not record["install_date"]:
             return
 
@@ -36,7 +38,7 @@ class ConfigurationProfiles(IOSExtraction):
             "data": f"{record['plist']['PayloadType']} installed: {record['plist']['PayloadUUID']} - {payload_name}: {payload_description}"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -58,7 +60,7 @@ class ConfigurationProfiles(IOSExtraction):
                     self.detected.append(result)
                     continue
 
-    def run(self):
+    def run(self) -> None:
         for conf_file in self._get_backup_files_from_manifest(domain=CONF_PROFILES_DOMAIN):
             conf_rel_path = conf_file["relative_path"]
             # Filter out all configuration files that are not configuration profiles.

mvt/ios/modules/backup/manifest.py

@@ -5,6 +5,7 @@
 
 import datetime
 import io
+import logging
 import os
 import plistlib
 import sqlite3
@@ -18,10 +19,11 @@ from ..base import IOSExtraction
 class Manifest(IOSExtraction):
     """This module extracts information from a backup Manifest.db file."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
     def _get_key(self, dictionary, key):
@@ -47,7 +49,7 @@ class Manifest(IOSExtraction):
             timestamp = datetime.datetime.utcfromtimestamp(timestamp_or_unix_time_int)
             return convert_timestamp_to_iso(timestamp)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
         if "modified" not in record or "status_changed" not in record:
             return
@@ -67,7 +69,7 @@ class Manifest(IOSExtraction):
 
         return records
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -92,8 +94,8 @@ class Manifest(IOSExtraction):
                                      ioc["value"], rel_path)
                     self.detected.append(result)
 
-    def run(self):
-        manifest_db_path = os.path.join(self.base_folder, "Manifest.db")
+    def run(self) -> None:
+        manifest_db_path = os.path.join(self.target_path, "Manifest.db")
         if not os.path.isfile(manifest_db_path):
             raise DatabaseNotFoundError("unable to find backup's Manifest.db")
 

mvt/ios/modules/backup/profile_events.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import plistlib
 
 from mvt.common.utils import convert_timestamp_to_iso
@@ -18,14 +19,14 @@ class ProfileEvents(IOSExtraction):
 
 
     """
-
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record.get("timestamp"),
             "module": self.__class__.__name__,
@@ -33,7 +34,7 @@ class ProfileEvents(IOSExtraction):
             "data": f"Process {record.get('process')} started operation {record.get('operation')} of profile {record.get('profile_id')}"
         }
 
-    def run(self):
+    def run(self) -> None:
         for events_file in self._get_backup_files_from_manifest(relative_path=CONF_PROFILES_EVENTS_RELPATH):
             events_file_path = self._get_backup_file_from_id(events_file["file_id"])
             if not events_file_path:

mvt/ios/modules/base.py

@@ -4,6 +4,7 @@
 #   https://license.mvt.re/1.1/
 
 import glob
+import logging
 import os
 import shutil
 import sqlite3
@@ -16,10 +17,11 @@ from mvt.common.module import (DatabaseCorruptedError, DatabaseNotFoundError,
 class IOSExtraction(MVTModule):
     """This class provides a base for all iOS filesystem/backup extraction modules."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.is_backup = False
@@ -73,7 +75,7 @@ class IOSExtraction(MVTModule):
         :param domain: Domain to use as filter from Manifest.db. (Default value = None)
 
         """
-        manifest_db_path = os.path.join(self.base_folder, "Manifest.db")
+        manifest_db_path = os.path.join(self.target_path, "Manifest.db")
         if not os.path.exists(manifest_db_path):
             raise DatabaseNotFoundError("unable to find backup's Manifest.db")
 
@@ -101,7 +103,7 @@ class IOSExtraction(MVTModule):
             }
 
     def _get_backup_file_from_id(self, file_id):
-        file_path = os.path.join(self.base_folder, file_id[0:2], file_id)
+        file_path = os.path.join(self.target_path, file_id[0:2], file_id)
         if os.path.exists(file_path):
             return file_path
 
@@ -109,7 +111,7 @@ class IOSExtraction(MVTModule):
 
     def _get_fs_files_from_patterns(self, root_paths):
         for root_path in root_paths:
-            for found_path in glob.glob(os.path.join(self.base_folder, root_path)):
+            for found_path in glob.glob(os.path.join(self.target_path, root_path)):
                 if not os.path.exists(found_path):
                     continue
 

mvt/ios/modules/fs/__init__.py

@@ -4,6 +4,7 @@
 #   https://license.mvt.re/1.1/
 
 from .analytics import Analytics
+from .analytics_ios_versions import AnalyticsIOSVersions
 from .cache_files import CacheFiles
 from .filesystem import Filesystem
 from .net_netusage import Netusage
@@ -14,6 +15,6 @@ from .webkit_indexeddb import WebkitIndexedDB
 from .webkit_localstorage import WebkitLocalStorage
 from .webkit_safariviewservice import WebkitSafariViewService
 
-FS_MODULES = [CacheFiles, Filesystem, Netusage, Analytics, SafariFavicon, ShutdownLog,
-              IOSVersionHistory, WebkitIndexedDB, WebkitLocalStorage,
-              WebkitSafariViewService]
+FS_MODULES = [CacheFiles, Filesystem, Netusage, Analytics, AnalyticsIOSVersions,
+              SafariFavicon, ShutdownLog, IOSVersionHistory, WebkitIndexedDB,
+              WebkitLocalStorage, WebkitSafariViewService]

mvt/ios/modules/fs/analytics.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import plistlib
 import sqlite3
 
@@ -18,21 +19,22 @@ ANALYTICS_DB_PATH = [
 class Analytics(IOSExtraction):
     """This module extracts information from the private/var/Keychains/Analytics/*.db files."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
-            "timestamp": record["timestamp"],
+            "timestamp": record["isodate"],
             "module": self.__class__.__name__,
             "event": record["artifact"],
             "data": f"{record}",
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -94,31 +96,35 @@ class Analytics(IOSExtraction):
 
         for row in cur:
             if row[0] and row[1]:
-                timestamp = convert_timestamp_to_iso(convert_mactime_to_unix(row[0], False))
+                isodate = convert_timestamp_to_iso(convert_mactime_to_unix(row[0], False))
                 data = plistlib.loads(row[1])
-                data["timestamp"] = timestamp
+                data["isodate"] = isodate
             elif row[0]:
-                timestamp = convert_timestamp_to_iso(convert_mactime_to_unix(row[0], False))
+                isodate = convert_timestamp_to_iso(convert_mactime_to_unix(row[0], False))
                 data = {}
-                data["timestamp"] = timestamp
+                data["isodate"] = isodate
             elif row[1]:
-                timestamp = ""
+                isodate = ""
                 data = plistlib.loads(row[1])
-                data["timestamp"] = timestamp
+                data["isodate"] = isodate
 
             data["artifact"] = artifact
 
             self.results.append(data)
 
-        self.results = sorted(self.results, key=lambda entry: entry["timestamp"])
-
         cur.close()
         conn.close()
 
-        self.log.info("Extracted information on %d analytics data from %s", len(self.results), artifact)
-
-    def run(self):
+    def process_analytics_dbs(self):
         for file_path in self._get_fs_files_from_patterns(ANALYTICS_DB_PATH):
             self.file_path = file_path
             self.log.info("Found Analytics database file at path: %s", file_path)
             self._extract_analytics_data()
+
+    def run(self) -> None:
+        self.process_analytics_dbs()
+
+        self.log.info("Extracted %d records from analytics databases",
+                      len(self.results))
+
+        self.results = sorted(self.results, key=lambda entry: entry["isodate"])

mvt/ios/modules/fs/analytics_ios_versions.py

@@ -0,0 +1,73 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+from datetime import datetime
+
+from mvt.ios.versions import find_version_by_build
+
+from ..base import IOSExtraction
+from .analytics import Analytics
+
+
+class AnalyticsIOSVersions(IOSExtraction):
+    """This module leverages the Analytics module in order to extract
+    a timeline of build numbers from the private/var/Keychains/Analytics/*.db
+    files."""
+
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def serialize(self, record: dict) -> None:
+        return {
+            "timestamp": record["isodate"],
+            "module": self.__class__.__name__,
+            "event": "analytics_ios_version",
+            "data": f"Seen iOS version {record['version']} ({record['build']})",
+        }
+
+    def run(self):
+        anl = Analytics(target_path=self.target_path, log=self.log)
+        anl.process_analytics_dbs()
+
+        dt_format = "%Y-%m-%d %H:%M:%S.%f"
+
+        builds = {}
+        for result in anl.results:
+            build = result.get("build")
+            if not build:
+                continue
+
+            ts = result.get("isodate", None)
+            if not ts:
+                continue
+
+            if build not in builds.keys():
+                builds[build] = ts
+                continue
+
+            result_dt = datetime.strptime(ts, dt_format)
+            cur_dt = datetime.strptime(builds[build], dt_format)
+
+            if result_dt < cur_dt:
+                builds[build] = ts
+
+        for build, ts in builds.items():
+            version = find_version_by_build(build)
+
+            self.results.append({
+                "isodate": ts,
+                "build": build,
+                "version": version,
+            })
+
+        self.results = sorted(self.results, key=lambda entry: entry["isodate"])
+        for result in self.results:
+            self.log.info("iOS version %s (%s) first appeared on %s",
+                          result["version"], result["build"], result["isodate"])

mvt/ios/modules/fs/cache_files.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import os
 import sqlite3
 
@@ -11,13 +12,14 @@ from ..base import IOSExtraction
 
 class CacheFiles(IOSExtraction):
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
         for item in self.results[record]:
             records.append({
@@ -29,7 +31,7 @@ class CacheFiles(IOSExtraction):
 
         return records
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -55,7 +57,7 @@ class CacheFiles(IOSExtraction):
         except sqlite3.OperationalError:
             return
 
-        key_name = os.path.relpath(file_path, self.base_folder)
+        key_name = os.path.relpath(file_path, self.target_path)
         if key_name not in self.results:
             self.results[key_name] = []
 
@@ -69,9 +71,9 @@ class CacheFiles(IOSExtraction):
                 "isodate": row[5],
             })
 
-    def run(self):
+    def run(self) -> None:
         self.results = {}
-        for root, dirs, files in os.walk(self.base_folder):
+        for root, dirs, files in os.walk(self.target_path):
             for file_name in files:
                 if file_name != "Cache.db":
                     continue

mvt/ios/modules/fs/filesystem.py

@@ -4,6 +4,7 @@
 #   https://license.mvt.re/1.1/
 
 import datetime
+import logging
 import os
 
 from mvt.common.utils import convert_timestamp_to_iso
@@ -18,13 +19,14 @@ class Filesystem(IOSExtraction):
 
     """
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["modified"],
             "module": self.__class__.__name__,
@@ -32,7 +34,7 @@ class Filesystem(IOSExtraction):
             "data": record["path"],
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -57,13 +59,13 @@ class Filesystem(IOSExtraction):
                     result["matched_indicator"] = ioc
                     self.detected.append(result)
 
-    def run(self):
-        for root, dirs, files in os.walk(self.base_folder):
+    def run(self) -> None:
+        for root, dirs, files in os.walk(self.target_path):
             for dir_name in dirs:
                 try:
                     dir_path = os.path.join(root, dir_name)
                     result = {
-                        "path": os.path.relpath(dir_path, self.base_folder),
+                        "path": os.path.relpath(dir_path, self.target_path),
                         "modified": convert_timestamp_to_iso(datetime.datetime.utcfromtimestamp(os.stat(dir_path).st_mtime)),
                     }
                 except Exception:
@@ -75,7 +77,7 @@ class Filesystem(IOSExtraction):
                 try:
                     file_path = os.path.join(root, file_name)
                     result = {
-                        "path": os.path.relpath(file_path, self.base_folder),
+                        "path": os.path.relpath(file_path, self.target_path),
                         "modified": convert_timestamp_to_iso(datetime.datetime.utcfromtimestamp(os.stat(file_path).st_mtime)),
                     }
                 except Exception:

mvt/ios/modules/fs/net_netusage.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 
 from ..net_base import NetBase
@@ -20,13 +21,14 @@ class Netusage(NetBase):
 
     """
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def run(self):
+    def run(self) -> None:
         for netusage_path in self._get_fs_files_from_patterns(NETUSAGE_ROOT_PATHS):
             self.file_path = netusage_path
             self.log.info("Found NetUsage database at path: %s", self.file_path)

mvt/ios/modules/fs/safari_favicon.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 
 from mvt.common.utils import convert_mactime_to_unix, convert_timestamp_to_iso
@@ -18,13 +19,14 @@ SAFARI_FAVICON_ROOT_PATHS = [
 class SafariFavicon(IOSExtraction):
     """This module extracts all Safari favicon records."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -32,7 +34,7 @@ class SafariFavicon(IOSExtraction):
             "data": f"Safari favicon from {record['url']} with icon URL {record['icon_url']} ({record['type']})",
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -92,7 +94,7 @@ class SafariFavicon(IOSExtraction):
         cur.close()
         conn.close()
 
-    def run(self):
+    def run(self) -> None:
         for file_path in self._get_fs_files_from_patterns(SAFARI_FAVICON_ROOT_PATHS):
             self.log.info("Found Safari favicon cache database at path: %s", file_path)
             self._process_favicon_db(file_path)

mvt/ios/modules/fs/shutdownlog.py

@@ -3,6 +3,8 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
+
 from mvt.common.utils import convert_mactime_to_unix, convert_timestamp_to_iso
 
 from ..base import IOSExtraction
@@ -15,13 +17,14 @@ SHUTDOWN_LOG_PATH = [
 class ShutdownLog(IOSExtraction):
     """This module extracts processes information from the shutdown log file."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -29,7 +32,7 @@ class ShutdownLog(IOSExtraction):
             "data": f"Client {record['client']} with PID {record['pid']} was running when the device was shut down",
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -83,7 +86,7 @@ class ShutdownLog(IOSExtraction):
 
         self.results = sorted(self.results, key=lambda entry: entry["isodate"])
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(root_paths=SHUTDOWN_LOG_PATH)
         self.log.info("Found shutdown log at path: %s", self.file_path)
         with open(self.file_path, "r", encoding="utf-8") as handle:

mvt/ios/modules/fs/version_history.py

@@ -5,6 +5,7 @@
 
 import datetime
 import json
+import logging
 
 from mvt.common.utils import convert_timestamp_to_iso
 
@@ -18,13 +19,14 @@ IOS_ANALYTICS_JOURNAL_PATHS = [
 class IOSVersionHistory(IOSExtraction):
     """This module extracts iOS update history from Analytics Journal log files."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -32,7 +34,7 @@ class IOSVersionHistory(IOSExtraction):
             "data": f"Recorded iOS version {record['os_version']}",
         }
 
-    def run(self):
+    def run(self) -> None:
         for found_path in self._get_fs_files_from_patterns(IOS_ANALYTICS_JOURNAL_PATHS):
             with open(found_path, "r", encoding="utf-8") as analytics_log:
                 log_line = json.loads(analytics_log.readline().strip())

mvt/ios/modules/fs/webkit_base.py

@@ -14,7 +14,7 @@ from ..base import IOSExtraction
 class WebkitBase(IOSExtraction):
     """This class is a base for other WebKit-related modules."""
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -26,7 +26,7 @@ class WebkitBase(IOSExtraction):
 
     def _process_webkit_folder(self, root_paths):
         for found_path in self._get_fs_files_from_patterns(root_paths):
-            key = os.path.relpath(found_path, self.base_folder)
+            key = os.path.relpath(found_path, self.target_path)
 
             for name in os.listdir(found_path):
                 if not name.startswith("http"):

mvt/ios/modules/fs/webkit_indexeddb.py

@@ -3,6 +3,8 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
+
 from .webkit_base import WebkitBase
 
 WEBKIT_INDEXEDDB_ROOT_PATHS = [
@@ -19,13 +21,14 @@ class WebkitIndexedDB(WebkitBase):
 
     slug = "webkit_indexeddb"
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -33,7 +36,7 @@ class WebkitIndexedDB(WebkitBase):
             "data": f"IndexedDB folder {record['folder']} containing file for URL {record['url']}",
         }
 
-    def run(self):
+    def run(self) -> None:
         self._process_webkit_folder(WEBKIT_INDEXEDDB_ROOT_PATHS)
         self.log.info("Extracted a total of %d WebKit IndexedDB records",
                       len(self.results))

mvt/ios/modules/fs/webkit_localstorage.py

@@ -3,6 +3,8 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
+
 from .webkit_base import WebkitBase
 
 WEBKIT_LOCALSTORAGE_ROOT_PATHS = [
@@ -17,13 +19,14 @@ class WebkitLocalStorage(WebkitBase):
 
     """
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -31,7 +34,7 @@ class WebkitLocalStorage(WebkitBase):
             "data": f"WebKit Local Storage folder {record['folder']} containing file for URL {record['url']}",
         }
 
-    def run(self):
+    def run(self) -> None:
         self._process_webkit_folder(WEBKIT_LOCALSTORAGE_ROOT_PATHS)
         self.log.info("Extracted a total of %d records from WebKit Local Storages",
                       len(self.results))

mvt/ios/modules/fs/webkit_safariviewservice.py

@@ -3,6 +3,8 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
+
 from .webkit_base import WebkitBase
 
 WEBKIT_SAFARIVIEWSERVICE_ROOT_PATHS = [
@@ -17,13 +19,14 @@ class WebkitSafariViewService(WebkitBase):
 
     """
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def run(self):
+    def run(self) -> None:
         self._process_webkit_folder(WEBKIT_SAFARIVIEWSERVICE_ROOT_PATHS)
         self.log.info("Extracted a total of %d records from WebKit SafariViewService WebsiteData",
                       len(self.results))

mvt/ios/modules/mixed/calls.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 
 from mvt.common.utils import convert_mactime_to_unix, convert_timestamp_to_iso
@@ -20,13 +21,14 @@ CALLS_ROOT_PATHS = [
 class Calls(IOSExtraction):
     """This module extracts phone calls details"""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -34,7 +36,7 @@ class Calls(IOSExtraction):
             "data": f"From {record['number']} using {record['provider']} during {record['duration']} seconds"
         }
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=CALLS_BACKUP_IDS,
                                 root_paths=CALLS_ROOT_PATHS)
         self.log.info("Found Calls database at path: %s", self.file_path)

mvt/ios/modules/mixed/chrome_favicon.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 
 from mvt.common.utils import (convert_chrometime_to_unix,
@@ -23,13 +24,14 @@ CHROME_FAVICON_ROOT_PATHS = [
 class ChromeFavicon(IOSExtraction):
     """This module extracts all Chrome favicon records."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -37,7 +39,7 @@ class ChromeFavicon(IOSExtraction):
             "data": f"{record['icon_url']} from {record['url']}"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -50,7 +52,7 @@ class ChromeFavicon(IOSExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=CHROME_FAVICON_BACKUP_IDS,
                                 root_paths=CHROME_FAVICON_ROOT_PATHS)
         self.log.info("Found Chrome favicon cache database at path: %s", self.file_path)

mvt/ios/modules/mixed/chrome_history.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 
 from mvt.common.utils import (convert_chrometime_to_unix,
@@ -22,13 +23,14 @@ CHROME_HISTORY_ROOT_PATHS = [
 class ChromeHistory(IOSExtraction):
     """This module extracts all Chome visits."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -36,7 +38,7 @@ class ChromeHistory(IOSExtraction):
             "data": f"{record['id']} - {record['url']} (visit ID: {record['visit_id']}, redirect source: {record['redirect_source']})"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -46,7 +48,7 @@ class ChromeHistory(IOSExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=CHROME_HISTORY_BACKUP_IDS,
                                 root_paths=CHROME_HISTORY_ROOT_PATHS)
         self.log.info("Found Chrome history database at path: %s", self.file_path)

mvt/ios/modules/mixed/contacts.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 
 from ..base import IOSExtraction
@@ -18,13 +19,14 @@ CONTACTS_ROOT_PATHS = [
 class Contacts(IOSExtraction):
     """This module extracts all contact details from the phone's address book."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=CONTACTS_BACKUP_IDS, root_paths=CONTACTS_ROOT_PATHS)
         self.log.info("Found Contacts database at path: %s", self.file_path)
 

mvt/ios/modules/mixed/firefox_favicon.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 from datetime import datetime
 
@@ -21,13 +22,14 @@ FIREFOX_HISTORY_ROOT_PATHS = [
 class FirefoxFavicon(IOSExtraction):
     """This module extracts all Firefox favicon"""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -35,7 +37,7 @@ class FirefoxFavicon(IOSExtraction):
             "data": f"Firefox favicon {record['url']} when visiting {record['history_url']}",
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -48,7 +50,7 @@ class FirefoxFavicon(IOSExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=FIREFOX_HISTORY_BACKUP_IDS,
                                 root_paths=FIREFOX_HISTORY_ROOT_PATHS)
         self.log.info("Found Firefox favicon database at path: %s", self.file_path)

mvt/ios/modules/mixed/firefox_history.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 from datetime import datetime
 
@@ -25,13 +26,14 @@ class FirefoxHistory(IOSExtraction):
 
     """
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -39,7 +41,7 @@ class FirefoxHistory(IOSExtraction):
             "data": f"Firefox visit with ID {record['id']} to URL: {record['url']}",
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -49,7 +51,7 @@ class FirefoxHistory(IOSExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=FIREFOX_HISTORY_BACKUP_IDS, root_paths=FIREFOX_HISTORY_ROOT_PATHS)
         self.log.info("Found Firefox history database at path: %s", self.file_path)
 

mvt/ios/modules/mixed/idstatuscache.py

@@ -4,6 +4,7 @@
 #   https://license.mvt.re/1.1/
 
 import collections
+import logging
 import plistlib
 
 from mvt.common.utils import convert_mactime_to_unix, convert_timestamp_to_iso
@@ -22,13 +23,14 @@ IDSTATUSCACHE_ROOT_PATHS = [
 class IDStatusCache(IOSExtraction):
     """Extracts Apple Authentication information from idstatuscache.plist"""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -36,7 +38,7 @@ class IDStatusCache(IOSExtraction):
             "data": f"Lookup of {record['user']} within {record['package']} (Status {record['idstatus']})"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -83,7 +85,7 @@ class IDStatusCache(IOSExtraction):
             entry["occurrences"] = entry_counter[entry["user"]]
             self.results.append(entry)
 
-    def run(self):
+    def run(self) -> None:
 
         if self.is_backup:
             self._find_ios_database(backup_ids=IDSTATUSCACHE_BACKUP_IDS)

mvt/ios/modules/mixed/interactionc.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 
 from mvt.common.utils import convert_mactime_to_unix, convert_timestamp_to_iso
@@ -20,10 +21,11 @@ INTERACTIONC_ROOT_PATHS = [
 class InteractionC(IOSExtraction):
     """This module extracts data from InteractionC db."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.timestamps = [
@@ -39,7 +41,7 @@ class InteractionC(IOSExtraction):
             "last_outgoing_recipient_date",
         ]
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
         processed = []
         for ts in self.timestamps:
@@ -63,8 +65,9 @@ class InteractionC(IOSExtraction):
 
         return records
 
-    def run(self):
-        self._find_ios_database(backup_ids=INTERACTIONC_BACKUP_IDS, root_paths=INTERACTIONC_ROOT_PATHS)
+    def run(self) -> None:
+        self._find_ios_database(backup_ids=INTERACTIONC_BACKUP_IDS,
+                                root_paths=INTERACTIONC_ROOT_PATHS)
         self.log.info("Found InteractionC database at path: %s", self.file_path)
 
         conn = sqlite3.connect(self.file_path)

mvt/ios/modules/mixed/locationd.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import plistlib
 
 from mvt.common.utils import convert_mactime_to_unix, convert_timestamp_to_iso
@@ -21,10 +22,11 @@ LOCATIOND_ROOT_PATHS = [
 class LocationdClients(IOSExtraction):
     """Extract information from apps who used geolocation."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.timestamps = [
@@ -39,7 +41,7 @@ class LocationdClients(IOSExtraction):
             "BeaconRegionTimeStopped",
         ]
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         records = []
         for timestamp in self.timestamps:
             if timestamp in record.keys():
@@ -52,7 +54,7 @@ class LocationdClients(IOSExtraction):
 
         return records
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -108,8 +110,7 @@ class LocationdClients(IOSExtraction):
 
             self.results.append(result)
 
-    def run(self):
-
+    def run(self) -> None:
         if self.is_backup:
             self._find_ios_database(backup_ids=LOCATIOND_BACKUP_IDS)
             self.log.info("Found Locationd Clients plist at path: %s", self.file_path)

mvt/ios/modules/mixed/net_datausage.py

@@ -3,6 +3,8 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
+
 from ..net_base import NetBase
 
 DATAUSAGE_BACKUP_IDS = [
@@ -20,13 +22,14 @@ class Datausage(NetBase):
 
     """
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=DATAUSAGE_BACKUP_IDS,
                                 root_paths=DATAUSAGE_ROOT_PATHS)
         self.log.info("Found DataUsage database at path: %s", self.file_path)

mvt/ios/modules/mixed/osanalytics_addaily.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import plistlib
 
 from mvt.common.utils import convert_timestamp_to_iso
@@ -20,13 +21,14 @@ OSANALYTICS_ADDAILY_ROOT_PATHS = [
 class OSAnalyticsADDaily(IOSExtraction):
     """Extract network usage information by process, from com.apple.osanalytics.addaily.plist"""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         record_data = f"{record['package']} WIFI IN: {record['wifi_in']}, WIFI OUT: {record['wifi_out']} - "  \
                       f"WWAN IN: {record['wwan_in']}, WWAN OUT: {record['wwan_out']}"
         return {
@@ -36,7 +38,7 @@ class OSAnalyticsADDaily(IOSExtraction):
             "data": record_data,
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -46,7 +48,7 @@ class OSAnalyticsADDaily(IOSExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=OSANALYTICS_ADDAILY_BACKUP_IDS,
                                 root_paths=OSANALYTICS_ADDAILY_ROOT_PATHS)
         self.log.info("Found com.apple.osanalytics.addaily plist at path: %s", self.file_path)

mvt/ios/modules/mixed/safari_browserstate.py

@@ -4,6 +4,7 @@
 #   https://license.mvt.re/1.1/
 
 import io
+import logging
 import os
 import plistlib
 import sqlite3
@@ -23,15 +24,16 @@ SAFARI_BROWSER_STATE_ROOT_PATHS = [
 class SafariBrowserState(IOSExtraction):
     """This module extracts all Safari browser state records."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self._session_history_count = 0
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["last_viewed_timestamp"],
             "module": self.__class__.__name__,
@@ -39,7 +41,7 @@ class SafariBrowserState(IOSExtraction):
             "data": f"{record['tab_title']} - {record['tab_url']}"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -115,10 +117,10 @@ class SafariBrowserState(IOSExtraction):
                 "tab_visible_url": row[2],
                 "last_viewed_timestamp": convert_timestamp_to_iso(convert_mactime_to_unix(row[3])),
                 "session_data": session_entries,
-                "safari_browser_state_db": os.path.relpath(db_path, self.base_folder),
+                "safari_browser_state_db": os.path.relpath(db_path, self.target_path),
             })
 
-    def run(self):
+    def run(self) -> None:
         if self.is_backup:
             for backup_file in self._get_backup_files_from_manifest(relative_path=SAFARI_BROWSER_STATE_BACKUP_RELPATH):
                 self.file_path = self._get_backup_file_from_id(backup_file["file_id"])

mvt/ios/modules/mixed/safari_history.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import os
 import sqlite3
 
@@ -25,13 +26,14 @@ class SafariHistory(IOSExtraction):
 
     """
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -73,7 +75,7 @@ class SafariHistory(IOSExtraction):
                 if elapsed_time.seconds == 0:
                     self.log.warning("Redirect took less than a second! (%d milliseconds)", elapsed_ms)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         self._find_injections()
 
         if not self.indicators:
@@ -111,13 +113,13 @@ class SafariHistory(IOSExtraction):
                 "isodate": convert_timestamp_to_iso(convert_mactime_to_unix(row[3])),
                 "redirect_source": row[4],
                 "redirect_destination": row[5],
-                "safari_history_db": os.path.relpath(history_path, self.base_folder),
+                "safari_history_db": os.path.relpath(history_path, self.target_path),
             })
 
         cur.close()
         conn.close()
 
-    def run(self):
+    def run(self) -> None:
         if self.is_backup:
             for history_file in self._get_backup_files_from_manifest(relative_path=SAFARI_HISTORY_BACKUP_RELPATH):
                 history_path = self._get_backup_file_from_id(history_file["file_id"])

mvt/ios/modules/mixed/shortcuts.py

@@ -5,6 +5,7 @@
 
 import io
 import itertools
+import logging
 import plistlib
 import sqlite3
 
@@ -24,13 +25,14 @@ SHORTCUT_ROOT_PATHS = [
 class Shortcuts(IOSExtraction):
     """This module extracts all info about SMS/iMessage attachments."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         found_urls = ""
         if record["action_urls"]:
             found_urls = "- URLs in actions: {}".format(", ".join(record["action_urls"]))
@@ -50,7 +52,7 @@ class Shortcuts(IOSExtraction):
             "data": f"iOS Shortcut '{record['shortcut_name'].decode('utf-8')}': {desc} {found_urls}"
         }]
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -60,7 +62,7 @@ class Shortcuts(IOSExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=SHORTCUT_BACKUP_IDS,
                                 root_paths=SHORTCUT_ROOT_PATHS)
         self.log.info("Found Shortcuts database at path: %s", self.file_path)

mvt/ios/modules/mixed/sms.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 from base64 import b64encode
 
@@ -22,13 +23,14 @@ SMS_ROOT_PATHS = [
 class SMS(IOSExtraction):
     """This module extracts all SMS messages containing links."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         text = record["text"].replace("\n", "\\n")
         return {
             "timestamp": record["isodate"],
@@ -37,7 +39,7 @@ class SMS(IOSExtraction):
             "data": f"{record['service']}: {record['guid']} \"{text}\" from {record['phone_number']} ({record['account']})"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -48,7 +50,7 @@ class SMS(IOSExtraction):
                 result["matched_indicator"] = ioc
                 self.detected.append(result)
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=SMS_BACKUP_IDS,
                                 root_paths=SMS_ROOT_PATHS)
         self.log.info("Found SMS database at path: %s", self.file_path)

mvt/ios/modules/mixed/sms_attachments.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 from base64 import b64encode
 
@@ -21,13 +22,14 @@ SMS_ROOT_PATHS = [
 class SMSAttachments(IOSExtraction):
     """This module extracts all info about SMS/iMessage attachments."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -36,7 +38,7 @@ class SMSAttachments(IOSExtraction):
                     f"with {record['total_bytes']} bytes (is_sticker: {record['is_sticker']}, has_user_info: {record['has_user_info']})"
         }
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=SMS_BACKUP_IDS,
                                 root_paths=SMS_ROOT_PATHS)
         self.log.info("Found SMS database at path: %s", self.file_path)

mvt/ios/modules/mixed/tcc.py

@@ -3,6 +3,7 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import logging
 import sqlite3
 from datetime import datetime
 
@@ -47,13 +48,14 @@ AUTH_REASONS = {
 class TCC(IOSExtraction):
     """This module extracts records from the TCC.db SQLite database."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         if "last_modified" in record:
             if "allowed_value" in record:
                 msg = f"Access to {record['service']} by {record['client']} {record['allowed_value']}"
@@ -66,7 +68,7 @@ class TCC(IOSExtraction):
                 "data": msg
             }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -156,7 +158,7 @@ class TCC(IOSExtraction):
         cur.close()
         conn.close()
 
-    def run(self):
+    def run(self) -> None:
         self._find_ios_database(backup_ids=TCC_BACKUP_IDS, root_paths=TCC_ROOT_PATHS)
         self.log.info("Found TCC database at path: %s", self.file_path)
         self.process_db(self.file_path)

mvt/ios/modules/mixed/webkit_resource_load_statistics.py

@@ -4,6 +4,7 @@
 #   https://license.mvt.re/1.1/
 
 import datetime
+import logging
 import os
 import sqlite3
 
@@ -22,15 +23,16 @@ class WebkitResourceLoadStatistics(IOSExtraction):
     """This module extracts records from WebKit ResourceLoadStatistics observations.db."""
     # TODO: Add serialize().
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = {} if not results else results
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -73,7 +75,7 @@ class WebkitResourceLoadStatistics(IOSExtraction):
         if len(self.results[key]) > 0:
             self.log.info("Extracted a total of %d records from %s", len(self.results[key]), db_path)
 
-    def run(self):
+    def run(self) -> None:
         if self.is_backup:
             try:
                 for backup_file in self._get_backup_files_from_manifest(relative_path=WEBKIT_RESOURCELOADSTATICS_BACKUP_RELPATH):
@@ -85,4 +87,4 @@ class WebkitResourceLoadStatistics(IOSExtraction):
                 self.log.info("Unable to search for WebKit observations.db: %s", e)
         elif self.is_fs_dump:
             for db_path in self._get_fs_files_from_patterns(WEBKIT_RESOURCELOADSTATICS_ROOT_PATHS):
-                self._process_observations_db(db_path=db_path, key=os.path.relpath(db_path, self.base_folder))
+                self._process_observations_db(db_path=db_path, key=os.path.relpath(db_path, self.target_path))