27bd5f03a8
Merge pull request #359 from mvt-project/optimise-domain-checking
...
Optimise domain checking performance
2023-06-29 14:56:50 +02:00
3babbadc1d
Add docs for the profiling feature
2023-06-29 14:55:09 +02:00
41db117168
Improve performance when checking URLs and domains
...
Some MVT modules such as the WhatsApp module can be very slow as it was taking a naive approach to look for IOCs. The code was checking URLs (potentially more than 100k) against
1000's of IOC domains resulting in a quadratic run-time with hundreds of millions of comparisons as the number of IOCs increases.
This commit add an Aho-Corasick library which allows the efficient search in a string (the URL in this case) for all matches in set of keys (the IOCs). This data structure is perfect for this use case.
A quick measurement shows a 80% performance improvement for a WhatsApp database with 100k entries. The slow path is now the time spent fetching and expanding short URLs found in the database. This
can also be sped up significantly by fetching each URL asynchronously. This would require reworking modules to split the URL expansion from the IOC check so I will implement in a separate PR.
2023-06-29 14:14:44 +02:00
2b01ed7179
Add optional profiling for MVT modules
2023-06-29 13:31:13 +02:00
78d493b17e
Merge pull request #356 from mvt-project/auto/add-new-ios-releases
...
[auto] Update iOS releases and versions
2023-06-22 11:06:45 +02:00
473c80009b
Add new iOS versions and build numbers
2023-06-22 00:17:52 +00:00
a1481683e3
Adapts linter workflow to black
2023-06-14 01:05:14 +02:00
bdd36a9179
Merge pull request #349 from mvt-project/code-cleanup
...
Linted code using isort + autoflake + black
2023-06-08 21:12:34 +02:00
e1677639c4
Linted code using isort + autoflake + black, fixed wrong use of Optional[bool]
2023-06-01 23:40:26 +02:00
c2d740ed36
Handle better some empty database issues in iOS backups
2023-05-25 00:24:34 +02:00
d0e24c6369
Fixes a bug in the applications module
2023-05-24 12:04:03 +02:00
a1994079b1
Sort imports
2023-05-24 12:03:49 +02:00
289b7efdeb
Add missing iOS build numbers
2023-05-21 17:11:07 +01:00
166a63e14c
Merge pull request #347 from mvt-project/auto/add-new-ios-releases
...
[auto] Update iOS releases and versions
2023-05-21 17:54:25 +02:00
1b933fdb12
Add new iOS versions and build numbers
2023-05-21 15:53:45 +00:00
0c0ff7012b
Set branch number for auto-generated pull request
2023-05-21 16:52:47 +01:00
f9b0d07a81
Don't include information beta's in the version JSON
2023-05-21 16:49:14 +01:00
d14bcdd05f
Update title used in auto PR for new iOS versions
2023-05-21 16:47:56 +01:00
e026bb0a76
Fix path to script in workflow
2023-05-21 16:44:17 +01:00
253b4f031a
Allow workflow to be triggered manually
2023-05-21 16:42:54 +01:00
ec14297643
Merge pull request #345 from mvt-project/feature/auto-update-version-info
...
Add workflow to auto-update iOS builds and version numbers
2023-05-21 17:38:46 +02:00
3142d86edd
Fix path to include version JSON files in built package
2023-05-21 16:37:36 +01:00
c18998d771
Add version 16.5 to resolve merge conflict from main
2023-05-21 16:26:12 +01:00
22fd794fb8
Fix python style and setup.cfg syntax
2023-05-21 16:15:49 +01:00
27c5c76dc2
Add script and worker to auto-update build and version info
2023-05-21 16:09:50 +01:00
fafbac3545
Fix sorting of version numbers
2023-05-20 21:49:27 +01:00
bbfaadd297
Load iOS device and build information from a JSON file.
2023-05-20 21:24:14 +01:00
85abed55b6
Merge branch 'main' of github.com:mvt-project/mvt
2023-05-20 00:14:01 +02:00
2fbd7607ef
Adds latest iOS version
2023-05-20 00:11:16 +02:00
3787dc48cd
Fix bug where getprop sections where missing due to non-standard section header
2023-05-18 11:28:10 +02:00
f814244ff8
Fixes bug in bugreport getprop module
2023-05-06 11:20:10 -04:00
11730f164f
Fixes an issue in androidqf SMS module
2023-05-06 11:04:42 -04:00
912fb060cb
Fix error when creating report: csv.Error ( #341 )
2023-05-02 17:09:16 +02:00
a9edf4a9fe
Bumps version
2023-04-25 12:20:45 +02:00
ea7b9066ba
Improves iOS app detection
2023-04-25 11:21:55 +02:00
fd81e3aa13
Adds verbose mode
2023-04-25 11:13:46 +02:00
15477cc187
Bumps version
2023-04-13 17:59:05 +02:00
551b95b38b
Improves documentation
2023-04-13 16:11:55 +02:00
d767abb912
Fixes a bug in the calendar plugin
2023-04-13 13:21:33 +02:00
8a507b0a0b
Fixes a bug in WhatsApp iOS module
2023-04-13 09:26:52 +02:00
63b95ee6a5
Bumps version
2023-04-12 12:52:57 +02:00
c8ae495971
Extract all messages from SMS and WhatsApp ( #337 )
2023-04-12 12:39:25 +02:00
33d092692e
Adds calendar iOS plugin
2023-04-12 10:21:17 +02:00
b1e5dc715f
Adds latest iOS version
2023-04-07 22:22:44 +02:00
1dc1ee2238
Improves Indicator object
2023-04-07 15:07:45 +02:00
a2cbaacfce
Fixes hashing issue
2023-04-07 14:51:54 +02:00
801fe367ac
Improves WebkitResourceLoadStatistics module
2023-04-07 14:43:20 +02:00
0d653be4dd
Adds Applications iOS module
2023-04-07 14:10:24 +02:00
179b6976fa
Improves interactionc module
2023-04-07 12:25:30 +02:00
577fcf752d
Fixes issues in analytics module
2023-04-07 12:25:17 +02:00
Donncha Ó Cearbhaill