CalvinBackup/mvt
1
0
Fork 0
mirror of https://github.com/mvt-project/mvt.git synced 2026-06-13 10:17:51 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
c782d799745286ab4582af311a6b1b864afafcdf
mvt/src/mvt/common/url.py
T
besendorf c782d79974 V3 (#716) 
* Run bugreport and backup modules during check-androidqf

Adding support to automatically run ADB backup and bugreport modules
automatically when running the check-androidqf command. This is a first
step to deduplicate the code for Android modules.

* Deduplicate modules which are run by the sub-commands.

* Raise the proper NoAndroidQFBackup exception when a back-up isn't found

* Remove check-adb command and update docs

* Remove check-apk code and old dependencies

* Major refactor to add structured alerting and typed indicators

This commit makes a structural change to MVT by changing binary
detected/not detected logic into a structured multi-level system
of alerts. This gives far more power to extend MVT and manage
alerts.

This commit also begins the process of adding proper typing for
key objects used in MVT including Indicators, IndicatorMatches,
and ModuleResults. This will also be keep to programmatically using
the output of MVT.

* Fix up, remove ADB module base

* Rework old detections tracking into stuctured alert levels

* Quote STIX path in log line

* Fix profile events log line

* Close open archive (zip/tar) file handles

* Fix root_binaries and mounts modules to use alertstore

* Update tests to use alertstore instead of detected attribute

* Fix alertstore method calls - use high() instead of warning()

* Fix remaining test errors

- Add log_latest() call in root_binaries to log each alert
- Fix UnboundLocalError in cmd_check_androidqf by initializing bugreport variable
- Remove incorrect backup.close() call since load_backup() returns bytes
- Remove duplicate from_ab method in cmd_check_backup that was using old attributes

* Log alerts on add

* Remove slug from alertstore calls

* update alerts.py

* update alerts.py

* move indicator_match to alert object

* .

* - Remove timeline_detected and route to alertstore

* fix typing for mypy

* Remove unused type imports

* Fix check_receiver_prefix and check_android_property_name

- check_receiver_prefix() used dict syntax (ioc["value"]) on Indicator
  dataclass objects from get_iocs(). Changed to ioc.value/ioc.name.
- check_receiver_prefix() returned raw ioc instead of IndicatorMatch.
  Now returns IndicatorMatch with descriptive message.
- Fixed return type annotations on both methods to Optional[IndicatorMatch].
- Removed unused Union import.

* Fix residual self.detected usage in packages and dumpsys_receivers

These modules still used self.detected.append() which no longer exists
after the alertstore migration. Converted to alertstore calls:
- packages.py: ROOT_PACKAGES detection → alertstore.high()
- dumpsys_receivers.py: receiver IOC match → alertstore.critical()

* Fix SMS module alertstore.high() call passing slug as message

The first argument was self.get_slug() (module slug) instead of a
human-readable message. The module is already auto-detected via
AlertStore._get_calling_module(). Also removed redundant log_latest().

* Apply suggestions from code review

Fix JSON serialization in `module.save_to_json` and fix argument order in iOS alertstore calls.

Co-authored-by: tes <tesitura@users.noreply.github.com>

* Remove unsupported ADB modules

* Fail removed check-adb command

* Fix alert serialization and logging

* Close sqlite connections in iOS modules

* Fix DEBUG messages not reaching handlers, save_to_json for dictionary results and TypeError on mixed event_time types in safary_history

* add matched_indicator via alertstore instead of directly modifying json objects

* Alert on battery daily uninstall and downgrade

* Lower alert severity to medium for suspicious items

* Switch version to 2026.4.28 CalVer

---------

Co-authored-by: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
Co-authored-by: tes <tesitura@users.noreply.github.com>
Co-authored-by: Janik Besendorf <janik.besendorf@reporter-ohne-grenzen.de>
2026-04-29 14:32:29 +02:00

383 lines
6.7 KiB
Python
Raw Blame History

# Mobile Verification Toolkit (MVT)
# Copyright (c) 2021-2023 The MVT Authors.
# Use of this software is governed by the MVT License 1.1 that can be found at
# https://license.mvt.re/1.1/
from typing import Optional
import requests
from tld import get_tld
SHORTENER_DOMAINS = [
"0rz.tw",
"1drv.ms",
"1link.in",
"1url.com",
"2big.at",
"2.gp",
"2pl.us",
"2tu.us",
"2ya.com",
"3.ly",
"4sq.com",
"4url.cc",
"6url.com",
"7.ly",
"a2a.me",
"abbrr.com",
"adf.ly",
"adjix.com",
"a.gg",
"alturl.com",
"a.nf",
"anon.to",
"apple.news",
"atu.ca",
"b23.ru",
"bacn.me",
"bc.vc",
"bfy.tw",
"binged.it",
"bit.do",
"bit.ly",
"bizj.us",
"bkite.com",
"bloat.me",
"budurl.com",
"buff.ly",
"buk.me",
"burnurl.com",
"chilp.it",
"chn.ge",
"clck.ru",
"clickmeter.com",
"cli.gs",
"c-o.in",
"cort.as",
"cut.ly",
"cutt.ly",
"cuturl.com",
"dai.ly",
"dailym.ai",
"db.tt",
"decenturl.com",
"dfl8.me",
"digbig.com",
"digg.com",
"disq.us",
"dlvr.it",
"doiop.com",
"do.my",
"dwarfurl.com",
"dy.fi",
"easyuri.com",
"easyurl.net",
"eepurl.com",
"esyurl.com",
"ewerl.com",
"fa.b",
"fa.by",
"fb.me",
"fff.to",
"ff.im",
"fhurl.com",
"fire.to",
"firsturl.de",
"firsturl.net",
"flic.kr",
"flq.us",
"fly2.ws",
"fon.gs",
"forms.gle",
"fwd4.me",
"gdurl.com",
"gg.gg",
"gl.am",
"go2cut.com",
"go2.me",
"go.9nl.com",
"goo.gl",
"goshrink.com",
"got.by",
"gowat.ch",
"gri.ms",
"gurl.es",
"hellotxt.com",
"hex.io",
"hongkiat.shorturl.com",
"hover.com",
"href.in",
"ht.ly",
"htxt.it",
"hugeurl.com",
"hurl.it",
"hurl.me",
"hurl.ws",
"ibb.co",
"icanhaz.com",
"idek.net",
"inreply.to",
"iscool.net",
"is.gd",
"iterasi.net",
"jijr.com",
"j.mp",
"jmp2.net",
"just.as",
"kissa.be",
"kl.am",
"klck.me",
"korta.nu",
"krunchd.com",
"lat.ms",
"liip.to",
"liltext.com",
"lin.cr",
"linkbee.com",
"linkbun.ch",
"liurl.cn",
"lnkd.in",
"lnk.gd",
"lnk.in",
"ln-s.net",
"ln-s.ru",
"loopt.us",
"lru.jp",
"lt.tl",
"lurl.no",
"lyhyt.eu",
"metamark.net",
"migre.me",
"minilien.com",
"miniurl.com",
"minurl.fr",
"moourl.com",
"myurl.in",
"nbcnews.to",
"ne1.net",
"njx.me",
"nn.nf",
"notlong.com",
"n.pr",
"nsfw.in",
"nyti.ms",
"om.ly",
"onforb.es",
"on.mktw.net",
"ow.ly",
"o-x.fr",
"pca.st",
"pd.am",
"pic.gd",
"ping.fm",
"piurl.com",
"pnt.me",
"politi.co",
"poprl.com",
"posted.at",
"post.ly",
"profile.to",
"q.gs",
"qicute.com",
"qlnk.net",
"qr.ae",
"qte.me",
"quip-art.com",
"rb6.me",
"rb.gy",
"read.bi",
"redir.ec",
"redirx.com",
"redr.me",
"reut.rs",
"rickroll.it",
"r.im",
"ri.ms",
"riz.gd",
"rsmonkey.com",
"rubyurl.com",
"ru.ly",
"s7y.us",
"safe.mn",
"sharein.com",
"sharetabs.com",
"shorl.com",
"short.ie",
"shortlinks.co.uk",
"shortna.me",
"short.to",
"shorturl.at",
"shorturl.com",
"shoturl.us",
"shout.to",
"shrinkify.com",
"shrinkster.com",
"shrten.com",
"shrt.st",
"shrunkin.com",
"shw.me",
"simurl.com",
"smsh.me",
"sn.im",
"snipr.com",
"snipurl.com",
"snurl.com",
"sp2.ro",
"spedr.com",
"sqrl.it",
"starturl.com",
"sturly.com",
"su.pr",
"t.cn",
"t.co",
"tcrn.ch",
"tgr.ph",
"thrdl.es",
"tighturl.com",
"tiny123.com",
"tinyarro.ws",
"tiny.cc",
"tinylink.in",
"tiny.pl",
"tiny.tw",
"tinytw.it",
"tinyuri.ca",
"tinyurl.com",
"tinyvid.io",
"t.me",
"tnij.org",
"tnw.to",
"togoto.us",
"to.ly",
"traceurl.com",
"tr.im",
"tr.my",
"turo.us",
"tweetburner.com",
"twirl.at",
"twit.ac",
"twitterpan.com",
"twitthis.com",
"twiturl.de",
"twurl.cc",
"twurl.nl",
"u6e.de",
"ub0.cc",
"ukl.me.uk",
"u.mavrev.com",
"u.nu",
"updating.me",
"ur1.ca",
"url4.eu",
"urlao.com",
"urlbrief.com",
"url.co.uk",
"urlcover.com",
"urlcut.com",
"urlenco.de",
"urlhawk.com",
"url.ie",
"urlkiss.com",
"urlot.com",
"urlpire.com",
"urlx.ie",
"urlx.org",
"urlzen.com",
"use.my",
"u.to",
"v.gd",
"virl.com",
"vl.am",
"vurl.com",
"vzturl.com",
"w3t.org",
"wapo.st",
"wapurl.co.uk",
"wipi.es",
"wp.me",
"xaddr.com",
"x.co",
"xeeurl.com",
"xr.com",
"xrl.in",
"xrl.us",
"x.se",
"xurl.es",
"xurl.jp",
"xzb.cc",
"ye.pe",
"yep.it",
"yfrog.com",
"yhoo.it",
"ymlp.com",
"yuarel.com",
"yweb.com",
"zi.ma",
"zi.pe",
"zipmyurl.com",
"zurl.to",
"zurl.ws",
"zz.gd",
]
class URL:
def __init__(self, url: str) -> None:
if isinstance(url, bytes):
url = url.decode()
self.url = url
self.domain = self.get_domain()
self.top_level = self.get_top_level()
self.is_shortened = False
def get_domain(self) -> str:
"""Get the domain from a URL.
:returns: Domain name extracted from URL
:rtype: str
"""
tld_obj = get_tld(self.url, as_object=True, fix_protocol=True)
if isinstance(tld_obj, str):
return tld_obj
if tld_obj is None:
return ""
return tld_obj.parsed_url.netloc.lower().lstrip("www.")
def get_top_level(self) -> str:
"""Get only the top-level domain from a URL.
:returns: Top-level domain name extracted from URL
:rtype: str
"""
tld_obj = get_tld(self.url, as_object=True, fix_protocol=True)
if isinstance(tld_obj, str):
return tld_obj
if tld_obj is None:
return ""
return tld_obj.fld.lower()
def check_if_shortened(self) -> bool:
"""Check if the URL is among list of shortener services.
:returns: True if the URL is shortened, otherwise False
:rtype: bool
"""
if self.domain.lower() in SHORTENER_DOMAINS:
self.is_shortened = True
return self.is_shortened
def unshorten(self) -> Optional[str]:
"""Unshorten the URL by requesting an HTTP HEAD response."""
res = requests.head(self.url)
if str(res.status_code).startswith("30"):
return res.headers["Location"]
return ""
Reference in New Issue View Git Blame Copy Permalink