6 releases (breaking)
0.5.0 | Sep 9, 2024 |
---|---|
0.4.0 | Aug 9, 2024 |
0.3.2 | Jul 25, 2024 |
0.2.0 | Jun 29, 2024 |
0.1.0 | Apr 24, 2024 |
#568 in Command line utilities
53 downloads per month
525KB
5K
SLoC
URL Cleaner
Websites often put unique identifiers into URLs so that, when you send a link to a friend and they open it, the website knows it was you who sent it to them.
As most people do not understand and therefore cannot consent to this, it is polite to remove the spytext query parameters before sending URLs to people.
URL Cleaner is an extremely versatile tool designed to make this process as comprehensive, fast, and easy as possible.
C dependencies
These packages are required on Kubuntu 2024.04 (and probably therefore all Debian based distros.):
libssl-dev
for thehttp
feature flag.libsqlite3-dev
for thecaching
feature flag.
There are likely plenty more dependencies required that various Linux distros may or may not pre-install.
If you can't compile it I'll try to help you out. And if you can make it work on your own please let me know so I can add to this list.
Anonymity
In theory, if you're the only one on a website sharing posts from that website without URL trackers, the website could realize that and track you in the same way ("if no trackers then assume sender is Jared".).
In practise you are very unlikely to be the only one sharing clean URLs. Search engines generally provide URLs without trackers[citation needed], some people manually remove trackers, and some websites like vxtwitter.com automatically strip URL trackers.
However, for some websites the default config strips more stuff than search engines. In this case anonymity does fall back to many people using URL Cleaner and providing cover for each other.
As with Tor, protests, and anything else where privacy matters, safety comes in numbers.
Canonicalization
There is a vague notion of "canonical" URLs. One where every URL with its semantics can be converted into the canonical form with no external details.
For example, https://amazon.com/product-name-here/dp/PRODUCTID
can be "canonicalized" to https://amazon.com/dp/PRODUCTID
.
I have not yet decided if I want to remove the minimize
flag and/or just make it the default behavior.
Input on the matter would be appreciated.
Basic usage
By default, compiling URL Cleaner includes the default-config.json
file in the binary. Because of this, URL Cleaner can be used simply with url-cleaner "https://example.com/of?a=dirty#url"
.
The default config
The default config is intended to always obey the following rules:
- "Meaningful semantic changes"[definition?] should only ever occur as a result of a flag being enabled.
- Insignificant details like the item categories navbar on amazon listings being slightly different are insignificant.
- URLs that are "semantically valid"[definition?] shouldn't ever throw an error.
- URLs that aren't semantically valid also shouldn't ever throw an error but that is generally less important.
- URLs that are semantically valid should never change semantics and/or become semantically invalid.
- URLs that are semantically invalid may become semantically valid if there is an obvious way to do so. See the
unmangle
flag for details.
- URLs that are semantically invalid may become semantically valid if there is an obvious way to do so. See the
- Outside of long (>10)/infinite redirects/shortlinks, it should always be idempotent.
- Outside of redirect sites changing behavior, network connectivity issues, or other similarly difficult things to guarantee determinism for, it should always be deterministic.
- The
command
anddebug
features, as well as any features starting withexperiment-
/experimental-
are never expected to be enabled. Thecommand
feature is enabled by default for convenience but, for situations where untrusted/user-provided configs have a chance to be run, should be disabled.
Currently no guarantees are made, though when the above rules are broken it is considered a bug and I'd appreciate being told about it.
Additionally, these rules may be changed at any time for any reason. Usually just for clarification.
Flags
Flags let you specify behaviour with the --flag name --flag name2
command line syntax.
Various flags are included in the default config for things I want to do frequently.
no-https-upgrade
: Disable replacinghttp://
withhttps://
.no-http
: Don't make any HTTP requests.assume-1-dot-2-is-shortlink
: Treat all that match the Regex^.\...$
as shortlinks. Let's be real, they all are.no-unmangle
: Disable all unmangling.no-unmangle-host-is-http-or-https
: Don't converthttps://https//example.com/abc
tohttps://example.com/abc
.no-unmangle-path-is-url
: Don't converthttps://example1.com/https://example2.com/user
tohttps://example2.com/abc
.no-unmangle-path-is-url-encoded-url
: TODOno-unmangle-second-path-segment-is-url
: Don't converthttps://example1.com/profile/https://example2.com/profile/user
tohttps://example2.com/profile/user
.no-unmangle-subdomain-ends-in-not-subdomain
: Don't converthttps://profile.example.com.example.com
tohttps://profile.example.com
.no-unmangle-subdomain-starting-with-www-segment
: Don't converthttps://www.username.example.com
tohttps://username.example.com
.unmobile
: Converthttps://m.example.com
,https://mobile.example.com
,https://abc.m.example.com
, andhttps://abc.mobile.example.com
intohttps://example.com
andhttps://abc.example.com
.minimize
: Remove non-essential parts of that are likely not tracking related.youtube-unshort
: Turnshttps://youtube.com/shorts/abc
intohttps://youtube.com/watch?v=abc
.discord-unexternal
: Replaceimages-ext-1.discordapp.net
with the original images they refer to.discord-compatibility
: Sets the domain oftwitter.com
to the domain specified by thetwitter-embed-domain
variable.deadname-twitter
: Changex.com
totwitter.com
.breezewiki
: Sets the domain offandom.com
and BreezeWiki to the domain specified by thebreezewiki-domain
variable.unbreezewiki
: Turn BreezeWiki intofandom.com
.tor2web
: Append the suffix specified by thetor2web-suffix
variable to.onion
domains.tor2web2tor
: Replace**.onion.**
domains with**.onion
domains.bypass.vip
: Use bypass.vip to expand linkvertise links. Currently untestable as the API is down.
If a flag is enabled in a config's "params"
field, it can be disabled using --unflag flag1 --unflag flag1
.
Variables
Variables let you specify behaviour with the --var name value --var name2 value2
command line syntax.
Various variables are included in the default config for things that have multiple useful values.
twitter-embed-domain
: The domain to use for twitter when thediscord-compatibility
flag is specified. Defaults tovxtwitter.com
.breezewiki-domain
: The domain to use to turnfandom.com
and BreezeWiki into BreezeWiki. Defaults tobreezewiki.com
tor2web-suffix
: The suffix to append to the end of.onion
domains if the flagtor2web
is enabled. Should not start with.
as that's added automatically. Left unset by default.bypass.vip-api-key
: The API key used for bypass.vip's premium backend. Overrides theURL_CLEANER_BYPASS_VIP_API_KEY
environment variable.
If a variable is specified in a config's "params"
field, it can be unspecified using --unvar var1 --unvar var2
.
Environment variables
There are some things you don't want in the config, like API keys, that are also a pain to repeatedly insert via --var abc xyz
. For this, URL Cleaner does make use of native environment variables.
URL_CLEANER_BYPASS_VIP_API_KEY
: The API key used for bypass.vip's premium backend. Can be overridden with thebypass.vip-api-key
variable.
Sets
Sets let you check if a string is one of many specific strings very quickly.
Various sets are included in the default config.
https-upgrade-host-blacklist
: Hosts to not upgrade fromhttp
tohttps
even when theno-https-upgrade
flag isn't enabled.shortlink-hosts
: Hosts that are considered shortlinks in the sense that they return HTTP 3xx status codes. URLs with hosts in this set (as well as URLs with hosts that are "www." then a host in this set) will have theExpandShortLink
mapper applied.utps-host-whitelist
: Hosts to never remove universal tracking parameters from.utps
: The set of "universal tracking parameters" that are always removed for any URL with a host not in theutp-host-whitelist
set. Please note that theutps
common mapper in the default config also removes any parameter starting withcm_mmc
,__s
,at_custom
, andutm_
and thus parameters starting with those can be omitted from this set.unmangle-path-is-url-host-whitelist
: Effectively theno-unmangle-path-is-url
for specified hosts.unmangle-subdomain-ends-in-not-subdomain-not-subdomain-whitelist
: Effectivelyno-unmangle-subdomain-ends-in-not-subdomain-not-subdomain-whitelist
for specified not subdomains.breezewiki-hosts
: Hosts to replace with thebreezewiki-domain
variable when thebreezewiki
flag is enabled.fandom.com
is always replaced and is therefore not in this set.lmgtfy-hosts
: Hosts to replace withgoogle.com
.
Sets can have elements inserted into them using --insert-into-set name1 value1 value2 --insert-into-set name2 value3 value4
.
Sets can have elements removed from them using --remove-from-set name1 value1 value2 --remove-from-set name2 value3 value4
.
Lists
Lists allow you to iterate over strings for things like checking if another string contains any of them.
Currently only one list is included in the default config:
utp-prefixes
: If a query parameter starts with any of the strings in this list (such asutm_
) it is removed.
Currently there is no command line syntax for them. There really should be.
Citations
The people and projects I have stolen various parts of the default config from.
- Mozilla Firefox's Extended Tracking Protection's query stripping
- Brave Browser's query filter
- AdGuard's Tracking Parameters Filter
- FastForward
Custom rules
Although proper documentation of the config schema is pending me being bothered to do it, the url_cleaner
crate itself is reasonably well documented and the various types are (I think) fairly easy to understand.
The main files you want to look at are conditions.rs
and mappers.rs
.
Additionally url_part.rs
, string_location.rs
, and string_modification.rs
are very important for more advanced rules.
Footguns
There are various things in/about URL Cleaner that I or many consider stupid but for various reasons cannot/should not be "fixed". These include but are not limited to:
- For
UrlPart
s andMapper
s that use "suffix" semantics (the idea that the '.co.uk" in "google.co.uk" is semantically the same as the ".com" in "google.com"'), the psl crate is used which in turn uses Mozilla's Public Suffix List. Various suffixes are included that one may expect to be normal domains, such as blogspot.com. If for some reason a domain isn't working as expected, that may be the issue.
Reference for people who don't know Rust's syntax:
Option<...>
just means a value can benull
in the JSON.{"abc": "xyz"}
and{"abc": null}
are both valid states for aabc: Option<String>
field.Box<...>
has no bearing on JSON syntax or possible values. It's just used so Rust can put types inside themselves.Vec<...>
andHashSet<...>
are written as lists.HashMap<..., ...>
andHeaderMap
are written as maps.HeaderMap
keys are always lowercase.
u8
,u16
,u32
,u64
,u128
, andusize
are unsigned (never negative) integers.i8
,i16
,i32
,i64
,i128
, andisize
are signed (maybe negative) integers.usize
is au32
on 32-bit computers andu64
on 64-bit computers. Likewiseisize
isi32
andi64
under the same conditions. In practice, if a number makes sense to be used in a field then it'll fit.Duration
is written as{"secs": u64, "nanos": 0..1000000000}
.- If a field starts with
r#
you write it without ther#
(like"else"
).r#
is just Rust syntax for "this isn't a keyword". StringSource
,GlobWrapper
,RegexWrapper
,RegexParts
, andCommandWrapper
can be written as both strings and maps.RegexWrapper
andRegexParts
don't do any handling of/.../i
-style syntax.CommandWrapper
doesn't do any argument parsing.
#[serde(default)]
and#[serde(default = "...")]
allow for a field to be omitted when the desired value is almost always the same.- For
Option<...>
fields, the default isnull
. - For
bool
fields, the default isfalse
.
- For
#[serde(skip_serializing_if = "...")]
lets the--print-config
CLI flag omit unnecessary details (like when a field's value is its default value).#[serde(from = "...")]
,#[serde(into = "...")]
,#[serde(remote = "...")]
,#[serde(serialize_with = "...")]
,#[serde(deserialize_with = "...")]
, and#[serde(with = "...")]
are implementation details that can be mostly ignored.#[serde(remote = "Self")]
is a very strange way to allow a struct to be deserialized from a map or a string. See serde_with#702 for details.
Additionally, regex support uses the regex crate, which doesn't support look-around and backreferences.
Certain common regex operations are not possible to express without those, but this should never come up in practice.
MSRV
The Minimum Supported Rust Version is the latest stable release. URL Cleaner may or may not work on older versions, but there's no guarantee.
If this is an issue I'll do what I can to lower it but Diesel also keeps a fairly recent MSRV so you may lose caching.
Untrusted input
Although URL Cleaner has various feature flags that can be disabled to make handling untrusted input safer, no guarantees are made. Especially if the config file being used is untrusted.
That said, if you notice any rules that use but don't actually need HTTP requests or other data-leaky features, please let me know.
CLI
Parsing output
Note: JSON output is supported.
Unless a Debug
variant is used, the following should always be true:
-
Input URLs are a list of URLs starting with URLs provided as command line arguments then each line of the STDIN.
-
The nth line of STDOUT corresponds to the nth input URL.
-
If the nth line of STDOUT is empty, then something about reading/parsing/cleaning the URL failed.
-
The nth non-empty line of STDERR corresponds to the nth empty line of STDOUT.
- Currently empty STDERR lines are not printed when a URL succeeds. While it would make parsing the output easier it would cause visual clutter on terminals. While this will likely never change by default, parsers should be sure to follow 4 strictly in case this is added as an option.
JSON output
The --json
/-j
flag can be used to have URL Cleaner output JSON instead of lines.
The exact format is currently in flux.
Panic policy
URL Cleaner should only ever panic under the following circumstances:
-
Parsing the CLI arguments failed.
-
Loading/parsing the config failed.
-
Printing the config failed. (Shouldn't be possible.)
-
Testing the config failed.
-
Reading from/writing to STDIN/STDOUT/STDERR has a catastrophic error.
-
Running out of memory resulting in a standard library function/method panicking. This should be extremely rare.
-
(Only possible when the
debug
feature is enabled) The mutex controlling debug printing indenting is poisoned and a lock is attempted. This should only be possible when URL Cleaner is used as a library.
Outside of these cases, URL Cleaner should never panic. However as this is equivalent to saying "URL Cleaner has no bugs", no actual guarantees can be made.
Funding
URL Cleaner does not accept donations. If you feel the need to donate please instead donate to The Tor Project and/or The Internet Archive.
Dependencies
~7–25MB
~456K SLoC