8 releases
0.0.8 | Jan 14, 2023 |
---|---|
0.0.7 | Jan 3, 2023 |
0.0.5 | Dec 25, 2022 |
#1313 in Parser implementations
26 downloads per month
36KB
969 lines
adblock-list-compiler
CLI tool to compile multiple adblock list sources into a single adblock list file
Introduction
An adblock dns server is a DNS server that resolves most domain names to ip address as usual, but also blocks known domain for ads.
In order to function properly, an adblock DNS server needs an adblock list, which is basically a list of domains blacklist that it should block. Depending on what service is used to run the DNS server, the adblock list file needs to be in a specific format.
Online, there are plenty of projects dedicated to maintaining freely available adblock list. While it is entirely possible to use just a single adblock list file online, most of the time we want to compile our own custom adblock list by combining multiple blacklist sources based on our individual needs. The online adblock list might also be written in specific formats, that we need to combine and adapt to the right format that we can use.
This is where this project comes in. This project helps you fetch adblock list from multiple sources, parses them, compiles them into a single list, deduplicates, and writes it into a single file with the right format. At the moment, only bind zone output files are supported, with plans to add support for other file formats.
This project was created mainly to support compiling adblock list of my Adblock DNS Server project, but you are free to use it in any other Adblock DNS project as you see fit.
Getting Started
Installation
You will need to have rust
and cargo
installed on your system.
Head over to rustup.sh to get that done.
Install the project with cargo
:
cargo install adblock-list-compiler
Check the version you have
ablc -v
# output
adblock-list-compiler 0.0.7
Configuration File
In order to start compiling and adblock list, you need to start with a valid configuration yaml file. To get started quickly, you can refer to the Adblock DNS Server example here.
A typical file should look as follows:
# file: configuration.yaml
blacklist:
- format: hosts
path: https://sebsauvage.net/hosts/hosts
- format: hosts
path: https://raw.githubusercontent.com/r-a-y/mobile-hosts/master/AdguardDNS.txt
- format: domains
path: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
- format: hosts
path: ./blacklist.d/ads_custom.hosts
whitelist:
- format: domains
path: ./whitelist.d/blogspot_whitelist.txt
- format: domains
path: ./whitelist.d/facebook_whitelist.txt
- format: domains
path: https://raw.githubusercontent.com/raghavdua1995/DNSlock-PiHole-whitelist/master/whitelist.list
overrides:
- format: cname
path: ./overrides.d/bing-safesearch.zone
- format: cname
path: ./overrides.d/duckduckgo-safesearch.zone
- format: cname
path: ./overrides.d/google-safesearch.zone
- format: cname
path: ./overrides.d/ignore-whitelist.zone
The value for path
can support both local file and web locations, and can also support relative paths.
Some adblock list online are published using either plain domain list, or hosts file.
This tool can adapt to both formats.
The blacklist are combined together and deduplicated.
The whitelist is used to exclude any legit domains that were somehow added to any of the blacklist.
The overrides are used mainly to rewrite some domains to an alternative hosts, mainly used to adapt forced safe search on some supported search engines.
Validating the config file
Run the following to make sure the config file can be parsed correctly:
ablc config check -c /path/to/configuration.yaml
It should say the file can be parsed correctly, and print out some debug messages of the configuration.
Compiling the adblock list
Run the following to compile your adblock list:
# Compile with your own configuration file
ablc compile -c /path/to/configuration.yaml -o blacklist.zone -f zone
# Try the compile with an upstream configuration file from the Adblock DNS Server project!
ablc compile -c https://raw.githubusercontent.com/ragibkl/adblock-dns-server/master/data/configuration.yaml -o blacklist.zone -f zone
Dependencies
~13–28MB
~507K SLoC