Blockliste für Router/pi-hole, Adblock, etc

Antworten
Benutzeravatar
benny
Forum Junkie
Beiträge: 471
Registriert: 11.06.2016 - 11:51
Hat sich bedankt: 12 Mal
Geschlecht:

Blockliste für Router/pi-hole, Adblock, etc

Beitrag von benny »

Ich betreibe seit Jahren eine eigene, kleine Liste um mir unliebsame Verbindungen und Werbung aus meinen Netzwerken vom Leibe zu halten.
Die vollständige Blockliste liegt dann auf meinem Server unter https://hosts.servername.de/hosts.txt (Leider kann ich nicht die richtige URL angeben)
Diese aktualisiere ich bei jedem anmelden an meinem Rechner automatisch mittels einem leicht veränderten Script blocklist_update.sh:

Code: Alles auswählen

#!/bin/bash
#blocklist_update.sh

# First run? Save a copy of the system's original hosts file and set to read-only for safety
if [ ! -f ~/hosts-system ]
then
 echo "Saving copy of actual hosts file..."
 cp /home/user/hosts.txt ~/hosts-system
 chmod 444 ~/hosts-system
fi

# Perform work in temporary files
temphosts1=`mktemp`
temphosts2=`mktemp`
temphosts3=`mktemp`

# Obtain various hosts files and merge into one
echo "Downloading ad-blocking hosts files..."
wget -nv -O - "http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=1&mimetype=plaintext" >> $temphosts1
wget -nv -O - http://www.hostsfile.org/Downloads/hosts.txt >> $temphosts1
wget -nv -O - http://www.mvps.org/winhelp2002/hosts.txt >> $temphosts1
wget -nv -O - http://adaway.org/hosts.txt >> $temphosts1
wget -nv -O - http://hosts-file.net/ad_servers.asp >> $temphosts1
wget -nv -O - --no-check-certificate http://hosts.servername.de/hosts_custom.txt >> $temphosts1
echo

# Do some work on the file:
# 1. Remove MS-DOS carriage returns
# 2. Delete all lines that don't begin with 127.0.0.1 or 0.0.0.0 or 85.13.135.179 (servername.de)
# 3. Delete any lines containing the word localhost because we'll obtain that from the original hosts file
# 4. Replace 127.0.0.1 with 0.0.0.0 because then we don't have to wait for the resolver to fail 
# 5. Scrunch extraneous spaces separating address from name into a single tab
# 6. Delete any comments on lines
# 7. Clean up leftover trailing blanks
# Pass all this through sort with the unique flag to remove duplicates and save the result
echo "Parsing, cleaning, de-duplicating, sorting..."
echo
sed -e 's/\r//' -e '/^0.0.0.0\|^127.0.0.1\|^85.13.135.179/!d' -e '/localhost/d' -e 's/127.0.0.1/0.0.0.0/' -e 's/ \+/\t/' -e 's/#.*$//' -e 's/[ \t]*$//' < $temphosts1 | sort -u > $temphosts2

# Combine system hosts with adblocks
echo Merging with original system hosts...
echo -e "\n# Ad blocking hosts generated "`date` | cat ~/hosts-system - $temphosts2 > $temphosts3
echo
echo "de-duplicating once more"
sort -u $temphosts3 > /home/user/hosts.txt
echo

# Clean up temp files and remind user to copy new file
echo "Cleaning up..."
rm $temphosts1 $temphosts2 $temphosts3
echo
echo "Done."
echo
echo
echo "Copy ad-blocking hosts file to hosts.servername.de/hosts.txt"
echo
# Hochladen
lftp -u Password ftp://hosts.servername.de <<EOF
set ssl:check-hostname false;
set ftp:ssl-force true
set ftp:ssl-protect-data true
mput /home/user/hosts.txt 
exit
EOF

echo
echo "Complete..."
Das Script nimmt bekannte Blocklisten auf und erweitert sie durch Einträge die ich individuell in der Datei hosts_custom.txt auf meinem Server hoste.

Code: Alles auswählen

#hosts_custom.txt
# Zusatz user

# Facebook SDK die viele APPS zu facebook Daten senden lässt
127.0.0.1   graph.facebook.com

127.0.0.1   functionalclam.com
127.0.0.1   ggpht.com
127.0.0.1   ads.samsungads.com
127.0.0.1   samsungads.com
127.0.0.1   prod-kami.wuaki.tv
127.0.0.1   prod-tvplus-pmd.akamai.cdn.wuaki.tv
127.0.0.1   config.samsungads.com
127.0.0.1   gpm.samsungqbe.com
127.0.0.1   log-config.samsungacr.com
127.0.0.1   samsung.com
127.0.0.1   samsungacr.com
127.0.0.1   samsungcloudsolution.com
127.0.0.1   samsungcloudsolution.net
127.0.0.1   samsungotn.net
127.0.0.1   www.samsungotn.net
127.0.0.1   shop.rewe.de
127.0.0.1   shop.rewe-static.de
127.0.0.1   lcprd1.samsungcloudsolution.net
127.0.0.1   keymaker.synology.com
127.0.0.1   info.cspserver.net
127.0.0.1   gld.push.samsungosp.com
127.0.0.1   eu-odc.tizenstore.com
127.0.0.1   esdk-ffl.spotify.com
127.0.0.1   configprd.samsungcloudsolution.net
127.0.0.1   checkipv6.synology.com
127.0.0.1   checkip.synology.com
127.0.0.1   checkip.dyndns.org
127.0.0.1   bip.presage.io
127.0.0.1   ap-odc.tizenstore.com
Auf unseren Mobiltelefonen läuft ein Adaway welches die Blocklist enthält und somit auch außerhalb des eigenen Netzes funktioniert.
Zu Hause erledigt das ein pi-hole oder ein Script, das in einem DD-WRT läuft.

Code: Alles auswählen

#DD-WRT
#!/bin/sh
logger WAN up script executing
if test -s /tmp/hosts0
then
        rm /tmp/hosts0
fi

sleep 3
logger Downloading http://server.de/hosts.txt
wget -O - http://server.de/hosts.txt > /tmp/hosts0

grep addn-hosts /tmp/dnsmasq.conf || echo "addn-hosts=/tmp/hosts0" >>/tmp/dnsmasq.conf
logger Restarting DNSMasq
killall dnsmasq
dnsmasq --conf-file=/tmp/dnsmasq.conf
Oder auf OpenWRT in /etc/

Auszug der Entwickler:
The script must be copied to an OpenWRT router (gargoyle firmware works fine, too).

For example, if the router is located at 192.168.1.1:

# scp adblock.sh root@192.168.1.1:/etc/adblock.sh

Make the script executable:

# chmod +x /etc/adblock.sh

Basic usage

If you are running the script for the first time:

# sh /etc/adblock.sh -f

There should be status updates in the output, but there should be no errors. If these commands complete without errors, the adblocking is active. You can test it by looking up, say, google analytics.
Das Script wie ich es vor pi-hole verwandte:

Code: Alles auswählen

#!/bin/sh
#Put in /etc/adblock.sh
#Block ads, malware, etc.

#### CONFIG SECTION ####

# Only block wireless ads? Y/N
ONLY_WIRELESS="N"

# IPv6 support? Y/N
IPV6="Y"

# Need SSL websites?
SSL="Y"

# Try to transparently serve pixel response?
#   If enabled, understand the consequences and mechanics of this setup
TRANS="N"

# Exempt an ip range
EXEMPT="N"
START_RANGE="192.168.1.0"
END_RANGE="192.168.1.255"

# Redirect endpoint
ENDPOINT_IP4="0.0.0.0"
ENDPOINT_IP6="::"

#Change the cron command to what is comfortable, or leave as is
CRON="0 4 * * 0,3 sh /etc/adblock.sh"

#### END CONFIG ####

#### FUNCTIONS ####

cleanup()
{
    #Delete files used to build list to free up the limited space
    echo 'Cleaning up...'
    rm -f /tmp/block.build.list
    rm -f /tmp/block.build.before
}

install_dependencies()
{
    #Need iptables-mod-nat-extra installed
    if opkg list-installed | grep -q iptables-mod-nat-extra
    then
        echo 'iptables-mod-nat-extra is installed!'
    else
        echo 'Updating package list...'
        opkg update > /dev/null
        echo 'Installing iptables-mod-nat-extra...'
        opkg install iptables-mod-nat-extra > /dev/null
    fi

    #Need iptable-mod-iprange for exemption
    if [ "$EXEMPT" = "Y" ]
    then 
        if opkg list-installed | grep -q iptables-mod-iprange
        then
            echo 'iptables-mod-iprange installed'
        else
            echo 'Updating package list...'
            opkg update > /dev/null
            echo 'Installing iptables-mod-iprange...'
            opkg install iptables-mod-iprange > /dev/null
        fi
    fi

    #Need wget for https websites
    if [ "$SSL" = "Y" ]
    then
        if opkg list-installed wget | grep -q wget
        then
            if wget --version | grep -q +ssl
            then
                echo 'wget (with ssl) found'
            else
                # wget without ssl, need to reinstall full wget
                opkg update > /dev/null
                opkg install wget --force-reinstall > /dev/null
            fi
        else
            echo 'Updating package list...'
            opkg update > /dev/null
            echo 'Installing wget (with ssl)...'
            opkg install wget > /dev/null
        fi
    fi
}

add_config()
{
    if [ "$ONLY_WIRELESS" = "Y" ]
    then
        echo 'Wireless only blocking!'
        if [ "$EXEMPT" = "Y" ]
        then
            echo 'Exempting some ips...'
            FW1="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -i wlan+ -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -i wlan+ -p udp --dport 53 -j REDIRECT --to-ports 53"
        else
            FW1="iptables -t nat -I PREROUTING -i wlan+ -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -i wlan+ -p udp --dport 53 -j REDIRECT --to-ports 53"
        fi
    else
        if [ "$EXEMPT" = "Y" ]
        then
            echo "Exempting some ips..."
            FW1="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -p udp --dport 53 -j REDIRECT --to-ports 53"
        else
            FW1="iptables -t nat -I PREROUTING -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 53"
        fi
    fi

    echo 'Updating config...'

    #Update DHCP config
    uci add_list dhcp.@dnsmasq[0].addnhosts=/etc/block.hosts > /dev/null 2>&1 && uci commit

    #Add to crontab
    echo "$CRON" >> /etc/crontabs/root

    #Update dnsmasq config for Tor
    TOR=`uci get tor.global.enabled 2> /dev/null`
    if [ "$TOR" == "1" ]
    then
        TORPORT=`uci get tor.client.dns_port`
        TORIP="127.0.0.1:$TORPORT"
        uci set dhcp.@dnsmasq[0].noresolv='1' > /dev/null &2>1 && uci commit
        uci add_list dhcp.@dnsmasq[0].server="$TORIP" > /dev/null &2>1 && uci commit
    fi
    
    # Add firewall rules
    echo "$FW1" >> /etc/firewall.user
    echo "$FW2" >> /etc/firewall.user

    # Provide hint if localservice is 1
    LS=`uci get dhcp.@dnsmasq[0].localservice 2> /dev/null`
    if [ "$LS" == "1" ]
    then
        echo "HINT: localservice is set to 1"
        echo "    Adblocking (and router DNS) over a VPN may not work"
        echo "    To allow VPN router DNS, manually set localservice to 0"
    fi

    
    # Determining uhttpd/httpd_gargoyle for transparent pixel support
    if [ "$TRANS" = "Y" ]
    then
        if [ ! -e "/www/1.gif" ]
        then
            /usr/bin/wget -O /www/1.gif http://upload.wikimedia.org/wikipedia/commons/c/ce/Transparent.gif  > /dev/null
        fi
        if [ -s "/usr/sbin/uhttpd" ]
        then
            #The default is none, so I don't want to check for it, so just write it
            echo "uhttpd found..."
            echo "updating server error page to return transparent pixel..."
            uci set uhttpd.main.error_page="/1.gif" && uci commit
        elif [ -s "/usr/sbin/httpd_gargoyle" ]
        then
            # Write without testing
            echo "httpd_gargoyle found..."
            echo "updating server error page to return transparent pixel..."
            uci set httpd_gargoyle.server.page_not_found_file="1.gif" && uci commit
        else
            echo "Cannot find supported web server..."
        fi
    fi
}

update_blocklist()
{
    #Delete the old block.hosts to make room for the updates
    rm -f /etc/block.hosts

    # Correct endpoint for transparent pixel response
    if [ "$TRANS" = "Y" ] && [ -e "/www/1.gif" ] && ([ -s "/usr/sbin/uhttpd" ] || [ -s "/usr/sbin/httpd_gargoyle" ])
    then
        ENDPOINT_IP4=$(uci get network.lan.ipaddr)
        if [ "$IPV6" = "Y" ]
        then
            ENDPOINT_IP6=$(uci get network.lan6.ipaddr)
        fi
    fi
    
    echo 'Downloading hosts lists...'
    #Download and process the files needed to make the lists (enable/add more, if you want)
    echo 'https://hosts.server.de/hosts.txt'
    wget -qO- --no-check-certificate "https://hosts.server.de/hosts.txt"| awk -v r="$ENDPOINT_IP4" '{sub(/^0.0.0.0/, r)} $0 ~ "^"r' > /tmp/block.build.list
    echo 'http://winhelp2002.mvps.org/hosts.txt'
    wget -qO- --no-check-certificate "http://winhelp2002.mvps.org/hosts.txt"| awk -v r="$ENDPOINT_IP4" '{sub(/^0.0.0.0/, r)} $0 ~ "^"r' >> /tmp/block.build.list
    echo 'http://adaway.org/hosts.txt'
    wget -qO- --no-check-certificate "http://adaway.org/hosts.txt"| awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
    #wget -qO- http://www.malwaredomainlist.com/hostslist/hosts.txt|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
    #wget -qO- "http://hosts-file.net/.\ad_servers.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list

    #Add black list, if non-empty
    if [ -s "/etc/black.list" ]
    then
        echo 'Adding blacklist...'
        awk -v r="$ENDPOINT_IP4" '/^[^#]/ { print r,$1 }' /etc/black.list >> /tmp/block.build.list
    fi

    echo 'Sorting lists...'

    #Sort the download/black lists
    awk '{sub(/\r$/,"");print $1,$2}' /tmp/block.build.list|sort -u > /tmp/block.build.before

    #Filter (if applicable)
    if [ -s "/etc/white.list" ]
    then
        #Filter the blacklist, supressing whitelist matches
        #  This is relatively slow =-(
        echo 'Filtering white list...'
        egrep -v "^[[:space:]]*$" /etc/white.list | awk '/^[^#]/ {sub(/\r$/,"");print $1}' | grep -vf - /tmp/block.build.before > /etc/block.hosts
    else
        cat /tmp/block.build.before > /etc/block.hosts
    fi

    if [ "$IPV6" = "Y" ]
    then
        safe_pattern=$(printf '%s\n' "$ENDPOINT_IP4" | sed 's/[[\.*^$(){}?+|/]/\\&/g')
        safe_addition=$(printf '%s\n' "$ENDPOINT_IP6" | sed 's/[\&/]/\\&/g')
        echo 'Adding ipv6 support...'
        sed -i -re "s/^(${safe_pattern}) (.*)$/\1 \2\n${safe_addition} \2/g" /etc/block.hosts
    fi
}

restart_firewall()
{
    echo 'Restarting firewall...'
    if [ -s "/usr/lib/gargoyle/restart_firewall.sh" ]
    then
        /usr/lib/gargoyle/restart_firewall.sh > /dev/null 2>&1
    else
        /etc/init.d/firewall restart > /dev/null 2>&1
    fi
}

restart_dnsmasq()
{
    if [ "$1" -eq "0" ]
    then
        echo 'Re-reading blocklist'
        killall -HUP dnsmasq
    else
        echo 'Restarting dnsmasq...'
        /etc/init.d/dnsmasq restart
    fi
}

restart_http()
{
    if [ -s "/usr/sbin/uhttpd" ]
    then
        echo 'Restarting uhttpd...'
        /etc/init.d/uhttpd restart
    elif [ -s "/usr/sbin/httpd_gargoyle" ]
    then
        echo 'Restarting httpd_gargoyle...'
        /etc/init.d/httpd_gargoyle restart
    fi
}
restart_cron()
{
    echo 'Restarting cron...'
    /etc/init.d/cron restart > /dev/null 2>&1
}

remove_config()
{
    echo 'Reverting config...'

    # Remove addnhosts
    uci del_list dhcp.@dnsmasq[0].addnhosts=/etc/block.hosts > /dev/null 2>&1 && uci commit

    # Remove cron entry
    sed -i '/adblock/d' /etc/crontabs/root

    # Remove firewall rules
    sed -i '/--to-ports 53/d' /etc/firewall.user
    
    # Remove Tor workarounds
    uci del_list dhcp.@dnsmasq[0].server > /dev/null 2>&1 && uci commit
    uci set dhcp.@dnsmasq[0].noresolv='0' > /dev/null 2>&1 && uci commit

    # Remove proxying
    uci delete uhttpd.main.error_page > /dev/null 2>&1 && uci commit
    uci set httpd_gargoyle.server.page_not_found_file="login.sh" > /dev/null 2>&1 && uci commit
}


toggle()
{
    # Check for cron as test for on/off
    if grep -q "adblock" /etc/crontabs/root
    then
        # Turn off
        echo 'Turning off!'
        remove_config
    else
        # Turn on
        echo 'Turning on!'
        add_config
    fi

    # Restart services
    restart_firewall
    restart_dnsmasq 1
    restart_http
    restart_cron
}

#### END FUNCTIONS ####

### Options parsing ####

case "$1" in
    # Toggle on/off
    "-t")
        toggle
        ;;
    #First time run
    "-f")
        install_dependencies
        add_config
        update_blocklist
        restart_firewall
        restart_dnsmasq 1
        restart_http
        restart_cron
        cleanup
        ;;
    #Reinstall
    "-r")
        remove_config
        install_dependencies
        add_config
        update_blocklist
        restart_firewall
        restart_dnsmasq 1
        restart_http
        restart_cron
        cleanup
        ;;
    #Default updates blocklist only
    *)
        update_blocklist
        restart_dnsmasq 0
        cleanup
        ;;
esac

#### END OPTIONS ####

exit 0
Auf dem Router gebe ich die IP des pi-hole an alle DHCP Clients weiter und habe eine zusätzliche Regel in die Firewall gebaut, damit nicht andere DNS-Server gefragt werden können.

Code: Alles auswählen

# DNS zum Pi-Hole umbiegen ausser vom pi-hole selbst
iptables -t nat -A PREROUTING ! -s IP_vom_pi-hole -p udp --dport 53 -j DNAT --to IP_vom_pi-hole
iptables -t nat -A PREROUTING ! -s IP_vom_pi-hole -p tcp --dport 53 -j DNAT --to IP_vom_pi-hole
Bis dann,
benny

PS: Ich werde in Zukunft keinen Support mehr per PN leisten. Bitte eröffnet ein Thema im Hilfebereich, oder wo auch immer eure Fragen hin passen.
Nach oben
Irino
Anfänger
Beiträge: 5
Registriert: 16.09.2022 - 11:10
Danksagung erhalten: 1 Mal
Geschlecht:

Re: Blockliste für Router/pi-hole, Adblock, etc

Beitrag von Irino »

Das ist ne mega gute Idee. Danke für die Info.
Nach oben
Benutzeravatar
benny
Forum Junkie
Beiträge: 471
Registriert: 11.06.2016 - 11:51
Hat sich bedankt: 12 Mal
Geschlecht:

Re: Blockliste für Router/pi-hole, Adblock, etc

Beitrag von benny »

Ich habe mittlerweile eine etwas andere Version laufen.
Auf meinem Rechner läuft nun:
blocklist.sh

Code: Alles auswählen

#!/bin/bash
###################################################################
#
# Dateien:
#   blocksites.txt - Blocklist URLs (eine URL pro Zeile)
#   domainlist.txt - Optionale Liste mit manuell hinzugefügten Einträgen
# Erzeugte Datei:
#   hosts.txt - Liste aller zu blockenden Domains
#
###################################################################
#
BLOCKSITES="blocksites.txt"
DOMAINLIST="domainlist.txt"
BLOCKLIST="merged_blocklist.txt"
PURGEDLIST="purged_blocklist.txt"
HOSTFILE="hosts.txt"

{

echo "# Letzte aktualisierung: $(date)"
echo "Lade ad-blocking hosts Dateien herunter..."
while  IFS='\n' read -r URL; do # lese block sites
  {
  echo "# Von: ${URL}"
  curl ${URL}
  echo                          
  }                            
done < "$BLOCKSITES"            

if [ -f "$DOMAINLIST" ]
then
  cat "$DOMAINLIST"
fi

} > "$BLOCKLIST"

echo "ausführen, säubern, Dubletten entfernen, sortieren..."
# Arbeiten für die Datei:
# 1. Entferne MS-DOS carriage returns
# 2. Lösche alle Zeilen die mit einem Kommentar (#) beginnen
# 3. Lösche alle IP-ähnlichen Zahlen wie 127.0.0.1 1.2.3.4
# 4. Lösche localhost am Anfang
# 5. Lösche alle leerzeilen
# 6. Lösche alle verbliebenen Leerzeichen am Zeilenanfang
# 7. 0.0.0.0 an jeden Zeilenanfang schreiben

sed 's/\r//;/^#/d;/^$/d;s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}//g;s/^localhost//g;s/^ *//g;s/[ \t]*$//;s/^/0.0.0.0	/' $BLOCKLIST > $PURGEDLIST

# Lösche Dubletten und sortiere
awk '!seen[$0]++' $PURGEDLIST | sort -u  > $HOSTFILE

echo "Schreibe Datum in die Datei"
echo "#Erstellt um "`date +%H:%M` "am" `date +%d.%m.%Y` von Benny >> $HOSTFILE

# Lösche temporäre Dateien 
rm $BLOCKLIST $PURGEDLIST

# Hochladen (lftp muss installiert sein)
lftp -u user,passwort ftp://webseite.de <<EOF
set ssl:check-hostname false;
set ftp:ssl-force true
set ftp:ssl-protect-data true
set ssl:verify-certificate no # Bei selbst signiertem ssl
mput hosts.txt 
exit
EOF

echo
echo "Fertig..."
Meine blocksites.txt:

Code: Alles auswählen

https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts;showintro=0
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
http://meine.webseite.de/hosts_custom.txt
Bis dann,
benny

PS: Ich werde in Zukunft keinen Support mehr per PN leisten. Bitte eröffnet ein Thema im Hilfebereich, oder wo auch immer eure Fragen hin passen.
Nach oben
Antworten

Zurück zu „Projekte“