Fail2Ban is a very nice little log monitoring tool that is used to detect cracking attempts on servers and to extract the malicious IPs and—do the things to them—usually temporarily adding the IP address of the source of badness to the server’s firewall “drop” list so that IP’s bad packets are lost in the aether. This is great, but it’d be cool to, instead of running a firewall on every server each locally detecting and blocking malicious actors, to instead detect across all services and servers on the LAN and push the results up to a central firewall so the bad IPs can’t reach the network at all. This is one method to achieve that goal.
NOTE: pfBlockerNG v3.2.0_3 adding a “_v4” suffix to the auto-generated IPv4 aliases. The shell script that runs on pfSense to update the alias via pfctl should be modified to match.
I like pfSense as a firewall and run FreeBSD on my servers; I couldn’t find a prebuilt tool to integrate F2B with pfSense, but it wasn’t hard to hack something together so it worked. Basically I have F2B maintain a local “block list” of bad IPs as a simple text file which is published via Apache from where pfSense’s grabs it and applies it as a LAN-wide IP filter. I use the pfSense package pfBlockerNG to set up the tables but in the end a custom script running on the pfSense server actually grabs the file and updates the pfSense block lists from it on a 1 minute cron job.
There are plenty of well-written guides for getting F2B working and how to configure it for jails; I found the following useful:
- https://www.digitalocean.com/community/tutorials/how-fail2ban-works-to-protect-services-on-a-linux-server
- https://dan.langille.org/2015/05/10/wordpress-and-fail2ban/
- https://forums.freebsd.org/threads/fail2ban-with-jails.49150/
- https://protectli.com/kb/how-to-setup-pfblockerng/
- https://synaptica.info/en/2019/06/09/pfsense-cron-iterface/
Note that this how-to assumes you have or can get F2B working and that you’ve got pfSense working and know how to install packages (like pfBlockerNG). I did not find sufficient detail in any one of the above sources to make setting it up a copy-pasta experience, but in aggregate it worked out.
The basic model is:
- Internet miscreants try to hack your site leaving clues in the log files
- Fail2Ban combs the log files to determine which IPs are are associated with sufficiently bad (and sufficiently frequent) behavior, which is normally used to update the host’s local firewall block list, adding and removing miscreant IPs according to rules defined in
.localjail scripts. - Instead of updating the firewall’s
iptableslist, a custombanactionscript (see below) instead writes the IPs to a list that is published to the LAN by a web server (or to the world, if you want to share). - pfSense, running on a different server, is configured to pull that list of miscreant IPs into pfBlockerNG as a standard IPv4 (in my case, IPv6 is also possible) block list.
- To get around pfBlockerNG’s too slow maximum update rate of 1 hour, a bash script runs on an every minute cron job on the pfSense server to
curlthe list over and update pfSense’spfctl(packet filter control) directly, which to some extent bypasses fail2ban other than letting it maintain the aliases and stats. - Packets from would be evildoers evaporate at the firewall.
This model is federatable – that is sites can make their lists accessible either via authenticated (e.g. client cert or scp) or open sharing of dynamic lists. This might be a nice thing as some IP block lists have gone offline or become subscription only. Hourly (or less frequent) updates would require only subscribing to someone’s HTTP/FTP published F2B dynamic miscreant list in pfBlockerNG or by adding bash/cron jobs to update more frequently.
I do not publish my list because it would seem to provide a bit of extra information to an attacker, but if someone with a specific IP that can be allowed wants it, I’m happy to except that IP.
The custom bits I did to get it to work with pfSense are:
Custom F2B Action
On the protected side, I modified the “dummy.conf” script to maintain a list of bad IPs as a banaction in an Apache served location that pfSense could reach. F2B manages that list, putting bad IPs in “jail” and letting them out as in any normal F2B installation—but instead of using the local server’s packet filter as the banaction, they’re pushed to a web-published text list. This list contains any IP that F2B has jailed, whether in the SSH jail or the Apache jail or the Postfix jail or whatnot based on banactions per jail.local config. Note that until the pfSense part of the process is set up, F2B is only generating a web-published list of miscreants trying to hack your system.
# Fail2Ban configuration file
#
# Author: David Gessel
# Based on: dummy.conf by Cyril Jaquier
#
[Definition]
# Option: actionstart
# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
# Values: CMD
#
actionstart = if [ -z '<target>' ]; then
touch <target>
printf %%b "# <init>\n" <to_target>
fi
chmod 755 <target>
echo "%(debug)s started"
# Option: actionflush
# Notes.: command executed once to flush (clear) all IPS, by shutdown (resp. by stop of the jail or this action)
# Values: CMD
#
actionflush = if [ ! -z '<target>' ]; then
rm -f <target>
touch <target>
printf %%b "# <init>\n" <to_target>
fi
chmod 755 <target>
echo "%(debug)s clear all"
# Option: actionstop
# Notes.: command executed at the stop of jail (or at the end of Fail2Ban)
# Values: CMD
#
actionstop = if [ ! -z '<target>' ]; then
rm -f <target>
touch <target>
printf %%b "# <init>\n" <to_target>
fi
chmod 755 <target>
echo "%(debug)s stopped"
# Option: actioncheck
# Notes.: command executed once before each actionban command
# Values: CMD
#
actioncheck =
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionban = printf %%b "<ip>\n" <to_target>
sed -i '' '/^$/d' <target>
sort -u -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 <target> -o <target>
chmod 755 <target>
echo "%(debug)s banned <ip> (family: <family>)"
# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
# flush the IP using grep which is supposed to be about 15x faster than sed
# grep -v "pattern" filename > filename2; mv filename2 filename
actionunban = grep -v "<ip>" <target> > <temp>
mv <temp> <target>
chmod 755 <target>
echo "%(debug)s unbanned <ip> (family: <family>)"
debug = [<name>] <actname> <target> --
[Init]
init = BRT-DNSBL
target = /usr/jails/claudel/usr/local/www/data-dist/brt/dnsbl/brtdnsbl.txt
temp = <target>.tmp
to_target = >> <target>
The target has to be set for your web served environment (this would be FreeBSD default host root). I’ve configured mine to be visible on the LAN only via .htaccess in the webserverroot/dnsbl/ directory.
AuthType Basic Order deny,allow Deny from all allow from 127.0.0.1/8 10
Then you need to call this as a banaction for the infractions that will get miscreants blocked at pfSense, for example in ./jail.local you might modify the default ban action to be:
# Default banning action (e.g. iptables, iptables-new, # iptables-multiport, shorewall, etc) It is used to define # action_* variables. Can be overridden globally or per # section within jail.local file banaction = brtdnsbl #banaction_allports = iptables-allports
or say in ./jail.d/sshd.local you might set
[sshd] enabled = true filter = sshd banaction = brtdnsbl maxretry = 2 findtime = 2d bantime = 30m bantime.increment = true bantime.factor = 2 bantime.maxtime = 10w logpath = /var/log/auth.log
But do remember to set your ignoreip as appropriate to prevent locking yourself out by having your own IP end up on the bad guy list. You can make a multi-line ignoreip block like so:
ignoreip = 127.0.0.1/8 ::1 10.0.0.0/8 12.114.97.224/27 15.24.60.0/23 22.31.214.140 13.31.114.141 75.106.28.144 125.136.28.0/23 195.170.192.0/19 72.91.136.167 24.43.220.14 18.198.235.171 17.217.65.19
That is put a leading space on continuation lines following the ignoreip directive. This isn’t documented AFAIK, so it might break, but works as of fail2ban version 0.11.2_3
Once this list is working (check by browsing to your list):
then move to the pfSense side to actually block these would be evildoers and scriptkiddies.
Set up pfBlockerNG
The basic setup of pfBlockerNG is well described, for example in https://protectli.com/kb/how-to-setup-pfblockerng/ and it provides a lot of useful blocking options, particularly with externally maintained lists of internationally recognized bad actors. There are two basic functions, related but different:
DNSBL
Domain Name Service Block Lists are lists of domains associated with unwanted activity and blocking them at the DNS server level (via Unbound) makes it hard for application level services to reach them. A great use of DNSBLs is to block all of Microsoft’s telemetry sites, which makes it much harder for Microsoft to steal all your files and data (which they do by default on every “free” Windows 10 install, including actually copying your personal files to their servers without telling you! Seriously. That’s pretty much the definition of spyware.)
It also works for non-corporate-sponsored spyware, for example lists of command and control servers found for botnets or ransomware servers. This can help prevent such attacks by denying trojans and viruses access to their instruction servers. It can also easily help identify infected computers on the LAN as any blocked requests are sent to what was supposed to be a null address and logged (to 1.1.1.1 at the moment, which is an unfortunate choice given that is now a well-reputed DNS server like Google’s 8.8.8.8 but, it seems, without all the corporate spying.) There is a bit of irony in blocking lists of telemetry gathering IPs using lists that are built using telemetry.
Basically DNSBLs prevent services on the LAN from reaching nasty destinations on the internet by returning any DNS request to look up a malicious domain name with a dead-end IP address. When your windows machine wants to report your web browsing habits to microsoft, it instead gets a “page not found” error.
IPBL: what this process uses to block baddies
This integration concept uses an IPBL, a list of IP addresses to block. An IPBL works at a lower level than a DNSBL and typically is set up to block traffic in both directions—a script kiddie trying to brute force a password can be blocked from reaching the target services on the LAN, but so too can the reverse direction be blocked—if a malicious entity trips F2B, not only are they blocked from trying to reach in, so too are any sneaky services on your LAN blocked from reaching out to them on the internet.
All we need to do is get the block list F2B is maintaining into pfSense. pfBlockerNG can subscribe to the list URL you set above easily enough just like any other IPv4 block list but the minimum update time is an hour.
My IPv4 Settings for the Fail2Ban list look like this:
An hour is an awfully long time to let someone try to guess passwords or flood your servers with 404 requests or whatever else you’re using F2B to detect and stop. So I wrote a simple script that lives on the pfSense server in the/root/custom directory (which isn’t flushed on update) and that executes a few simple commands to grab the IP list F2B maintains, do a fairly trivial grep to exclude any non-IP entries, and use it to update the packet filter drop lists via pfctl:
/root/custom/brtblock.sh
#!/usr/bin/env sh
# set -x # uncomment for "debug"
# Get latest block list
/usr/local/bin/curl -m 15 -s https://brt.llc/dnsbl/brtdnsbl.txt > /var/db/pfblockerng/original/BRTDNSBL.orig
# filter for at least semi-valid IPs.
/usr/bin/grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' /var/db/pfblockerng/original/BRTDNSBL.orig > /var/db/pfblockerng/native/BRTDNSBL.txt
# update pf tables
# for pfBlockerNG ≤ v3.2.0_3, omit the _v4 suffix shown below
/sbin/pfctl -t pfB_BRTblock_v4 -T replace -f /var/db/pfblockerng/native/BRTDNSBL.txt > /dev/null 2>&1HT to Jared Davenport for helping to debug the weird /env issues that arise when trying to call these commands directly from cron with the explicit env declaration in the shebang. Uncommenting the set -x directive will generate a verbose output for debugging. Getting this script onto your server requires ssh access or console access. If you’re adventurous it can be entered from the Diagnostics→Command Prompt tool.
Preventing Self-Lockouts at pfSense
One of the behaviors of pfBlockerNG that the dev seems to think is a feature is automatic filter order management. This overrides manually set filter orders and will put pfB’s block filters ahead of all other filters, including, say, allow filters of your own IPs that you don’t want to ever be locked out in case you forget your passwords and accidentally trigger F2B on yourself. This will override your own reordering. If this default behavior doesn’t work for you, then use a non-default setting for IPv4 block lists and make all IP block list “action” types “Alias_Native.”
To use Alias_Native lists, you write your own per-pfBlockerNG alias filter (typically “drop” or “reject”) and then pfBlockerNG won’t auto-order them for you on update. We let pfBlockerNG maintain the alias list at something like pfB_BRTblock (the pfB_ prefix is added by pfBlockerNG) which we then use like any other alias in manual firewall rule:
So the list of rules looks like this:
Cron Plugin
The last ingredient is to update the list on pfSense quickly. pfSense is designed to be pretty easy to maintain so it overwrites most of the file structure on upgrade, making command line modifications frustratingly transient. I understand that /root isn’t flushed on an upgrade so the above script should persist inside the /root directory. But crontab -e modifications just don’t stick around. To have cron modifications persist, install the “Cron” package with the pfSense package manager. Then just set up a cron job to run the script above to keep the block list updated. */1 means run the script once a minute.
Results
The system seems to be working well enough; the list of miscreants as small, but effectively targeted: 11,840 packets dropped from an average of about 8-10 bad IPs at any given time.
[…] connection (5,569,029,076 people as of today), and every target is attacked constantly (my own Fail2Ban has blocked 2,899,324 malicious packets) and even if they’re Google, they’re not […]