What do all these have in common? SECURITY!
When you run multiple websites and you want to split those off to different computers or containers for the purpose of security or load balancing you need to run a reverse proxy. Apache has a mod for that. The problem is that this mod does not forward the actual IP of the computer visiting the site, instead it sends to the container/computer (let’s call them “containers” from this point forward) the IP of the reverse proxy. This means you can’t use fail2ban to scan the logs to block bad actors. BIG SECURITY ISSUE HERE.
In order to scan for bad actors you use fail2ban. It has jails that look for specific types of activity such as failed login attempts. If it finds them it uses the jail configuration to ban that IP for a period of time. There are various jails that you can set up for things like MySQL, WordPress, Nextcloud, email, etc. When bad guy’s attempts to break in are found by fail2ban it adds an entry to whatever firewall you are using such as iptables/UFW. The next time that bad actor tries to come back in the system sees they are banned and just won’t respond to them. This stops their nefarious activity on your site.
If you have multiple websites as I said above and those are in separate containers then you need a reverse proxy. This reverse proxy will look at the incoming request and direct it to the appropriate website’s container. Apache has provided for this functionality with a few addons. A problem occurs because when they send the user to the target machine they indicate to that target the IP address of the proxy server. What we really want is the visitor’s public IP address. If it does something nefarious we want fail2ban to ban that IP. Since it isn’t sent to the container you can’t use fail2ban to secure your server, and the bad guys are free to just keep trying to break in.
Part of the problem also is the fact that you can have a lot of containers running various things such as websites, email servers, database servers, etc. So, lots of targets where you need to secure them and they aren’t going to be secured if you can’t read the actual IP address of the visitor.
I had a bit of a discussion with someone where he points out that a reverse proxy works for the most part with the web services and not necessarily with the other services that you might be running in containers. My point is that it isn’t that you need a reverse proxy for email (unless you have a web interface in that same container). His point was that you can just use fail2ban in those containers that don’t rely on the reverse proxy. In the case he outlines you would just need to run fail2ban separately there. My point is that certainly is necessary. The point I’m also trying to make is that if the bad guys first focus on one class of services then when they switch to trying to break into other services they’ll get another chance. My conclusion is that if you send all the logs over to the reverse proxy when you ban the bad guy attacking the other services they’ll automatically get banned from the web services too.
One program that you can use to accomplish this is rsyslog. You can set rsyslog up so that all the containers (and other computers, no matter what they are) send their logs to the reverse proxy. This is not hard, though much of the web guides are incomplete in that they don’t speak fully on the subject, instead they jump around with the assumption you know everything — bad.
Rsyslog allows each container to send their logs to another container. In my case the logs from those containers would go back to the proxy server. That container isn’t really doing a whole lot and is the perfect place for it, and it really does demand that since that’s where all the requests to access web services come from — that one place.
When you set up rsyslog logs are sent over. There’s a big flub here though. The log files don’t have the same names. This seems utterly silly — why not? The auth.log doesn’t end up in the reverse proxy with the name auth.log. Now there’s a reason for this. I believe when the log files are created on the websites they are created by a program that sends what I’ll call “events” to “various/numerous” log files. The email error events are supposed to go into the mail.err log, warnings go to the mail.warn, etc. That is, if your syslog program is set up that way. The mail.log is the important one as it can have entries for each of the events found in the other email log files. You can tell this wastes space.
When rsyslog does it’s job it sends the events to the rsyslog server (in my case the reverse proxy) where all the logs are “accumulated”. Those events go into the log files that the rsyslog is configured to handle. This results in fewer files, less waste, and file names that don’t match those in the containers themselves. You just have to work around that.
Note: I’ve learned there are ways to send actual log files with matching log names. I just wish this was set up this way to begin with.
Once rsyslog is set up the reverse proxy container running fail2ban looks at all the logs and scans for malicious activity in each log and bans the bad guys. However, because the reverse proxy sends the visitor initially to the target container and it sends with the reverse proxy’s IP address when the logs are then sent back to the proxy container for scanning by fail2ban there is no visitor IP address to ban and hence it fails.
The answer is mod_remoteip in Apache (sudo a2enmod remoteip). The problem is that it is nearly as convoluted to set up as the story I outlined above. Apache should just have mod_remoteip integrated and always send the public IP of the visitor instead of jerking us around making us figure out how to get the public IP into the target container ourselves.
Anyway, bah.
One final note. As you enable more containers to send their logs to your rsyslog server remember to restart fail2ban on the rsyslog server. You can look and see that the logs from the container are making their way to the rsyslog server because there’s a folder with the IP address of the containers in the /var/log/RemLogs/ folder. If you look at the fail2ban-client status <jail> you’ll note that it doesn’t have the reference to the newly added container that’s sending the logs.