I have a few comments about their short segment on email self-hosting, specifically what they all agreed about it. While listening I couldn’t help but think to myself “these people are talking about a subject with which they have almost zero experience”. I could hear myself thinking as each person spoke that they were wrong on virtually every point. Most of that complexity is easily overcome once you spend the time to do the work. When I say “do the work” I’m actually talking about doing the work in putting it together, trying to figure out the various tools, and understanding the links between everything. This is just like what you did when you started to learn Linux. Once you figure that out you realize your reticence was grossly over valued.
Admittedly for me it was complex and time consuming to learn Linux. Today as well as when I started it is difficult to get the information you need. Back when I first got started in Linux there were only so many people available that were capable of understanding the environment well enough to properly explain it. As a result I spent years learning just enough to make working with the environment easier. Even after almost 20 years with Linux I still am looking up things and finding it difficult to get answers. I do find commands documented yet context is always missing. Much of the guides and documentation found on the web are too old. They don’t cover changes in context. Things in Linux are often so interlinked that when you have one problem it leads to a cause by another package and that package has issues stemming from another package. Resolving issues in Linux is never a straight forward as it’s a chain of fixes.
It was the same for Windows. In the day it was considered hugely complex, especially for programmers. Learning the API was certainly something that could drive you nuts.
I’ve been in the PC industry since about 1986. I worked doing system administration for several companies, did some development work, managed groups of people, and have now run my own business for quite some time – probably as long as Noah has. I’ve also had to deal with the quandary named privacy. Because I started so long ago, the issue of privacy to me is much more ingrained. I grew up knowing that my privacy was important.
I’ve kept up with the trends of how companies have whittled away at the privacy of people, MOST of which are not even their customers (Facebook and Twitter tracking everyone even though you may not even have an account on their servers). For instance, there was a time when if a company collected anything about us, tracked anything we did, or tried to advertise to us over the internet it drew a large backlash. Most companies would back off. It wasn’t until Google (specifically gmail) came into being that we had a serious uptick in data collection and privacy violations. A privacy violation is what happens when we don’t give permission to an entity to collect and redistribute to others information about us.
BTW, on one of Ryan’s points, the PiHole was designed to block ads and stop tracking. The DNS stuff is actually a late edition. For Noah as he uses pfsense, pfsense comes with pfblocker-ng that does essentially the same thing as the pihole except it is integrated into the pfsense router firewall itself. I used the PiHole for a long time and switched to pfblocker-ng when it came out. I find that it performs exceptionally well.
Pfsense can also be configured to be your own DNS server. I’m sure Noah knows about it. Kill two birds with one stone and use pfsense with pfblocker-ng. On top of that squid which comes with a virus scanner can be integrated into pfsense. This gives you a virus scanner at the router, another at the Proxmox Mail Gateway (PMG) (which scans incoming and outgoing emails), and yet another at the workstation. Pfblocker-ng also has a GeoIP block list that allows you to block access from whole continents of the world if you so choose.
Getting on with it… I’d encourage you to study what happened in the uptake of email — from a historical perspective – as it pertains to ISPs and the growth of email aggregators such as gmail, hotmail, etc. I know you’ll have difficulty getting the whole picture as a lot of this stuff has either dropped off the web or it is on sites that you don’t know about. Anyone that lived the history is far more educated about it than someone going back to study it – revisionist history being all the rage these days.
Over the past 10 years I have run various email servers for myself self-hosting them at my location. I’ve had to deal with learning its complexities just as I had to learn Linux (and Windows and DOS before that).
As an example of how complex Linux has become I’ll use the screen command. You may not put any weight into it but you should take it for what it is worth. Noah recently offered “screen” as one of his picks. When I heard him talk about it I wanted to say “go to the command line and type “man screen””. When you do this, tap the page down key once a second. Count how many times you have to press the page down key just to get to the bottom of the page.
Another is SSH. SSH is my favorite program. I couldn’t get along without it. Well, you might think that it isn’t complex, that you can just ssh <user>@<ip address> with possibly some additional parameters. OK, now take some time and look at implementing the config files, encryption keys, proxyjump, and look at the server. Then try to look at all the SSH options. I recently found issues in using config files with the dolphin file manager (via sftp://). If you use a wildcard in your config file it won’t work with sftp in conjunction with dolphin. In this case the KDE guys blame how sftp works.
SSH can quickly become quite cumbersome if you try to remember every connection, that is, unless you try to organize with config files (yes files). I manage several businesses’ servers. I installed with Linux and set them up with email servers, web and other servers. I need to be able to get into containers on each server and move around quickly. I also use ansible to update these all at the same time. This means I need config files. I’ve created a folder and added a config file for each business and under that I have some sections of those individual config files that are repetitive. To keep them in order and minimal I will often use a wildcard entry at the bottom to set the username and a couple of other parameters. When doing this dolphin will fail to connect. If I remove the wild card and have a section for each server/container dolphin has no issues. See, it starts simple as SSH and then turns complex quite quickly.
Try using ssh -L or some of the other parameters that allow you bind a remote port of a host to a local host’s port. Combined with your browser you can create a sort of a mini-VPN. I’ve used this as a way to debug issues with port forwarding/firewall rules in a router.
My point is that Linux is complex and you learn it little by little over time. Same with learning and implementing a self-hosting of email.
One thing that troubles me with the email servers it seems that they have (seemingly intentionally) made working with them complex. Think back to when email was relatively new. We had people use it to spam. This was bad news and there was a need to overcome the spam issue. However, we also had companies such as the ISPs try to charge you for email addresses. To this day some still charge for email addresses and for storage. There’s no incentive to make it easy and to solve certain types of problems. I think of the mail administrator at a business who knows that their skills are valuable enough to keep them employed even during tough times. Spreading that knowledge in an effort to educate others is self-defeating to them. Without encouraging people to actively implement on their own we’ll only see slow advances. We can’t let a relatively small group of overlords decide what is best for us. That’s not how open source works.
Think of Google and understand as Ryan said that Google knows that if they get you on their system you are locked into it, and they can read your emails at will. This is the same for Microsoft and Yahoo, and the plethora of others. With PMG you can tell it to log in to gmail and fetch your mail thus breaking that lock in. A great thing is that every mail that is fetched goes to a target email address which PMG will process against your rules/filters.
Here’s something you don’t know. If your email is on a 3rd party’s server you have no expectation of privacy. Under Smith v Maryland the courts ruled that if your data is on a 3rd party server there’s no expectation of privacy. There is also a law (I’m pretty sure it is still a valid law) that was passed that states that any opened email on a 3rd party server after 180 days is considered abandoned and the government doesn’t need a warrant to gain access to it. Your only recourse is to read the mail and then mark it as unread or to stop using a 3rd party server. In case you weren’t aware these are the types of laws that the Obama administration used to permit the government to collect your emails en-mass. It was the government trying to solidify this type of law (and Snowden) that ultimately brought it to the general public’s attention. The police were using this “type” of law to allow them to track individuals around (from GPS data to copying your phone during a traffic stop to searching you and your equipment within 200 miles of any US border). This should incentivize you all to do your own email hosting.
Getting back to email. Setting up an email server is not difficult to say the least. There are plenty of guides. On top of that there are containers based on virtualization technology, such as docker (I’m not a fan of docker) that are fully implemented. Personally, I use Proxmox. I set up a container and get the whole email system as secure as possible and then I can clone it to a new container and do it for another entity’s domain. (Note: what’s neat about this is that I can use HA Proxy (from Pfsense in my case) and have multiple containers with different domains and pfsense knows how to direct email appropriately).
Each container is set up so that the email server within the container can handle multiple domains and has the proper administrative tools. Then, along with that I use a tool provided by Proxmox called the PMG – dead simple service to install, and easier to do in an LXC container). This is a powerful tool that seems overly simplistic, but it has some really great features meant to keep you out of trouble (e.g., relays, spam, virus). It has DKIM, SPF and other stuff built right in.
About the lists. This is regarding Noah’s comments about having two guys spending multiple hours a day dealing with the red tape to get their servers removed from the black lists. I’m sorry, and no malice intended, but that’s by far a stretch and completely unrealistic. The lists are not governing. They are just a group of people that created a list and formed a joint effort. If this were not the case and they were effective there would be no need for so many. For the most part you can just ignore them. Often it only matters if you are on a lot of the lists. As well, not everyone uses those lists. In PMG you can use the barracuda and Spamhaus RBLs for free. This aids not only in banning but in identifying possible spam. Since email is not evaluated as spam till it reaches a specific spam score those lists will only contribute to the score and luckily not be the only deciding factor.
I can’t figure out why people consider installing and taking care of their mail server as anything different than taking care of their Linux install. You spend time learning, you spend time configuring, and you spend time enhancing your Linux desktop or server. Why can’t you learn to put that kind of effort into managing this communication channel to ensure you have the proper privacy?
Keep in mind, any of your listeners (those that you were addressing for the most part) maintaining their own email server will likely be doing it for themselves or a small group. Thus the job of maintaining it isn’t going to require a herculean effort. Being that it is going to be a small email implementation it is going to be very hard for the administrator to miss when the server has been sending spam or doing something nefarious.
In case you didn’t know it, gmail is used to generate a lot of spam. Services such as amazonses.com sends out a lot of spam for 3rd parties. Same with sendgrid.net. There are plenty of others. Lots of businesses use these services including IBM and Reddit. I think Twitter also.
As you work with your email server you’re going to know if you are being manipulated. If you put in something like the PMG you can see (using the tracking center) all the email going to and from your server. In your router (for example pfsense) you can tell the router to only allow email out on port 25 only from specific IP address(es) on your network, and in doing so you push all your internal computers that send out emails through the PMG.
In PMG you can create filters, etc and you can stop the spam in its tracks. You can configure it to send a response to blocked or quarantined emails. This can be done by optionally sending an email to the “sender” and to the administrator, be the sender internal or external to the organization.
An email server without a virus scanner makes the maintainer negligent. PMG has a virus scanner. It also has greylisting. The idea behind greylisting is that it rejects all emails coming in unless they were previously known. This keeps spam from getting in. It rejects the email asking the sender to resend it. If the email isn’t resent it means the email server that sent it was not legit and in all likelihood it means that this is a computer that has become part of a botnet. Either way if the email isn’t resent it will never get to the user’s inbox. Most manually configured email servers also (by best practices) pass all their mails through a virus scanner.
My point here is that much of the concerns about running your own email server is just fear that is generated by a lack of knowledge. The more people that learn to use it the better the documentation is and the better tools get. This also means that more entities such as businesses, ISPs, etc will have to deal with the reality that denying people the capability to run their own systems is a bad thing. ISPs that deny you the right to run your own email server is like when an internet cafe refused to allow Linux users to connect to their router. It’s also like having websites only permit Internet Explorer to connect. It’s like saying that you can only use Windows. All of these were requirements at one point.
Another benefit is that the more we all host our own email servers the smaller a target we become. When hackers break into Google, Microsoft (think about Microsoft’s exchange server nightmares of not so long ago), or Yahoo (think of the 1 billion + user’s credentials and information that was leaked onto the dark web), it means that they can get at us all at once. If everyone ran their own email systems we wouldn’t be subject to that kind of nightmare. We shouldn’t forget that exposing your email to any email aggregator can leave you in a precarious position as far as privacy goes, even ones such as Proton mail. As I’ve said and has been proven true time and again, they will not do something until they do.