Any modern UNIX or UNIX-like OS can do internet mail out of the box. Just give it a domain while setting up the mail software, set up an MX record in the DNS for that domain, and you’re well on your way to sending and receiving mail.
Normally with the basic setup just mentioned, incoming mail gets delivered to standard UNIX accounts. At the most basic level, mail will not be delivered to email@example.com unless you add a user to the server with the username joe. You can also set up aliases, so that email to firstname.lastname@example.org also gets delivered to the user joe. Typically you get many pre-defined aliases that deliver email like postmaster, hostmaster, root, and a bunch of the other system accounts to the first user created during OS installation. More complicated aliases are possible, allowing you to deliver email to a completely different domain, or to send the email to a program for serious application processing.
If you’ve only got one or two domains and your needs are not very complicated, that’s all you’ll probably ever need. The difficulty with this arrangement appears when you start adding domains, particularly if the domains belong to different people and those people want to have the same prefix on their addresses – like email@example.com and firstname.lastname@example.org. You can tackle the problem by setting up aliases and creating UNIX users like frodo-sales, frodo-marketing, example-sales, example-marketing, and so on, but this becomes extremely cumbersome for the system administrator, the individual domain administrators, and end users. It also requires every user to have a UNIX account, opening up a potentially large attack surface for people with malicious intent.
For many years I have been running all my domain services on my own server, which has spent most if its life in my basement. The software suite that handles this for me has evolved over those years. At the heart of this are two major programs. One is called postfix, which is an MTA, or mail transport agent. This program sends and receives email using the SMTP protocol, speaking to other MTAs around the Internet. The other is dovecot, which is an IMAP and POP3 server. A mail client like Thunderbird uses IMAP or POP3 to log in and retrieve mail for reading, and uses SMTP to send mail.
Both of these major programs talk to a MySQL database back-end that stores the actual user email addresses, passwords, and preferences. A variety of other software provides ancillary services like spam and virus detection, webmail, etc. What I am running now is a more capable system than what I had built when I was actually running an ISP several years ago, and many times easier to maintain.
There is one real problem I face, and that’s a lack of documentation. I have never written down the steps taken and configuration changes required to get a fully functional server. I aim to change that, and will maintain it on this static blog page. This will make it much easier to recreate the work, as I’ve had to do a few times in a professional capacity.
EDIT: The blog software is not well-suited to documenting commands and config files, so I have moved the documentation to my wiki.