How to Set Up VPS Mail Server | Postfix, Dovecot, Roundcube

I have been working as a system administrator for over a decade. During this time, I have seen many people struggle with third-party email providers. The costs go up. The rules change. Your data is scanned for advertising.

You might be tired of paying high monthly fees per user. Or maybe you want complete control over your own data. The good news is that you can build a system yourself. You just need a virtual private server (VPS) and some open-source software.

Building your own VPS email server takes a little patience. You will need to configure Postfix, Dovecot, and Roundcube. It sounds highly technical, but I will break it down into easy steps for you.

By the end of this guide, you will know exactly how to set up a mail server on a VPS. You will understand how these tools work together. More importantly, you will learn how to get your emails safely into the inbox. Let us get started.

What Is a Self-Hosted Mail Server?

A self-hosted mail server is an email system you build and manage yourself. You do not rely on big companies like Google or Microsoft. You rent a server, install the software, and control everything.

How email servers work

Think of an email server like a digital post office. When you send a message, your mail client drops it off at your server. Your server then looks up the recipient’s address. It finds their server and delivers the message. When someone replies, your server holds the message until you log in to read it.

Why businesses use VPS mail servers

Businesses often outgrow standard email hosting. They might need hundreds of mailboxes. Paying per user gets very expensive quickly. A self-hosted system on a VPS solves this problem. You can create as many addresses as your server storage allows. It is a very cost-effective way to scale your business communications.

Advantages over third-party email services

The biggest advantage is privacy. Nobody reads your emails to sell you ads. You also get total control over your security rules. You decide how much space each user gets. If you run a web agency, you can even explore white label email hosting and sell mailboxes to your clients.

What Are Postfix, Dovecot, and Roundcube?

You need three main pieces of software to run a modern email server. They work together to send, receive, and display your messages.

Postfix as a mail transfer agent (SMTP)

Postfix is a Mail Transfer Agent (MTA). Its job is to handle the Simple Mail Transfer Protocol (SMTP). This means Postfix is responsible for sending your emails out to the world. It also receives incoming emails from other servers. Postfix is known for being fast, secure, and very reliable.

Dovecot for IMAP/POP3 access

Dovecot is an IMAP and POP3 server. Postfix receives the mail, but Dovecot stores it and lets you read it. When you open your mail app on your phone, it connects to Dovecot. Dovecot checks your password and shows you your inbox. It keeps your folders synced across all your devices.

Roundcube as a webmail client

Roundcube is a free webmail interface. It gives you a nice visual dashboard to read and write emails in your web browser. It looks and feels a lot like standard commercial email services. Roundcube connects to Dovecot and Postfix behind the scenes to show your messages.

What Do You Need Before Setting Up a Mail Server?

You cannot just start installing software randomly. You need a solid foundation first. Here is what you must have ready.

VPS requirements and specifications

Email servers need memory. I highly recommend starting with an Ubuntu 24.04 VPS with at least 2 GB of RAM. If you plan to run strong spam filters, 4 GB of RAM is much better. You should also choose the right VPS plan that offers clean IP addresses.

Domain and DNS configuration

You must own a domain name, like yourdomain.com. You will also need to configure your DNS records properly. You will create an A record pointing to your server’s IP address. You will also need an MX record pointing to your mail server domain (like mail.yourdomain.com).

Static IP and reverse DNS

Your VPS must have a static IP address. It also needs a valid Reverse DNS (rDNS) record. This is also called a PTR record. This record must match your server’s hostname. If you skip this step, other servers will reject your emails immediately as spam.

Step 1: Installing and Configuring Postfix

Now we get to the fun part. Let us start by installing Postfix to handle message routing.

Installing Postfix on Linux VPS

First, log into your server as the root user. You want to update your system packages. Run the simple apt update command. Next, install Postfix. The setup wizard will ask you a few questions. Select “Internet Site” when prompted. Then, enter your primary domain name.

Basic SMTP configuration

You need to edit the main Postfix configuration file. You will find it at /etc/postfix/main.cf. You must set your hostname, domain, and origin here. You also need to configure your network settings so Postfix knows who is allowed to send mail. Always make a backup of the original file before you make changes.

Testing mail sending

Once you save your changes, restart Postfix. You should test it right away. Use the command line to send a basic text email to your personal Gmail account. Check your inbox. It might land in the spam folder right now, but that is perfectly normal at this stage. We just want to make sure the message leaves your server.

Step 2: Installing and Configuring Dovecot

Postfix is running. Now we need Dovecot so you can actually log in and read your mail.

Setting up IMAP/POP3 services

Install the Dovecot core, IMAP, and LMTP packages. Dovecot needs to know where Postfix is dropping the incoming emails. Open the 10-mail.conf file in the Dovecot folder. Tell Dovecot to look for the Maildir format. This format is much faster and safer than the older mbox format.

User authentication setup

Next, we configure how users log in. Open the 10-auth.conf file. You need to disable plain text authentication. This forces your users to log in securely. You will also configure Dovecot to use standard system users or a custom password file.

Securing mail access

Security is vital. You must set up SSL certificates. You can use a free tool called Certbot to get Let’s Encrypt certificates. Open Dovecot’s 10-ssl.conf file. Point it to your shiny new SSL certificate files. This ensures that no one can spy on your emails while you read them on your phone or laptop.

Step 3: Setting Up Roundcube Webmail

Now let us give you a nice visual interface. We will install Roundcube.

Installing Roundcube

Roundcube runs on PHP and needs a database. You will install Apache, PHP, and MariaDB (or MySQL). Once those are running, install the Roundcube package. The system will ask you to configure the database automatically. Say yes, and create a strong database password.

Connecting to mail server

Roundcube needs to talk to Postfix and Dovecot. You will edit Roundcube’s config.inc.php file. Set the default host to your secure mail domain (like ssl://mail.yourdomain.com). Set the SMTP server to use TLS on port 587. This connects the web interface to your underlying mail engine securely.

Configuring webmail interface

Finally, you need to set up a web server block in Apache. This tells the server to load Roundcube when you visit webmail.yourdomain.com. Restart your web server. Go to your browser and log in with your email user credentials. You will see a clean, professional webmail dashboard.

DNS Configuration for Mail Deliverability

Installing the software is only half the battle. Getting your emails into the inbox is the real challenge. You must configure your DNS properly.

Setting MX records

An MX (Mail Exchanger) record tells the internet where to send emails for your domain. Go to your domain registrar. Add an MX record. Set the priority to 10 and point it to your mail server’s hostname.

SPF, DKIM, and DMARC setup

You absolutely need SPF, DKIM, and DMARC records today. SPF tells the world your server is allowed to send mail for your domain. DKIM adds a digital signature to every email to prove it was not altered. DMARC tells receiving servers what to do if an email fails the SPF or DKIM checks.

Avoiding spam folder issues

Clean DNS records are your best defense against the spam folder. You should also check your server’s IP address against major blacklists. Keep your sending volume low at first. Warm up your server slowly. Send real emails to real people who will reply to you.

Security Best Practices for VPS Mail Servers

A mail server on the open internet is a huge target for hackers. You must protect it carefully. If you prefer to manage everything yourself, unmanaged VPS hosting gives you total root access to lock things down.

SSL/TLS encryption

Never allow unencrypted connections. Force all incoming and outgoing connections to use TLS. Disable old protocols like SSLv2 and SSLv3 in both Postfix and Dovecot. This keeps your passwords and messages safe from eavesdroppers on public Wi-Fi networks.

Firewall configuration

You need a strict firewall. Configuring firewalls correctly is essential. Only open the ports you actually need. Open port 25 for server-to-server mail. Open port 587 for secure sending. Open port 993 for secure IMAP reading. Open 80 and 443 for your webmail. Block everything else.

Anti-spam and brute-force protection

You will face password guessing attacks constantly. Install a tool called Fail2ban. It monitors your log files. If someone fails to log in five times, Fail2ban blocks their IP address automatically. You should also install a spam filter like Rspamd to catch junk mail before it clutters your inbox.

Common Problems and How to Fix Them

Running a mail server means fixing things when they break. Here are the most common issues you will face.

Emails not sending or receiving

If emails will not send, check if port 25 is blocked. Some VPS providers block this port to prevent spam. If you cannot receive emails, verify your MX records using a DNS lookup tool. Make sure your firewall allows traffic on port 25. Check the mail.log file to see exactly where the connection drops.

Spam delivery issues

If your mail goes to the junk folder, your DNS is usually the problem. Run your domain through an online mail tester. Ensure your Reverse DNS (PTR record) perfectly matches your Postfix hostname. Ensure your DKIM signature is passing validation.

Authentication failures

If your email client says “password incorrect”, check Dovecot. Look at the authentication logs. Make sure you set up the password file correctly. Ensure Dovecot has permission to read the authentication socket created by Postfix.

How Does SkyNetHosting.Net Inc. Support VPS Email Hosting?

Having the right hosting partner makes building a mail server much easier. SkyNetHosting.net offers the exact tools you need for this project.

High-performance VPS infrastructure

Mail servers need fast storage. Our systems use powerful NVMe drives. If you are migrating from shared hosting, you will notice a huge speed boost immediately. Your webmail will load faster, and incoming messages will process instantly.

Reliable IP reputation and uptime

A bad IP address ruins your email delivery. We take spam abuse very seriously. This keeps our IP ranges clean and respected by major email providers. Our servers also boast high uptime, ensuring you never miss an important message.

Scalable server configurations

You can start small and grow. If you decide to start selling mailboxes, reseller hosting or upgrading to a larger VPS is easy. You can add more RAM or storage as your mailbox sizes increase over time. It is always wise to choose the best VPS hosting that lets you scale on demand.

When Should You Use a Self-Hosted Mail Server?

Self-hosting is a big commitment. It is not for everyone. But for certain groups, it is the only logical choice.

Business email control requirements

If your business deals with highly sensitive data, you need total control. Law firms, medical clinics, and financial offices often cannot risk storing data on third-party clouds. A self-hosted VPS mail server ensures the data never leaves your direct control.

Privacy-focused organizations

Privacy advocates refuse to let major tech companies read their communications. A self-hosted system guarantees complete data sovereignty. You own the keys. You own the storage. Your private conversations stay strictly private.

Developer and enterprise use cases

Developers love self-hosted mail. It lets them build custom applications that generate thousands of emails. Enterprises use it to bypass the strict sending limits imposed by commercial providers. It gives technical teams the freedom to customize the routing rules exactly how they want.

Ready to Build Your Own Mail Server?

Setting up your own email system takes effort. But the rewards are massive. You get complete privacy. You stop paying per-user monthly fees. You take total ownership of your digital communications.

Self-hosted mail servers provide full control but require technical skill

You have seen that installing Postfix, Dovecot, and Roundcube requires command-line work. It is highly technical. But if you follow instructions carefully, it is very manageable. You just need to respect the process and check your logs when things go wrong.

Proper configuration is critical for deliverability and security

Never skip the DNS steps. Your SPF, DKIM, and DMARC records are your passport to the inbox. Lock down your firewall. Enforce strong passwords. Keep your server software updated to patch security holes.

SkyNetHosting.net provides reliable VPS infrastructure suitable for mail server setups

A great mail server starts with a great host. SkyNetHosting.net offers the clean IPs and fast storage you need for success. Grab a high-performance VPS today. Take the time to set up your mail server properly. You will be amazed at the freedom and control you gain over your own email.