Problem

The client wanted a custom email infrastructure comprising:

  1. SMTP server for receiving emails.
  2. IMAP server for email accessibility via email clients like Mozilla Thunderbird.
  3. Mail Transfer Agent server for outbound emails.
  4. An admin panel with Role-Based Access Control for user and email management. The main goal was a flexible and scalable system that could be extended easily as business needs evolved.

Challenges

  1. Technical Complexity: Popular open-source SMTP and IMAP servers, such as Postfix and Dovecot (C/C++ based), are really hard to customize and modify.
  2. Data Management: Popular open source SMTP servers store emails as plain text .eml files, complicating email handling, encryption, and system scaling.
  3. System Integrity: Ensuring no email loss, logging issues, and notifying relevant users in case of an issue.
  4. Security: Ensuring robust protection against breaches, spoofing, and phishing.

Research

Understanding the SMTP ecosystem from the first principals was essential to come up with the best solution for the client. This needed a lot of research. The best place to get the most complete information was the RFCs. RFCs are formal documents maintained by The Internet Engineering Task Force (IETF) that defines protocols and specs in painful detail so that engineers can implement those technologies. The team did a deep dive into the SMTP ecosystem which spans over 10s of major and few minor RFCs. For example this is the most fundamental RFC document that defines how email transactions happen: RFC5321.

Also a lot of research and experimentation was done to find out best open-source tools that were developer friendly, well-maintained, and customizable.

Solution

The chosen tech stack ensured:

  • Future adaptability.
  • Scalability.
  • Extensibility.
  • Developer-friendly technology.

Architecture

The Tech stack:

  • NodeJS: Popular, performant, and supported by a vast community and massive collection of libraries.
  • Haraka SMTP Server: A modern, well-maintained open-source SMTP server in NodeJS. Its plugin based architecture offers high adaptability. It does the heavy lifting of receiving and sending emails allowing us to create custom plugins for specific business needs.
  • MongoDB: A flexible NoSQL database, making it easy to extend our data schema as the product evolves in the future.
  • AWS: Enough said :)

The IMAP server was built from the ground up. Building it from scratch was ambitious but it was a great experience. We got to learn from popular open source IMAP server implementations, porting the best parts into our project, while building out the custom features from scratch, getting the best of both worlds.

Security Measures:

  • Database-level encryption.

  • TLS for SMTP and IMAP communication.

  • Modern email security mechanisms:

    • SPF records: Protects against email forging.
    • DKIM: Ensures email content integrity during transit.
    • DMARC Record: Dictates actions for emails failing SPF and DKIM checks

    These modern security mechanism achieved multiple objectives:

    • Prevents spoofing and protects against phishing and fraud.
    • Allows other email providers to attach reputation to our client’s domain.
    • Clients can deploy additional servers under the same reputed domain thus making the whole infrastructure extremely scalable.
    • Reputation guarantees email deliverability from the client’s domain, it makes sure that emails sent from the client’s domain are not rejected by other email providers or does not end up in the recipient’s spam folder.