Troubleshooting Course Content Rough Drafts-How Does Email Work
What is e-mail?
Defining E-mail:
Often referred to as "e-mail", this term is frequently used or associated with the discussion of not only actual "mail" composition and retrieval, but is also used in the explanation of different protocols, standards, and architecture(s) associated with the electronic creation, submission, and retrieval of electronic messages.
We can see how "e-mail" is defined below;
e-mail: noun messages distributed by electronic means from one computer user to one or more recipients via a network. verb send an e-mail to (someone).
In this training unit, we'll look at how e-mail came about, the industry standards and protocols used today as well as the architecture and vernacular of how e-mail works within your Zimbra deployment.
The origin of e-mail and industry standard protocols
History and Background:
Dating back to the early 1960's, the first form(s) of e-mail sprouted up in the form of services like AUTODIN, a legacy data communications service developed by the U.S. Department of Defense, and MIT's CTSS Mail, where remote terminals could dial-in to share and store files on a central disk(s). Informal methods of using these type(s) of systems later evolved to pass messages between terminal users.
The first "e-mail" systems used different features and ran on different systems that were not compatible with each other; coined "host-based e-mail systems". Most of these systems required users to be logged into in the same "terminal" or "mainframe" in order for communication to occur.
Subsequent to "host-based e-mail systems" came "LAN e-mail systems", those that ran on a local area network, which still required users to be logged into the same infrastructure. Some examples of "LAN e-mail systems" include Microsoft Mail and Lotus Notes. Further evolution allowed for these types of systems to communicate outside of an organization, assuming the same email system and proprietary protocols were in use.
Circa the 1970's, more of what we see in use today was in the origin of development. It was essential that as organizations grew larger and as technology advanced, those methods of mail exchange between remote sites or other organizations provided an avenue to transport messages of text globally via telecommunication links.
Internet Message Format:
Alas, through RFC, an "Internet Message Format" was derived and published by Qualcomm. RFC5322 is not the first RFC to establish an "IMF", albeit it's the most current.
From RFC5322:
This document specifies the Internet Message Format (IMF), a syntax for text messages that are sent between computer users, within the framework of "electronic mail" messages. This specification is a revision of Request For Comments (RFC) 2822, which itself superseded Request For Comments (RFC) 822, "Standard for the Format of ARPA Internet Text Messages", updating it to reflect current practice and incorporating incremental changes that were specified in other RFCs.
You can read more about this RFC here; RFC5322.
Beyond Text...Multimedia Attachments (MIME):
Further expansion led to "e-mail" that not only included text, but multimedia attachments as well. Established in RFC2045, through RFC2049, Multipurpose Internet Mail Extensions or "MIME" were derived.
Message Format:
Consisting of two major components, e-mail messages include both a "header" and a "body" section.
The Header:
This section is structured into fields, such as From, To, CC, Date, Subject, and other information pertaining to the message. An example "header" section is shown below:
Received: from zimbra.server.com (LHLO zmail.zimbra.com) (10.137.27.36) by zimbra.server.com with LMTP; Sat, 31 Jan 2015 01:05:53 -0800 (PST) Received: from localhost (localhost [IPv6:::1]) by zmail.zimbra.com (Postfix) with ESMTP id 7970049F7 for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:53 -0800 (PST) by localhost (zimbra.server.com [IPv6:::1]) (amavisd-new, port 10032) with ESMTP id rPQ4Stk2X1Sy for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:52 -0800 (PST) Received: from localhost (localhost [IPv6:::1]) by zmail.zimbra.com (Postfix) with ESMTP id 3E2384A4C for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:52 -0800 (PST) Received: from zmail.zimbra.com ([IPv6:::1]) by localhost (zimbra.server.com [IPv6:::1]) (amavisd-new, port 10026) with ESMTP id 24DhbFXYNwUn for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:51 -0800 (PST) Received: from zimbra.server.com (localhost [127.0.0.1]) by zmail.zimbra.com (Postfix) with ESMTP id E388C4A46 for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:51 -0800 (PST) Date: Sat, 31 Jan 2015 01:05:51 -0800 (PST) From: admin <admin@zimbra.server.com> To: admin@zimbra.server.com Message-ID: <1205797309.15.1422695151122.JavaMail.zimbra@zimbra.server.com> In-Reply-To: <843455228.13.1422695143436.JavaMail.zimbra@zimbra.server.com> Subject: Re: ZCS Backup Report: SUCCESS MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Auto-Submitted: auto-replied (zimbra; vacation) Precedence: bulk X-Mailer: Zimbra 8.0.6_GA_5922 Thread-Topic: ZCS Backup Report: SUCCESS Thread-Index: FP1J+yf4TQ4duzXiVDb92nMu0M2BmQ==
The Body:
Included in the body is basic content in form of unstructured text; possible containing a signature block at the end. The content of this section would be comparable to the "body" of a regular letter.
An example "body" section can be seen below:
Hello, During the Zimbra Admin Training class, we will be installing Zimbra on Amazon EC2 instances. Sincerely, Your Zimbra Training Administrator Zimbra | Community & Collaboration 1855 S Grant Street, Suite 200, San Mateo, CA 94402 USA
The Message Header Explained
Every message has only one header structured into fields. Every field has a name and a value as outlined in RFC5322.
Header Fields:
A message header mustinclude the following fields:
•From: The email address, and optionally the name of the author(s). In many email clients not changeable except through changing account settings.
•Date: The local time and date when the message was written. Like the From: field, many email clients fill this in automatically when sending. The recipient's client may then display the time in the format and time zone local to him/her.
A message header should include the following fields:
•Message-ID: Also an automatically generated field; used to prevent multiple delivery and for reference in In-Reply-To: (see below).
•In-Reply-To: Message-ID of the message that this is a reply to. Used to link related messages together. This field only applies for reply messages.
Please refer to RFC3864 for elaboration on message header fields.
Simple Mail Transport Protocol
SMTP Explained:
The Simple Mail Trasnport Protocol, more commonly known as "SMTP", is an internet standard for e-mail transmission, first established in RFC821 and later expanded to include "Extended SMTP" via RFC5321; the protocol most widely used today.
SMTP by default uses port 25 for mail submission. Secure SMTP, known as SMTPS, defaults to port 465 in legacy configurations and port 587 in more current configurations.
The default values in a Zimbra deployment are port 25 for SMTP and port 587 for SMTP over TLS (SMTPS).
SMTP: Server vs. Client
Server:
Zimbra architecture employs a server side application called Postfix, which is an open source Mail Transfer Agent (MTA), for sending and receiving messages. The implementation of Postfix included with a Zimbra deployment is customized and not interchangeable with "standard" or "vanilla" Postfix installation(s).
Zimbra translates Postfix configuration variables for both users and server values into LDAP attributes and local configuration variables (zmlocalconifg read from /opt/zimbra/conf/localconfig.xml). It's important to keep in mind that any changes made to Postfix, outside of the Zimbra local configuration or modification of LDAP variables will not remain in place post upgrade.
The following files are customized for Zimbra's Postfix installation:
•main.cf - Modified to include the LDAP tables. The configuration script in the Zimbra MTA pulls data from the Zimbra LDAP and modifies the Postfix configuration files. •master.cf - Modified to use Amavisd-New.
Client:
Despite the primary focus of SMTP protocols being related to server side operations, Client applications can also use SMTP for the submission of mail to an MTA server, or relay. In contrast, Client applications would use POP or IMAP for the retrieval of mail.
Zimbra: SMTP vs. LMTP:
While SMTP is employed by Zimbra to communicate and send mail to both Zimbra and non-Zimbra mail environments, Zimbra also uses another protocol, LMTP (Local Mail Transfer Protocol), to route messages to the appropriate mailstore.
Postfix performs the Zimbra mail transfer and relay of mail. Postfix receives inbound messages via SMTP (or SMTPS), and hands off the mail message(s) to the Zimbra server via LMTP, as shown in the following figure. The Zimbra MTA can also perform anti-virus and anti-spam filtering.
Zimbra Mail Processing:
Mail is submitted to the Zimbra Server (Mail Submission Agent) via User Agents (UA's) on either port 25 (SMTP) or port 587 (SMTP w/TLS), traditionally. From there, messages are routed from the Mail Submission Agent to the Zimbra MTA (Mail Transfer Agent). Both submission and transfer occur within the same application, albeit use different operational procedures.
As we previously discussed, mail is routed from the Zimbra back-end (mailstore) to the Zimbra MTA with the LMTP protocol. Once the mail is received by the Zimbra MTA, SMTP is used to route the messages for both local and external delivery. All mail, whether internal or external, is routed through the Zimbra MTA; this happens to ensure that the archive function (if licensed) works accurately to save a copy of every message transmitted through the Zimbra MTA, regardless of whether a message was internal or external.
Message Queues:
During the receipt of mail, the Zimbra MTA uses a series of queues (envision a series of buckets) to manage delivery. The Zimbra MTA maintains four queues, where mail is placed temporarily during processing; incoming, active, deferred, and hold.
Queue Definitions:
The qmgr(8) daemon maintains the following queues:
incoming Inbound mail from the network, or mail picked up by the local pickup(8) daemon from the maildrop directory.
active Messages that the queue manager has opened for delivery. Only a limited number of messages is allowed to enter the active queue (leaky bucket strategy, for a fixed delivery rate).
deferred Mail that could not be delivered upon the first attempt. The queue manager implements exponential backoff by doubling the time between delivery attempts.
corrupt Unreadable or damaged queue files are moved here for inspection.
hold Messages that are kept "on hold" are kept here until someone sets them free.
There are a few tools available to us to manage the Postfix mail queues; either from the command line or the admin interface. The screenshot below shows us the queues from the Admin Console; where you can also move mail between queues and re-queue mail for submission.
From the command line, tools such as 'postqueue' can be used to manipulate/move messages between queues:
postqueue usage: postqueue -f | postqueue -i queueid | postqueue -p | postqueue -s site
Other tools, such as 'qshape' or 'mailq' can be used to view the mail queues on the command line.
$ qshape
$ mailq
In a default installation of Postfix, you'd manipulate the configuration parameters by modifying the Postfix configuration files directly, or through the use of 'postconf'. In terms of your Zimbra deployment, we use both 'zmlocalconfig' and LDAP attributes to modify these variables; this ensures any changes we've made survive upgrades and future installations.
The following variables are tracked the 'zmlocalconfig' for Postfix:
[zimbra@zcs807 ~]$ zmlocalconfig |grep -i postfix ldap_postfix_password = * postfix_alias_maps = hash:/etc/aliases postfix_always_add_missing_headers = yes postfix_bounce_notice_recipient = postmaster postfix_bounce_queue_lifetime = 5d postfix_broken_sasl_auth_clients = yes postfix_command_directory = ${zimbra_home}/postfix/sbin postfix_daemon_directory = ${zimbra_home}/postfix/libexec postfix_delay_warning_time = 0h postfix_enable_smtpd_policyd = no postfix_header_checks = pcre:${zimbra_home}/conf/postfix_header_checks postfix_import_environment = postfix_in_flow_delay = 1s postfix_lmtp_connection_cache_destinations = postfix_lmtp_connection_cache_time_limit = 4s postfix_lmtp_host_lookup = dns postfix_mail_owner = postfix postfix_mailq_path = ${zimbra_home}/postfix/sbin/mailq postfix_manpage_directory = ${zimbra_home}/postfix/man postfix_maximal_backoff_time = 4000s postfix_milter_command_timeout = 30s postfix_milter_connect_timeout = 30s postfix_milter_content_timeout = 300s postfix_milter_default_action = tempfail postfix_minimal_backoff_time = 300s postfix_newaliases_path = ${zimbra_home}/postfix/sbin/newaliases postfix_notify_classes = resource,software postfix_policy_time_limit = 3600 postfix_propagate_unmatched_extensions = canonical postfix_queue_directory = ${zimbra_home}/data/postfix/spool postfix_queue_run_delay = 300s postfix_sender_canonical_maps = proxy:ldap:${zimbra_home}/conf/ldap-scm.cf postfix_sendmail_path = ${zimbra_home}/postfix/sbin/sendmail postfix_setgid_group = postdrop postfix_smtp_cname_overrides_servername = no postfix_smtp_helo_name = $myhostname postfix_smtp_sasl_auth_enable = no postfix_smtp_sasl_security_options = noplaintext,noanonymous postfix_smtpd_banner = $myhostname ESMTP $mail_name postfix_smtpd_client_restrictions = reject_unauth_pipelining postfix_smtpd_data_restrictions = reject_unauth_pipelining postfix_smtpd_error_sleep_time = 1s postfix_smtpd_hard_error_limit = 20 postfix_smtpd_helo_required = yes postfix_smtpd_proxy_timeout = 100s postfix_smtpd_reject_unlisted_recipient = no postfix_smtpd_reject_unlisted_sender = no postfix_smtpd_sasl_authenticated_header = no postfix_smtpd_sasl_security_options = noanonymous postfix_smtpd_sasl_tls_security_options = $smtpd_sasl_security_options postfix_smtpd_soft_error_limit = 10 postfix_smtpd_tls_cert_file = ${zimbra_home}/conf/smtpd.crt postfix_smtpd_tls_key_file = ${zimbra_home}/conf/smtpd.key postfix_smtpd_tls_loglevel = 1 postfix_transport_maps = proxy:ldap:${zimbra_home}/conf/ldap-transport.cf postfix_virtual_alias_domains = proxy:ldap:${zimbra_home}/conf/ldap-vad.cf postfix_virtual_alias_expansion_limit = 10000 postfix_virtual_alias_maps = proxy:ldap:${zimbra_home}/conf/ldap-vam.cf postfix_virtual_mailbox_domains = proxy:ldap:${zimbra_home}/conf/ldap-vmd.cf postfix_virtual_mailbox_maps = proxy:ldap:${zimbra_home}/conf/ldap-vmm.cf postfix_virtual_transport = error
How Zimbra Receives Mail:
The diagram above shows how Zimbra receives mail from the "outside world", or Internet. The message originates from a senders user agent. A sender's UA is either a web or desktop Client used in the composition of mail (i.e. Zimbra Web Client, Zimbra Desktop, Outlook). Once composed, the users message is routed to an MTA (Mail Transfer Agent). The MTA is responsible for DNS lookups and for locating the receiving MTA, which resides elsewhere on the Internet.
The communication that occurs between the originating and receiving MTA uses the Simple Mail Transport Protocol. Once the Zimbra MTA (Postfix) receives the message from the Internet, an LDAP lookup is performed to determine what mail host the Zimbra user resides on. The 'zimbraMailHost' attribute set for each account is responsible for providing the users store. Once a users store is identified, messages make their way from the Zimbra MTA to the mailstore via LMTP.
How Zimbra Fights Spam - AS/AV:
Zimbra & Postfix:
As outlined before, Zimbra uses an adaptation of an open source software called postfix. There are several ways to customize and make modifications to postfix, however, it should be noted that only those postfix configuration variables tracked in the Zimbra localconfig (zmlocalconfig) and in the Zimbra LDAP (OpenLDAP) will remain in place during upgrades
Click here for more information on the Zimbra MTA
DNS and MX Records
Proper configuration of both DNS and MX records is essential in order for Zimbra to send and receive mail on the internet. As well as for a successful installation of your Zimbra deployment. Many Zimbra services rely on DNS to lookup and resolve host names both internally and externally.
DNS Requirements
Before any Zimbra installation occurs, appropriate 'A' and 'MX' records must be configured for your Zimbra hostname(s). The same holds true whether you configure a service such as 'bind' or 'named' to handle name resolution, or if you use an existing DNS service already deployed in your environment.
The A Record:
An "A" record is configured for each host that resides in your Zimbra deployment. An "A" record tells the name service the fully qualified domain name and IP of a server. This information is used to internally and externally to identify where servers are located. Often, split DNS is required should your server(s) not be resolvable from outside of your network.