Troubleshooting Course Content Rough Drafts-How Does Email Work

Revision as of 20:39, 15 April 2015 by Haynesc (talk | contribs) (Simple Mail Transport Protocol)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

What is e-mail?

Defining E-mail:

Often referred to as "e-mail", this term is frequently used or associated with the discussion of not only actual "mail" composition and retrieval, but is also used in the explanation of different protocols, standards, and architecture(s) associated with the electronic creation, submission, and retrieval of electronic messages.


We can see how "e-mail" is defined below;

e-mail:
noun
messages distributed by electronic means from one computer user to one or more recipients via a   network.
verb
send an e-mail to (someone).


In this training unit, we'll look at how e-mail came about, the industry standards and protocols used today as well as the architecture and vernacular of how e-mail works within your Zimbra deployment.



The origin of e-mail and industry standard protocols

History and Background:

Dating back to the early 1960's, the first form(s) of e-mail sprouted up in the form of services like AUTODIN, a legacy data communications service developed by the U.S. Department of Defense, and MIT's CTSS Mail, where remote terminals could dial-in to share and store files on a central disk(s). Informal methods of using these type(s) of systems later evolved to pass messages between terminal users.

The first "e-mail" systems used different features and ran on different systems that were not compatible with each other; coined "host-based e-mail systems". Most of these systems required users to be logged into in the same "terminal" or "mainframe" in order for communication to occur.

Subsequent to "host-based e-mail systems" came "LAN e-mail systems", those that ran on a local area network, which still required users to be logged into the same infrastructure. Some examples of "LAN e-mail systems" include Microsoft Mail and Lotus Notes. Further evolution allowed for these types of systems to communicate outside of an organization, assuming the same email system and proprietary protocols were in use.

Circa the 1970's, more of what we see in use today was in the origin of development. It was essential that as organizations grew larger and as technology advanced, those methods of mail exchange between remote sites or other organizations provided an avenue to transport messages of text globally via telecommunication links.


Internet Message Format:

Alas, through RFC, an "Internet Message Format" was derived and published by Qualcomm. RFC5322 is not the first RFC to establish an "IMF", albeit it's the most current.

From RFC5322:

  This document specifies the Internet Message Format (IMF), a syntax
  for text messages that are sent between computer users, within the
  framework of "electronic mail" messages.  This specification is a
  revision of Request For Comments (RFC) 2822, which itself superseded
  Request For Comments (RFC) 822, "Standard for the Format of ARPA
  Internet Text Messages", updating it to reflect current practice and
  incorporating incremental changes that were specified in other RFCs.

You can read more about this RFC here; RFC5322.


Beyond Text...Multimedia Attachments (MIME):

Further expansion led to "e-mail" that not only included text, but multimedia attachments as well. Established in RFC2045, through RFC2049, Multipurpose Internet Mail Extensions or "MIME" were derived.


Message Format:

Consisting of two major components, e-mail messages include both a "header" and a "body" section.

The Header:

This section is structured into fields, such as From, To, CC, Date, Subject, and other information pertaining to the message. An example "header" section is shown below:


Received: from zimbra.server.com (LHLO zmail.zimbra.com)
(10.137.27.36) by zimbra.server.com with LMTP; Sat, 31 Jan 2015
01:05:53 -0800 (PST)
Received: from localhost (localhost [IPv6:::1])
by zmail.zimbra.com (Postfix) with ESMTP id 7970049F7
for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:53 -0800 (PST)
by localhost (zimbra.server.com [IPv6:::1]) (amavisd-new, port 10032)
with ESMTP id rPQ4Stk2X1Sy for <admin@zimbra.server.com>;
Sat, 31 Jan 2015 01:05:52 -0800 (PST)
Received: from localhost (localhost [IPv6:::1])
by zmail.zimbra.com (Postfix) with ESMTP id 3E2384A4C
for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:52 -0800 (PST)
Received: from zmail.zimbra.com ([IPv6:::1])
by localhost (zimbra.server.com [IPv6:::1]) (amavisd-new, port 10026)
with ESMTP id 24DhbFXYNwUn for <admin@zimbra.server.com>;
Sat, 31 Jan 2015 01:05:51 -0800 (PST)
Received: from zimbra.server.com (localhost [127.0.0.1])
by zmail.zimbra.com (Postfix) with ESMTP id E388C4A46
for <admin@zimbra.server.com>; Sat, 31 Jan 2015 01:05:51 -0800 (PST)
Date: Sat, 31 Jan 2015 01:05:51 -0800 (PST)
From: admin <admin@zimbra.server.com>
To: admin@zimbra.server.com
Message-ID: <1205797309.15.1422695151122.JavaMail.zimbra@zimbra.server.com>
In-Reply-To: <843455228.13.1422695143436.JavaMail.zimbra@zimbra.server.com>
Subject: Re: ZCS Backup Report: SUCCESS
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Auto-Submitted: auto-replied (zimbra; vacation)
Precedence: bulk
X-Mailer: Zimbra 8.0.6_GA_5922
Thread-Topic: ZCS Backup Report: SUCCESS
Thread-Index: FP1J+yf4TQ4duzXiVDb92nMu0M2BmQ==


The Body:

Included in the body is basic content in form of unstructured text; possible containing a signature block at the end. The content of this section would be comparable to the "body" of a regular letter.

An example "body" section can be seen below:

Hello, 
During the Zimbra Admin Training class, we will be installing Zimbra on Amazon EC2 instances. 
Sincerely, 
Your Zimbra Training Administrator 
Zimbra | Community & Collaboration 
1855 S Grant Street, Suite 200, San Mateo, CA 94402 USA


The Message Header Explained

Every message has only one header structured into fields. Every field has a name and a value as outlined in RFC5322.


Header Fields:

A message header mustinclude the following fields:

•From: The email address, and optionally the name of the author(s). In many email clients not changeable except through changing account settings.

•Date: The local time and date when the message was written. Like the From: field, many email clients fill this in automatically when sending. The recipient's client may then display the time in the format and time zone local to him/her.


A message header should include the following fields:

•Message-ID: Also an automatically generated field; used to prevent multiple delivery and for reference in In-Reply-To: (see below).

•In-Reply-To: Message-ID of the message that this is a reply to. Used to link related messages together. This field only applies for reply messages.


Please refer to RFC3864 for elaboration on message header fields.

Simple Mail Transport Protocol

SMTP Explained:

The Simple Mail Trasnport Protocol, more commonly known as "SMTP", is an internet standard for e-mail transmission, first established in RFC821 and later expanded to include "Extended SMTP" via RFC5321; the protocol most widely used today.

SMTP by default uses port 25 for mail submission. Secure SMTP, known as SMTPS, defaults to port 465 in legacy configurations and port 587 in more current configurations.

The default values in a Zimbra deployment are port 25 for SMTP and port 587 for SMTP over TLS (SMTPS).

SMTP: Server vs. Client

Server:

Zimbra architecture employs a server side application called Postfix, which is an open source Mail Transfer Agent (MTA), for sending and receiving messages. The implementation of Postfix included with a Zimbra deployment is customized and not interchangeable with "standard" or "vanilla" Postfix installation(s).

Zimbra translates Postfix configuration variables for both users and server values into LDAP attributes and local configuration variables (zmlocalconifg read from /opt/zimbra/conf/localconfig.xml). It's important to keep in mind that any changes made to Postfix, outside of the Zimbra local configuration or modification of LDAP variables will not remain in place post upgrade.

The following files are customized for Zimbra's Postfix installation:

•main.cf - Modified to include the LDAP tables. The configuration script in the Zimbra MTA pulls   data from the Zimbra LDAP and modifies the Postfix configuration files.
•master.cf - Modified to use Amavisd-New.


Client:

Despite the primary focus of SMTP protocols being related to server side operations, Client applications can also use SMTP for the submission of mail to an MTA server, or relay. In contrast, Client applications would use POP or IMAP for the retrieval of mail.


Zimbra: SMTP vs. LMTP:

While SMTP is employed by Zimbra to communicate and send mail to both Zimbra and non-Zimbra mail environments, Zimbra also uses another protocol, LMTP (Local Mail Transfer Protocol), to route messages to the appropriate mailstore.

Postfix performs the Zimbra mail transfer and relay of mail. Postfix receives inbound messages via SMTP (or SMTPS), and hands off the mail message(s) to the Zimbra server via LMTP, as shown in the following figure. The Zimbra MTA can also perform anti-virus and anti-spam filtering.


ZimbraMTA.png


Zimbra Mail Processing:

Mail is submitted to the Zimbra Server (Mail Submission Agent) via User Agents (UA's) on either port 25 (SMTP) or port 587 (SMTP w/TLS), traditionally. From there, messages are routed from the Mail Submission Agent to the Zimbra MTA (Mail Transfer Agent). Both submission and transfer occur within the same application, albeit use different operational procedures.

As we previously discussed, mail is routed from the Zimbra back-end (mailstore) to the Zimbra MTA with the LMTP protocol. Once the mail is received by the Zimbra MTA, SMTP is used to route the messages for both local and external delivery. All mail, whether internal or external, is routed through the Zimbra MTA; this happens to ensure that the archive function (if licensed) works accurately to save a copy of every message transmitted through the Zimbra MTA, regardless of whether a message was internal or external.


Message Queues:

During the receipt of mail, the Zimbra MTA uses a series of queues (envision a series of buckets) to manage delivery. The Zimbra MTA maintains four queues, where mail is placed temporarily during processing; incoming, active, deferred, and hold.


MtaQueues.png


From Postfix:


Queue Definitions:

The qmgr(8) daemon maintains the following queues:

      incoming
             Inbound mail from the network, or mail picked up  by  the  local
             pickup(8) daemon from the maildrop directory.
      active Messages  that the queue manager has opened for delivery. Only a
             limited number of messages is allowed to enter the active  queue
             (leaky bucket strategy, for a fixed delivery rate).
      deferred
             Mail  that  could  not  be delivered upon the first attempt. The
             queue manager implements exponential  backoff  by  doubling  the
             time between delivery attempts.
      corrupt
             Unreadable or damaged queue files are moved here for inspection.
      hold   Messages that are kept "on hold" are  kept  here  until  someone
             sets them free.

There are a few tools available to us to manage the Postfix mail queues; either from the command line or the admin interface. The screenshot below shows us the queues from the Admin Console; where you can also move mail between queues and re-queue mail for submission.


AdminQueue.png


From the command line, tools such as 'postqueue' can be used to manipulate/move messages between queues:

postqueue usage: postqueue -f | postqueue -i queueid | postqueue -p | postqueue -s site

Other tools, such as 'qshape' or 'mailq' can be used to view the mail queues on the command line.


$ qshape

Qshape.png


$ mailq

Mailq.png


In a default installation of Postfix, you'd manipulate the configuration parameters by modifying the Postfix configuration files directly, or through the use of 'postconf'. In terms of your Zimbra deployment, we use both 'zmlocalconfig' and LDAP attributes to modify these variables; this ensures any changes we've made survive upgrades and future installations.

The following variables are tracked the 'zmlocalconfig' for Postfix:

[zimbra@zcs807 ~]$ zmlocalconfig |grep -i postfix
ldap_postfix_password = *
postfix_alias_maps = hash:/etc/aliases
postfix_always_add_missing_headers = yes
postfix_bounce_notice_recipient = postmaster
postfix_bounce_queue_lifetime = 5d
postfix_broken_sasl_auth_clients = yes
postfix_command_directory = ${zimbra_home}/postfix/sbin
postfix_daemon_directory = ${zimbra_home}/postfix/libexec
postfix_delay_warning_time = 0h
postfix_enable_smtpd_policyd = no
postfix_header_checks = pcre:${zimbra_home}/conf/postfix_header_checks
postfix_import_environment = 
postfix_in_flow_delay = 1s
postfix_lmtp_connection_cache_destinations = 
postfix_lmtp_connection_cache_time_limit = 4s
postfix_lmtp_host_lookup = dns
postfix_mail_owner = postfix
postfix_mailq_path = ${zimbra_home}/postfix/sbin/mailq
postfix_manpage_directory = ${zimbra_home}/postfix/man
postfix_maximal_backoff_time = 4000s 
postfix_milter_command_timeout = 30s
postfix_milter_connect_timeout = 30s
postfix_milter_content_timeout = 300s
postfix_milter_default_action = tempfail
postfix_minimal_backoff_time = 300s
postfix_newaliases_path = ${zimbra_home}/postfix/sbin/newaliases
postfix_notify_classes = resource,software
postfix_policy_time_limit = 3600
postfix_propagate_unmatched_extensions = canonical
postfix_queue_directory = ${zimbra_home}/data/postfix/spool
postfix_queue_run_delay = 300s
postfix_sender_canonical_maps = proxy:ldap:${zimbra_home}/conf/ldap-scm.cf
postfix_sendmail_path = ${zimbra_home}/postfix/sbin/sendmail
postfix_setgid_group = postdrop
postfix_smtp_cname_overrides_servername = no
postfix_smtp_helo_name = $myhostname
postfix_smtp_sasl_auth_enable = no
postfix_smtp_sasl_security_options = noplaintext,noanonymous
postfix_smtpd_banner = $myhostname ESMTP $mail_name
postfix_smtpd_client_restrictions = reject_unauth_pipelining
postfix_smtpd_data_restrictions = reject_unauth_pipelining
postfix_smtpd_error_sleep_time = 1s
postfix_smtpd_hard_error_limit = 20
postfix_smtpd_helo_required = yes
postfix_smtpd_proxy_timeout = 100s
postfix_smtpd_reject_unlisted_recipient = no
postfix_smtpd_reject_unlisted_sender = no
postfix_smtpd_sasl_authenticated_header = no
postfix_smtpd_sasl_security_options = noanonymous
postfix_smtpd_sasl_tls_security_options = $smtpd_sasl_security_options
postfix_smtpd_soft_error_limit = 10
postfix_smtpd_tls_cert_file = ${zimbra_home}/conf/smtpd.crt
postfix_smtpd_tls_key_file = ${zimbra_home}/conf/smtpd.key
postfix_smtpd_tls_loglevel = 1
postfix_transport_maps = proxy:ldap:${zimbra_home}/conf/ldap-transport.cf
postfix_virtual_alias_domains = proxy:ldap:${zimbra_home}/conf/ldap-vad.cf
postfix_virtual_alias_expansion_limit = 10000
postfix_virtual_alias_maps = proxy:ldap:${zimbra_home}/conf/ldap-vam.cf
postfix_virtual_mailbox_domains = proxy:ldap:${zimbra_home}/conf/ldap-vmd.cf
postfix_virtual_mailbox_maps = proxy:ldap:${zimbra_home}/conf/ldap-vmm.cf
postfix_virtual_transport = error

The following are Zimbra LDAP attributes used to manipulate Postfix configuration variables as well as AS/AV configuration(s):

amavisSpamTag2Level
amavisSpamTag3Level (no attribute info)
amavisSpamTagLevel
zimbraMtaAntiSpamLockMethod
zimbraMtaAuthEnabled
zimbraMtaAuthHost
zimbraMtaAuthTarget
zimbraMtaAuthURL
zimbraMtaBlockedExtension
zimbraMtaBlockedExtensionWarnAdmin
zimbraMtaBlockedExtensionWarnRecipient
zimbraMtaCommonBlockedExtension
zimbraMtaDnsLookupsEnabled
zimbraMtaFallbackRelayHost
zimbraMtaMaxMessageSize
zimbraMtaMyDestination
zimbraMtaMyHostname
zimbraMtaMyNetworks
zimbraMtaMyOrigin
zimbraMtaNonSmtpdMilters
zimbraMtaRecipientDelimiter
zimbraMtaRelayHost
zimbraMtaRestriction
zimbraMtaSaslAuthEnable
zimbraMtaSmtpdMilters
zimbraMtaTlsAuthOnly
zimbraMtaTlsSecurityLevel
zimbraShareNotificationMtaAuthAccount
zimbraShareNotificationMtaAuthPassword
zimbraShareNotificationMtaAuthRequired
zimbraShareNotificationMtaConnectionType
zimbraShareNotificationMtaEnabled
zimbraShareNotificationMtaHostname
zimbraShareNotificationMtaPort
zimbraSpamTagPercent

For a detailed description and definition of 'zmlocalconfig' and LDAP attributes for Postfix, please refer to the following link: Postfix Configuration Variables. You can look up LDAP attributes meanings and definitions here.


How Zimbra Receives Mail:


HowMailWorks.png


The diagram above shows how Zimbra receives mail from the "outside world", or Internet. The message originates from a senders user agent. A sender's UA is either a web or desktop Client used in the composition of mail (i.e. Zimbra Web Client, Zimbra Desktop, Outlook). Once composed, the users message is routed to an MTA (Mail Transfer Agent). The MTA is responsible for DNS lookups and for locating the receiving MTA, which resides elsewhere on the Internet.

The communication that occurs between the originating and receiving MTA uses the Simple Mail Transport Protocol. Once the Zimbra MTA (Postfix) receives the message from the Internet, an LDAP lookup is performed to determine what mail host the Zimbra user resides on. The 'zimbraMailHost' attribute set for each account is responsible for providing the users store. Once a users store is identified, messages make their way from the Zimbra MTA to the mailstore via LMTP.


How Zimbra Fights Spam - AS/AV:

FightingSpam.png


Similar to Postifx, Zimbra's AS/AV service have their own configuration files and attributes or local configuration variables to manipulate their functionality. More information about how to control, train, or configure the AS/AV components can be found on the [wiki.zimbra.com/wiki/Zimbra_MTA Zimbra MTA Wiki].


Zimbra & Postfix:

As outlined before, Zimbra uses an adaptation of an open source software called postfix. There are several ways to customize and make modifications to postfix, however, it should be noted that only those postfix configuration variables tracked in the Zimbra localconfig (zmlocalconfig) and in the Zimbra LDAP (OpenLDAP) will remain in place during upgrades.

Often times as System Administrator's, we'll find the necessity to dig deeper into the configuration or take advantage of log files and log levels to help us identify and resolve problem(s). You can refer to the following link for more information of debugging Postfix and increasing or decreasing log levels here.


Click here for more information on the Zimbra MTA

DNS and MX Records

Proper configuration of both DNS and MX records is essential in order for Zimbra to send and receive mail on the internet. As well as for a successful installation of your Zimbra deployment. Many Zimbra services rely on DNS to lookup and resolve host names both internally and externally.

DNS Requirements

Before any Zimbra installation occurs, appropriate 'A' and 'MX' records must be configured for your Zimbra hostname(s). The same holds true whether you configure a service such as 'bind' or 'named' to handle name resolution, or if you use an existing DNS service already deployed in your environment.

The A Record:

An "A" record is configured for each host that resides in your Zimbra deployment. An "A" record tells the name service the fully qualified domain name and IP of a server. This information is used to internally and externally to identify where servers are located. Often, split DNS is required should your server(s) not be resolvable from outside of your network.

The MX Record:

A 'Mail Exchanger' or 'MX' record is required in order for your Zimbra server (MTA) to appropriately route mail. An 'MX' record for a domain specifies how mail should be routed (via SMTP) for a particular domain. If multiple domains reside in your Zimbra configuration, multiple MX records are required.

You can verify or view the MX record for your domain on the CLI using 'dig':

DigMta.png


You can view both 'MX' and 'A' records of a domain with 'dig' by specifying the 'any' flag as shown below:

DigAny.png


Verified Against: Zimbra Collaboration Suite 8.6 Date Created: 01/22/2015
Article ID: https://wiki.zimbra.com/index.php?title=Troubleshooting_Course_Content_Rough_Drafts-How_Does_Email_Work Date Modified: 2015-04-15



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search