Ajcody-Hardlinks-And-Postfix-default destination recipient limit

1. Zimbra Tech Center
2. Zeta Alliance
3. Ajcody-Hardlinks-And-Postfix-default destination recipient limit

Is Zimbra Really Using Single-Copy Message Storage

 - This article is a Work in Progress, and may be unfinished or missing sections.

Zimbra Server's Single-Copy Message Storage says : Single-Copy Message Storage - “Single copy storage” allows messages with multiple recipients to be stored only once in the file system. On UNIX systems, the mailbox directory for each user contains a hard link to the actual file.

I would say the above statement needs to be clarified by including details on postfix's default_destination_recipient_limit variable since a "message" might actually be multiple messages for the same Message-Id .

Details On Postfix Variable default_destination_recipient_limit

• default_destination_recipient_limit = 50
  • (default: 50)
  • The default maximal number of recipients per message delivery. This is the default limit for delivery via the lmtp(8), pipe(8), smtp(8) and virtual(8) delivery agents.
  • http://www.postfix.org/postconf.5.html#default_destination_recipient_limit

$ postconf default_destination_recipient_limit
default_destination_recipient_limit = 50 

Hardlinks And How the Variable default_destination_recipient_limit From Postfix Interact

When an email comes in for multiple recipients on the same mailstore, Zimbra will use hard links to save the message blob on the filesystem under the zimbra volume [default primary message store path is /opt/zimbra/store]. This will generally be true until the postfix variable default_destination_recipient_limit is exceeded - the default is 50. After this limit, you'll see more inodes being used for message.

Example When Default Destination Recipient Limit Is NOT Exceeded

An example of a message with the number of recipients under the default_destination_recipient_limit limit. I sent an email to a DL with four users in it and then grep'ed the /opt/zimbra/log/mailbox.log with the Message-ID.

Please note - the blob id's on the filesystem can be different for the different users, you'll notice this in the example below.

[root@zcs806 0]# zgrep dl /opt/zimbra/log/mailbox.log.2014-04-* | grep Add | grep "20140425192200.44AA4346F"

/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,431 INFO  [LmtpServer-1]
 [name=dluser1@zcs806.DOMAIN.com;mid=60;ip=10.137.27.36;] mailop - 
 Adding Message: id=321, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.
/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,516 INFO  [LmtpServer-1] 
 [name=dluser2@zcs806.DOMAIN.com;mid=61;ip=10.137.27.36;] mailop - 
 Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.
/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,535 INFO  [LmtpServer-1] 
 [name=dluser3@zcs806.uDOMAIN.com;mid=62;ip=10.137.27.36;] mailop - 
 Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.
/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,546 INFO  [LmtpServer-1] 
 [name=restore_dluser1@zcs806.DOMAIN.com;mid=70;ip=10.137.27.36;] mailop - 
 Adding Message: id=301, Message-ID<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.

Get the mailbox Id for one of the users and identify the location path of the blob under their account directory and then do the hard link search.

[root@zcs806 0]# find /opt/zimbra/store -xdev -samefile ./70/msg/0/301-1501.msg | xargs ls -ai --full-time
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg

The 13430 above states the inode [option -i in the ls] and 4 is the number of 'hard links' to the same file.

If you have the Message-ID and want to search upon that, you could do the following below. Note, I included the size also with -size 1158c .

[root@zcs806 0]# for hardlink in `find /opt/zimbra/store -type f -size 1158c -print0 | xargs -0 grep -l "20140425192200.44AA4346F"`
> do
>  ls -ai --full-time $hardlink
> done
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg

Detailed Walk-through Example When Default Destination Recipient Limit Is Exceeded

Test Setup Commands

$ su - zimbra

$ for i in $(seq 1 1000); do echo "ca largedl-user$i@zcs806.DOMAIN.com PaSSW0RD" >> /tmp/dl-users.zmp ; done 

$ time zmprov -f /tmp/dl-users.zmp 

$ for i in $(seq 1 1000); do printf "largedl-user$i@zcs806.DOMAIN.com " >> /tmp/dl-users ; done 

$ zmprov adlm large-dl@`zmhostname` `cat /tmp/dl-users` 

# Confirm we have 1000 users in the DL we made
$ zmprov gdlm large-dl@`zmhostname` | tail -n +3 | grep '@' | wc -l
  1000

# Create A Test Email
vi /tmp/test-email-DL.txt
To: large-dl@zcs806.DOMAIN.com
Subject: Test message
From: admin@zcs806.DOMAIN.com
test

$ /opt/zimbra/postfix/sbin/sendmail -Am -t < /tmp/test-email-DL.txt

Gathering Necessary Message Details

• Get Message-Id from one of the messages delivered.
  • See http://wiki.zimbra.com/wiki/Ajcody-MTA-Postfix-Topics#Postfix_Queue_ID_vs._message-id

Message-ID is in the header information of the email delivered and it's also logged in /var/log/zimbra.log .

Reporting Message Blob And Hard Link Details

Get the number of message blobs that have the Message-Id of 20140428212948.74DF13484

$ cd /opt/zimbra/store/
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | wc -l
  1000

Get the total unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory]

$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort | wc -l
  28

Get the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory]

$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print "Inode Number = " $1 " Number of Hard Links To Inode = " $3 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort
Inode Number = 13400 Number of Hard Links To Inode = 50
Inode Number = 13444 Number of Hard Links To Inode = 50
Inode Number = 13449 Number of Hard Links To Inode = 50
Inode Number = 13451 Number of Hard Links To Inode = 1
Inode Number = 13452 Number of Hard Links To Inode = 50
Inode Number = 13453 Number of Hard Links To Inode = 49
Inode Number = 13454 Number of Hard Links To Inode = 48
Inode Number = 13455 Number of Hard Links To Inode = 50
Inode Number = 13456 Number of Hard Links To Inode = 50
Inode Number = 13457 Number of Hard Links To Inode = 50
Inode Number = 13458 Number of Hard Links To Inode = 50
Inode Number = 13459 Number of Hard Links To Inode = 1
Inode Number = 13460 Number of Hard Links To Inode = 49
Inode Number = 13461 Number of Hard Links To Inode = 2
Inode Number = 13462 Number of Hard Links To Inode = 1
Inode Number = 13464 Number of Hard Links To Inode = 1
Inode Number = 13465 Number of Hard Links To Inode = 2
Inode Number = 13467 Number of Hard Links To Inode = 1
Inode Number = 13469 Number of Hard Links To Inode = 1
Inode Number = 13473 Number of Hard Links To Inode = 49
Inode Number = 13474 Number of Hard Links To Inode = 49
Inode Number = 13475 Number of Hard Links To Inode = 49
Inode Number = 13476 Number of Hard Links To Inode = 50
Inode Number = 13477 Number of Hard Links To Inode = 48
Inode Number = 13478 Number of Hard Links To Inode = 50
Inode Number = 13479 Number of Hard Links To Inode = 50
Inode Number = 13480 Number of Hard Links To Inode = 50
Inode Number = 13481 Number of Hard Links To Inode = 49

Save the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes.txt] - [still in the /opt/zimbra/store directory]

$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print $1 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort > /tmp/inodes.txt

Save all ls details and md5sums of the inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes-details.txt]

$ for i in `cat /tmp/inodes.txt`; do find /opt/zimbra/store -inum $i -exec ls -ai --full-time {} \; -exec md5sum {} \; ; done

I placed the report on a dedicated wiki page because of it's length:

• https://wiki.zimbra.com/wiki/Ajcody-inodes-details-report

Let's now show the differences between two of those files in the report above:

13400 -rw-r----- 50 zimbra zimbra 1160 2014-04-28 14:31:24.000000000 -0700 /opt/zimbra/store/0/714/msg/0/257-2.msg
ca68f4ca1a1c198be3f3bf4746ae635a  /opt/zimbra/store/0/714/msg/0/257-2.msg
 -- vs --
13481 -rw-r----- 49 zimbra zimbra 1160 2014-04-28 14:31:48.000000000 -0700 /opt/zimbra/store/0/944/msg/0/257-2.msg
0e729c9bbad30c94219154617a64db09  /opt/zimbra/store/0/944/msg/0/257-2.msg

And now the message differences:

]$ diff /opt/zimbra/store/0/944/msg/0/257-2.msg /opt/zimbra/store/0/714/msg/0/257-2.msg
4c4
<  14:31:47 -0700 (PDT)
---
>  14:31:24 -0700 (PDT)
6,7c6,7
<       by zcs806.us.zimbralab.com (Postfix) with ESMTP id C95D634A0;
<       Mon, 28 Apr 2014 14:30:49 -0700 (PDT)
---
>       by zcs806.us.zimbralab.com (Postfix) with ESMTP id 8619E3493;
>       Mon, 28 Apr 2014 14:30:22 -0700 (PDT)
17c17
<       with ESMTP id 9v2smobzMGcy; Mon, 28 Apr 2014 14:30:29 -0700 (PDT)
---
>       with ESMTP id ONyocz3V6Pmq; Mon, 28 Apr 2014 14:30:08 -0700 (PDT)

What About zmdedupe

zmdedupe doesn't apply here because the blobs in questions are actually different.

$ zmdedupe stats
usage: zmdedupe [options] start/status/stop
  -h,--help                  Display this help message.
  -v,--verbose               Display stack trace on error.
     --volumes <volume-ids>  Specify which volumes to dedupe.  If not specified,
                             dedupe all volumes.

The "start/stop" command is required, to avoid unintentionally running a blob
dedupe.  Id values are separated by commas.

$ zmdedupe status
Status = stopped
Total links created = 0
Total size saved = 0

$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print "Inode Number = " $1 " Number of Hard Links To Inode = " $3 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort
Inode Number = 13400 Number of Hard Links To Inode = 50
Inode Number = 13444 Number of Hard Links To Inode = 50
Inode Number = 13449 Number of Hard Links To Inode = 50
Inode Number = 13451 Number of Hard Links To Inode = 1
Inode Number = 13452 Number of Hard Links To Inode = 50
Inode Number = 13453 Number of Hard Links To Inode = 49
Inode Number = 13454 Number of Hard Links To Inode = 48
Inode Number = 13455 Number of Hard Links To Inode = 50
Inode Number = 13456 Number of Hard Links To Inode = 50
Inode Number = 13457 Number of Hard Links To Inode = 50
Inode Number = 13458 Number of Hard Links To Inode = 50
Inode Number = 13459 Number of Hard Links To Inode = 1
Inode Number = 13460 Number of Hard Links To Inode = 49
Inode Number = 13461 Number of Hard Links To Inode = 2
Inode Number = 13462 Number of Hard Links To Inode = 1
Inode Number = 13464 Number of Hard Links To Inode = 1
Inode Number = 13465 Number of Hard Links To Inode = 2
Inode Number = 13467 Number of Hard Links To Inode = 1
Inode Number = 13469 Number of Hard Links To Inode = 1
Inode Number = 13473 Number of Hard Links To Inode = 49
Inode Number = 13474 Number of Hard Links To Inode = 49
Inode Number = 13475 Number of Hard Links To Inode = 49
Inode Number = 13476 Number of Hard Links To Inode = 50
Inode Number = 13477 Number of Hard Links To Inode = 48
Inode Number = 13478 Number of Hard Links To Inode = 50
Inode Number = 13479 Number of Hard Links To Inode = 50
Inode Number = 13480 Number of Hard Links To Inode = 50
Inode Number = 13481 Number of Hard Links To Inode = 49

---