Ajcody-Hardlinks-And-Postfix-default destination recipient limit: Difference between revisions
Line 5: | Line 5: | ||
=Is Zimbra Really Using Single-Copy Message Storage= | =Is Zimbra Really Using Single-Copy Message Storage= | ||
https://wiki.zimbra.com/wiki/Zimbra_Server#Single-Copy_Message_Storage says: | [https://wiki.zimbra.com/wiki/Zimbra_Server#Single-Copy_Message_Storage Zimbra Server's Single-Copy Message Storage] says : ''Single-Copy Message Storage - “Single copy storage” allows messages with multiple recipients to be stored only once in the file system. On UNIX systems, the mailbox directory for each user contains a hard link to the actual file.'' | ||
' | I would say the above statement needs to be clarified by including details on postfix's default_destination_recipient_limit variable since a "message" might actually be multiple messages for the same Message-Id . | ||
==Details On Postfix Variable default_destination_recipient_limit== | ==Details On Postfix Variable default_destination_recipient_limit== |
Revision as of 07:24, 30 April 2014
- This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information. |
Is Zimbra Really Using Single-Copy Message Storage
Zimbra Server's Single-Copy Message Storage says : Single-Copy Message Storage - “Single copy storage” allows messages with multiple recipients to be stored only once in the file system. On UNIX systems, the mailbox directory for each user contains a hard link to the actual file.
I would say the above statement needs to be clarified by including details on postfix's default_destination_recipient_limit variable since a "message" might actually be multiple messages for the same Message-Id .
Details On Postfix Variable default_destination_recipient_limit
- default_destination_recipient_limit = 50
- (default: 50)
- The default maximal number of recipients per message delivery. This is the default limit for delivery via the lmtp(8), pipe(8), smtp(8) and virtual(8) delivery agents.
- http://www.postfix.org/postconf.5.html#default_destination_recipient_limit
$ postconf default_destination_recipient_limit default_destination_recipient_limit = 50
Hardlinks And How the Variable default_destination_recipient_limit From Postfix Interact
When an email comes in for multiple recipients on the same mailstore, Zimbra will use hard links to save the message blob on the filesystem under the zimbra volume [default primary message store path is /opt/zimbra/store]. This will generally be true until the postfix variable default_destination_recipient_limit is exceeded - the default is 50. After this limit, you'll see more inodes being used for message.
Example When Default Destination Recipient Limit Is NOT Exceeded
An example of a message with the number of recipients under the default_destination_recipient_limit limit. I sent an email to a DL with four users in it and then grep'ed the /opt/zimbra/log/mailbox.log with the Message-ID.
Please note - the blob id's on the filesystem can be different for the different users, you'll notice this in the example below.
[root@zcs806 0]# zgrep dl /opt/zimbra/log/mailbox.log.2014-04-* | grep Add | grep "20140425192200.44AA4346F" /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,431 INFO [LmtpServer-1] [name=dluser1@zcs806.DOMAIN.com;mid=60;ip=10.137.27.36;] mailop - Adding Message: id=321, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox. /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,516 INFO [LmtpServer-1] [name=dluser2@zcs806.DOMAIN.com;mid=61;ip=10.137.27.36;] mailop - Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox. /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,535 INFO [LmtpServer-1] [name=dluser3@zcs806.uDOMAIN.com;mid=62;ip=10.137.27.36;] mailop - Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox. /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,546 INFO [LmtpServer-1] [name=restore_dluser1@zcs806.DOMAIN.com;mid=70;ip=10.137.27.36;] mailop - Adding Message: id=301, Message-ID<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox.
Get the mailbox Id for one of the users and identify the location path of the blob under their account directory and then do the hard link search.
[root@zcs806 0]# find /opt/zimbra/store -xdev -samefile ./70/msg/0/301-1501.msg | xargs ls -ai --full-time 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg
The 13430 above states the inode [option -i in the ls] and 4 is the number of 'hard links' to the same file.
If you have the Message-ID and want to search upon that, you could do the following below. Note, I included the size also with -size 1158c .
[root@zcs806 0]# for hardlink in `find /opt/zimbra/store -type f -size 1158c -print0 | xargs -0 grep -l "20140425192200.44AA4346F"` > do > ls -ai --full-time $hardlink > done 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg
Detailed Walk-through Example When Default Destination Recipient Limit Is Exceeded
Test Setup Commands
$ su - zimbra $ for i in $(seq 1 1000); do echo "ca largedl-user$i@zcs806.DOMAIN.com PaSSW0RD" >> /tmp/dl-users.zmp ; done $ time zmprov -f /tmp/dl-users.zmp $ for i in $(seq 1 1000); do printf "largedl-user$i@zcs806.DOMAIN.com " >> /tmp/dl-users ; done $ zmprov adlm large-dl@`zmhostname` `cat /tmp/dl-users` # Confirm we have 1000 users in the DL we made $ zmprov gdlm large-dl@`zmhostname` | tail -n +3 | grep '@' | wc -l 1000 # Create A Test Email vi /tmp/test-email-DL.txt To: large-dl@zcs806.DOMAIN.com Subject: Test message From: admin@zcs806.DOMAIN.com test $ /opt/zimbra/postfix/sbin/sendmail -Am -t < /tmp/test-email-DL.txt
Gathering Necessary Message Details
- Get Message-Id from one of the messages delivered.
Message-ID is in the header information of the email delivered and it's also logged in /var/log/zimbra.log .
Reporting Message Blob And Hard Link Details
Get the number of message blobs that have the Message-Id of 20140428212948.74DF13484
$ cd /opt/zimbra/store/ $ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | wc -l 1000
Get the total unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory]
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort | wc -l 28
Get the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory]
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print "Inode Number = " $1 " Number of Hard Links To Inode = " $3 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort Inode Number = 13400 Number of Hard Links To Inode = 50 Inode Number = 13444 Number of Hard Links To Inode = 50 Inode Number = 13449 Number of Hard Links To Inode = 50 Inode Number = 13451 Number of Hard Links To Inode = 1 Inode Number = 13452 Number of Hard Links To Inode = 50 Inode Number = 13453 Number of Hard Links To Inode = 49 Inode Number = 13454 Number of Hard Links To Inode = 48 Inode Number = 13455 Number of Hard Links To Inode = 50 Inode Number = 13456 Number of Hard Links To Inode = 50 Inode Number = 13457 Number of Hard Links To Inode = 50 Inode Number = 13458 Number of Hard Links To Inode = 50 Inode Number = 13459 Number of Hard Links To Inode = 1 Inode Number = 13460 Number of Hard Links To Inode = 49 Inode Number = 13461 Number of Hard Links To Inode = 2 Inode Number = 13462 Number of Hard Links To Inode = 1 Inode Number = 13464 Number of Hard Links To Inode = 1 Inode Number = 13465 Number of Hard Links To Inode = 2 Inode Number = 13467 Number of Hard Links To Inode = 1 Inode Number = 13469 Number of Hard Links To Inode = 1 Inode Number = 13473 Number of Hard Links To Inode = 49 Inode Number = 13474 Number of Hard Links To Inode = 49 Inode Number = 13475 Number of Hard Links To Inode = 49 Inode Number = 13476 Number of Hard Links To Inode = 50 Inode Number = 13477 Number of Hard Links To Inode = 48 Inode Number = 13478 Number of Hard Links To Inode = 50 Inode Number = 13479 Number of Hard Links To Inode = 50 Inode Number = 13480 Number of Hard Links To Inode = 50 Inode Number = 13481 Number of Hard Links To Inode = 49
Save the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes.txt] - [still in the /opt/zimbra/store directory]
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print $1 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort > /tmp/inodes.txt
Save all ls details and md5sums of the inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes-details.txt]
$ for i in `cat /tmp/inodes.txt`; do find /opt/zimbra/store -inum $i -exec ls -ai --full-time {} \; -exec md5sum {} \; ; done
I placed the report on a dedicated wiki page because of it's length:
Let's now show the differences between two of those files in the report above:
13400 -rw-r----- 50 zimbra zimbra 1160 2014-04-28 14:31:24.000000000 -0700 /opt/zimbra/store/0/714/msg/0/257-2.msg ca68f4ca1a1c198be3f3bf4746ae635a /opt/zimbra/store/0/714/msg/0/257-2.msg -- vs -- 13481 -rw-r----- 49 zimbra zimbra 1160 2014-04-28 14:31:48.000000000 -0700 /opt/zimbra/store/0/944/msg/0/257-2.msg 0e729c9bbad30c94219154617a64db09 /opt/zimbra/store/0/944/msg/0/257-2.msg
And now the message differences:
]$ diff /opt/zimbra/store/0/944/msg/0/257-2.msg /opt/zimbra/store/0/714/msg/0/257-2.msg 4c4 < 14:31:47 -0700 (PDT) --- > 14:31:24 -0700 (PDT) 6,7c6,7 < by zcs806.us.zimbralab.com (Postfix) with ESMTP id C95D634A0; < Mon, 28 Apr 2014 14:30:49 -0700 (PDT) --- > by zcs806.us.zimbralab.com (Postfix) with ESMTP id 8619E3493; > Mon, 28 Apr 2014 14:30:22 -0700 (PDT) 17c17 < with ESMTP id 9v2smobzMGcy; Mon, 28 Apr 2014 14:30:29 -0700 (PDT) --- > with ESMTP id ONyocz3V6Pmq; Mon, 28 Apr 2014 14:30:08 -0700 (PDT)