Ajcody-Hardlinks-And-Postfix-default destination recipient limit

- This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.

Hardlinks And How the Variable default_destination_recipient_limit From Postfix Interact

When an email comes in to multiple recipients on the same mailstore, Zimbra will use hard links to save the message blob on the filesystem under the zimbra volume [default primary message store path is /opt/zimbra/store].

For example, I sent an email to a DL with four users in it and grep the /opt/zimbra/log/mailbox.log with the Message-ID. Please note - the blob id's on the filesystem can be different for the different users, you'll notice this in the example below.

[root@zcs806 0]# zgrep dl /opt/zimbra/log/mailbox.log.2014-04-* | grep Add | grep "20140425192200.44AA4346F"

/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,431 INFO  [LmtpServer-1]
 [name=dluser1@zcs806.DOMAIN.com;mid=60;ip=10.137.27.36;] mailop - 
 Adding Message: id=321, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.
/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,516 INFO  [LmtpServer-1] 
 [name=dluser2@zcs806.DOMAIN.com;mid=61;ip=10.137.27.36;] mailop - 
 Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.
/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,535 INFO  [LmtpServer-1] 
 [name=dluser3@zcs806.uDOMAIN.com;mid=62;ip=10.137.27.36;] mailop - 
 Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.
/opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,546 INFO  [LmtpServer-1] 
 [name=restore_dluser1@zcs806.DOMAIN.com;mid=70;ip=10.137.27.36;] mailop - 
 Adding Message: id=301, Message-ID<20140425192200.44AA4346F@zcs806.DOMAIN.com>, 
 parentId=-1, folderId=2, folderName=Inbox.

Get the mailbox Id for one of the users and identify the location path of the blob under their account directory and then do the hard link search.

[root@zcs806 0]# find /opt/zimbra/store -xdev -samefile ./70/msg/0/301-1501.msg | xargs ls -ai --full-time
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg

The 13430 above states the inode [option -i in the ls] and 4 is the number of 'hard links' to the same file.

If you have the Message-ID and want to search upon that, you could do the following below. Note, I included the size also with -size 1158c .

[root@zcs806 0]# for hardlink in `find /opt/zimbra/store -type f -size 1158c -print0 | xargs -0 grep -l "20140425192200.44AA4346F"`
> do
>  ls -ai --full-time $hardlink
> done
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg

Details On Postfix Variable default_destination_recipient_limit

• default_destination_recipient_limit = 50
  • (default: 50)
  • The default maximal number of recipients per message delivery. This is the default limit for delivery via the lmtp(8), pipe(8), smtp(8) and virtual(8) delivery agents.
  • http://www.postfix.org/postconf.5.html#default_destination_recipient_limit

$ postconf default_destination_recipient_limit
default_destination_recipient_limit = 50 

Detailed Walk-through Example When Default Destination Recipient Limit Is Exceeded

Test Setup Commands

$ su - zimbra

$ for i in $(seq 1 1000); do echo "ca largedl-user$i@zcs806.DOMAIN.com PaSSW0RD" >> /tmp/dl-users.zmp ; done 

$ time zmprov -f /tmp/dl-users.zmp 

$ for i in $(seq 1 1000); do printf "largedl-user$i@zcs806.DOMAIN.com " >> /tmp/dl-users ; done 

$ zmprov adlm large-dl@`zmhostname` `cat /tmp/dl-users` 

# Confirm we have 1000 users in the DL we made
$ zmprov gdlm large-dl@`zmhostname` | tail -n +3 | grep '@' | wc -l
  1000

# Create A Test Email
vi /tmp/test-email-DL.txt
To: large-dl@zcs806.DOMAIN.com
Subject: Test message
From: admin@zcs806.DOMAIN.com
test

$ /opt/zimbra/postfix/sbin/sendmail -Am -t < /tmp/test-email-DL.txt

Gathering Necessary Message Details

• Get Message-Id from one of the messages delivered.
  • See http://wiki.zimbra.com/wiki/Ajcody-MTA-Postfix-Topics#Postfix_Queue_ID_vs._message-id

Message-ID is in the header information of the email delivered and it's also logged in /var/log/zimbra.log .

Reporting Message Blob And Hard Link Details

$ cd /opt/zimbra/store/

# Number of message blobs that have the Message-Id of 20140428212948.74DF13484
# In the /opt/zimbra/store directory
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | wc -l
  1000

# The total unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484
# In the /opt/zimbra/store directory
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort | wc -l
  28

# The unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484
# In the /opt/zimbra/store directory
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort
13400
13444
13449
13451
13452
13453
13454
13455
13456
13457
13458
13459
13460
13461
13462
13464
13465
13467
13469
13473
13474
13475
13476
13477
13478
13479
13480
13481

# Save the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes.txt]
# In the /opt/zimbra/store directory
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort > /tmp/inodes.txt

# Save all ls details and md5sums of the inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes-details.txt]
$ for i in `cat /tmp/inodes.txt`; do find /opt/zimbra/store -inum $i -exec ls -ai --full-time {} \; -exec md5sum {} \; ; done

I place the report on a dedicated wiki page because of it's length:

• https://wiki.zimbra.com/wiki/Ajcody-inodes-details-report

Let's now show the differences between two of those files in the report above:

13400 -rw-r----- 50 zimbra zimbra 1160 2014-04-28 14:31:24.000000000 -0700 /opt/zimbra/store/0/714/msg/0/257-2.msg
ca68f4ca1a1c198be3f3bf4746ae635a  /opt/zimbra/store/0/714/msg/0/257-2.msg
 -- vs --
13481 -rw-r----- 49 zimbra zimbra 1160 2014-04-28 14:31:48.000000000 -0700 /opt/zimbra/store/0/944/msg/0/257-2.msg
0e729c9bbad30c94219154617a64db09  /opt/zimbra/store/0/944/msg/0/257-2.msg

And now the message differences:

]$ diff /opt/zimbra/store/0/944/msg/0/257-2.msg /opt/zimbra/store/0/714/msg/0/257-2.msg
4c4
<  14:31:47 -0700 (PDT)
---
>  14:31:24 -0700 (PDT)
6,7c6,7
<       by zcs806.us.zimbralab.com (Postfix) with ESMTP id C95D634A0;
<       Mon, 28 Apr 2014 14:30:49 -0700 (PDT)
---
>       by zcs806.us.zimbralab.com (Postfix) with ESMTP id 8619E3493;
>       Mon, 28 Apr 2014 14:30:22 -0700 (PDT)
17c17
<       with ESMTP id 9v2smobzMGcy; Mon, 28 Apr 2014 14:30:29 -0700 (PDT)
---
>       with ESMTP id ONyocz3V6Pmq; Mon, 28 Apr 2014 14:30:08 -0700 (PDT)