Ajcody-Hardlinks-And-Postfix-default destination recipient limit: Difference between revisions
mNo edit summary |
|||
(13 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
{| | {{BC|Zeta Alliance}} <!-- Note, this will also add [[Category: Zeta Alliance]] to bottom of wiki page. --> | ||
__FORCETOC__ <!-- Will force a TOC regards of size of article. __NOTOC__ if no TOC is wanted. --> | |||
|} | <div class="col-md-12 ibox-content"> | ||
=Is Zimbra Really Using Single-Copy Message Storage= <!-- Normally will reflect page title. Is listed at very top of page. --> | |||
{{KB|{{ZETA}}|{{ZCS 8.5}}|{{ZCS 8.0}}|{{ZCS 7.0}}|}} <!-- Can only handle 3 ZCS versions. --> | |||
{{WIP}} <!-- For pages that are "work in progress". --> | |||
[https://wiki.zimbra.com/wiki/Zimbra_Server#Single-Copy_Message_Storage Zimbra Server's Single-Copy Message Storage] says : ''Single-Copy Message Storage - “Single copy storage” allows messages with multiple recipients to be stored only once in the file system. On UNIX systems, the mailbox directory for each user contains a hard link to the actual file.'' | |||
I would say the above statement needs to be clarified by including details on postfix's default_destination_recipient_limit variable since a "message" might actually be multiple messages for the same Message-Id . | |||
==Details On Postfix Variable default_destination_recipient_limit== | |||
* default_destination_recipient_limit = 50 | |||
** (default: 50) | |||
** The default maximal number of recipients per message delivery. This is the default limit for delivery via the lmtp(8), pipe(8), smtp(8) and virtual(8) delivery agents. | |||
** http://www.postfix.org/postconf.5.html#default_destination_recipient_limit | |||
<pre> | |||
$ postconf default_destination_recipient_limit | |||
default_destination_recipient_limit = 50 | |||
</pre> | |||
==Hardlinks And How the Variable default_destination_recipient_limit From Postfix Interact== | |||
When an email comes in for multiple recipients on the same mailstore, Zimbra will use hard links to save the message blob on the filesystem under the zimbra volume [default primary message store path is /opt/zimbra/store]. This will generally be true until the postfix variable default_destination_recipient_limit is exceeded - the default is 50. After this limit, you'll see more inodes being used for message. | |||
===Example When Default Destination Recipient Limit Is NOT Exceeded=== | |||
An example of a message with the number of recipients under the default_destination_recipient_limit limit. I sent an email to a DL with four users in it and then grep'ed the /opt/zimbra/log/mailbox.log with the Message-ID. | |||
'''Please note - the blob id's on the filesystem can be different for the different users, you'll notice this in the example below.''' | |||
<pre> | <pre> | ||
Line 53: | Line 76: | ||
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg | 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg | ||
13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg | 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg | ||
</pre> | </pre> | ||
Line 105: | Line 116: | ||
===Reporting Message Blob And Hard Link Details=== | ===Reporting Message Blob And Hard Link Details=== | ||
Get the number of message blobs that have the Message-Id of 20140428212948.74DF13484 | |||
<pre> | <pre> | ||
$ cd /opt/zimbra/store/ | $ cd /opt/zimbra/store/ | ||
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | wc -l | $ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | wc -l | ||
1000 | 1000 | ||
</pre> | |||
Get the total unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory] | |||
<pre> | |||
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort | wc -l | $ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort | wc -l | ||
28 | 28 | ||
</pre> | |||
Get the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory] | |||
<pre> | |||
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print "Inode Number = " $1 " Number of Hard Links To Inode = " $3 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort | |||
Inode Number = 13400 Number of Hard Links To Inode = 50 | |||
Inode Number = 13444 Number of Hard Links To Inode = 50 | |||
Inode Number = 13449 Number of Hard Links To Inode = 50 | |||
Inode Number = 13451 Number of Hard Links To Inode = 1 | |||
Inode Number = 13452 Number of Hard Links To Inode = 50 | |||
Inode Number = 13453 Number of Hard Links To Inode = 49 | |||
Inode Number = 13454 Number of Hard Links To Inode = 48 | |||
Inode Number = 13455 Number of Hard Links To Inode = 50 | |||
Inode Number = 13456 Number of Hard Links To Inode = 50 | |||
Inode Number = 13457 Number of Hard Links To Inode = 50 | |||
Inode Number = 13458 Number of Hard Links To Inode = 50 | |||
Inode Number = 13459 Number of Hard Links To Inode = 1 | |||
Inode Number = 13460 Number of Hard Links To Inode = 49 | |||
Inode Number = 13461 Number of Hard Links To Inode = 2 | |||
Inode Number = 13462 Number of Hard Links To Inode = 1 | |||
Inode Number = 13464 Number of Hard Links To Inode = 1 | |||
Inode Number = 13465 Number of Hard Links To Inode = 2 | |||
Inode Number = 13467 Number of Hard Links To Inode = 1 | |||
Inode Number = 13469 Number of Hard Links To Inode = 1 | |||
Inode Number = 13473 Number of Hard Links To Inode = 49 | |||
Inode Number = 13474 Number of Hard Links To Inode = 49 | |||
Inode Number = 13475 Number of Hard Links To Inode = 49 | |||
Inode Number = 13476 Number of Hard Links To Inode = 50 | |||
Inode Number = 13477 Number of Hard Links To Inode = 48 | |||
Inode Number = 13478 Number of Hard Links To Inode = 50 | |||
Inode Number = 13479 Number of Hard Links To Inode = 50 | |||
Inode Number = 13480 Number of Hard Links To Inode = 50 | |||
Inode Number = 13481 Number of Hard Links To Inode = 49 | |||
</pre> | |||
Save the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes.txt] - [still in the /opt/zimbra/store directory] | |||
<pre> | |||
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | | $ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print $1 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort > /tmp/inodes.txt | ||
</pre> | |||
Save all ls details and md5sums of the inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes-details.txt] | |||
<pre> | |||
$ for i in `cat /tmp/inodes.txt`; do find /opt/zimbra/store -inum $i -exec ls -ai --full-time {} \; -exec md5sum {} \; ; done | $ for i in `cat /tmp/inodes.txt`; do find /opt/zimbra/store -inum $i -exec ls -ai --full-time {} \; -exec md5sum {} \; ; done | ||
</pre> | </pre> | ||
I placed the report on a dedicated wiki page because of it's length: | |||
* https://wiki.zimbra.com/wiki/Ajcody-inodes-details-report | |||
Let's now show the differences between two of those files in the report above: | Let's now show the differences between two of those files in the report above: | ||
Line 474: | Line 203: | ||
> with ESMTP id ONyocz3V6Pmq; Mon, 28 Apr 2014 14:30:08 -0700 (PDT) | > with ESMTP id ONyocz3V6Pmq; Mon, 28 Apr 2014 14:30:08 -0700 (PDT) | ||
</pre> | </pre> | ||
===What About zmdedupe=== | |||
zmdedupe doesn't apply here because the blobs in questions are actually different. | |||
<pre> | |||
$ zmdedupe stats | |||
usage: zmdedupe [options] start/status/stop | |||
-h,--help Display this help message. | |||
-v,--verbose Display stack trace on error. | |||
--volumes <volume-ids> Specify which volumes to dedupe. If not specified, | |||
dedupe all volumes. | |||
The "start/stop" command is required, to avoid unintentionally running a blob | |||
dedupe. Id values are separated by commas. | |||
$ zmdedupe status | |||
Status = stopped | |||
Total links created = 0 | |||
Total size saved = 0 | |||
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print "Inode Number = " $1 " Number of Hard Links To Inode = " $3 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort | |||
Inode Number = 13400 Number of Hard Links To Inode = 50 | |||
Inode Number = 13444 Number of Hard Links To Inode = 50 | |||
Inode Number = 13449 Number of Hard Links To Inode = 50 | |||
Inode Number = 13451 Number of Hard Links To Inode = 1 | |||
Inode Number = 13452 Number of Hard Links To Inode = 50 | |||
Inode Number = 13453 Number of Hard Links To Inode = 49 | |||
Inode Number = 13454 Number of Hard Links To Inode = 48 | |||
Inode Number = 13455 Number of Hard Links To Inode = 50 | |||
Inode Number = 13456 Number of Hard Links To Inode = 50 | |||
Inode Number = 13457 Number of Hard Links To Inode = 50 | |||
Inode Number = 13458 Number of Hard Links To Inode = 50 | |||
Inode Number = 13459 Number of Hard Links To Inode = 1 | |||
Inode Number = 13460 Number of Hard Links To Inode = 49 | |||
Inode Number = 13461 Number of Hard Links To Inode = 2 | |||
Inode Number = 13462 Number of Hard Links To Inode = 1 | |||
Inode Number = 13464 Number of Hard Links To Inode = 1 | |||
Inode Number = 13465 Number of Hard Links To Inode = 2 | |||
Inode Number = 13467 Number of Hard Links To Inode = 1 | |||
Inode Number = 13469 Number of Hard Links To Inode = 1 | |||
Inode Number = 13473 Number of Hard Links To Inode = 49 | |||
Inode Number = 13474 Number of Hard Links To Inode = 49 | |||
Inode Number = 13475 Number of Hard Links To Inode = 49 | |||
Inode Number = 13476 Number of Hard Links To Inode = 50 | |||
Inode Number = 13477 Number of Hard Links To Inode = 48 | |||
Inode Number = 13478 Number of Hard Links To Inode = 50 | |||
Inode Number = 13479 Number of Hard Links To Inode = 50 | |||
Inode Number = 13480 Number of Hard Links To Inode = 50 | |||
Inode Number = 13481 Number of Hard Links To Inode = 49 | |||
</pre> | |||
---- | |||
[[Category: Community Sandbox]] | |||
[[Category: Author:Ajcody]] | |||
[[Category: Zeta Alliance]] |
Latest revision as of 17:06, 20 June 2016
Is Zimbra Really Using Single-Copy Message Storage
Zimbra Server's Single-Copy Message Storage says : Single-Copy Message Storage - “Single copy storage” allows messages with multiple recipients to be stored only once in the file system. On UNIX systems, the mailbox directory for each user contains a hard link to the actual file.
I would say the above statement needs to be clarified by including details on postfix's default_destination_recipient_limit variable since a "message" might actually be multiple messages for the same Message-Id .
Details On Postfix Variable default_destination_recipient_limit
- default_destination_recipient_limit = 50
- (default: 50)
- The default maximal number of recipients per message delivery. This is the default limit for delivery via the lmtp(8), pipe(8), smtp(8) and virtual(8) delivery agents.
- http://www.postfix.org/postconf.5.html#default_destination_recipient_limit
$ postconf default_destination_recipient_limit default_destination_recipient_limit = 50
Hardlinks And How the Variable default_destination_recipient_limit From Postfix Interact
When an email comes in for multiple recipients on the same mailstore, Zimbra will use hard links to save the message blob on the filesystem under the zimbra volume [default primary message store path is /opt/zimbra/store]. This will generally be true until the postfix variable default_destination_recipient_limit is exceeded - the default is 50. After this limit, you'll see more inodes being used for message.
Example When Default Destination Recipient Limit Is NOT Exceeded
An example of a message with the number of recipients under the default_destination_recipient_limit limit. I sent an email to a DL with four users in it and then grep'ed the /opt/zimbra/log/mailbox.log with the Message-ID.
Please note - the blob id's on the filesystem can be different for the different users, you'll notice this in the example below.
[root@zcs806 0]# zgrep dl /opt/zimbra/log/mailbox.log.2014-04-* | grep Add | grep "20140425192200.44AA4346F" /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,431 INFO [LmtpServer-1] [name=dluser1@zcs806.DOMAIN.com;mid=60;ip=10.137.27.36;] mailop - Adding Message: id=321, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox. /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,516 INFO [LmtpServer-1] [name=dluser2@zcs806.DOMAIN.com;mid=61;ip=10.137.27.36;] mailop - Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox. /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,535 INFO [LmtpServer-1] [name=dluser3@zcs806.uDOMAIN.com;mid=62;ip=10.137.27.36;] mailop - Adding Message: id=301, Message-ID=<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox. /opt/zimbra/log/mailbox.log.2014-04-25.gz:2014-04-25 12:22:16,546 INFO [LmtpServer-1] [name=restore_dluser1@zcs806.DOMAIN.com;mid=70;ip=10.137.27.36;] mailop - Adding Message: id=301, Message-ID<20140425192200.44AA4346F@zcs806.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox.
Get the mailbox Id for one of the users and identify the location path of the blob under their account directory and then do the hard link search.
[root@zcs806 0]# find /opt/zimbra/store -xdev -samefile ./70/msg/0/301-1501.msg | xargs ls -ai --full-time 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg
The 13430 above states the inode [option -i in the ls] and 4 is the number of 'hard links' to the same file.
If you have the Message-ID and want to search upon that, you could do the following below. Note, I included the size also with -size 1158c .
[root@zcs806 0]# for hardlink in `find /opt/zimbra/store -type f -size 1158c -print0 | xargs -0 grep -l "20140425192200.44AA4346F"` > do > ls -ai --full-time $hardlink > done 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/70/msg/0/301-1501.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/60/msg/0/321-1602.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/61/msg/0/301-1336.msg 13430 -rw-r----- 4 zimbra zimbra 1158 2014-04-25 12:22:16.000000000 -0700 /opt/zimbra/store/0/62/msg/0/301-1036.msg
Detailed Walk-through Example When Default Destination Recipient Limit Is Exceeded
Test Setup Commands
$ su - zimbra $ for i in $(seq 1 1000); do echo "ca largedl-user$i@zcs806.DOMAIN.com PaSSW0RD" >> /tmp/dl-users.zmp ; done $ time zmprov -f /tmp/dl-users.zmp $ for i in $(seq 1 1000); do printf "largedl-user$i@zcs806.DOMAIN.com " >> /tmp/dl-users ; done $ zmprov adlm large-dl@`zmhostname` `cat /tmp/dl-users` # Confirm we have 1000 users in the DL we made $ zmprov gdlm large-dl@`zmhostname` | tail -n +3 | grep '@' | wc -l 1000 # Create A Test Email vi /tmp/test-email-DL.txt To: large-dl@zcs806.DOMAIN.com Subject: Test message From: admin@zcs806.DOMAIN.com test $ /opt/zimbra/postfix/sbin/sendmail -Am -t < /tmp/test-email-DL.txt
Gathering Necessary Message Details
- Get Message-Id from one of the messages delivered.
Message-ID is in the header information of the email delivered and it's also logged in /var/log/zimbra.log .
Reporting Message Blob And Hard Link Details
Get the number of message blobs that have the Message-Id of 20140428212948.74DF13484
$ cd /opt/zimbra/store/ $ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | wc -l 1000
Get the total unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory]
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | cut -d' ' -f-1 | awk '!seen[$0]++ { print; }' | uniq -u | sort | wc -l 28
Get the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 [still in the /opt/zimbra/store directory]
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print "Inode Number = " $1 " Number of Hard Links To Inode = " $3 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort Inode Number = 13400 Number of Hard Links To Inode = 50 Inode Number = 13444 Number of Hard Links To Inode = 50 Inode Number = 13449 Number of Hard Links To Inode = 50 Inode Number = 13451 Number of Hard Links To Inode = 1 Inode Number = 13452 Number of Hard Links To Inode = 50 Inode Number = 13453 Number of Hard Links To Inode = 49 Inode Number = 13454 Number of Hard Links To Inode = 48 Inode Number = 13455 Number of Hard Links To Inode = 50 Inode Number = 13456 Number of Hard Links To Inode = 50 Inode Number = 13457 Number of Hard Links To Inode = 50 Inode Number = 13458 Number of Hard Links To Inode = 50 Inode Number = 13459 Number of Hard Links To Inode = 1 Inode Number = 13460 Number of Hard Links To Inode = 49 Inode Number = 13461 Number of Hard Links To Inode = 2 Inode Number = 13462 Number of Hard Links To Inode = 1 Inode Number = 13464 Number of Hard Links To Inode = 1 Inode Number = 13465 Number of Hard Links To Inode = 2 Inode Number = 13467 Number of Hard Links To Inode = 1 Inode Number = 13469 Number of Hard Links To Inode = 1 Inode Number = 13473 Number of Hard Links To Inode = 49 Inode Number = 13474 Number of Hard Links To Inode = 49 Inode Number = 13475 Number of Hard Links To Inode = 49 Inode Number = 13476 Number of Hard Links To Inode = 50 Inode Number = 13477 Number of Hard Links To Inode = 48 Inode Number = 13478 Number of Hard Links To Inode = 50 Inode Number = 13479 Number of Hard Links To Inode = 50 Inode Number = 13480 Number of Hard Links To Inode = 50 Inode Number = 13481 Number of Hard Links To Inode = 49
Save the unique inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes.txt] - [still in the /opt/zimbra/store directory]
$ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print $1 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort > /tmp/inodes.txt
Save all ls details and md5sums of the inodes used for blobs that have Message-Id of 20140428212948.74DF13484 to a temp file [/tmp/inodes-details.txt]
$ for i in `cat /tmp/inodes.txt`; do find /opt/zimbra/store -inum $i -exec ls -ai --full-time {} \; -exec md5sum {} \; ; done
I placed the report on a dedicated wiki page because of it's length:
Let's now show the differences between two of those files in the report above:
13400 -rw-r----- 50 zimbra zimbra 1160 2014-04-28 14:31:24.000000000 -0700 /opt/zimbra/store/0/714/msg/0/257-2.msg ca68f4ca1a1c198be3f3bf4746ae635a /opt/zimbra/store/0/714/msg/0/257-2.msg -- vs -- 13481 -rw-r----- 49 zimbra zimbra 1160 2014-04-28 14:31:48.000000000 -0700 /opt/zimbra/store/0/944/msg/0/257-2.msg 0e729c9bbad30c94219154617a64db09 /opt/zimbra/store/0/944/msg/0/257-2.msg
And now the message differences:
]$ diff /opt/zimbra/store/0/944/msg/0/257-2.msg /opt/zimbra/store/0/714/msg/0/257-2.msg 4c4 < 14:31:47 -0700 (PDT) --- > 14:31:24 -0700 (PDT) 6,7c6,7 < by zcs806.us.zimbralab.com (Postfix) with ESMTP id C95D634A0; < Mon, 28 Apr 2014 14:30:49 -0700 (PDT) --- > by zcs806.us.zimbralab.com (Postfix) with ESMTP id 8619E3493; > Mon, 28 Apr 2014 14:30:22 -0700 (PDT) 17c17 < with ESMTP id 9v2smobzMGcy; Mon, 28 Apr 2014 14:30:29 -0700 (PDT) --- > with ESMTP id ONyocz3V6Pmq; Mon, 28 Apr 2014 14:30:08 -0700 (PDT)
What About zmdedupe
zmdedupe doesn't apply here because the blobs in questions are actually different.
$ zmdedupe stats usage: zmdedupe [options] start/status/stop -h,--help Display this help message. -v,--verbose Display stack trace on error. --volumes <volume-ids> Specify which volumes to dedupe. If not specified, dedupe all volumes. The "start/stop" command is required, to avoid unintentionally running a blob dedupe. Id values are separated by commas. $ zmdedupe status Status = stopped Total links created = 0 Total size saved = 0 $ fgrep -lr "20140428212948.74DF13484" ./ | xargs ls -li --full-time | awk '{ print "Inode Number = " $1 " Number of Hard Links To Inode = " $3 }' | awk '!seen[$0]++ { print; }' | uniq -u | sort Inode Number = 13400 Number of Hard Links To Inode = 50 Inode Number = 13444 Number of Hard Links To Inode = 50 Inode Number = 13449 Number of Hard Links To Inode = 50 Inode Number = 13451 Number of Hard Links To Inode = 1 Inode Number = 13452 Number of Hard Links To Inode = 50 Inode Number = 13453 Number of Hard Links To Inode = 49 Inode Number = 13454 Number of Hard Links To Inode = 48 Inode Number = 13455 Number of Hard Links To Inode = 50 Inode Number = 13456 Number of Hard Links To Inode = 50 Inode Number = 13457 Number of Hard Links To Inode = 50 Inode Number = 13458 Number of Hard Links To Inode = 50 Inode Number = 13459 Number of Hard Links To Inode = 1 Inode Number = 13460 Number of Hard Links To Inode = 49 Inode Number = 13461 Number of Hard Links To Inode = 2 Inode Number = 13462 Number of Hard Links To Inode = 1 Inode Number = 13464 Number of Hard Links To Inode = 1 Inode Number = 13465 Number of Hard Links To Inode = 2 Inode Number = 13467 Number of Hard Links To Inode = 1 Inode Number = 13469 Number of Hard Links To Inode = 1 Inode Number = 13473 Number of Hard Links To Inode = 49 Inode Number = 13474 Number of Hard Links To Inode = 49 Inode Number = 13475 Number of Hard Links To Inode = 49 Inode Number = 13476 Number of Hard Links To Inode = 50 Inode Number = 13477 Number of Hard Links To Inode = 48 Inode Number = 13478 Number of Hard Links To Inode = 50 Inode Number = 13479 Number of Hard Links To Inode = 50 Inode Number = 13480 Number of Hard Links To Inode = 50 Inode Number = 13481 Number of Hard Links To Inode = 49