Ajcody-Lucene-Topics
- This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information. |
Lucene
Actual Lucene Topics Homepage
Please see Ajcody-Lucene-Topics
Other References to Lucene Index
Please see:
Some General Questions On The Lucene Indexing
Index Directory Numbering
We know the directory under the index volume path is like the following:
- /opt/zimbra/index/ "X" / "Y" / index / "Z" /
We believe that "X" is the number which is determined by bitshifting the mailbox_id to the right by 12 bits. That the "Y" is the maibox_id of the user. However, how do you get "Z"?
- Answer: It's always '0'.
When Is User Message Store Directory Created
When is the directory for the message data for an account (/opt/zimbra/store/0/...) created? When a message is stored for the first time? Which would also mean, that it would not be created if message data does not exist, correct?
- Answer: Yes.
When Is User Index Directory And Index Files Created
Concerning the index directory. We know the index directory already exists even when mail data does not exist [see question above]. When will the index directory be created? With the account's first login?
- Answer: The directory is created with user creation. The index files that will exist in the user's directory are created with the first indexing or search event.
Example Walk Through
On my 5.0.24 test box.
Create a test account:
[zimbra@mail37 ~]$ zmprov gmi index-test@`zmhostname` mailboxId: 6 quotaUsed: 0
Notice that the 'store' directory ISN'T automatically created for the user upon user creation :
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0 total 20 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1 drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 .. drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3 drwxr-x--- 5 zimbra zimbra 4096 Sep 13 15:05 .
Notice that the 'index' directory IS automatically created for the user upon user creation but the actual indexing files are not :
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 8 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .
Let's see what changes when the user is sent and email BUT they still have not logged in yet:
[zimbra@mail37 ~]$ mail index-test@`zmhostname` tSubject:test from localhost test . Cc:
Checking the relevant user directory for changes. Notice we now have a 6 directory, matching the users mailboxId. We don't see any index files though under their index directory:
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0/ mailboxId: 6 quotaUsed: 1563 total 24 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1 drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 .. drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:13 6 drwxr-x--- 6 zimbra zimbra 4096 Oct 21 12:13 . [zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 8 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .
Let's see if logging into the webclient as the user changes anything. Log into the webclient and then check the user directories again. Still no change, no index files created.
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0/ mailboxId: 6 quotaUsed: 1563 total 24 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1 drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 .. drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:13 6 drwxr-x--- 6 zimbra zimbra 4096 Oct 21 12:13 . [zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 8 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .
Let's do a manual index of the user account and confirm index files are made.
[zimbra@mail37 ~]$ zmprov rim index-test@`zmhostname` start status: started [zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 20 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. -rw-r----- 1 zimbra zimbra 45 Oct 21 12:15 segments_2 -rw-r----- 1 zimbra zimbra 20 Oct 21 12:15 segments.gen -rw-r----- 1 zimbra zimbra 2455 Oct 21 12:15 _0.cfs drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:15 .
So far, we've confirm user creation doesn't create the store directory until a message or something similar is processed. That the users index directory path will be created with user creation but the index files will not be. That the index files aren't created when the user first logs in but are created with a manual index [zmprov rim user@domain].
Let's confirm if a 'search' creates the index files. First, I'll remove the existing index files that were made. Then log into the webclient as the user and do an email search. Confirming after words that the index files were made from that search - which it does.
[zimbra@mail37 ~]$ cd /opt/zimbra/index/0/6/index/0/ [zimbra@mail37 0]$ ls _0.cfs segments.gen segments_2 [zimbra@mail37 0]$ rm -rf * [zimbra@mail37 0]$ ls
Perform webclient search and check index directory again.
[zimbra@mail37 0]$ ls segments.gen segments_1
Delete Flag
Does lucene create delete flag when index is deleted?
- Answer: Yes.
Delete Flag Operational Details
If so, we believe that it creates only delete flag, and files having an actual index (such as segment file) will be deleted (reuse of disk space) only when segment is merged, or any function for optimization is called, is this correct?
- Answer: Yes.
Is It The Same For zmmailboxmove With purgeOld
The above behavior is the same when executing zmmailboxmove with purgeOld?
- Answer: No, it physically deletes the entire files.
Manually Deleting Lucene Index Directories
Please see King0770-Notes#Manually_Delete_Index_Directories
Performance Tuning
Please see Performance_Tuning_Guidelines_for_Large_Deployments#Lucene_Index
Some smaller notes:
- Upgrade to 6.0.8:
- See Bug 47890
- http://bugzilla.zimbra.com/show_bug.cgi?id=47890
- Search in soap SendMsg dosen't scale
- FIXED: 6.0.8 or 6.0.9
- See Bug 47890
- These last 2 will decrease Indexing overhead, but obviously with a loss of functionality
- set zimbraPrefAutoAddAddressEnabled to FALSE
- set zimbraAttachmentsIndexingEnabled to FALSE