Ajcody-Lucene-Topics

Attention.png - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.

Lucene

Actual Lucene Topics Homepage

Please see Ajcody-Lucene-Topics

Other References to Lucene Index

Please see:

Some General Questions On The Lucene Indexing

Index Directory Numbering

We know the directory under the index volume path is like the following:

/opt/zimbra/index/ "X" / "Y" / index / "Z" /

We believe that "X" is the number which is determined by bitshifting the mailbox_id to the right by 12 bits. That the "Y" is the maibox_id of the user. However, how do you get "Z"?

Answer: It's always '0'.

When Is User Message Store Directory Created

When is the directory for the message data for an account (/opt/zimbra/store/0/...) created? When a message is stored for the first time? Which would also mean, that it would not be created if message data does not exist, correct?

Answer: Yes.

When Is User Index Directory Created

Concerning the index directory. We know the index directory already exists even when mail data does not exist [see question above]. When will the index directory be created? With the account's first login?

Answer: It's created with the first indexing or search event.
Example Walk Through

On my 5.0.24 test box.

Create a test account:

[zimbra@mail37 ~]$ zmprov gmi index-test@`zmhostname`
mailboxId: 6
quotaUsed: 0

Notice that the 'store' directory ISN'T automatically created for the user upon user creation :

[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0
total 20
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1
drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 ..
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3
drwxr-x--- 5 zimbra zimbra 4096 Sep 13 15:05 .

Notice that the 'index' directory IS automatically created for the user upon user creation but the actual indexing files are not :

[zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/
total 8
drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 ..
drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .

Let's see what changes when the user is sent and email BUT they still have not logged in yet:

[zimbra@mail37 ~]$ mail index-test@`zmhostname`
tSubject:test from localhost
test
.
Cc: 

Checking the relevant user directory for changes. Notice we now have a 6 directory, matching the users mailboxId. We don't see any index files though under their index directory:

[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0/
mailboxId: 6
quotaUsed: 1563
total 24
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1
drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 ..
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3
drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:13 6
drwxr-x--- 6 zimbra zimbra 4096 Oct 21 12:13 .

[zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/
total 8
drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 ..
drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .
<pre>

Let's see if logging into the webclient as the user changes anything. Log into the webclient and then check the user directories again. Still no change, no index files created.
<pre>
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0/
mailboxId: 6
quotaUsed: 1563
total 24
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1
drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 ..
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2
drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3
drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:13 6
drwxr-x--- 6 zimbra zimbra 4096 Oct 21 12:13 .

[zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/
total 8
drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 ..
drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .

Let's do a manual index of the user account and confirm index files are made.

[zimbra@mail37 ~]$ zmprov rim index-test@`zmhostname` start
status: started
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/
total 20
drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 ..
-rw-r----- 1 zimbra zimbra   45 Oct 21 12:15 segments_2
-rw-r----- 1 zimbra zimbra   20 Oct 21 12:15 segments.gen
-rw-r----- 1 zimbra zimbra 2455 Oct 21 12:15 _0.cfs
drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:15 .

So far, we've confirm user creation doesn't create the store directory until a message or something similar is processed. That the users index directory path will be created with user creation but the index files will not be. That the index files aren't created when the user first logs in but are created with a manual index [zmprov rim user@domain].

Let's confirm if a 'search' creates the index files. First, I'll remove the existing index files that were made. Then log into the webclient as the user and do an email search. Confirming after words that the index files were made from that search - which it does.

[zimbra@mail37 ~]$ cd /opt/zimbra/index/0/6/index/0/
[zimbra@mail37 0]$ ls
_0.cfs  segments.gen  segments_2
[zimbra@mail37 0]$ rm -rf *
[zimbra@mail37 0]$ ls

Perform webclient search and check index directory again.

[zimbra@mail37 0]$ ls
segments.gen  segments_1

Delete Flag

Does lucene create delete flag when index is deleted?

Answer: Yes.
Delete Flag Operational Details

If so, we believe that it creates only delete flag, and files having an actual index (such as segment file) will be deleted (reuse of disk space) only when segment is merged, or any function for optimization is called, is this correct?

Answer: Yes.
Is It The Same For zmmailboxmove With purgeOld

The above behavior is the same when executing zmmailboxmove with purgeOld?

Answer: No, it physically deletes the entire files.

Manually Deleting Lucene Index Directories

Please see King0770-Notes#Manually_Delete_Index_Directories

Performance Tuning

Please see Performance_Tuning_Guidelines_for_Large_Deployments#Lucene_Index

Jump to: navigation, search