Mgolfieri-zmmboxsearchx alternative

Cross Mailbox Search never yields more than 500 results in ZCS 7.x.

There is a bug report for this, and it's being targeted to get fixed for ZCS 8 (no ETA yet, unfortunately):

http://bugzilla.zimbra.com/show_bug.cgi?id=58901

Here follows a workaround for this situation.

At this URL we have an attachment called zmmboxsearchx, https://bugzilla.zimbra.com/show_bug.cgi?id=43265) which is somehow similar to what zmmboxsearch does and was developed for workarounding a similar situation. It circumvents the design limitation of zmmboxsearch which is now hard limited to 500. Usage is similar to zmmboxsearch, but with some differences. .

This script requires the following installed via CPAN as root and accept the default options suggested (http://www.cpan.org/modules/INSTALL.html):

Parallel::ForkManager

SOAP::Lite

To install them (as root):

cpan Parallel::ForkManager
cpan SOAP::Lite

Once you are done, untar the xmboxsearchx package somewhere (/tmp/ in this case). From now on, every operation should be executed as the zimbra user as well:

su - zimbra
cd /tmp
tar zxvf zmmboxsearchx-20100625.tgz

The script is within the bin folder. To get the usage, launch the script without args:

zimbra@zcs7-ga:/tmp/zmmboxsearchx-20100625/bin$ ./zmmboxsearchx 
zmmboxsearchx: --query is required
zmmboxsearchx: use of --account, --searchdirectory or <userlistfile> is required
zmmboxsearchx: try --help option for more information

Usage:
   zmmboxsearchx [options] [<userlistfile> ...]

     Options: [*] == Required, [+] == Multiple vals OK, (val) == Default
       --query <query_string>   [*]query string
       --dir <directory>        directory to write messages to (no directory)
       --account <acct>         [+]account to include, comma separated values ok
       --exclude <acct>         [+]account to exclude, comma separated values ok
       --exclude-file <file>    [+]file listing accounts to exclude
       --searchdirectory <filt> ldap filter to search for accounts
       --server <hostname>      mailbox server (LC:zimbra_zmprov_default_soap_server)
       --limit <num>            limit the total number of results returned (25)
       --offset <num>           offset in hit list to start at (0)
       --proc <num>             maximum number of child/worker processes to use (4)
       --url <url>              URL of soap service (LC:...)
       --authuser <user>        account for authentication (LC:zimbra_ldap_user)
       --password <password>    password for authuser (LC:zimbra_ldap_password)
       --help                   display a brief help message
       --man                    display the entire man page
       --debug [<num>]          verbose output useful for debugging
       --verbose                increase verbosity (increments --debug)

     Note: LC:<key> means the default is read from localconfig if possible.

Note, you should be just interested in using the bolded options, since as you can see usage varies, about the proc option, please try to stick to 4 initially. You should get started with the above options, and start verifying its functionality with just a subset of the accounts on the system. Should it work, try broaden the search to the widest one you need. The usage itself as you can see slightly varies from the stock one we ship by default.

Example:

./zmmboxsearchx --proc 4 --query "whatever" --account "*" --d outputDir

or (notice the quotes, single ones go outside, double ones go inside when multiple words have to be searched as a whole):

zmmboxsearchx -m "*" -q '"customer list"' -l 999999 -d outputDir

Should you then want to have those messages imported within a destination account so that you can read the message via a mail/web client, you can script the following command to go over the whole outputDir and inject every single message you've found:

# zmmailbox -z -m mailsearchresults@example.com am "/Inbox" /var/tmp/test.txt 
480 (/var/tmp/test.txt)


This will add within the Inbox folder the email msg file test.txt. The 480 return code is the message id, but I doubt you'll need that. This might be pretty slow, since each time you call zmmailbox from the shell, the java env has to be set up. Therefore -pretty much as in: http://wiki.zimbra.com/wiki/Bulk_Provisioning - you should script so that you create a command file that then has to be fed to zmmailbox. Something like:

cmd.txt:

am "/Inbox" /var/tmp/msg1
am "/Inbox" /var/tmp/msg2
am "/Inbox" /var/tmp/msg3
...

And then finally:

zmmailbox -z -m mailsearchresults@example.com < cmd.txt


This way, the message injection should be considerably faster. Once done, accessing the mailbox will let you easily manage the outcome of your search.


Troubleshooting

If you see the following error while executing the script:

SOAPAction: "urn:zimbraAdmin#AuthRequest"

<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soapenc="http://www.w3.org/2003/05/soap-encoding"  xmlns:xsd="http://www.w3.org/2001/XMLSchema" soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding" xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Header><context  xmlns="urn:zimbraAdmin" /></soap:Header><soap:Body><AuthRequest xmlns="urn:zimbraAdmin"> <name>zimbra</name><password>*****</password></AuthRequest></soap:Body></soap:Envelope>
SOAP::Transport::HTTP::Client::send_receive: 500 Can't connect to localhost:7071 (certificate verify failed)
Content-Type: text/plain
Client-Date: Wed, 07 Dec 2011 19:49:40 GMT
Client-Warning: Internal response

Can't connect to localhost:7071 (certificate verify failed)

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /opt/zimbra/zimbramon/lib//LWP/Protocol/http.pm line 51.
zmmboxsearchx: sleep(3): try#2 soap error: 500 Can't connect to localhost:7071 (certificate verify failed) at ./zmmboxsearchx line 851

Then you need to prefix the following when calling the script:

zimbra@zimbra:~$ PERL_LWP_SSL_VERIFY_HOSTNAME=0 zmmboxsearch ...
Jump to: navigation, search