Courier-IMAP Maildir to zmmailbox

From Zimbra :: Wiki

Jump to: navigation, search

This script is a significant augmentation of NERvOus' script. The export data used and tested came from an H-Sphere mail server. This should work with other platforms but modifications may be needed due to inconsistent file arrangement among Maildir implementations.

A colleague of mine pointed out that the Courier IMAP server could be configured to use a delimiter other than "." for folder hierarchy. If this is the case in your instance, adjusting the code which sets the value of folder should help deal with this. If subfolders are arranged into a real subdirectory tree on the filesystem, you might also have to change how folderdir is determined.

Features

This adds the following features not present in the previous versions of the script it was based on:

  • Support for Courier-IMAP/vpopmail style Maildir folder structure
  • Importation of all known message flags from the Maildir filenames as defined by DJB
  • Importation of all Courier-specific "courierimapkeywords" flags
    • Unknown flags are imported as tags so the user can deal with these messages manually.
  • Automatic creation of subfolder heirarchy whenever needed, handling the possibility of "non-folders" which have children.
  • Automatic migration of built-in folders to the top level if they exist as children of the Inbox.
  • Automatic processing of a vpasswd list and support to specify a single user or user and folder name.
  • To support tagging and flagging we process every message individually, but a speedier import is accomplished by queuing up the whole import to pass to zmmailbox in one shot. zmmailbox startup is by far the slowest part of importing messages.
  • "Pretty" progress status output!

Usage

The script assumes your current working directory is named after the domain you want to work on. It also assumes the directory contains a vpasswd file containing a line-separated list of username definitions for Courier. If you wish, you can simply create a file named vpasswd with a line-separated list of users in the working directory to let the script do its work normally.

To just run a full domain import under the above conditions, you simply run:

./import.sh

If you wish to run a specific user only, you can run:

./import.sh username

You can also run an import for a specific folder only (specified by courier IMAP path name):

./import.sh username ".Sent"

The path name should be quoted if it contains spaces and should match the directory name Courier IMAP used. If the folder path is multiple levels deep, it might looks something like ".Archived Items.2009.Work"

The Script

#!/bin/bash
#
# courier/vpopmail Maildir to Zimbra Import
#
# This script can be stored anywhere, but you should run it while in the root
# of the domain's users.  It looks for the file vpasswd which contains a
# line-separated list of users and uses that to import.  You can also run the
# script with a user name to process a single user.  Additionally, you can
# specify a folder name (courier format) to process a single folder for that
# user.

# We assume the folder structure is like this:
# Inbox: <working directory>/<user>/Maildir/<cur|new>
# Subfolder: <working directory>/<user>/Maildir/.Subfolder/<cur|new>
# If this is not what your structure looks like, you need to change the
# "folderpath" variable construction down further in this script.

# This is the command to run to run mailbox commands.
ZMCMD='/opt/zimbra/bin/zmmailbox -z'

# This will be used for temporary/log files during the import process
TEMP='/tmp/'

# We assume the working directory's name is the domain.
# Otherwise, override this with your actual domain name.
domain=`basename ${PWD}`

echo Process ID: $$

if [[ $1 != "" ]] ; then
  USERS=$1;
else
  USERS=`cat vpasswd | cut -f1 -d:`
fi

for user in ${USERS}; do
  echo "Beginning User: $user..."

  if [[ $2 != "" ]] ; then
    FOLDERS="$user/Maildir/$2/cur";
  else
    FOLDERS=`find $user -type d -name cur | sort`
  fi

  echo "$FOLDERS" | while read line; do
    folderdir=`echo ${line} | cut -f3 -d"/"`
    if [[ ${folderdir} == "cur" ]] ; then
      folderdir="";
    fi 
    folder=`echo ${folderdir} | sed 's/^\.//; s%\.%/%g; s%\&-%\&%g'`
    folderpath=${PWD}/${user}/Maildir/${folderdir}/
    
    # If the folder name is blank, this is the top level folder,
    # Zimbra calls it "Inbox" (so do most clients/servers).
    if [[ $folder == "" ]] ; then
      folder="Inbox";
    fi
    # In Courier IMAP, all folders must be children of the root
    # folder, which means Trash, Junk, Sent, Drafts are typically
    # under Inbox. This is not the case with Zimbra, so we will
    # slide these mailboxes to the top level so they behave properly,
    # For all "non-special" mailboxes, we will keep them as children
    # so they remain where the user had them before.
    if [[ $folder != "Trash" && $folder != "Junk" && $folder != "Sent"
       && $folder != "Drafts" && $folder != "Inbox" ]] ; then
      folder="Inbox/${folder}";
    fi
    echo "* Working on Folder $folder..."

    # Courier allows heirarchy where non-folders (literally nothing) are
    # able to have children.  Zimbra does not.  It's also possible that
    # we will process the folders out of heirarchical order for some reason
    # Here we separate the path and make sure all the parent folders exist
    # before trying to create the folder we're working on.
    parts=(`echo $folder | sed 's% %\x1a%g; s%/% %g'`);
    hier="";
    for i in "${parts[@]}"; do
      hier=`echo ${hier}/$i | sed 's%^/%%; s%\x1a% %g'`;
      ${ZMCMD} -m ${user}@${domain} getFolder "/${hier}" >/dev/null 2>&1 ||
      ( echo -n "  + Creating folder $hier... " &&
      ${ZMCMD} -m ${user}@${domain} createFolder "/${hier}" )
    done

    # Figure out how many messages we have
    count=`find "${folderpath}new/" "${folderpath}cur/" -type f | wc -l`;
    imported=0;
    echo "  * $count messages to process..."

    # Define the temporary file names we will need
    importfn="${TEMP}/import-$domain-$user-$folderdir-$$"
    implogfn="${TEMP}/import-$domain-$user-$folderdir-$$-log"
    impflogfn="${TEMP}/import-$domain-$user-$folderdir-$$-flaglog"
    impflagfn="${TEMP}/import-$domain-$user-$folderdir-$$-flags"
    touch "$importfn"

    # Determine the courier extended flag identifiers ("keywords")
    flagid=0
    if [[ -f "${folderpath}courierimapkeywords/:list" ]] ; then
      extflags="YES"
      cat "${folderpath}courierimapkeywords/:list" 2>/dev/null | while read line; do
        # A blank line indicates the end of the definitions.
        if [[ "${line}" == "" ]]; then break; fi

        # To avoid escape character madness, I'm swapping $ with % here.
        flag=`echo ${line} | sed 's/\\\$/%/'`
        echo courierflag[${flagid}]="'$flag'";
        flagid=$(( flagid + 1 ));

        # Create the tag if it doesn't start with '%'
        if [[ `echo ${flag} | grep '%'` == "" ]] ; then
	  echo -n "  + Attemping to create tag ${flag}... " >&2
          ${ZMCMD} -m ${user}@${domain} createTag "${flag}" >&2
        fi

      done > "$impflagfn"
      source "$impflagfn"
    fi

    echo -n "  * Queuing messages for import...        " 

    # Find all "cur" or "new" messages in this folder and import them.
    find "${folderpath}new/" "${folderpath}cur/" -type f | while read msg; do
      flags="";
      tags="";
      msgid=`echo $msg | cut -d: -f1 | sed s%.*/%%`

      # Determine the old maildir style flags
      oldflags=`echo $msg | cut -d: -f2`
      # Replied
      if [[ `echo ${oldflags} | grep 'R'` != "" ]] ; then flags="${flags}r"; fi
      # Seen
      if [[ `echo ${oldflags} | grep 'S'` == "" ]] ; then flags="${flags}u"; fi
      # Trashed
      if [[ `echo ${oldflags} | grep 'T'` != "" ]] ; then flags="${flags}x"; fi
      # Draft
      if [[ `echo ${oldflags} | grep 'D'` != "" ]] ; then flags="${flags}d"; fi
      # Flagged
      if [[ `echo ${oldflags} | grep 'F'` != "" ]] ; then flags="${flags}f"; fi

      # Determine the courier-imap extended flags for this message
      if [[ ${extflags} == "YES" ]] ; then
        oldflags2=`grep $msgid "${folderpath}courierimapkeywords/:list" 2>/dev/null | cut -d: -f2`
        for flag in ${oldflags2}; do
          # Forwarded
          if [[ ${courierflag[$flag]} == '%Forwarded' ]] ; then flags="${flags}w"; fi
          # Sent by me
          if [[ ${courierflag[$flag]} == '%MDNSent' ]] ;   then flags="${flags}s"; fi
          # Convert non-system flags to Zimbra tags
          if [[ `echo ${courierflag[$flag]} | grep '%'` == "" ]] ; then
            tags="${tags},${courierflag[$flag]}"
          fi
        done
        # Clean up the tag list for the command line
        if [[ ${tags} != "" ]]; then
          tags=`echo ${tags} | sed "s/^,\?/--tags \'/; s/\$/\'/"`;
        fi
      fi

      # Log the result of flag processing for debugging
      if [[ $flags != "" || $tags != "" ]] ; then
        echo `date +%c` "$msg had flags $oldflags and $oldflags2, now $flags and $tags in folder $folder" >> "$impflogfn"
      fi

      # Add the command to the queue file to import this message
      echo "addMessage --flags \"${flags}\" ${tags} --noValidation \"/$folder\" \"${msg}\"" >> "$importfn"

      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    echo "...done";

    # Since we redirect the queue file to the mailbox tool, we end with "quit"
    echo "quit" >> "$importfn"

    # We're counting "prompts" from the zmmailbox utility here.  The first
    # one comes up before a message is imported, so we start at -1 to offset
    # its existence.
    imported=-1;

    # We do this redirect because running the command for each message is very
    # slow.  We can't just pass the directory to the command, despite Zimbra's
    # support because we can't tag or flag the messages that way.
    echo -n "  * Running import process...             "
    ${ZMCMD} -m $user@$domain < "${importfn}" 2> "${implogfn}" | while read; do
      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    if [[ -s "${implogfn}" ]]; then 
      echo "...some messages did not import correctly: check $importfn";
    else
      echo "...done";
    fi
  done
done
echo "Import Process Complete!"


Verified Against: unknown Date Created: 6/16/10
Article ID: http://wiki.zimbra.com/index.php?title=Courier-IMAP_Maildir_to_zmmailbox Date Modified: 07/16/2010
Personal tools