Building Zimbra with Maven

Revision as of 19:25, 9 February 2015 by Jflanigan (talk | contribs) (Direct Maven Usage)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

This wiki is a collection of notes, tips, and tricks related to the migration to the Maven build system for the KISS/9.0 release. Readers are highly encouraged to add new tips, correct outdated information, and expand on the content.

Benefits

The first question an engineer asks before changing anything should always be 'what problem are we trying to solve here?'. In this case, there are many inefficiencies to the current Ant build process which can be improved by using Maven. The collaboration server is comprised of dozens of third party Java libraries, and managing those libraries is a cumbersome process.

  • Cleaner dependency hierarchy - We often have to trace through Bugzilla history to determine what project/module uses a particular jar file. We have a number of jar files which are most likely unused but we have no clear way to determine which ones are safe to remove. Maven requires each project to explicitly list dependencies, and gives us a tool to understand where dependencies are inherited vs. where they are directly used in our codebase.
  • Standard declarative project structure - Our current Ant projects are a mishmash of repeated copy/paste scripting with small variance for each project. Maven establishes a standard project object model (POM) which is then inherited by all sub projects. This means that we do not have to reinvent standard build functions such as compile and jar for each new project and that we will reduce the number of naming variances for example where we have 'dev-dist' in one project and 'dist-dev' in another performing a similar function.
  • Easier ability to trace/debug into third party libraries - Currently we maintain a collection of the source for several of the most commonly used third party jar files for use in debugging the inner workings of those libraries; however obtaining the source for additional libraries can be cumbersome; and we often overlook updating the source when we update the jar. Maven and the m2e plugin can automatically download the sources for most jar files, making it much easier to debug across third party code.
  • Easier ability to upgrade in development - We frequently need to determine if a particular bug in a third party library exists in more recent versions. Manually downloading the new jar and installing in a development environment is time consuming. With Maven the same can be accomplished by updating a single version number in a single file.
  • Easier ability to automate unit tests - Our Hudson continuous integration server currently has to call the 'ant test' target in a number of individual directories, and our current configuration appears to exclude a few directories which would be desirable to cover in CI. Maven allows us to create a hierarchy so 'mvn test' is called in a single directory which then automatically

Maven Installation

We require Maven 3; and all testing to date has been done with Maven 3.2.2 and 3.0.5 versions.

To check if you have Maven installed; execute $mvn -version

This should return Apache Maven 3.0.0 or a higher version. If you do not have Maven, you will need to install it.

Mac OS X

Maven was included by default in 10.8 and earlier, but is not included from 10.9 on. The easiest way to install it is via homebrew: $brew install maven

If you do not already have homebrew installed, see the homebrew website for instructions http://brew.sh/.

If you would like to attempt to install Maven by hand here is a discussion which includes some suggestions http://stackoverflow.com/questions/8826881/maven-install-on-mac-os-x

Linux

Available in apt-get/yum package managers.

Windows

If anyone is still building the server on Windows they can please fill in these details.


Configuration

Maven uses the MAVEN_OPTS environment variable to pass arguments to the underlying Java process.

It sometimes helps to set a higher max memory size than default:

export MAVEN_OPTS="-Xmx1g"

It also helps to set the JAVA_HOME environment variable:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home

Eclipse

Now is a great time to upgrade to Eclipse Luna if you have not done so already.

Two plugins which are highly recommended (i.e. required):

m2e: https://www.eclipse.org/m2e/download/

m2e dynamic source lookup: https://marketplace.eclipse.org/content/m2e-dynamic-sources-lookup

The .project/.classpath files have been updated to use the Maven build nature for projects which have been converted. The m2e plugin will allow Eclipse to interact with Maven, and actually may be installed by default in new downloads of Eclipse Luna.

The m2e dynamic source lookup plugin which allows Eclipse to resolve/fetch the sources for Maven dependencies when debugging (most Maven dependencies publish sources, although we will likely find some do not). Without this plugin you can still debug your own source but cannot easily step into third party libraries. This plugin appears to require Eclipse Luna.

Legacy Eclipse Support

If you are running an old version of Eclipse and/or cannot install the m2e plugin, you can ask Maven to generate .project and .classpath files for you. This not not ideal or recommended, but if you are stuck on an old version this could be a viable workaround. These instructions are mutually exclusive with m2e, so use one or the other but not both.

To do this, first open the files in question so they are writable, then run the eclipse:clean and eclipse:eclipse goals

cd ZimbraServer
p4 open .classpath
p4 open .project
mvn eclipse:clean eclipse:eclipse

Now when you open Eclipse you'll see each dependency referenced via a M2_REPO classpath variable. If Eclipse has not configured this for you, you can add it in Preferences->Java->Build Path->Classath Variables, or through the mvn eclipse:configure-workspace goal as described here: http://www.mkyong.com/maven/how-to-configure-m2_repo-variable-in-eclipse-ide/

Maven Usage

Usage from Ant

If you don't want to use Maven directly, just call your normal Ant targets which should still work.

ant reset-all

ant compile

One note on this is that some Ant subprojects call compile/jar in parent projects and can create a bit of a cycle compiling the various Maven projects repeatedly. To avoid this pass the -Dskip.maven=true argument on the ant command line.

ant -Dskip.maven=true compile-just-my-jar

Probably will be useful with targets like dev-sync. The normal reset-all takes care of this implicitly by calling Maven once at the start of the build and then skipping it on subsequent calls in the same ant run.

Direct Maven Usage

As mentioned in the section covering the migration from Ant, most developers can simply continue to use Ant and the build process will trigger Maven when required. However, Java developers are encouraged to use new features provided by Maven to streamline their workflow.

  • Build everything which is Maven enabled: From the basedir (i.e. //depot/zimbra/main) invoke $mvn install
  • Run unit tests $mvn test
  • Run a test server with $mvn jetty:run. This works in the ZimbraServer module once you have initialized everything with the standard 'ant reset-all'.
  • Debug a test server in Eclipse by launching ZimbraServer jetty:run. Eclipse Luna seems to have the best support for debugging into Maven dependencies with the appropriate plugins.
  • Developers add your favorite goal/target/tip/trick here.

Jars which are not available in Maven

Several existing Zimbra dependencies are not available in Maven public repositories, for example jar files which Zimbra has patched or jar files which simply are not published in Maven by their author. These are kept in Perforce under ZimbraCommon/jars-bootstrap. During an 'ant reset-all' invocation, the 'maven-seed-local-repo' target is called to install these jars in the local Maven repository. While it should generally not be necessary; if you happen to remove your Maven local repository (e.g. by deleting ~/.m2/repository) you will also want to remove ZimbraServer/mvn.seed.properties; as this is used by Ant to determine which bootstrap jars have been installed in earlier Ant runs.

You may also run the mvn-local-jars shell script to populate the local repository with these jars.

Troubleshooting

Perforce client workspace inclusions

You may run into Maven errors building the first time if you do not have all of the standard directories in your P4 client workspace. For example you may see errors such as this:
[exec] [ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/Users/devuser/zimbra/p4/main). Please verify you invoked Maven from the correct directory. -> [Help 1]
Child module /home/devuser/p4/devuser-u14-main/main/ZimbraNative does not exist

If you see these errors make sure you have the following in your P4 client specification.

  • //depot/zimbra/main/pom.xml
  • //depot/zimbra/main/ZimbraNative/...
  • //depot/zimbra/main/ZimbraCharset/...

Test Failures

One of the nice features of Maven's standard lifecycle is that unit tests are run automatically during every build. In general this is great because it prevents developers from accidentally forgetting to run unit tests and submitting bad code. However, it may be necessary to skip tests under some circumstances; for example if there are sporadic/environmental failures, or if you need to build your component even though some unrelated test is failing.

In these cases, the -DskipTests=true argument can be passed to Maven or Ant.

$mvn -DskipTests=true clean install
$ant -DskipTests=true reset-all

Jump to: navigation, search