Difference between revisions of "Building FRANKLIN on CentOS 5"

Revision as of 08:27, 3 December 2008

Introduction

Building Zimbra from source is necessary if we want to make modifications to it. The binary release of the so-called open source version is released under a restrictive ZEULA (Zimbra End-user License Agreement), which disallows any modifications. Building Zimbra from source allows us to use the less restrictive YPL 1.1 (Yahoo Public License). Even though the YPL 1.1 is not an OSI-approved license, it still allows modifications. It is "badgeware", though, meaning that Zimbra/Yahoo logos need to be included in any modifications.

The build process for Zimbra is poorly documented. You won't find many useful articles from the Zimbra wiki or from the forums. Lots of the articles are obsolete. The only really good instructions are here, provided by the Zimbra Community Builds project. Currently (3rd Dec 2008) the only other useful articles in the Zimbra wiki are located here

Alternatives to source install

• If ZEULA restrictions are not a concern, use the ZCS "Open Source Edition" binary installer, which is trivial to install
• If you're willing to run Zimbra on Ubuntu or Solaris, get the appropriate packages from Zimbra Community Builds. They might (or might not) work.
• If you require Outlook support, Symbian mobile clients or such, get the trial version for a commercial version of Zimbra.

Getting Zimbra sources with Perforce

Getting the source

There are two main places where to get the Zimbra sources:

• Zimbra Community Builds (http://sourceforge.net/projects/zimbracommunity)
• Zimbra's official Perforce repositories (instructions below)

The latest Zimbra code - including the ThirdParty packages - is located in a proprietary Perforce (p4) repository. If you're doing a production build, you should try to download a stable branch. In case you're going to do development, you should probably get HEAD (=development branch). I probably downloaded HEAD at first, as there were numerous stupid errors in the build files that I had to fix.

To get FRANKLIN (ZCS 5.0.x) from Perforce repositories, you need to get the proprietary Perforce (P4) client from http://www.perforce.com/perforce/downloads/index.html

Once p4 binary is installed, do the following:

$ mkdir -p /home/public/p4
$ export P4EDITOR=nano
$ export P4PORT=codes.zimbra.com:2666
$ p4 -u public -P public1234 -c public-view sync -f //depot/zcs/FRANKLIN/...

The mkdir is required, because that's where Perforce wants it's stuff to go. It's probably possible to change it's behavior, but hacking around it is faster. After running this command you should have Zimbra sources in /home/public/p4/FRANKLIN/.

Getting Zimbra sources from Zimbra Community Builds

An alternative to getting sources from Zimbra's Perforce repository is to get the Zimbra Community Build sources which are located here:

• http://sourceforge.net/project/showfiles.php?group_id=224243

Make sure you download FRANKLIN-5010_2660.tar.gz. The FRANKLIN-5010.tar.gz is full of files with Windows linefeeds, which makes it useless.

Overview of the build process

The Zimbra build process well described in these two articles:

• http://varlogmessages.vroomvroom.org/documentation/ubuntu-port/ubuntu-build-instructions
• http://varlogmessages.vroomvroom.org/documentation/solaris-port/build-instructions

Use the Solaris instructions only as a reference. According to dijichi, Zimbra Community Builds project manager, the Solaris instructions are more up-to-date.

The basic steps are these:

• Set up the build environment
• Get the source (p4 or zimbra community builds)
• Build ThirdParty modules
• Build memcached, nginx and tcmalloc in ThirdParty
• Build Zimbra modules

Even though the instructions are good, a couple of notes are in place:

• You should probably add the environment variables to /etc/profile.d/zimbra.sh instead of /etc/profile directly.
• Remove the perforce sync statements from ThirdParty/buildThirdParty.sh. The p4 public account keeps track of checkouts on the server side, so it's useless when several people use the same account.

Installing dependencies

Build-time dependencies

Run-time dependencies

Run-time dependencies are checked when you run the install.sh in ZimbraBuild/zcs-5.0.10_GA_2660.CentOS5.FRANKLIN. These are all installable via yum if you have Dag repositories configured:

• NPTL
• sudo
• libidn
• fetchmail
• gmp
• compat-libstdc++-296
• compat-libstdc++-33
• libtool-ltdl
• /usr/lib/libstdc++.so.6
• perl-5.8.8
• perl-LDAP
• perl-Net-DNS

Debugging the build process

Build problem categories

You'll most likely encounter lots of errors in the build process. Instead of trying to cover all the cases, which is impossible, I'll outline the debugging process. Causes for Zimbra build problems can be divided into several categories:

• False assumptions about the build system. For example, the build files assume /home/public/... exists.
• Undocumented build-time dependencies (there are lots of these)
• Missing definitions files or directories, in case your OS version is not supported
• Outdated definition files, such as RPM spec files
• Wrong program version numbers or paths in build files
• Version mismatches between the included third party software and OS'es *-devel packages

RPM spec files

During actual Zimbra module build you might get errors related to RPM spec files. Most likely you'll encounter problem with legacy SPEC-file format:

#
# spec file for zimbra.rpm
#
Summary: Zimbra LDAP
Name: zimbra-ldap
Version: @@VERSION@@
Release: @@RELEASE@@
# Copyright statement is no longer valid, we need to
# change it to License instread
License: OpenLDAP
Group: Applications/Messaging
URL: http://www.zimbra.com
Vendor: Zimbra, Inc.
Packager: Zimbra, Inc.
BuildRoot: /opt/zimbra
AutoReqProv: no
requires: zimbra-core

The spec files are in ZimbraBuild/rpmconf/Spec.

Missing or faulty file and directories

Zimbra build makes a lot of (false) assumptions about how the system is configured and what directories are available. Luckily the error messages will be rather clear, so these problems are easy to fix. During my CentOS5 install I had to do a couple of things to get things running. Here I assume you've followed the Ubuntu/Solaris build howtos while setting up your system (see above).

P4 downloads fail without this:

$ mkdir -p /home/public/p4

Some build scripts fail without these:

$ mkdir -p /home/public/p4/zcs 
$ ln -s /opt/build.zimbra/source/FRANKLIN /home/public/p4/zcs/main

ThirdParty modules won't build without this:

$ mkdir -p /opt/build.zimbra/source/FRANKLIN/ThirdPartyBuilds&CentOS5

Zimbra modules won't build without these:

$ cd /opt/build.zimbra/source/FRANKLIN/ZimbraBuild/defs/
$ cp RHEL5.def CentOS5.def

You also need to create a JDK tar.gz package and put it to the correct directory:

$ mkdir -p /opt/build.zimbra/source/FRANKLIN/ThirdPartyBuilds/i386/java
$ cd /opt/build.zimbra/source/FRANKLIN/ThirdPartyBuilds/i386/java
$ tar -C /opt -zcf jdk1.5.0_16.tgz jdk1.5.0_16

You probably need to change the Java version in ZimbraBuild/defs/plat_common.def too:

JAVA_VERSION    := 1.5.0_16
JAVA_FILE       := jdk
JAVA_DIR        := java
JAVA_SOURCE     := $(THIRD_PARTY_BUILDS)/i386/$(JAVA_DIR)/$(JAVA_FILE)$(JAVA_VERSION)

You might run into trouble when ThirdParty build creates files which are not the same name/version as those defined in actual Zimbra build files. In this case make a symbolic link or copy the file over from older/newer Zimbra source tree (if available).

Missing dependencies

During ThirdParty build process you'll eventually stumble upon missing dependencies. When a component fails, interrupt the build process with CTRL-C, enter the failed component's directory and run the build manually from there. For most components using make is enough, but some use custom build scripts. In that case launch the script instead. Once make has finished and failed, you'll most likely see an error related to a missing header files (something.h).

If Zimbra bundle already contains the necessary header file, there's most likely a problem with some Makefile. For example, the FRANKLIN-5010_2660 has a bug in curl Makefile which caused header file (e.g. curl.h) installation to fail silently. For more information take a look here:

• http://www.zimbra.com/forums/installation/23102-fc8-installation-source.html)

I fixed that issue by copying over ThirdParty/curl from newer sources obtained from Perforce repository, which seemed to work.

If you verify that an external dependency is missing, you must install the correct RPM development package. The easiest to accomplish by installing yum-utils (yum install yum-utils) and then using repoquery:

$ repoquery -f /usr/include/ldap.h
openldap-devel-0:2.3.27-8.el5_2.4.i386
openldap-devel-0:2.3.27-8.el5_1.3.i386

Note that you need to include the full path to the file. Alternatively you can use wildcards like this:

$ repoquery -f "*ldap.h"

Once you know the package name, install it with yum install packagename.

Then enter that component's directory, run make (or in some cases ant) and see that it builds cleanly. Then you can restart the build process. If you're trying to fix a ThirdParty build failure, you might encounter these errors:

1 out of 1 hunk ignored -- saving rejects to file servers/slapd/back-meta/search.c.rej
patching file servers/slapd/back-sql/config.c
Reversed (or previously applied) patch detected!  Assume -R? [n] n 
Apply anyway? [n] n

These are caused by the fact that Zimbra-specific patches have already been applied, but the build file tried to apply them anyway. You can safely say n to all of these. Or just tap return repeatedly

Tips to making the build process faster

It's highly unlikely that Zimbra build will success on the first try. By default the Zimbra build scripts are pretty stupid, as they remove all that was left in the previous run and recompile everything from start. This means that when a ThirdParty module fails, the script recompiles everything all over again. To avoid this, build try building the modules individually until they all succeed.

The correct compile order for Zimbra ThirdParty software can be seen from the ThirdParty/Makefile. In FRANKLIN-5010_2660 community build the correct order is this:

DIRS := openssl \
        mysql \
        sleepycat \
        libxml2 \
        heimdal \
        curl \
        cyrus-sasl \
        openldap \
        aspell \
        clamav \
        apache-httpd \
        php \
        pcre \
        expat \
        popt \
        PostFix \
        dspam \
        rrdtool \
        snmp

When you fix the failed module, enter the next one's directory, build it and fix it (if necessary) and so on. In the end all modules should have been built. You can use the same technique for building the actual Zimbra modules. You just need to follow this order:

ore
proxy
mta
store
ldap
snmp
logger
apache
spell
zcs_stage

This order was taken from http://varlogmessages.vroomvroom.org/documentation/solaris-port/build-instructions/build-zimbra-server.

Introduction

This HOWTO covers the basic steps required to build Zimbra. For more information, see these essential documents:

• Building Zimbra on Solaris
• Building Zimbra using Perforce
• Franklin README

These might be helpful, too:

• Building Zimbra on FreeBSD
• Building Zimbra on Ubuntu Hardy

Preparing the build environment

Stock CentOS5 install has only a few basic repositories configured. You need to add rpmforge to your list of yum repositories. You can get the repository information and instructions from http://dag.wieers.com/rpm/packages/rpmforge-release.

Troubleshooting

Zimbra build problems can be divided into a couple main categories:

• Undocumented, missing dependencies
• Faulty assumptions about the build environment
• Problems with the sources or the downloaded source tree

Instead of trying to document every problem I encountered, I'll simply describe how these problems can be circumvented.

Finding and installing missing dependencies

It's certain that Zimbra won't build cleanly if you just follow the various Wiki articles and README files. When the build fails, it's usually due to a missing dependency. If the error message tells you which software package is missing, you can just guess the package name

$ yum search package
$ yum install package

Most likely you need lots of development packages. They are named something-devel in CentOS. If the build process complains about a single missing file, you can find it by installing yum-utils and then using repoquery to locate the package containing the missing file:

$ yum install yum-utils
$ repoquery -a -f *missing-file

This is similar to using apt-file on Debian-based systems.