JDunphy-build.pl-guide

   KB 24653        Last updated on 2025-04-8  




1.00
(one vote)

Zimbra build.pl Build Script – Developer Guide

The following bash script can be used to build any version of zimbra and/or generate the build.pl syntax required.

Overview of build.pl and the Zimbra Build Process

The build.pl script is the central orchestrator for building the Zimbra Collaboration Suite from source. Zimbra’s source code is split across many git repositories (components), and build.pl automates the end-to-end process of fetching all repositories, building each component in the correct order, and packaging the results into a full Zimbra release. In a typical full build, build.pl will:

  • Clone or update each required git repository (using lists defined for FOSS vs Network editions).
  • Check out the appropriate branches or tags for the desired release.
  • Compile each component (using Ant, Maven, or Make as needed) and run any staging steps.
  • Package the components into installable packages (e.g. tar.gz installers or Debian/RPM packages) and assemble the final distribution bundle.

This script ensures that all components use consistent versions and targets, making it easier to produce official release builds. By default, it performs a clean build – it assumes a fresh environment and will clean out previous build artifacts to avoid inconsistencies. It’s designed to be run either in a prepared build environment (such as Zimbra’s Docker build containers) or on a properly configured system with all necessary build tools installed (Java JDK, Ant, Maven, etc.) (zm-build/README.md at develop · Zimbra/zm-build · GitHub) (zm-build/README.md at develop · Zimbra/zm-build · GitHub).

Command-Line Switches and Configuration Options

build.pl accepts a variety of command-line switches (flags and options) to configure the build. These correspond to internal configuration variables and allow you to specify what to build and how. Below is a breakdown of all major options, grouped by purpose. (All long option names can also be set in a config.build file or via environment, but command-line switches take precedence (zm-build/config.build.in at develop · Zimbra/zm-build · GitHub).)

Release Identification (Required)

These options tell the script which Zimbra release you are building. They must be provided (the script will error if they are missing (zm-build/build.pl at develop · Zimbra/zm-build · GitHub)):

  • --build-release <Name> – The release name or code name. This is typically a word like IRONMAIDEN, JUDASPRIEST, or LIBERTY corresponding to the major release series. It will be included in output artifact names.
  • --build-release-candidate <RC|GA|BetaX> – The release stage or type. Common values are GA (General Availability) for a final release, or RC/Beta for pre-releases. This is appended to the version (e.g. 9.0.0_GA).
  • --build-type <FOSS|NETWORK> – Which edition to build: FOSS (open-source edition) or NETWORK (Network Edition with proprietary components). This determines which repositories are included. For example, FOSS will build the open-source components only.
  • --build-thirdparty-server <URL> – The URL or identifier of the third-party package server. Zimbra depends on pre-built third-party libraries; this tells the build where to retrieve them. For official builds this is often set to files.zimbra.com (the Zimbra artifact server) (zm-build/README.md at develop · Zimbra/zm-build · GitHub). (The script will pass this to packaging scripts via an environment variable ThirdPartyServer.)

Example: For Zimbra 10.1.0 GA FOSS, code-named “LIBERTY”, using Zimbra’s third-party server, you would provide:

<syntaxhighlight lang="bash">--build-release=LIBERTY \ --build-release-no=10.1.0 \ --build-release-candidate=GA \ --build-type=FOSS \ --build-thirdparty-server=files.zimbra.com</syntaxhighlight> These core parameters will be incorporated into the build output naming. For instance, the script will compose a destination name like <OS>-<ReleaseName>-<Version>-<Timestamp>-<Type>-<BuildNo> for the build output directory (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). (The OS tag and build number are explained below.)

Build Version and Environment Options

These options control the build numbering, timestamp, and environment paths. In many cases you can accept the defaults, but they are available for customization:

  • --build-no <number> – The build number (an integer) for this build. By default, the script will auto-generate the next build number (e.g., increment from the last) if not provided. This is typically used to distinguish multiple builds of the same version (e.g. build 1, build 2, etc.).
  • --build-ts <timestamp> – A build timestamp. Defaults to the current time (seconds since epoch) if not specified. This helps ensure unique output directories and can track when the build was produced.
  • --build-os <name> – The operating system identifier for the build target. The script will try to detect the OS automatically (e.g., UBUNTU18_64 or RHEL7_64) if you don’t supply this. It’s used in packaging names (for example, package filenames include the OS tag).
  • --build-arch <arch> – The CPU architecture, e.g. x64. This is usually auto-detected as well.
  • --build-dev-tool-base-dir <path> – Directory for development tools. Default is $HOME/.zm-dev-tools. The build process expects certain tools (Ant, Maven, etc.) to be available here. The script adds $BUILD_DEV_TOOL_BASE_DIR/bin (and subdirectories) to the PATH automatically (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). If you have your build tools installed in custom locations, you might adjust this.
  • --build-destination-base-dir <path> – Base directory where final build artifacts will be placed. Defaults to BUILDS/ under the zm-build directory (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). The actual output for this particular build will go into a subfolder named after the destination name (see above).
  • --build-sources-base-dir <path> – The root directory for source code checkouts. By default this is the top of the zm-build working copy (i.e., the directory containing build.pl) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). All repositories will be cloned as subdirectories of this path. If you want to keep source code in a separate location, you can set this accordingly.

Most of the time, you do not need to manually set --build-no or --build-ts – letting the script generate them is fine. The script also computes some convenience values internally, like BUILD_RELEASE_NO_SHORT (a compressed version number without dots) and DESTINATION_NAME (the composite name for the build) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub), but these are derived from the above inputs and rarely set manually.

Selective Build Controls

These options and flags let you include or exclude certain parts of the build, which is useful for partial builds or custom scenarios:

  • --exclude-git-repos <list> – Exclude specific repositories from the build. Provide a comma-separated list of repository names to skip. Any repo in this list will not be cloned or built (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). For example, --exclude-git-repos=zm-network-store,zm-network-modules would omit those components (useful if building FOSS only or temporarily skipping a broken component).
  • --git-overrides <key>=<value> – Override repository source details. This is a flexible option (can be used multiple times) that allows you to specify a custom branch, tag, remote, or repository suffix for a given repo. For example:

These overrides are applied per repository and let you mix and match non-standard code drops in a build. (They correspond to the script’s internal %GIT_OVERRIDES hash.)

  • --git-default-tag <tag[,tag2,...]> – Specify a default tag or a comma-separated list of tags to try for all repositories (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This is especially useful for building a release or patch: you can list the release tag and fallbacks so that each repo will use the highest available tag. For example, --git-default-tag=10.0.8,10.0.7,10.0.6,...10.0.0-GA ensures that if a component has a 10.0.8 tag it will use it, otherwise it tries 10.0.7, and so on down to GA (zm-build/README.md at develop · Zimbra/zm-build · GitHub). (More on this logic in the build instructions below.) If only one tag is given, the script will attempt to use that for every repo by default.
  • --git-default-branch <branch> – Default branch name to use for all repos if no specific tag is being applied. If not set, the script uses the branch specified in the repo list or defaults to “develop” (for mainline development). You might use this to globally build from a different branch (e.g., a maintenance branch) without editing the repo list.
  • --git-default-remote <name> – Default remote label to use for all repos. The repo lists define remotes (e.g. gh-zm for GitHub Zimbra, gh-ks for a community fork, etc.); if you wanted to switch all repos to a different remote source, you could set this. By default it’s not set, so each repo uses its listed remote or falls back to gh-zm (the official GitHub).
  • --git-default-repo-name-suffix <string> – An optional suffix to append to each repository name when constructing clone URLs. Default is none (for standard repo names). This could be used if your repositories have a suffix in their names on the remote. (For example, if all your repos are named zm-mailbox-fork.git you might set suffix -fork to generate those clone URLs.)

Using these switches, you can fine-tune which code is fetched and built. For instance, if you only care about building the core server and not extensions, you could exclude zimlet repos. Or if you are testing a bug fix on one component, you can override just that repo’s branch.

Build Execution Flags

These boolean switches modify how the build runs. They are provided as --option (to enable) or --no-option (to disable), since each default is defined in the script. Key flags include:

  • --build-prod-flag / --no-build-prod-flag – Toggle production build mode. By default, production mode is ON (equivalent to --build-prod-flag, which is the default) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). Production mode might, for example, strip debug symbols or use optimized compilation settings. If you specify --no-build-prod-flag, you effectively turn production mode off. (The script also has a BUILD_DEBUG_FLAG which is the inverse; enabling debug will typically disable prod – they are meant to be opposites.) In practice, you might not need to touch these unless you specifically want a debug-oriented build.
  • --build-debug-flag – Enable a debug build. By default this is off (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). Setting --build-debug-flag may compile components with debug symbols or non-optimized settings. (It’s mutually exclusive with BUILD_PROD_FLAG – turning on debug implies not production.)
  • --interactive / --no-interactive – Controls whether the script runs in interactive mode. By default it is interactive (--interactive is true) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub), meaning the script might prompt for input on certain errors or pauses. In automated environments (CI/CD or when you don’t want any prompts), use --no-interactive to run non-interactively. For example, the documented build commands for patches include --no-interactive (zm-build/README.md at develop · Zimbra/zm-build · GitHub) to ensure the script doesn’t wait for user input.
  • --disable-tar – If set, the script will skip creating tar archives of the packages. By default this is off (tar creation is enabled) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This flag is useful if you want to build everything but not compress the output into tarballs (perhaps for debugging or if you only need the raw packages).
  • --disable-bundle – If set, skip the final bundling steps. By default this is off (bundling runs) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). “Bundling” refers to running the packaging scripts that produce the installable packages (and the final combined installer bundle). Using --disable-bundle means the script will stop after compiling/staging the build outputs, without invoking the packaging scripts. This is helpful if you want to quickly test compile everything or if you intend to run packaging manually later.
  • --stop-after-checkout – Enable this flag to have the script stop execution after it has checked out (fetched) all the repositories, but before building them (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). By default it’s false. This can be handy if you only want to update the source repositories or if you want to manually inspect/modify code before the build continues. Essentially, build.pl --stop-after-checkout will do all git clone/checkout steps and then exit.
  • --ant-options "<opts>" – Pass additional options to the Ant build tool. Many Zimbra components use Ant, and this allows you to inject parameters (for example, --ant-options "-DskipTests=true" to skip running tests during the build) (zm-build/README.md at develop · Zimbra/zm-build · GitHub). The string you provide will be applied to all Ant build calls. In the documented examples, -DskipTests=true is commonly used to speed up the build by not executing unit tests.
  • --deploy-url-prefix <URL> – Optionally specify a URL prefix for deployment. By default, if not set, the script will set up a local HTTP deploy path for testing (on port 8008 of the build machine, using the destination name) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This is an advanced option typically not needed for the normal build process – it’s used if you want to simulate hosting the build artifacts on a server.
  • --dump-config-to <filepath> – If provided, the script will output all its resolved configuration values to the given file. This can be useful for debugging or record-keeping – it dumps the entire %CFG (the config hash) so you can see exactly what values are being used (including defaults and any environment influences).

As you run build.pl, if you ever need a quick reminder of the supported options, you can run ./build.pl --help. The script will print a usage summary and list all the --options it accepts (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). In summary, the above switches give you fine-grained control over the build configuration. Most developers will consistently use the core release identification options and perhaps --ant-options, and only use the others (exclusions, overrides, flags) as needed for special cases.

Full Release Build: Step-by-Step Instructions

Building a full Zimbra release from scratch involves preparing a workspace and running build.pl with the proper parameters. Below is a step-by-step guide, assuming you want to build an official release (for example, version 10.1.0 GA FOSS):

1. Prepare the Build Environment: Ensure you have all required build tools installed (JDK, Ant, Maven, etc., as per Zimbra’s build requirements). Zimbra provides Docker images for building, or you can install tools manually (zm-build/README.md at develop · Zimbra/zm-build · GitHub) (zm-build/README.md at develop · Zimbra/zm-build · GitHub). Also, make sure you have network access to the git repositories and the third-party server.

2. Obtain the zm-build repository: Create a working directory and clone the zm-build repo. You should check out the branch or tag of zm-build that corresponds to the release you intend to build (this repository contains build.pl and the instruction lists for that release). For example, for version 10.1.0, you might do:

<syntaxhighlight lang="bash">mkdir zimbra-build-10.1.0 cd zimbra-build-10.1.0 git clone https://github.com/Zimbra/zm-build.git cd zm-build git checkout 10.1.0 # checkout the tag or branch for the release, if applicable</syntaxhighlight> (In many cases, the develop branch of zm-build knows how to build the latest development version, whereas specific releases/patches may have their own branch or tag. Always use the appropriate one for reproducibility.)

3. Run the build script with release parameters: Use build.pl with the required switches discussed above. At minimum, you’ll provide --build-release, --build-release-no, --build-release-candidate, --build-type, and --build-thirdparty-server. If you want a production build without prompts, also add --no-interactive.

For example, to build Zimbra 10.1.0 GA FOSS on this machine, you would run:

<syntaxhighlight lang="bash">ENV_CACHE_CLEAR_FLAG=true ./build.pl \ 

 --ant-options "-DskipTests=true" \
 --git-default-tag=10.1.0 \
 --build-release=LIBERTY \
 --build-release-no=10.1.0 \
 --build-release-candidate=GA \
 --build-type=FOSS \
 --build-thirdparty-server=files.zimbra.com \
 --no-interactive</syntaxhighlight>

Let’s break down this example:

  • We set ENV_CACHE_CLEAR_FLAG=true in front of the command (more on environment flags in the next section) to clear any cached dependencies, ensuring a truly clean build.
  • We used --ant-options -DskipTests=true to skip running unit tests in each component, which significantly speeds up the process (optional but common).
  • We specified --git-default-tag=10.1.0 since this is a GA release and all repositories should have a 10.1.0 tag. The script will attempt to check out the 10.1.0 tag for each component.
  • The other options (LIBERTY, 10.1.0, GA, FOSS, files.zimbra.com) identify the release as explained earlier.
  • --no-interactive ensures the build runs without waiting for any user input.

When you run this, build.pl will first print a summary of configuration values it’s using (including any defaulted ones and any ENV_... variables). For instance, it will echo each BUILD_* setting and each ENV_* flag (with “(undef)” if not set) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This is a good sanity check to verify it picked up your parameters correctly.

4. Repositories checkout phase: The script will then process the list of repositories needed for FOSS or Network build (depending on BUILD_TYPE). For each repository, it determines which remote URL and what ref (branch/tag) to use. If you provided a --git-default-tag, the logic is as follows:

  • The script consults the per-repo instructions (which may specify a default branch or tag for that component). For official releases, these lists often point to a branch like “release/10.1” or similar. However, because we gave --git-default-tag=10.1.0, the script will prefer that tag. In fact, it will treat the provided tag as the highest priority in each repo unless an override for that repo is set.
  • It tries to find the tag in the remote. Under the hood, for each repo not already cloned, it runs a git ls-remote on the remote’s tags and heads to see if the specified tag exists (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). If found, it will use that. If a repository doesn’t have that tag (which can happen in patch or RC scenarios), and if multiple tags were provided in a comma-separated list, it will try each tag in order until one is found (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). (If none of the tags exist in that repo, it will fall back to using the default branch for that repo – or fail if neither is available.)

The script will then clone any repository that isn’t already present. By default, clones are done shallow (--depth=1) to save time and bandwidth, except for the zm-mailbox repository which is always cloned in full depth (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). (This is a quirk because zm-mailbox contains many tags and branches and often requires a full history for things like git describe – so the build doesn’t shallow-clone it.) If you need full clone for all repos (for example, if you plan to push commits or inspect history), you can export ENV_GIT_FULL_CLONE=true to disable shallow cloning (zm-build/build.pl at develop · Zimbra/zm-build · GitHub).

If a repository already exists from a previous run, by default the script will update it rather than clone again. Specifically, if the repo directory is there, it will either check out the requested tag or pull the latest updates on the branch:

  • If building by tag, the script attempts a git checkout of the desired tag in the existing repo (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). (It does not automatically fetch new tags from remote, so if the tag wasn’t present from an earlier fetch, this could fail. In practice, for a fresh release build you typically start from scratch or manually pull tags. Often, it’s simplest to start each full release build in a new workspace to avoid missing new tags.)
  • If building from a branch, it will perform a git pull --ff-only to fast-forward the local repository to the latest commit (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). If the pull brings in new commits (not “Already up-to-date”), the script will note that changes occurred and will later trigger a rebuild of that component’s code.

After each clone or update, the script cleans any old build artifacts for that repo. For example, right after cloning a repo or after pulling updates, it will remove that component’s output directory under the staging area (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) to ensure a fresh compile. This behavior guarantees that new code is built from scratch and no old binaries linger.

5. Build compilation phase: Once all repositories are checked out at the correct versions, build.pl proceeds to build each component. The order of building is predetermined by an instructions file (for example, FOSS_staging_list.pl for FOSS builds) which lists the components and any dependencies or ordering requirements. The script iterates through that list and runs the appropriate build tool in each repository directory. Most components use Ant (some use Maven or Make). The script will enter each repo’s directory and invoke the build tool with targets defined in the staging list (often something like ant package to compile and create a jar or other package).

By default, each component is cleaned before building. The script issues an Ant/Maven “clean” command for the component unless you tell it not to (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This is another aspect of ensuring a clean build – it’s essentially like running ant clean && ant assemble. If you set the environment flag to skip clean (discussed later), it would omit this step. But normally, you will see output showing a clean followed by the build of each module.

The script prints progress messages as it goes. For each component, it will log something like “BUILDING: zm-store (5 of 19)” to indicate which component it’s on, out of the total. If any build fails (the Ant/Maven process returns an error), build.pl will stop with an error at that point (since a failure likely means the package couldn’t be built).

6. Packaging and bundling phase: After compiling all components, if bundling is not disabled, the script will run a series of packaging scripts to create the distributable artifacts. These packaging scripts (found under instructions/bundling-scripts/) handle things like assembling .deb or .rpm packages for each service, creating ZIP or TGZ files, and finally creating the combined “zcs” installer tarball. For example, there are scripts for zimbra-store.sh, zimbra-mta.sh, etc., and a zcs-bundle.sh to gather them.

By default, build.pl will execute each packaging script unless --disable-bundle was used. During this step, it passes along parameters such as the ThirdPartyServer URL (so that package scripts can pull any required third-party binaries). The packaging step will place the finished packages into the BUILDS/ output directory (inside a subfolder named by the build’s destination name). If --disable-tar is not set (the default), the final installer will be tarred/compressed as well. If you did set --disable-tar, the final output might be left as an uncompressed directory of files.

7. Completion: When the process finishes, you will have a set of build artifacts in your output directory. Typically, for a FOSS build, this includes a tarball named something like zcs-<version>_<os>.tgz (the full installer), and possibly individual package files or other artifacts inside. The script will also output a summary or confirmation that the build completed successfully.

At this point, you have a full Zimbra release built. You can deploy or install it as needed for testing.

Logic for Selecting Latest Tags: One noteworthy aspect in release builds is ensuring each component uses the correct git tag. If all components have been tagged with the exact release (e.g., 10.1.0_GA tag in each repo), you can simply provide that tag. However, especially for patch releases, not every repository is updated for every patch. Some components might still be at an earlier patch level tag (because no changes were needed in that component for this patch). To handle this, the build process supports tag lists.

In our example above for 10.1.0 GA, we gave a single tag. Now consider building 9.0.0 Patch 40 (9.0.0.p40). The approach would be:

  • Use the zm-build branch or tag corresponding to 9.0.0 (say we use branch/tag 9.0.0.p38 as a base, since .p40 might not exist in zm-build) (zm-build/README.md at develop · Zimbra/zm-build · GitHub).
  • Run build.pl with --build-release-no=9.0.0.p40 and importantly --git-default-tag=9.0.0.p40,9.0.0.p39,9.0.0.p38,...,9.0.0_GA. This tag list (from latest patch back to GA) means: for each repo, try to use 9.0.0.p40 tag. If the repo doesn’t have that tag, use p39, if not, p38, … down to GA. In other words, each component will be built at the highest patch level it has (zm-build/README.md at develop · Zimbra/zm-build · GitHub). This ensures that components with no new changes since p38 will still build from their 9.0.0.p38 code, while those with newer fixes will use p40. The ordering of tags in the comma-separated list is critical – it should be from newest to oldest (the script does not automatically sort them; you must provide them in the correct order, which usually means descending version order) (zm-build/README.md at develop · Zimbra/zm-build · GitHub).

For a major release GA, usually a single tag (like 8.8.0_GA or 10.1.0) is sufficient because all relevant repos get that tag. For interim builds (nightlies or RCs), the default might be to use a branch (like “develop” or a release branch), possibly with no tag at all unless one is specified via overrides. In summary, for official releases it’s common to explicitly list tags so that you gather the correct code from each repo. This is a bit of a manual step – you often have to gather and sort the tags across all repos for a patch release. The example above (p40) shows how to encode that knowledge into the build command.

Default Behavior: Clean Builds and Repository Reset

By design, build.pl emphasizes reproducibility and consistency, which is why its default behavior is to perform a very clean build. In practice, this means:

  • It will update or clone each repository fresh (unless instructed otherwise). The script does not simply build whatever code is lying around; it ensures the code matches the requested tags/branches. This can give the impression that it “removes all repositories” each time, especially if you run it in a fresh directory for every build (as the documentation often suggests). Technically, the current implementation will reuse an existing clone if present, but it will reset it (checkout the specified commit or pull latest) and it will remove prior build outputs for that repo (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). The end result is equivalent to starting from a clean slate for each component.
  • It always cleans each component’s build artifacts before rebuilding (unless overridden). That means any compiled classes, jars, or previous outputs are wiped, either by deleting the staging output or by running the build tool’s “clean” target (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This prevents any stale artifacts from creeping into the new build. For example, if you had an old jar in the output directory, it gets removed so that only the newly built jar (if produced) will be present.
  • It (by default) uses shallow clones to speed up fetching source, but that doesn’t compromise the build output – it just means the git history not needed for compilation isn’t downloaded. The only exception, as noted, is zm-mailbox which is fully cloned to ensure tag availability.
  • It clears dependency caches when requested. For instance, setting ENV_CACHE_CLEAR_FLAG=true as we did in the example will purge certain caches such as Ivy/Maven caches (~/.ivy2/cache and ~/.zcs-deps) before building. By default this isn’t done, but using it ensures even third-party dependencies are fetched fresh (useful if you suspect a corrupted cache or want to ensure all dependencies are up to date).

Why does it remove/clean so much by default? The goal is to eliminate “it works on my machine” scenarios and ensure that each build is built from precisely the source intended, with no leftover artifacts. It provides confidence that if something was built, it was from the current code, not from something you forgot to rebuild. The trade-off is performance – a full clean build takes longer since nothing is reused. This is usually acceptable for official release builds (which you want to be pristine). When you are developing or iterating, however, you might not want this full reset each time. In the next section, we’ll cover how you can override these defaults for a faster iterative development cycle.

Rebuilding a Single Component (Partial Builds)

During development, you often want to rebuild just one component (one repository) after making changes, without going through the entire full build again. If you run build.pl normally with the same parameters, it will, by default, update everything and clean everything – essentially a full rebuild. However, build.pl provides mechanisms (via environment variables and some options) to support partial builds or incremental builds. Here’s how you can tweak the process to rebuild a single component in an existing workspace:

Scenario: Suppose you have just built Zimbra 9.0.0 GA. Now you modify the source code in the zm-mailbox repository to fix a bug. You want to compile and package a new server with just that change, without rebuilding all other components (which haven’t changed).

Key strategies for partial builds:

  1. Limit which components are built – to skip untouched components.
  1. Avoid re-fetching or cleaning repos that haven’t changed – to save time.
  1. Optionally, reuse the same build output directory – so that the final installer includes both new and previously built packages, and you don’t end up with duplicates or a new build number.

build.pl addresses these with a set of environment flags (prefixed ENV_...) that you can set before running the script. These are not command-line switches, but variables you export in your shell. The important ones for partial builds are:

  • ENV_BUILD_INCLUDE – Include only these components in the build. Provide a comma-separated list of repo names or patterns. If set, the script will only build components whose name matches one of the entries, and skip building all others (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This drastically reduces build time when you only need one or a few components. In our example, you would set ENV_BUILD_INCLUDE=zm-mailbox to restrict the build to just the zm-mailbox component. (The script does a substring match, so you could even use something like zm-mail and it would match zm-mailbox, but it’s safer to use the full name to avoid unintended matches.)
  • ENV_GIT_UPDATE_INCLUDE – Update only these repositories. With this set, the script will only perform git fetch/pull/checkout on the listed repos (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). Repos not in this list will be left as-is (no pull, no new checkout). This is crucial if you don’t want to disturb other repositories. In our scenario, you’d use ENV_GIT_UPDATE_INCLUDE=zm-mailbox so that the script doesn’t try to update all the other repos (which presumably are already on the correct tag from the previous full build). It will only interact with the zm-mailbox repo in terms of git operations.
  • ENV_SKIP_CLEAN_FLAG – Skip the clean step for each component’s build. By setting ENV_SKIP_CLEAN_FLAG=true, you tell the script not to run the build tool’s “clean” target before building (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This means an incremental compile is possible (only changed classes compile, etc.). Use this with caution: if you know your changes don’t require a full clean build, it can save time. In many cases, Ant or Maven will only rebuild what’s necessary even if you don’t clean, but leaving it out can sometimes cause inconsistencies if you changed something structural. However, for quick iteration, it’s a useful option.
  • ENV_FORCE_REBUILD – Force rebuild of specific components. This is somewhat the inverse of skip-clean. If set, it should contain a comma-separated list of component names for which any existing build output should be deleted, ensuring a full rebuild of those components (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). If you suspect that even with ENV_SKIP_CLEAN your component needs a full rebuild, you can list it here. In practice, if you’re including only one component and you skip clean, you might not need this. But if you had multiple and only want to force one to clean-build, this is the tool.
  • ENV_RESUME_FLAG – “Resume” a previous build. This flag tells the script not to start a brand new build number/timestamp, but to continue where it left off. When you run a full build, build.pl records the build number and timestamp used (in a file .build.last_no_ts). With ENV_RESUME_FLAG=true, the script will load that and reuse those values (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). The practical effect is that the output directory (which includes the timestamp and build number in its name) remains the same, and already-built components can be skipped. Specifically, the script will skip building a component if it sees a marker that it was already built in this build directory (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). It creates a file .built.<timestamp> in each component’s staging folder when that component is successfully built. On resume, if it finds that file for a given component (matching the current build timestamp), it will output “SKIPPING… [TO REBUILD REMOVE ’]” and not rebuild that component (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This is exactly what we want if we only changed zm-mailbox but, say, zm-store and others were already built – we can skip those. However, a caution: if you use ENV_RESUME_FLAG for a component that was already built, it will skip it too! So in our example, if zm-mailbox was built in the last run, resume mode would skip it (thinking nothing changed), which is not what we want after editing its code. To handle this, you have two options: (a) remove the .built.* marker file in the zm-mailbox build directory before running, or (b) do not use ENV_RESUME_FLAG and instead accept that the script will assign a new build number. Option (a) allows you to reuse the same build output directory and still rebuild zm-mailbox. This nuance is important – many developers simply avoid RESUME when actively changing something, and use it only to skip things that truly haven’t changed.
  • ENV_PACKAGE_INCLUDE – Include only certain packages in the packaging stage. Similar to BUILD_INCLUDE but for the packaging scripts, this lets you specify which package scripts to run. For instance, you could set ENV_PACKAGE_INCLUDE=zm-mailbox to only run the packaging for mailbox (and skip repackaging everything else). If you are doing a partial rebuild and you know other packages are already built and present from before, you might not need to re-run all packaging. On the other hand, if you want a new full installer tarball, you might still run all packaging so that the final bundle is created with all components (new and old). Often, developers will rebuild a component and re-run the final bundling to get a new installer that includes the updated component; in that case, you might not restrict PACKAGE_INCLUDE for the final assembly (so that zcs-bundle runs and includes everything). But you could use it if you only want to produce, say, a new zm-mailbox.jar or a specific .deb for testing.
  • ENV_ARCHIVE_SUFFIX_STR – Suffix for package archives. This is a minor option that allows adding a custom suffix to package filenames. For example, if set to “BETA”, it might turn a package name from zimbra-mta_10.1.0.GA_amd64.deb into zimbra-mta_10.1.0.GA_BETA_amd64.deb. The script appends this string to non-bundle package artifacts if provided. This is mainly useful if you want to label the output artifacts uniquely (for instance, if you are producing a special build and don’t want to overwrite or confuse with the official naming).

Using these environment flags, you can perform a targeted rebuild. Let’s apply them to our example:

Partial rebuild example: We want to rebuild zm-mailbox with our changes and update the installer. We have just run a full 9.0.0_GA build previously. We do:

<syntaxhighlight lang="bash"># Inside the zm-build directory of our previous build export ENV_BUILD_INCLUDE="zm-mailbox" export ENV_GIT_UPDATE_INCLUDE="zm-mailbox" export ENV_SKIP_CLEAN_FLAG=true # skip clean to speed up if safe export ENV_RESUME_FLAG=true # continue in same build directory ./build.pl --build-release=JUDASPRIEST \ 

          --build-release-no=9.0.0 \
          --build-release-candidate=GA \
          --build-type=FOSS \
          --build-thirdparty-server=files.zimbra.com \
          --no-interactive</syntaxhighlight>

A few things to note in this command: we export variables (in shell) before calling the script, and we reuse the same main options as our original build (release name, number, etc. should be identical to the previous run so that we’re targeting the same build output). We included only zm-mailbox in both update and build phases. We set resume to reuse the build context. We also turned off interactive again.

When this runs, build.pl will: check only the zm-mailbox repository for updates (e.g., you might have committed your bugfix locally or pulled a new commit – if you made the change locally without committing, ensure the working copy is at the right state because the script by itself won’t apply patches; it assumes code is in the repo). If zm-mailbox is already at the right ref, it won’t pull anything. Because resume is true, it will list all components but skip each one that was already built except zm-mailbox (assuming we removed its .built marker or it detects the code changed due to a new commit). It will then compile zm-mailbox (quickly, since we skipped the clean, Ant will likely just compile the changed classes into the existing classes directory) and then proceed to packaging.

If we didn’t restrict packaging, the script will attempt to package everything again. However, since we skipped building most components, their artifacts are still present from last time, so packaging scripts will find those and reuse them. The final zcs-bundle script will gather all the packages (most from last build, and the new zm-mailbox package) and create a new installer tarball. In essence, we get an updated full installer containing our modified component, without having recompiled the others. This is a very powerful way to iterate on a single component’s development.

Important: If you find that with ENV_RESUME_FLAG your target component was skipped (because it was marked built), simply delete the file .staging/<destname>/zm-mailbox/.built.<timestamp> and run again (still with resume flag). The script explicitly instructs this in its skip message: “SKIPPING… [TO REBUILD REMOVE ‘<target_dir>’]” (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). Removing the marker file for that component signals the script that it needs to rebuild it even in resume mode. Alternatively, you could run without ENV_RESUME_FLAG in the partial build (omitting it will force the script to generate a new timestamp/build and rebuild included components regardless of markers). The downside of not using resume is that it will increment the build number and create a new output directory – you’ll then have two build outputs (which may be fine; you can then take the needed packages from the new one). It’s up to your workflow; both approaches are valid.

Using --exclude-git-repos vs ENV_BUILD_INCLUDE: We described using include lists because it pairs well with environment flags for update. You could also achieve a similar effect by using the --exclude-git-repos option to leave out most repos. For example, --exclude-git-repos=<list of everything except zm-mailbox> (that would be a long list though and a bit impractical). Generally, it’s easier to specify what you do want (include) rather than what not to include, especially since the repository list is long. So the environment approach is convenient for iterative work. If you needed to exclude just one or two problematic repos while still building others, then --exclude-git-repos is handy. Remember that exclusion happens at the very start (the repo won’t even be cloned or considered), whereas ENV_BUILD_INCLUDE still clones all repos but simply skips building those not included. In a dev workflow, you usually already cloned everything in the initial build, so subsequent runs you’re not worried about cloning cost – you’re more concerned with skipping build steps.

Using --git-overrides for development: Another common scenario is you might want to test a change on a different branch or a pending PR. Instead of manually checking out that branch in the repo, you can invoke build.pl --git-overrides <repo>.branch=<branchName> as mentioned earlier. This pairs with partial builds too. For instance, if you want to build zm-mailbox from a feature branch, you could do:

<syntaxhighlight lang="bash">ENV_BUILD_INCLUDE=zm-mailbox ENV_GIT_UPDATE_INCLUDE=zm-mailbox ./build.pl ... --git-overrides zm-mailbox.branch=featureX</syntaxhighlight> The overrides take effect during the checkout phase – the script will switch zm-mailbox to featureX (fetching it if needed) and then build it. This saves you from manually managing that checkout. Just remember that --git-overrides should also be repeated in subsequent runs if you keep using the override, since the script resets to config defaults each run (unless you added it to a config file).

In summary, to modify a single component without wiping all repositories, use the include flags to limit the scope, use the update include to avoid re-cloning/updating everything, and leverage resume to keep the same build context. This will dramatically reduce build times for iterative development. You can mix and match the environment flags depending on your needs (for example, sometimes you might not skip tests or not skip clean if you want to be thorough for a final run). The flexibility is there to essentially turn off the heavy “clean everything” approach when you don’t need it.

Example Use Cases and Command Examples

To solidify the understanding, here are a few example scenarios with typical command invocations:

  • Full Clean Build of a Major Release (GA):
    Scenario: Build the GA release of Zimbra X.Y.Z FOSS from scratch on a new machine.
    Command:

    <syntaxhighlight lang="bash">mkdir build && cd build git clone --depth 1 --branch X.Y.Z https://github.com/Zimbra/zm-build.git # get the zm-build repo for that release cd zm-build ./build.pl --build-release=<CODENAME> \ 

             --build-release-no=X.Y.Z \
         --build-release-candidate=GA \
         --build-type=FOSS \
         --build-thirdparty-server=files.zimbra.com \
         --no-interactive</syntaxhighlight>

    Notes: This assumes all tests run and all components build. If you want to skip tests and speed it up, add ENV_CACHE_CLEAR_FLAG=true and --ant-options "-DskipTests=true" as shown in earlier examples. The result will be a full set of packages for version X.Y.Z GA in the BUILDS/ directory.

  • Building a Patch Release with Tag Fallbacks:
    Scenario: Build Zimbra 9.0.0 Patch 40 (9.0.0.p40) FOSS.
    Command:

    <syntaxhighlight lang="bash">mkdir patch40-build && cd patch40-build git clone --depth 1 --branch 9.0.0.p38 https://github.com/Zimbra/zm-build.git # using p38 as baseline if p40 branch not available cd zm-build ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options "-DskipTests=true" \ 

        --git-default-tag=9.0.0.p40,9.0.0.p39,9.0.0.p38,9.0.0.p37,...,9.0.0_GA \
    --build-release=JUDASPRIEST \
    --build-release-no=9.0.0.p40 \
    --build-release-candidate=GA \
    --build-type=FOSS \
    --build-thirdparty-server=files.zimbra.com \
    --no-interactive</syntaxhighlight>

    Notes: The --git-default-tag provides a list from .p40 down to .GA (zm-build/README.md at develop · Zimbra/zm-build · GitHub). The script will attempt p40 for each repo; if a repo lacks it, it tries p39, etc. This ensures each repo is built at the latest patch level it has. The output will be labeled 9.0.0.p40. Since we used --depth 1 on clone, only that branch’s history is present, but the script’s ls-remote calls will still detect tags on the remote. This is a typical approach QA or release engineers use to produce a patch build. Always verify afterwards that key components picked up the intended versions (the console output will show which tag/commit was used for each repository during checkout).

  • Partial Rebuild After a Change:
    Scenario: After building 10.1.0 GA, you updated the zm-web-client (UI) code and want to rebuild just that and then create a new installer.
    Command:

    <syntaxhighlight lang="bash"># In the same zm-build directory as the full 10.1.0 build export ENV_BUILD_INCLUDE="zm-web-client" export ENV_GIT_UPDATE_INCLUDE="zm-web-client" export ENV_RESUME_FLAG=true

    1. (optional: ENV_SKIP_CLEAN_FLAG=true if you want faster incremental compile)

    ./build.pl --build-release=LIBERTY \ 

             --build-release-no=10.1.0 \
         --build-release-candidate=GA \
         --build-type=FOSS \
         --build-thirdparty-server=files.zimbra.com \
         --no-interactive</syntaxhighlight>

    Notes: This will update the zm-web-client repo (if you committed changes locally, ensure they’re in the repo – if not, the script might not see a need to rebuild unless you force it or skip resume). It will then skip all other components and rebuild zm-web-client. Because resume is true, other components’ .built markers cause them to skip. After this, the packaging should ideally re-run for zm-web-client’s output (creating a new zimbra-web-client package) and then regenerate the final bundle. If the final bundle step was skipped (perhaps because ENV_PACKAGE_INCLUDE was unintentionally set or something), you could run the bundling step separately or just package that one component. But typically, leaving packaging to run normally will incorporate the new component into the existing build output and update the installer package.

  • Building with Custom Branch Overrides:
    Scenario: Test a feature branch for zm-mailbox on top of the current release.
    Command:

    <syntaxhighlight lang="bash">./build.pl --build-release=IRONMAIDEN \ 

             --build-release-no=8.8.15 \
         --build-release-candidate=GA \
         --build-type=FOSS \
         --build-thirdparty-server=files.zimbra.com \
         --git-overrides zm-mailbox.branch=feature/XYZ \
         --no-interactive</syntaxhighlight>

    Notes: This will build 8.8.15 GA but for the zm-mailbox component, instead of using the 8.8.15 tag, it will checkout the feature/XYZ branch (zm-build/config.build.in at develop · Zimbra/zm-build · GitHub). All other components will use their normal tags/branches for 8.8.15. The result is a hybrid build that includes the experimental zm-mailbox code. This is very useful for validating a fix or testing a component in isolation. If that branch lives on a fork or different remote, you could also specify zm-mailbox.remote=<name> in the overrides (assuming the remote is defined in FOSS_remote_list.pl). Alternatively, if the branch is in the main repo, no remote override is needed. After building, you get an installer you can deploy to test the mailbox changes.

In each scenario, tailor the environment flags and command switches to your needs. The examples above illustrate common patterns: full official build vs. patch build vs. incremental rebuild.

Additional Tips and Notable Quirks

Understanding the Output Directory: The build output by default goes into BUILDS/ under your zm-build directory. Inside, it creates a folder named something like UBUNTU18_64-Zimbra-10.1.0_GA-20250403-FOSS-1/ (this is just an illustrative format). This name is derived from <OS>-<ReleaseName>-<ReleaseNoShort>_<ReleaseCandidate>-<Timestamp>-<Type>-<BuildNo>. For example, if BUILD_OS is UBUNTU18_64, BUILD_RELEASE is LIBERTY, BUILD_RELEASE_NO_SHORT is 1010 (from 10.1.0), BUILD_TS is a timestamp (say 20250403 for date), BUILD_TYPE is FOSS, and BUILD_NO is 1, it concatenates to form the folder name (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). All packages, logs, and the final tarball will be found in that directory. If you rerun with a new build number or timestamp, a new directory is made. If you run with ENV_RESUME_FLAG, it reuses the existing directory (and just updates content within it). Keep an eye on these directories so you know where your latest artifacts went. It’s easy to accidentally accumulate multiple build directories if you forget resume vs. new runs.

Log Files: The script itself outputs to the console, but the build processes (Ant/Maven) often produce log files or you can tee the output. It’s wise to capture the console output to a file for later review, especially for long builds or CI logs.

Cleaning Up: If you want to truly start over, the simplest method is to wipe out the entire zm-build directory (or at least the cloned repos and the .staging directory). The script does not automatically delete the repository directories on disk unless you explicitly exclude them next time (it will reuse them if present). To force fresh clones, you can manually remove the subdirectories of zm-build corresponding to each repo. Sometimes, if you suspect your local repo is out of sync or has local changes that might interfere, it’s easiest to delete it and let build.pl clone afresh. (Or clone the whole zm-build anew in a new folder as the documentation often does for patches.)

Shallow Clone vs. Full Clone: By default, as noted, all repos except mailbox are cloned shallowly (--depth=1) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). This means you won’t have the full commit history. If you need to inspect history or if a build requires history (rarely, some builds might use git describe to get a version which needs tags – in such cases the script would have to fetch tags manually), you can set ENV_GIT_FULL_CLONE=true to force full clones of all repos. This of course makes the initial clone slower.

Speeding Up Subsequent Builds: Using the environment flags (like skipping tests, skipping clean, limiting components) is key to speeding up iterative development. Another tip: if you are going to run many builds, consider leaving out ENV_CACHE_CLEAR_FLAG after the first time. Clearing caches ensures clean dependency resolution, but it means Maven will re-download all jars for every build. If you trust your Ivy/Maven cache, you can omit that to save time on repeated builds (just be cautious if dependencies change).

Using Config Files: In addition to command-line switches and env variables, build.pl can read default values from a file named config.build in the same directory. You can define any of the BUILD_* variables or even GIT_OVERRIDES in that file (Perl syntax, as key=value or a hash for overrides, as shown in config.build.in template) (zm-build/config.build.in at develop · Zimbra/zm-build · GitHub) (zm-build/config.build.in at develop · Zimbra/zm-build · GitHub). This can be convenient if you have a set of options you always use – you put them in config.build so you don’t have to type them every time. Remember, anything on the command line will override the config file. The script also saves the last build’s number and timestamp in .build.last_no_ts, which is how it knows what to resume.

Troubleshooting Quirks:
- If the script exits early or you Ctrl-C it, you might end up with partial state. For example, a repo might be cloned but not built, or some .built markers might not be written. It’s usually safe to just run again with resume or do a clean-up as needed. The script is fairly robust in skipping things that are already done (especially in resume mode).
- Pay attention to any Die messages from the script. It uses a function Die() to halt on errors with messages. Common ones include missing required parameters (if you forgot one of the required BUILD_* options, it will complain and list usage), or failure to find a required tag/branch (e.g., if none of the tags you provided exist, it will Die("Clone Attempts Failed") for that repo (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) (zm-build/build.pl at develop · Zimbra/zm-build · GitHub)). In such cases, double-check your tag names or branch names. The error will usually indicate which repo failed.
- The environment variable listing at the start (the script prints all ENV_* vars it knows about) is very useful to confirm which env flags are in effect (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). If you expected something to be set and see “(undef)”, it means the script didn’t pick it up (perhaps you didn’t export it or spelled it wrong). - After a partial build, if you forget to include a component that needed rebuilding, you might end up with a bundle that still has the old component. For example, if you changed zm-store but only included zm-mailbox in the build, your new installer won’t have the zm-store changes. Always ensure your ENV_BUILD_INCLUDE covers all components you modified. If in doubt, you can include multiple (e.g., ENV_BUILD_INCLUDE="zm-mailbox,zm-store"). You can also use patterns if convenient.
- The ENV_FORCE_REBUILD can be used to clean out one component’s previous build completely. If you suspect an incremental build didn’t pick up something, try adding that component to the force rebuild list. This essentially does what a clean would have done for that component’s stage directory (zm-build/build.pl at develop · Zimbra/zm-build · GitHub). - The script might create intermediate files in a .staging directory (for compiled outputs before packaging) and uses zm-build/RE/ for some version markers (you may notice it writing files like BUILD, MAJOR, MINOR, MICRO under zm-build/RE/ (zm-build/build.pl at develop · Zimbra/zm-build · GitHub) – those are internal use, capturing the version parts). Typically you won’t interact with those, but good to know they exist.

Conclusion: The build.pl script is a powerful tool that automates the complex process of building Zimbra. Its default settings favor clean, repeatable builds (ideal for official releases), but it also provides flexibility for developers to iterate quickly on specific components via its many options and environment flags. By understanding the command-line switches, you can customize what you build, and by leveraging the include/exclude and resume features, you can save a lot of time during development. Always test your partial builds to ensure everything needed is included, and gradually you’ll develop a workflow that balances speed and thoroughness. With this guide, you should be able to confidently use and even modify build.pl to suit your release-building and development needs, whether you’re producing a full GA release or just experimenting with a single module.


More articles written by me, https://wiki.zimbra.com/wiki/JDunphy-Notes

Retrieved from "http://wiki.zimbra.com/index.php?title=JDunphy-build.pl-guide&oldid=71032"
Jump to: navigation, search