JDunphy-CompileZimbraScript

   KB 24615        Last updated on 2024-05-7  




1.00
(one vote)

Script to Compile Zimbra FOSS

It will install the development environment on any of the supported platforms and allow one to build a zimbra release for Version 8.8.15, 9, and 10. Script can be found here: https://raw.githubusercontent.com/JimDunphy/ZimbraScripts/master/src/build_zimbra.sh. To use this script, you must have previously setup your github account (free) and imported your ssh keys to be able to git clone the repositories and have zm-build/build.pl work. If unsure, see this link as Ian's script will be installing the dependencies. The script builds upon these two repositories from github.


Building zimbra amounts to creating a tarball which this script does. Installing this tarball amounts to understanding two parts.

  • you will extract the tar ball and issue install.sh - See zimbra wiki documentation
  • during the above installation, third party components are supplied by pre-built zimbra repositories. (nginx, ldap, etc)


Note: If you use this script and an official patch is released, you will need to repeat the above process. Be extra careful to grab the latest tags. One way is to create a new directory and start clean; otherwise use the tag option for your version. Hint: always do a --clean and view the tags as a sanity check that will be used before building.

Step 1

Install development environment for any supported Zimbra platforms. This will prompt for root via sudo to allow all software and components to be installed.

% ./build_zimbra --init

Step 2

Determine what tags will be present in your FOSS build. Let's build a 10.0 FOSS release.

% ./build_zimbra.sh --tags

Build the latest version from the tags present from the previous command:

% ./build_zimbra.sh --version 10

or Build a specific release.

% ./build_zimbra.sh --version 10.0.7

Tags

When Zimbra checks in their changes, they will tag the patch. Issue any of the following commands to see the tags for your release you want to build:

% ./build_zimbra.sh --tags
% ./build_zimbra.sh --tags9
% ./build_zimbra.sh --tags8
% ls tag*
tags_for_10.txt  tags_for_8.txt  tags_for_9.txt
% cat tags_for_10.txt 
10.0.8,10.0.7,10.0.6,10.0.5,10.0.4,10.0.2,10.0.1,10.0.0-GA,10.0.0

Building New Patch Releases

Your current directory contains a previous build for Zimbra 10.0.7. You can verify this by running any of the commands below:

% ./build_zimbra.sh --version 10.0.7 --dry-run
% tail -1 builds.log
% ls -lt BUILDS

A new patch 10.0.8 was announced and you now want to build this latest patch release.

% ./build_zimbra.sh --tags
% ./build_zimbra.sh --version 10.0.8 --clean 

Note: if you forget to --clean, the program will stop and warn you. It can take Zimbra up to a week to push their new patches to the FOSS github after they announce a new NETWORK release. Using the ./build_zimbra.sh --show-tags will provide a list of repositories, the tag and date to help determine when.

How it Works

The Zimbra build documenation describes doing the following:

mkdir installer-build
cd installer-build
git clone --depth 1 --branch 10.0.6 git@github.com:Zimbra/zm-build.git
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag=10.0.8,10.0.7,10.0.6,10.0.5,10.0.4,10.0.3,10.0.2,10.0.1,10.0.0-GA --build-release-no=10.0.8 --build-type=FOSS --build-release=LIBERTY --build-release-candidate=GA --build-thirdparty-server=files.zimbra.com --no-interactive

There are 2 parts to this complex command line.

  • determine which branch to clone for zm-build
  • the tags to use and supply with the --git-default-tag option

This script iterates through the github repository and builds a tags file for the version you want to build. It will then take the version you are attempting to build and find the highest branch for zm-build to checkout. Using the --dry-run option, you can see what it has determined and compare.

Useful Commands

% ./build_zimbra.sh --help


        ./build_zimbra.sh
        --init                     #first time to setup envioroment (only once)
        --version [10|9|8]         #build release 8.8.15 or 9.0.0 or 10.0.0
        --version 10.0.8           #build release 10.0.8
        --debug                    #extra output
        --clean                    #remove everything but BUILDS
        --tags                     #create tags for version 10
        --tags8                    #create tags for version 8
        --tags9                    #create tags for version 9
        --upgrade                  #echo what needs to be done to upgrade the script
        --builder foss             # an alphanumeric builder name, updates .build.builder file with value
        --builderID [\d\d\d]       # 3 digit value starting at 101-999, updates .build.number file with value
        -V                         #version of this program
        --dry-run                  #show what we would do
        --show-tags                #show latest tag for each repositories
        --show-tags | grep 10.0.8  #show latest tag for each repositories with 10.0.8
        --show-cloned-tags         #show tag of each cloned repository used for build
        --help

       Example usage:
       ./build_zimbra.sh --init               # first time only
       ./build_zimbra.sh --upgrade            # show how get latest version of this script
       ./build_zimbra.sh --upgrade | sh       # overwrite current version of script with latest version from github
       ./build_zimbra.sh --version 10         # build latest patch version 10 according to tags
       ./build_zimbra.sh --version 10.0.6     # build version 10.0.6

       ./build_zimbra.sh --clean; ./build_zimbra.sh --version 9  #build version 9 leaving version 10 around
       ./build_zimbra.sh --clean; ./build_zimbra.sh --version 8  #build version 8 leaving version 9, 10 around
       ./build_zimbra.sh --clean; ./build_zimbra.sh --version 10 --dry-run  #see how to build version 10
       ./build_zimbra.sh --clean; ./build_zimbra.sh --version 10  #build version 10

      WARNING: ********************************************************************************
        the tags are cached. If a new release comes out, you must explicity do this before building if you are using the same directory:

       ./build_zimbra.sh --clean; ./build_zimbra.sh --tags

      This is because the tags are cached in a file and need to recalculated again.
      *****************************************************************************************

See how it will build the latest version of 10.0: (Note: you could use --version 10.0.7 if you know the patch you want)

% ./build_zimbra.sh --version 10 --dry-run
#!/bin/sh

git clone --depth 1 --branch "10.0.6" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="10.0.7,10.0.6,10.0.5,10.0.4,10.0.2,10.0.1,10.0.0-GA,10.0.0" --build-release-no="10.0.7" --build-type=FOSS --build-release="DAFFODIL" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA

Now build version 10.0 patch 7 shown above:

% ./build_zimbra.sh --version 10
Attempting to clone branch 10.0.7...
Cloning into 'zm-build'...
warning: Could not find remote branch 10.0.7 to clone.
fatal: Remote branch 10.0.7 not found in upstream origin
Failed to clone branch 10.0.7. Trying the next tag...
Attempting to clone branch 10.0.6...
Cloning into 'zm-build'...
remote: Enumerating objects: 186, done.
remote: Counting objects: 100% (186/186), done.
remote: Compressing objects: 100% (154/154), done.
remote: Total 186 (delta 43), reused 93 (delta 18), pack-reused 0
Receiving objects: 100% (186/186), 213.70 KiB | 1.22 MiB/s, done.
Resolving deltas: 100% (43/43), done.
Note: switching to '63dacdd96e958c4c3a0493cbbae7ae113d3bb858'.
...
...
...sudo bash -s <<"EOM_SCRIPT"
[ -f /etc/redhat-release ] && ( yum install -y epel-release && yum install -y nginx && service nginx start )
[ -f /etc/redhat-release ] || ( apt-get -y install nginx && service nginx start )
tee /etc/nginx/conf.d/zimbra-pkg-archives-host.conf <<EOM
server {
  listen 8008;
  location / {
     root /home/jad/build-zimbra/zmbuild/my-automated-build/BUILDS;
     autoindex on;
  }
}
EOM
service httpd stop 2>/dev/null
service nginx restart
service nginx status
EOM_SCRIPT


=========================================================================================================

BUILDS/RHEL8_64-DAFFODIL-1007-20240319140057-FOSS-1000/zcs-10.0.7_GA_1000.RHEL8_64.20240319140057.tgz
BUILDS/RHEL8_64-KEPLER-900-20240319113455-FOSS-1000/zcs-9.0.0_GA_P39_1000.RHEL8_64.20240319113455.tgz
BUILDS/RHEL8_64-JOULE-8815-20240319111605-FOSS-1000/zcs-8.8.15_GA_P46_1000.RHEL8_64.20240319111605.tgz

See what it takes to build Zimbra version 8:

./build_zimbra.sh --version 8 --dry-run
#!/bin/sh

git clone --depth 1 --branch "8.8.15.p45" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="8.8.15.p46,8.8.15.p45,8.8.15.p44,8.8.15.p43,8.8.15.p41,8.8.15.p40,8.8.15.P40,8.8.15.p39.1,8.8.15.p39,8.8.15.p37,8.8.15.p36,8.8.15.p35,8.8.15.p34,8.8.15.p33,8.8.15.p32,8.8.15.p31.1,8.8.15.p31,8.8.15.p30,8.8.15.p29,8.8.15.p28,8.8.15.p27,8.8.15.p26,8.8.15.p25,8.8.15.p24,8.8.15.p17,8.8.15.p23,8.8.15.p22,8.8.15.p21,8.8.15.p20,8.8.15.p19,8.8.15.p18,8.8.15.p15.nysa,8.8.15.p16,8.8.15.p15,8.8.15.p14,8.8.15.p13,8.8.15.p12,8.8.15.p11,8.8.15.p10,8.8.15.p9,8.8.15.p8,8.8.15.p7,8.8.15.p6.1,8.8.15.p6,8.8.15.p5,8.8.15.p4,8.8.15.p3,8.8.15.p2,8.8.15.p1,8.8.15.0,8.8.15" --build-release-no="8.8.15" --build-type=FOSS --build-release="JOULE" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA_P46

Now Version 9.

./build_zimbra.sh --version 9 --dry-run
#!/bin/sh

git clone --depth 1 --branch "9.0.0.p38" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="9.0.0.p39,9.0.0.p38,9.0.0.p37,9.0.0.p36,9.0.0.p34,9.0.0.p33,9.0.0.P33,9.0.0.p32.1,9.0.0.p32,9.0.0.p30,9.0.0.p29,9.0.0.p28,9.0.0.p27,9.0.0.p26,9.0.0.p25,9.0.0.p24.1,9.0.0.p24,9.0.0.p23,9.0.0.p22,9.0.0.p21,9.0.0.p20,9.0.0.p19,9.0.0.p18,9.0.0.p17,9.0.0.p16,9.0.0.p15,9.0.0.p14,9.0.0.p13,9.0.0.p12,9.0.0.p11,9.0.0.p10,9.0.0.p9,9.0.0.p8,9.0.0.p7,9.0.0.p6,9.0.0.p5,9.0.0.p4,9.0.0.p3,9.0.0.p2,9.0.0.p1,9.0.0" --build-release-no="9.0.0" --build-type=FOSS --build-release="KEPLER" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA_P39

Show latest tags in repositories

./build_zimbra.sh --show-tags | head -10
Tag Name             Formatted Date                 Directory           
10.0.5               2023-09-20 07:30:53            zm-admin-ajax       
10.0.8               2024-04-11 23:38:04            zm-admin-console    
10.1.0.beta          2024-04-08 11:21:32            zm-admin-help-common
10.0.5               2023-09-20 07:51:32            zm-ajax             
10.0.1               2023-05-17 02:06:44            zm-amavis           
10.0.8               2024-04-12 01:09:56            zm-aspell           
10.0.6               2023-11-29 02:25:48            zm-build            
10.0.0-GA            2023-03-08 21:06:59            zm-bulkprovision-admin-zimlet
10.0.0-GA            2023-03-08 21:07:22            zm-bulkprovision-store

Show tags used to clone the repositories for the current build

./build_zimbra.sh --show-cloned-tags | head -10
Tag Name             Formatted Date                 Directory           
10.0.5               2023-09-20 07:30:53            zm-admin-ajax       
10.0.8               2024-04-11 23:38:04            zm-admin-console    
10.0.0-GA            2023-03-08 21:00:27            zm-admin-help-common
10.0.5               2023-09-20 07:51:32            zm-ajax             
10.0.1               2023-05-17 02:06:44            zm-amavis           
10.0.8               2024-04-12 01:09:56            zm-aspell           
10.0.6               2023-11-29 02:25:48            zm-build            
10.0.0-GA            2023-03-08 21:06:59            zm-bulkprovision-admin-zimlet
10.0.0-GA            2023-03-08 21:07:22            zm-bulkprovision-store

Universal Naming

This script will attempt to label the builds and document how the build was created. It will encode the tags used in the build what branch that zm-build.git was checked out to do the build. An example:

% ls -lt BUILDS/
drwxr-xr-x 3 jad jad 4096 Apr 22 09:29 RHEL8_64-DAFFODIL_T100008C100006FOSS-1008-20240422091945-FOSS-1131007
drwxr-xr-x 3 jad jad 4096 Apr 22 06:24 RHEL8_64-DAFFODIL_T100007C100006JAD-1007-20240422061457-FOSS-1131006
drwxr-xr-x 3 jad jad 4096 Apr 22 06:08 RHEL8_64-DAFFODIL_T100008C100006FOSS-1008-20240422055830-FOSS-1131005
drwxr-xr-x 3 jad jad 4096 Apr 21 07:55 RHEL8_64-KEPLER_T090000p39C090000p38JIM-900-20240421074316-FOSS-1131004
drwxr-xr-x 3 jad jad 4096 Apr 21 07:32 RHEL8_64-JOULE_T080815p46C080815p45JAD-8815-20240421071950-FOSS-1031003
drwxr-xr-x 3 jad jad 4096 Apr 21 07:05 RHEL8_64-DAFFODIL_T100008C100006FOSS-1008-20240421065519-FOSS-1011001

Any build in a new directory will have the default builder name of FOSS and a starting build number of 1011001 where 101 represents the builderID part encoded in the build number unless overridden with --builder or --builderID. Subsequent builds will be 1011002, etc, etc. The build number is passed through to the web client as a result. For example:

 
RHEL8_64-DAFFODIL_T100008C100006FOSS-1008-20240421065519-FOSS-1011001

contains a few bits of information. Here is what we know from T100008C100006FOSS:

  • built for RHEL8_64 and had tags T100008 - tags 10.0.8
  • Further it was built from clone branch of zm-build of 10.0.6
  • 101 is the builder id and the script or entity that built it.
  • 1001 is the build number and is incremented automatically by build.pl
  • builder name is FOSS which represents how/who built it


Here is how the build was previously created:

% ./build_zimbra.sh --version 10.0.8 --dry-run
#!/bin/sh

git clone --depth 1 --branch "10.0.6" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="10.0.8,10.0.7,10.0.6,10.0.5,10.0.4,10.0.2,10.0.1,10.0.0-GA,10.0.0" --build-release-no="10.0.8" --build-type=FOSS --build-release="DAFFODIL_T100008C100006FOSS" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA


To brand the build and create a new build, you can do this:

% ./build_zimbra.sh --clean --version 10.0.7 --builder JAD

And the FOSS will be removed and replaced with JAD. The encoding is still FOSS-1011006

drwxr-xr-x 3 jad jad 4096 Apr 22 06:24 RHEL8_64-DAFFODIL_T100007C100006JAD-1007-20240422061457-FOSS-1011006

Ref: https://forums.zimbra.org/viewtopic.php?p=313466#p313466

Important Files

The script maintains 3 state files and they are created automatically. You can changed the contents with:

  • .build.number --- changed with --builderID. Used by build.pl which increments the build number.
  • .build.builder --- changed with --builder
  • builds.logs
% cat builds.log
0240420-162530  4311006  RHEL8_64-DAFFODIL_T100007C100006FOSS-1007-20240420161606-FOSS-4311006
20240421-070503  1011001  RHEL8_64-DAFFODIL_T100008C100006FOSS-1008-20240421065519-FOSS-1011001
20240421-073241  1031003  RHEL8_64-JOULE_T080815p46C080815p45JAD-8815-20240421071950-FOSS-1031003
20240421-075503  1131004  RHEL8_64-KEPLER_T090000p39C090000p38JIM-900-20240421074316-FOSS-1131004
20240422-060824  1131005  RHEL8_64-DAFFODIL_T100008C100006FOSS-1008-20240422055830-FOSS-1131005
20240422-062417  1131006  RHEL8_64-DAFFODIL_T100007C100006JAD-1007-20240422061457-FOSS-1131006
20240422-092928  1131007  RHEL8_64-DAFFODIL_T100008C100006FOSS-1008-20240422091945-FOSS-1131007

Limitations

This script attempts to build it's releases based on the available tags at the time of compile. It will walk through attempting to guess at the tags used as Zimbra NETWORK builds are not based on the public git tree that we have access to. If you need a specific version built then specify it on the command line. Example: --version 10.0.5 instead of --version 10 (which would be 10.0.7 as this is written)

FOSS Builds and Projects


More articles written by me, https://wiki.zimbra.com/wiki/JDunphy-Notes

Jump to: navigation, search