Ajcody-Notes-ServerPlanning: Difference between revisions

Line 151: Line 151:
#** Comments:
#** Comments:
#* Sustained transfer rate:
#* Sustained transfer rate:
#** Ultra Wide SCSI 40 MB/s
#** Ultra Wide SCSI 40 (16 bits/20MHz) - 320 Mbit/s
#** Ultra2 SCSI 40 MB/s
#*** Real speed: 40 MB/s
#** Ultra2 Wide SCSI 80 MB/s
#** Ultra2 SCSI  
#** Ultra3 SCSI 160 MB/s (alternate names: Ultra-160, Fast-80 wide)
#*** Real speed: 40 MB/s
#** Ultra-320 SCSI 320 MB/s (alternate name: Ultra4 SCSI)
#** Ultra2 Wide SCSI 80 (16 bits/40 MHz) - 640 Mbit/s
#** Ultra-640 SCSI 640 MB/s
#*** Real speed: 80 MB/s
#** Ultra3 SCSI 160 (16 bits/80 MHz DDR) (alternate names: Ultra-160, Fast-80 wide) - 1,280 Mbit/s
#*** Real speed: 160 MB/s
#** Ultra-320 SCSI (alternate name: Ultra4 SCSI) (16 bits/80 MHz DDR) - 2,560 Mbit/s
#*** Real speed: 320 MB/s
#** Ultra-640 SCSI (16 bits/160 MHz DDR) - 5,120 Mbit/s
#*** Real speed: 640 MB/s
#* Latency:
#* Latency:
#** Comments:
#** Comments:

Revision as of 22:28, 19 September 2008

Server Performance Planning & Infrastructure - The Big Picture

Actual Server Planning Homepage

Please see Ajcody-Notes-ServerPlanning

Initial Comments

These are just my random thoughts. They have not been peer-reviewed yet.

Actual DR and Server restore issues are listed else where in my Ajcody-Server-Topics page.

Articles For Your Review

Redundancy - High Availability - And Other Related Questions

One might ask, "Is there any other way to have redundant zimbra email servers?"

The short answer is No, but if you have millions of dollars you can get really close.

Remember, redundancy always comes with a dollar amount cost. And this dollar amount will determine your range of options to implement redundant technologies or processes.

Redundancy isn't magical and all choices have a "time" for the operation to failover and the application being used can restrict what can be done or increase the length of "time" of the operation. Timing of the operation is where the service/server has a client side impact of being "unavailable".

So you break down the server and services into components and put them in a graph or excel sheet. This will help to make sure your not "over engineering" the system.

For example, disk and data.

Disks can use different raid systems to give redundancy. Channels to the disk (remote) can also have redundant paths. The disk chassis can have redundant power supplies which goto different UPS's on different circuit breakers. This data can also be sent to tape, rysnc'd/copied to another disk subsystem on another server, or "flashed" to another location if your filesystem or SAN unit supports it. There's allot that has to fail for you to completely use the data. When exploring these items, you want to have multiple channel paths to make sure copies, rsync's, flashing occurs differently as compared to the "production" path. Have your network backup occur on a different ethernet device.

Two big picture objectives for "redundancy" are "data redundancy" for DR situations and "application" redundancy for service availability. Our "cluster" situation is an active-passive "redundancy" for the purposes of the mailstore application layer. The raid level of the disks on the san servers the "redundancy" in regards to data recovery for DR.

When you introduce the objective of having off-site redundancy the costs and issues become huge. Remote locations introduce speed issue for data transfers which will also impact performance for most applications as it tries to stay in sync between the two machines for write and roll-back purposes.

My intent wasn't so much to give specific answers to this question but rather demonstrate that to answers for these questions you have to get down to the real specifics - it's simply impossible to answer the broad question of "how do I make redundant servers". It would take a book to fully explore all the issues and possibilities but then still come down to - how much money you have to spend on the project. I use to work with High Performance Clusters and HA Linux+Oracle+SAP global deployments with TB's of data - this issue would arise daily for us in those environments.

CPU And Motherboards

Other references: Performance_Tuning_Guidelines_for_Large_Deployments#RAM_and_CPU

This is your mail server, there is no higher profile application in your environment most likely than this box. You'll want to think 3 to 5 years down the road when you spec out the box. Make sure to confirm:

  • It will scale or offer:
    • Hotpluggable technology
      • You'll need to confirm the OS can handle this
    • Redundant and Hot-swap power supplies
      • You might need to upgrade your power supplies depending on the "accessories" you put in it.
    • CPU's (There is NO reason to use a 32bit chip - the memory limitations will kill you)
      • By default, a 32-bit Linux kernel only allows each process to address 2GB of space. Through PAE (Process Address Extension), a feature available in some CPUs, and a special 32-bit kernel that supports large address space for processes, it is possible to get a 32-bit mode kernel that really uses > 4GB of RAM, and get a per process 3-4GB address range
    • Memory
      • What are your onboard cache options?
      • How many slots are available? Does it force you to buy the large memory sticks to max out total memory - this increases cost?
      • Can you mix & match different memory size sticks? (This will increase costs when you goto scale)
      • Understand how memory interacts with multi-cpu's if your server has them
      • Understand the front side bus (FSB)
      • Does the motherboard offline for bad memory detection
    • Have growth for expansion cards and the right kind
      • What is PCI Express?
      • Know what slots you need for your network card, raid card, san card, etc.
        • Then do a sanity check against the motherboard and understand if the cpu and channels can allow the full throughout from all channels.
      • Is there room for redundancy with your expansion cards?

Memory

Other references: Performance_Tuning_Guidelines_for_Large_Deployments#RAM_and_CPU

When I was working with HPC, I found there was never a good reason to NOT at least start with 32GB's of RAM. Now, a mail server isn't a HPC compute node - I understand that. But I would still try to spec out a system that has at least 8 memory slots (usually a ratio of x slots per cpu on system) but allows me to use 4 4GB DIMMS giving me 16GB of memory and allows an upgrade path choice of 4 more 4GB or 8GB DIMMS. Get the higher speed DIMMS, you can't mixed memory of different speeds.

Bus For Expansion Cards

Typical uses are for network, san, raid cards. I'll deal with them separately under each section below.

Network Infrastructure

Network Cards

Most motherboards will have integrated ethernet ports. If they are the same chipset then you might want to just channel bond these. What I have done in the past, is used them for management ports and added in network cards for production activity.

Ideally, I would buy two cards that had two ports (Gb). I would then channel bond 2 of the ports across the cards, for 2 separate bondings. I would use one of those bonds for "front facing" traffic and the other for "backup" traffic. Remember to consider the bus infrastructure and other cards when deciding on what ones to get.

Channel Bonding

This will require you to confirm your switches can do this. There are different choices when it comes to channel bonding and some of them require the switch to support it - it usually involves "bonding" on the switch side by configuring the ports in question.

The Production Channel Bond

The "production" bond is more in regards to failover. Make sure this happens as expected through the application layer. That proxies, firewall, and so forth allow on of the ports in the channel bond to go down without any end-user impact.

The Backup Channel Bond

The backup port is for throughput. You'll want to map out the network path and switches to the end host that data is being moved to. You'll also need to confirm the network path actually gives you a gain in performance by doing the channel bonding.

This will give you an excellent way to off load your tape backups, rsyncs (as shown at Ajcody-Notes-ServerMove , and maybe nfs mounts if your using them.

You'll want to map out the network path and switches to the end host the data is being moved to. You'll also need to confirm the network path actually gives you a gain in performance by doing the channel bonding.

Disk Infrastructure

Other references: Performance_Tuning_Guidelines_for_Large_Deployments#Disk

Disk Types

SATA
  1. Interface Type
    • Comments:
  2. Performance
    • Access Time:
      • Comments:
    • I/Os per second & Sustained transfer rate:
      • SATA 1.5 Gbit/s (SATA/150)
        • Real speed: 150 MB/s
      • SATA 3.0 Gbit/s (SATA/300 - Sometimes referred to as SATA II or SATA2)
        • Real speed: 300 MB/s
      • SATA 6.0 Gbit/s (SATA 6Gb/s)
        • Standard is expected to be available before the end of 2008.
    • Latency:
      • Comments:
  3. Reliability (MTBF or unrecoverable read error rate)
    • Comments:
  4. Capacity
    • Comments:
  5. Price
    • Comments:
SCSI (Parallel SCSI)
  1. Interface Type
    • Comments:
    • Maximum Devices (On Single Channel)
      • Ultra Wide SCSI 16
      • Ultra2 SCSI 8
      • Ultra2 Wide SCSI 16
      • Ultra3 SCSI 16 (alternate names: Ultra-160, Fast-80 wide)
      • Ultra-320 SCSI 16 (alternate name: Ultra4 SCSI)
      • Ultra-640 SCSI 16
  2. Performance
    • Access Time:
      • Comments:
    • I/Os per second:
      • Comments:
    • Sustained transfer rate:
      • Ultra Wide SCSI 40 (16 bits/20MHz) - 320 Mbit/s
        • Real speed: 40 MB/s
      • Ultra2 SCSI
        • Real speed: 40 MB/s
      • Ultra2 Wide SCSI 80 (16 bits/40 MHz) - 640 Mbit/s
        • Real speed: 80 MB/s
      • Ultra3 SCSI 160 (16 bits/80 MHz DDR) (alternate names: Ultra-160, Fast-80 wide) - 1,280 Mbit/s
        • Real speed: 160 MB/s
      • Ultra-320 SCSI (alternate name: Ultra4 SCSI) (16 bits/80 MHz DDR) - 2,560 Mbit/s
        • Real speed: 320 MB/s
      • Ultra-640 SCSI (16 bits/160 MHz DDR) - 5,120 Mbit/s
        • Real speed: 640 MB/s
    • Latency:
      • Comments:
  3. Reliability (MTBF or unrecoverable read error rate)
    • Comments:
  4. Capacity
    • Comments:
  5. Price
    • Comments:
SAS (Serial Attached SCSI)
  1. Interface Type
    • Comments:
    • Maximum Devices:
      • SAS 16,256
  2. Performance
    • Access Time:
      • Comments:
    • I/Os per second:
      • Comments:
    • Sustained transfer rate:
      • SAS 300 MB/s (full duplex , per direction)
    • Latency:
      • Comments:
  3. Reliability (MTBF or unrecoverable read error rate)
    • Comments:
  4. Capacity
    • Comments:
  5. Price
    • Comments:
Fibre Channel
  1. Interface Type
    • Comments:
    • Maximum Devices
      • FC-AL 1Gb 127
      • FC-AL 2Gb 127
      • FC-AL 4GB 127
  2. Performance
    • Access Time:
      • Comments:
    • I/Os per second:
      • Comments:
    • Sustained transfer rate:
      • FC-AL 1Gb 100 MB/s (full duplex , per direction)
      • FC-AL 2Gb 200 MB/s (full duplex , per direction)
      • FC-AL 4GB 400 MB/s (full duplex , per direction)
    • Latency:
      • Comments:
  3. Reliability (MTBF or unrecoverable read error rate)
    • Comments:
  4. Capacity
    • Comments:
  5. Price
    • Comments:

Raid Cards (Not Raid Level)

  • If your raid chip on your motherboard goes out, what is your expect downtime to resolve it?
    • Isn't it easier to buy a hot space raid card and replace it in case of a raid card failing?

SAN

San Cards
  • Have you planned on multipathing?
    • You do have more than one port/card, right?
      • If not (because of cost), you do have an open slot on motherboard to allow another one?
      • Remember to consider this with your SAN switch purchase
    • Confirm that your HBA's, HBA drivers, SAN switches allow for the options you want?
    • Multipathing can be for failover
    • Multipathing can increase throughput for same partition mount
Multpathing
iSCSI
Fibre (or Fiber) Channel

Filesystem Choices

Other references: Performance_Tuning_Guidelines_for_Large_Deployments#File_System

Most configuration go with ext3 because of it's implied default. My own experience has me using xfs always except for the root ( / ) partition.

  1. LVM
    • 1.
      • A.
      • B.
    • 2.
      • A.
      • B.
  2. XFS
    • 1.
      • A.
      • B.
    • 2.
      • A.
      • B.
  3. Ext3
    • 1.
      • A.
      • B.
    • 2.
      • A.
      • B.
  4. Reiser
    • 1.
      • A.
      • B.
    • 2.
      • A.
      • B.

Local (Server) Disk Configuration

I would have 3 disk subsystems & partitions at a minimum.

OS
  1. For OS / - this could be whatever - local mirrored SATA/SCSI drives
    • A. Make sure /tmp has enough space for operations
    • B. Setup a proper mirror for /
      • 1. Most likely you'll use linux software raid for this
        • A. Make sure to put in a commented configuration line in /etc/fstab to use the non-active mirror
      • 2. Most motherboards will have at least two SATA ports and drive slots.
Backup
  1. /opt/zimbra/back
    • A. I would make sure the disk I/O is separate from /opt/zimbra . This way you minimize performance hits to your end-users. Do a review of the disk i/o | bus is clean as possible to the cpu/memory. Motherboard spec's should tell you want "slots" are on shared buses. Make sure you maximizing your raid/san cards performance to the bus path to the cpu.
    • B. I would make this a multiple of some degree of the /opt/zimbra space that your spec'ing. This default backup schedule is to purge sessions older than one month. This means you'll have 4 fulls and 24 incremental sessions.
    • C. If disks are local
      • 1. Raid Choices
        • A.
        • B.
      • 2. LVM
        • A.
        • B.
      • 2. Other Topics
        • A.
        • B.
    • D. If disks are on SAN
      • 1. Raid Choices
        • A.
        • B.
      • 2. LVM
        • A.
        • B.
      • 2. Other Topics
        • A.
        • B.
Zimbra
  1. /opt/zimbra
    • A. Remember you having logging data in here as well. If this partition becomes full, zimbra will hang and could cause database corruption as well.
    • C. If disks are local
      • 1. Raid Choices
        • A.
        • B.
      • 2. LVM
        • A.
        • B.
    • D. If disks are on SAN
      • 1. Raid Choices
        • A.
        • B.
      • 2. LVM
        • A.
        • B.
      • 2. Other Topics
        • A.
        • B.

NFS

What About Backups? I Need A Plan

Please Review Other Backup-Restore As Well

Please review the Ajcody-Backup-Restore-Issues section as well.

What Might Be Wrong?

First thing to check is the log file and see if anything notable stands out.

grep -i backup /opt/zimbra/log/mailbox.log

Use Auto-Group Backups Rather Than Default Style Backup

Please see the administrative manual page on this:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html

Compare the sections called "Standard Backup Method" & "Auto-Grouped Backup Method"

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html#1080744

Configuration details are here on that page:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html#1084777

See this for a really good explanation:

http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html

Enabling Auto-Group For Backups

Make sure your backup schedule was adjusted for this change (zmschedulebackup which basically dumps to zimbra's crontab). The url above mentions this and give an example.

First, basic reference:

  • Run zmschedulebackup without arguments to see your current schedule.
  • Here is a schedule without auto-grouped backups enabled:
    • Current Schedule:
f 0 1 * * 6 -a all
i 0 1 * * 0-5 -a all
d 1m 0 0 * * *
  • Here is a schedule with auto-grouped enabled:
    • Current Schedule:
f 0 23 * * * -z
d 7d 0 21 * * *
  • You'll need to make the variable changes as listed in the administrative guide and then the following for set the crontab correctly.
  • Run zmschedulebackup --help to see a list of options.
  • Run zmschedulebackup -D , which will now set a new default schedule (crontab) that uses auto-group settings now that your variable are set for it.
  • Incremental backups aren't performed as they were.
    • You'll see there's no longer the -i option in cron.
    • Incrementals are performed, in a manner, but the redologs are just copied into the full backup session

Two bugs to look at as well:

Some Variables For Auto-Group

The below might not be complete or the defaults, I just wanted to save this before I forget them. Try to get more complete details on these later.

zmprov gacf | grep Backup
zimbraBackupAutoGroupedInterval: 1d
zimbraBackupAutoGroupedNumGroups: 7
zimbraBackupAutoGroupedThrottled: FALSE
zimbraBackupMode: Auto-Grouped

Need To Write Fewer Files - Add The Zip Option To Your Backup Commands

There is very little details in the official documentation on this option unfortunately.

This does have a really good explanation though:

http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html

From the administrative manual on the Backup section:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html

It says,

"-zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space."

It's implied that instead of having all the individual message files in the backup that it will bunch them all together into zip files. This will be useful when the number of message files is causing disk i/o issues. Maybe your trying to rsync the backup session directories off to another server or your running a third party backup on it to save to tape. The default use of -zip will use compression, if this also causes overhead that you need to avoid you can use the -zipStore option.

Note about -zipStore:

"when used with the -zip option, it allows the backup to write fewer files (-zip), but not incur the compression overhead as well"
How To Use As A Default Option?

You'll add the options to the zimbra crontab file. This can be done with the zmschedulebackup command.

Run zmschedulebackup with help option:

zmschedulebackup --help

You'll see:

-z: compress - compress email blobs with zip

It appears that you'll need to manually add the options about -zipStore , if you want it, to the crontab file.

See bug :

http://bugzilla.zimbra.com/show_bug.cgi?id=30981

What Does It Look Like When I Use Zip?

Shared blobs are zipped and blobs (messages) are zipped per root store directory.

mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs 
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip

To Tape?

I would then use the non-rsync network ports for your traditional network backup software to run over to dump the data to tape. This way that activity doesn't effect prod performance at all. All full DR would use the backup/ data anyways (offsite DR).


Creating A Hot-spare server

Setup along the same lines... though you could cut out some of the HA/performance items if you only see this box being used for "short-term" use. Rsync's will occur over backup network port.

  1. Setup the hot-spare according to http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove
    • Basically install the zimbra packages. You'll want to do this with upgrades to production as the system evolves.
  2. I would do an initial rsync of /opt/zimbra (remember to use nice flag to not effect prod to much)
  3. I would then setup 2 daily rsync jobs (following the same wiki instructions)
    1. rsync /opt/zimbra/backup
      • This could be intergrated within the backup cron job so it kicks off after the backup is done. You'll need to monitor the times of backups and also the time for sync's so you can make sure you make it within the window of rotation - backup , rsync , next backup. Times will be different on diff and full backups.
      • rsync other necessary files:
        • /opt/zimbra/conf
        • /opt/zimbra/redolog
        • /opt/zimbra/log
      • This will give some "sanity" in case issues arise with the full restore. This part could use some better feedback from more experience Zimbra staff. I can think of some circumstances where his effort would prove useful.

A Real Hot-spare DR RFE

Note: Please add your votes to this RFE!

If It All Blows Up, Now What?

References:

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove

http://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery

Jump to: navigation, search