Ajcody-Server-Topics: Difference between revisions
Line 7: | Line 7: | ||
==Server Move== | ==Server Move== | ||
Please see | Please see [[Ajcody-Notes-ServerMove]] | ||
{{:Ajcody-Notes-BackupPlans|Backup Plans And Cheap HA/DR Secondary Server}} | |||
== | ==My Disaster Recovery Failed== | ||
Reference: [[Network_Edition_Disaster_Recovery#Restoring_to_the_new_server]] | |||
First, make sure your TIME is set right! See [[:Time_Zones_in_ZCS#The_server_OS]] | |||
Though I'm still investigating why this is happening for our customers, the root issues seem to be resolved by the following. Some of the auth errors will be logged to /var/log/zimbra.log and /opt/zimbra/log/mailbox.log | Though I'm still investigating why this is happening for our customers, the root issues seem to be resolved by the following. Some of the auth errors will be logged to /var/log/zimbra.log and /opt/zimbra/log/mailbox.log | ||
Line 130: | Line 126: | ||
zmprov ms -- -zimbraServiceEnabled ldap | zmprov ms -- -zimbraServiceEnabled ldap | ||
==Public Service Hostname== | ==A Proxy On You!== | ||
===Proxy References=== | |||
From Install Guide: | |||
* Configuring IMAP and POP Proxy Server | |||
** http://www.zimbra.com/docs/ne/latest/multi_server_install/planning%20a%20multi-server%20instalaltion.4.1.html#1073257 | |||
* Configuring ZCS HTTP Proxy (Beta 5.0.6) | |||
** http://www.zimbra.com/docs/ne/latest/multi_server_install/planning%20a%20multi-server%20instalaltion.4.1.html#1090706 | |||
* Configuring for Virtual Hosting | |||
** http://www.zimbra.com/docs/ne/latest/multi_server_install/planning%20a%20multi-server%20instalaltion.4.1.html#1090962 | |||
===Notable Proxy Bugs=== | |||
* "mailbox does not load with zimbraPublicServiceHostname and reverse proxy enabled" | |||
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=29138 | |||
* "WRONG_HOST error while viewing a public shared doc using http proxy" | |||
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=29204 | |||
* "In reverse-proxied mode, when redirecting with a 302, web client should set Location to proxy provided Host header" | |||
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=29053 | |||
* "ZCS Mailbox Server Proxy" | |||
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=9469 | |||
* "need to support multi-server (proxy) for mobile clients" | |||
** See: http://bugzilla.zimbra.com/show_bug.cgi?id=8174 | |||
===Public Service Hostname=== | |||
variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema | variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema | ||
Line 216: | Line 239: | ||
http://bugzilla.zimbra.com/show_bug.cgi?id=25135 | http://bugzilla.zimbra.com/show_bug.cgi?id=25135 | ||
{{:Ajcody-Notes-No-Such-Blob}} | |||
==Migration Issues== | ==Migration Issues== |
Revision as of 23:50, 18 July 2008
Server Topics
Actual Server Topics Homepage
Please see Ajcody-Server-Topics
Server Move
Please see Ajcody-Notes-ServerMove
Ajcody Notes Server Planning
Server Performance Planning & Infrastructure - The Big Picture
Actual Server Planning Homepage
Please see Ajcody-Notes-ServerPlanning
Initial Comments
These are just my random thoughts. They have not been peer-reviewed yet.
Actual DR and Server restore issues are listed in my Ajcody-Server-Topics page.
Items For Your Review
- Official Zimbra Performance Tuning Guidelines
- Red Hat Enterprise Linux 5 performance tuning: Monitoring and maximizing your system
- Linux Performance Tuning Part 1
- Linux Performance Tuning Part 2
- Finding The Bottleneck
- Improve Systems Sizing and Capacity Planning
- Selecting a Platform for your MySQL server
- RFE - "Customer Accessible Sizing Guide"
- RFE - "Installation guidelines cover basic do's and don'ts (RAID 5, GFS, XSAN, etc.)"
Redundancy - High Availability - And Other Related Questions
One might ask, "Is there any other way to have redundant zimbra email servers?"
The short answer is No, but if you have millions of dollars you can get really close.
Remember, redundancy always comes with a dollar amount cost. And this dollar amount will determine your range of options to implement redundant technologies or processes.
Redundancy isn't magical and all choices have a "time" for the operation to failover and the application being used can restrict what can be done or increase the length of "time" of the operation. Timing of the operation is where the service/server has a client side impact of being "unavailable".
So you break down the server and services into components and put them in a graph or excel sheet. This will help to make sure your not "over engineering" the system.
For example, disk and data.
- Disks can use different raid systems to give redundancy. Channels to the disk (remote) can also have redundant paths. The disk chassis can have redundant power supplies which goto different UPS's on different circuit breakers. This data can also be sent to tape, rysnc'd/copied to another disk subsystem on another server, or "flashed" to another location if your filesystem or SAN unit supports it. There's allot that has to fail for you to completely use the data. When exploring these items, you want to have multiple channel paths to make sure copies, rsync's, flashing occurs differently as compared to the "production" path. Have your network backup occur on a different ethernet device.
Two big picture objectives for "redundancy" are "data redundancy" for DR situations and "application" redundancy for service availability. Our "cluster" situation is an active-passive "redundancy" for the purposes of the mailstore application layer. The raid level of the disks on the san servers the "redundancy" in regards to data recovery for DR.
When you introduce the objective of having off-site redundancy the costs and issues become huge. Remote locations introduce speed issue for data transfers which will also impact performance for most applications as it tries to stay in sync between the two machines for write and roll-back purposes.
My intent wasn't so much to give specific answers to this question but rather demonstrate that to answers these questions you have to get down to the real specifics - it's simply impossible to answer the broad question of "how do I make redundant servers". It would take a book to fully explore all the issues and possibilities but it still comes down to - how much money you have to spend on the project. I use to work with High Performance Clusters and HA Linux+Oracle+SAP global deployments with TB's of data - this issue would arise daily for us in those environments.
HA Clustering Software
From the Single Node Cluster Installation Guide - Rev 1 June 2008:
- For ZCS 5.0.7 to ZCS 7. With ZCS 8, only Vmware HA is support by Zimbra. RHCS still can work but it would be hardware only monitoring and support should be directed to Redhat for the specifics.
- For cluster integration to provide high availability, Zimbra Collaboration Suite (ZCS) can integrate with either of the following:
- Red Hat® Enterprise Linux® Cluster Suite version 4, Update 5 or later update release. In the single-node cluster implementation, all Zimbra servers are part of a cluster under the control of the Red Hat Cluster Manager.
- Note: Red Hat Cluster Suite consists of Red Hat Cluster Manager and Linux Virtual Server Cluster. For ZCS, only Red Hat Cluster Manager is used. In this guide, Red Hat Cluster Suite refers only to Cluster Manager.
- Veritas™ Cluster Server by Symantec (VCS) version 5.0 with maintenance pack 1 or later.
References:
- Red Hat Cluster Suite
- Veritas Software Related to High Availability
- I have other notes about HA Clustering Software here:
CPU And Motherboards
Other references: Performance_Tuning_Guidelines_for_Large_Deployments#RAM_and_CPU
This is your mail server, there is no higher profile application in your environment most likely than this box. You'll want to think 3 to 5 years down the road when you spec out the box. Make sure to confirm:
- It will scale or offer:
- Hotpluggable technology
- You'll need to confirm the OS can handle this
- Redundant and Hot-swap power supplies
- You might need to upgrade your power supplies depending on the "accessories" you put in it.
- CPU's (There is NO reason to use a 32bit chip - the memory limitations will kill you)
- By default, a 32-bit Linux kernel only allows each process to address 2GB of space. Through PAE (Process Address Extension), a feature available in some CPUs, and a special 32-bit kernel that supports large address space for processes, it is possible to get a 32-bit mode kernel that really uses > 4GB of RAM, and get a per process 3-4GB address range
- Memory
- What are your onboard cache options?
- How many slots are available? Does it force you to buy the large memory sticks to max out total memory - this increases cost?
- Can you mix & match different memory size sticks? (This will increase costs when you goto scale)
- Understand how memory interacts with multi-cpu's if your server has them
- Understand the front side bus (FSB)
- Does the motherboard offline for bad memory detection
- Have growth for expansion cards and the right kind
- What is PCI Express?
- How PCI Express Works?
- Know what slots you need for your network card, raid card, san card, etc.
- Then do a sanity check against the motherboard and understand if the cpu and channels can allow the full throughout from all channels.
- Is there room for redundancy with your expansion cards?
- Hotpluggable technology
Memory
Other references: Performance_Tuning_Guidelines_for_Large_Deployments#RAM_and_CPU
When I was working with HPC, I found there was never a good reason to NOT at least start with 32GB's of RAM. Now, a mail server isn't a HPC compute node - I understand that. But I would still try to spec out a system that has at least 8 memory slots (usually a ratio of x slots per cpu on system) but allows me to use 4 4GB DIMMS giving me 16GB of memory and allows an upgrade path choice of 4 more 4GB or 8GB DIMMS. Get the higher speed DIMMS, you can't mixed memory of different speeds.
Chart Of Memory Speeds
Memory Interconnect Buses | Bit | Bytes |
---|---|---|
PC2100 DDR-SDRAM (single channel) | 16.8 Gbit/s | 2.1 GB/s |
PC1200 RDRAM (single-channel) | 19.2 Gbit/s | 2.4 GB/s |
PC2700 DDR-SDRAM (single channel) | 21.6 Gbit/s | 2.7 GB/s |
PC800 RDRAM (dual-channel) | 25.6 Gbit/s | 3.2 GB/s |
PC1600 DDR-SDRAM (dual channel) | 25.6 Gbit/s | 3.2 GB/s |
PC3200 DDR-SDRAM (single channel) | 25.6 Gbit/s | 3.2 GB/s |
PC2-3200 DDR2-SDRAM (single channel) | 25.6 Gbit/s | 3.2 GB/s |
PC1066 RDRAM (dual-channel) | 33.6 Gbit/s | 4.2 GB/s |
PC2100 DDR-SDRAM (dual channel) | 33.6 Gbit/s | 4.2 GB/s |
PC2-4200 DDR2-SDRAM (single channel) | 34.136 Gbit/s | 4.267 GB/s |
PC4000 DDR-SDRAM (single channel) | 34.3 Gbit/s | 4.287 GB/s |
PC1200 RDRAM (dual-channel) | 38.4 Gbit/s | 4.8 GB/s |
PC2-5300 DDR2-SDRAM (single channel) | 42.4 Gbit/s | 5.3 GB/s |
PC2-5400 DDR2-SDRAM (single channel) | 42.664 Gbit/s | 5.333 GB/s |
PC2700 DDR-SDRAM (dual channel) | 43.2 Gbit/s | 5.4 GB/s |
PC3200 DDR-SDRAM (dual channel) | 51.2 Gbit/s | 6.4 GB/s |
PC2-3200 DDR2-SDRAM (dual channel) | 51.2 Gbit/s | 6.4 GB/s |
PC2-6400 DDR2-SDRAM (single channel) | 51.2 Gbit/s | 6.4 GB/s |
PC4000 DDR-SDRAM (dual channel) | 67.2 Gbit/s | 8.4 GB/s |
PC2-4200 DDR2-SDRAM (dual channel) | 67.2 Gbit/s | 8.4 GB/s |
PC2-5300 DDR2-SDRAM (dual channel) | 84.8 Gbit/s | 10.6 GB/s |
PC2-5400 DDR2-SDRAM (dual channel) | 85.328 Gbit/s | 10.666 GB/s |
PC2-6400 DDR2-SDRAM (dual channel) | 102.4 Gbit/s | 12.8 GB/s |
PC2-8000 DDR2-SDRAM (dual channel) | 128.0 Gbit/s | 16.0 GB/s |
PC2-8500 DDR2-SDRAM (dual channel) | 136.0 Gbit/s | 17 GB/s |
PC3-8500 DDR3-SDRAM (dual channel) | 136.0 Gbit/s | 17 GB/s |
PC3-10600 DDR3-SDRAM (dual channel) | 165.6 Gbit/s | 21.2 GB/s |
PC3-12800 DDR3-SDRAM (dual channel) | 204.8 Gbit/s | 25.6 GB/s |
Bus For Expansion Cards [Peripheral buses]
Typical uses are for network, san, raid cards. I'll deal with them separately under each section below.
Chart Of Bus Speeds
Interconnect | Max speed (MB/s) | Comments |
---|---|---|
PCI 2.0 | 132.0 MB/s | |
PCI 2.1 | 264.0 MB/s | |
PCI 2.2 | 528 MB/s | |
PCI-X 1.0 | 1 GB/s | |
PCI-X 2.0 | 4 GB/s | |
PCI-E (Express) Ver. 1.1 | 250 MB/s x2 - bi-directional | These speeds are bi-directional per "lane". Meaning that they are the same going both ways and not shared. |
Ver. 1.1 @ 1x | 256 MB/s x2 - bi-directional | PCI-E Ver. 1.1 notes. |
Ver. 1.1 @ 2x | 512 MB/s x2 - bi-directional | PCI-E Ver. 1.1 notes. |
Ver. 1.1 @ 4x | 1 GB/s x2 - bi-directional | PCI-E Ver. 1.1 notes. |
Ver. 1.1 @ 8x | 2 GB/s x2 - bi-directional | PCI-E Ver. 1.1 notes. |
Ver. 1.1 @ 16x | 4 GB/s x2 - bi-directional | PCI-E Ver. 1.1 notes. |
PCI-E (Express) Ver. 2.0 | 400 MB/s x2 - bi-directional | 500 MBs but there's a 20% overhead hit. These speeds are bi-directional per "lane". Meaning that they are the same going both ways and not shared. |
Ver. 2 @ 1x | 400 MB/s x2 - bi-directional | PCI-E Ver. 2 notes. |
Ver. 2 @ 4x | 1600 MB/s x2 - bi-directional | PCI-E Ver. 2 notes. |
Ver. 2 @ 8x | 3200 MB/s x2 - bi-directional | PCI-E Ver. 2 notes. |
Ver. 2 @ 16x | 6400 MB/s x2 - bi-directional | PCI-E Ver. 2 notes. |
PCI-E (Express) Ver. 3.0 | 1 GB/s x2 - bi-directional | The final spec is due in 2009. These speeds are bi-directional per "lane". Meaning that they are the same going both ways and not shared. |
Front-side Bus replacements below | ||
HyperTransport 1.x | 6.4 GB/s x2 - bi-directional | Bidirectional per 32 bit link at 800MHz. Front-side bus replacement, see HyperTransport for more details . |
HyperTransport 2.0 | 11.2 GB/s x2 - bi-directional | Bidirectional per 32 bit link at 1.4GHz. Front-side bus replacement, see HyperTransport for more details . |
HyperTransport 3.0 | 20.8 GB/s x2 - bi-directional | Bidirectional per 32 bit link at 2.6GHz/ Front-side bus replacement, see HyperTransport for more details . |
HyperTransport 3.1 | 25.6 GB/s x2 - bi-directional | Bidirectional per 32 bit link at 3.2GHz. Front-side bus replacement, see HyperTransport for more details . |
QuickPath Interconnect | 12.8 GB/s x2 - bi-directional | Intel competitor to HyperTransport. Everything you wanted to know about QuickPath (QPI) |
Network Infrastructure
Network Cards
Most motherboards will have integrated ethernet ports. If they are the same chipset then you might want to just channel bond these. What I have done in the past, is used them for management ports and added in network cards for production activity.
Ideally, I would buy two cards that had two ports (Gb). I would then channel bond 2 of the ports across the cards, for 2 separate bondings. I would use one of those bonds for "front facing" traffic and the other for "backup" traffic. Remember to consider the bus infrastructure and other cards when deciding on what ones to get.
Channel Bonding
This will require you to confirm your switches can do this. There are different choices when it comes to channel bonding and some of them require the switch to support it - it usually involves "bonding" on the switch side by configuring the ports in question.
The Production Channel Bond
The "production" bond is more in regards to failover. Make sure this happens as expected through the application layer. That proxies, firewall, and so forth allow on of the ports in the channel bond to go down without any end-user impact.
The Backup Channel Bond
The backup port is for throughput. You'll want to map out the network path and switches to the end host that data is being moved to. You'll also need to confirm the network path actually gives you a gain in performance by doing the channel bonding.
This will give you an excellent way to off load your tape backups, rsyncs (as shown at Ajcody-Notes-ServerMove ), and maybe nfs mounts if your using them.
You'll want to map out the network path and switches to the end host the data is being moved to. You'll also need to confirm the network path actually gives you a gain in performance by doing the channel bonding.
Disk And Data Infrastructure
Other references: Performance_Tuning_Guidelines_for_Large_Deployments#Disk
Disk Types
References
- Source for some items below used statements from these resources*
- Reference Guide - Hard Disk Drives
- Storage Technology and Drive Interfaces
Description of Items Used Below For Specific HDD's:
- Notable Items
- Benefits
- Downsides
- Bus Interface
- Types:
- Maximum Devices:
- Bus Speeds:
- Performance
- I/Os per second & Sustained Transfer Rate:
- Spindle Speed: This is the speed the drives disk actually spins at. This ranges from 5400rpm to 15,000rpm. The higher the speed the more often the data on the disk will be in the right position to be read by the drive heads, and the faster data can be transferred.
- Average Access Time: Time. This is the average time is takes to position the heads so that data can be read. The faster the better.
- Cache Size: This is the size of the cache on board the disk drive itself. This does reach a point where doubling the size generates a very small boost in performance and is not worth the cost, but generally the bigger the better. It also assists in small bursts of data where you may actually achieve near maximum performance as data is read from cache rather than the drive.
- Internal Transfer Rate: This is the speed that data can be transferred within the drive. This speed will be higher than the actual transfer rate of the drive as there is some overhead for protocol handling as data is transferred to the SCSI or IDE bus.
- Latency: the time it takes for the selected sector to be positioned under the read/write head. Latency is directly related to the spindle speed of the drive and such is influenced solely by the drive's spindle characteristics.
- Reliability (MTBF or unrecoverable read error rate)
- Capacity
- Price
Ultra DMA ATA HDD's
- Notable Items
- Benefits
- Downsides
- Bus Interface
- Types : ATA
- Performance
- I/Os per second & Sustained transfer rate:
- Ultra DMA ATA 33 - 264 Mbit/s 33 MB/s
- Ultra DMA ATA 66 - 528 Mbit/s 66 MB/s
- Ultra DMA ATA 100 - 800 Mbit/s 100 MB/s
- Ultra DMA ATA 133 - 1,064 Mbit/s 133 MB/s
- Spindle Speed:
- Average Access Time:
- Cache Size:
- Internal Transfer Rate:
- Latency:
- I/Os per second & Sustained transfer rate:
- Reliability (MTBF or unrecoverable read error rate)
- Capacity
- Price
SATA HDD's
- Notable Items
- Benefits
- SATA drives typically draw less power than traditional SAS HDDs due to slower RPM speeds.
- SATA drives have the best dollar per gigabyte compared to SAS drives.
- SATA HHDs can work on a SAS interface.
- Downsides
- SATA HDDs are single port and not capable of being utilized in dual port environments without the addition of an interposer designed for this purpose.
- Benefits
- Bus Interface Type
- Types: SATA , SAS
- Performance
- I/Os per second & Sustained transfer rate:
- Serial ATA (SATA-150) - 1,500 Mbit/s 187.5 MB/s
- Real speed: 150 MB/s
- Serial ATA (SATA-300) - 3,000 Mbit/s 375 MB/s
- Real speed: 300 MB/s
- (alternate names: SATA II or SATA2)
- Serial ATA (SATA-600) - 4,800 Mbit/s 600 MB/s
- I've seen this listed as well though, SATA 6.0 Gbit/s (SATA 6Gb/s).
- Standard is expected to be available before the end of 2008.
- Serial ATA (SATA-150) - 1,500 Mbit/s 187.5 MB/s
- Spindle Speed: 7200 RPM , 5400 RPM
- Average Access Time:
- Cache Size:
- Internal Transfer Rate:
- Latency:
- I/Os per second & Sustained transfer rate:
- Reliability (MTBF or unrecoverable read error rate)
- Capacity
- Price
SCSI (Parallel SCSI) HDD's
- Notable Items
- Benefits
- Downsides
- Bus Interface Type
- Types: SCSI
- Maximum Devices (On Single Channel):
- Ultra Wide SCSI - 16
- Ultra2 SCSI - 8
- Ultra2 Wide SCSI - 16
- Ultra3 SCSI - 16
- (alternate names: Ultra-160, Fast-80 wide)
- Ultra-320 SCSI - 16
- (alternate name: Ultra4 SCSI)
- Ultra-640 SCSI - 16
- Bus Speeds:
- The total amount of data that can be transferred throughout the whole channel.
- Data transfers will step down to the rate speed of the drive. If you have a Ultra160 HDD on a Ultra320 controller it operate at 160 MB/s.
- SCSI-3 : Also known as Ultra SCSI and fast-20 SCSI. Bus speeds at 20 MB/s for narrow (8 bit) systems and 40 MB/s for wide (16-bit).
- Ultra-2 : Also know as LVD SCSI. Data transfer to 80 MB/s.
- Ultra-3 : Also known as Ultra-160 SCSI. Data transfer to 160 MB/s.
- Ultra-320 : Data transfer to 320 MB/s.
- Ultra-640 : Also known as Fast-320. Data transfer to 640 MB/s.
- Performance
- I/Os per second & Sustained Transfer Rate:
- Ultra Wide SCSI 40 (16 bits/20MHz) - 320 Mbit/s 40 MB/s
- Real speed: 40 MB/s
- Ultra2 SCSI
- Real speed: 40 MB/s
- Ultra2 Wide SCSI 80 (16 bits/40 MHz) - 640 Mbit/s 80 MB/s
- Real speed: 80 MB/s
- Ultra3 SCSI 160 (16 bits/80 MHz DDR) - 1,280 Mbit/s 160 MB/s
- (alternate names: Ultra-160, Fast-80 wide)
- Real speed: 160 MB/s
- Ultra-320 SCSI (16 bits/80 MHz DDR) - 2,560 Mbit/s 320 MB/s
- (alternate name: Ultra4 SCSI)
- Real speed: 320 MB/s
- Ultra-640 SCSI (16 bits/160 MHz DDR) - 5,120 Mbit/s 640 MB/s
- Real speed: 640 MB/s
- Ultra Wide SCSI 40 (16 bits/20MHz) - 320 Mbit/s 40 MB/s
- Spindle Speed:
- Average Access Time:
- Cache Size:
- Internal Transfer Rate:
- Latency:
- I/Os per second & Sustained Transfer Rate:
- Reliability (MTBF or unrecoverable read error rate)
- Capacity
- Price
SAS (Serial Attached SCSI) HDD's
- Notable Items
- Benefits
- SAS HDDs are true dual port, full duplex devices. This means SAS HDDs can simultaneously process commands on both ports.
- All SAS HDDs are hot-swap capable. Users can add or remove an HDD without disrupting the enterprise environment.
- SAS HDDs can support online firmware update (check vendor). This allows users to update firmware on the SAS HDD without having to schedule downtime.
- Downsides
- SAS HDDs cannot be used on the older architecture SCSI backplanes or cables.
- SAS HDDs typically draw more power than the equivalent SATA counterparts.
- Benefits
- Bus Interface Type
- Types: SAS
- Maximum Devices:
- SAS 16,256
- 128 devices per port expanders
- SAS 16,256
- Bus Speeds:
- Performance
- I/O per second & Sustained transfer rate:
- Serial Attached SCSI (SAS) - 3,000 Mbit/s 375 MB/s
- Real Speed: 300 MB/s (full duplex , per direction)
- Serial Attached SCSI 2 - 6,000 Mbit/s 750 MB/s
- Planned
- Serial Attached SCSI (SAS) - 3,000 Mbit/s 375 MB/s
- Spindle Speed: 15000 RPM , 10000 RPM , 7200 RPM
- Average Access Time:
- Cache Size:
- Internal Transfer Rate:
- Latency:
- I/O per second & Sustained transfer rate:
- Reliability (MTBF or unrecoverable read error rate)
- 10K & 15K SAS HDDs has been rated at 1.6 million hours MTBF
- Capacity
- Price
Fibre Channel - Arbitrated Loop (FC-AL) HDD's
- Notable Items
- Benefits
- FC-AL HDDs are dual ported, providing two simultaneous input/output sessions
- FC-AL HDDs are hot-swap capable so users can add and remove a hard drives without interrupting system operation.
- Downsides
- FC-AL HDDs are typically utilized in unique environments and are not compatible to be used on SAS or SATA interfaces
- Long term, SAS is projected to replace FC-AL HDDs within the IT industry
- Benefits
- Bus Interface Type
- Types: FC-AL
- Maximum Devices:
- FC-AL in a private loop with 8-bit ID's - 127
- FC-AL in a public loop with 24-bit ID's - +16 million
- Bus Speeds:
- Performance
- I/Os per second & Sustained transfer rate:
- FC-AL 1Gb 100 MB/s (full duplex , per direction)
- FC-AL 2Gb 200 MB/s (full duplex , per direction)
- FC-AL 4GB 400 MB/s (full duplex , per direction)
- Spindle Speed:
- Average Access Time:
- Cache Size:
- Internal Transfer Rate:
- Latency:
- I/Os per second & Sustained transfer rate:
- Reliability (MTBF or unrecoverable read error rate)
- Capacity
- Price
Raid Cards (Not Raid Level)
- If your raid chip on your motherboard goes out, what is your expect downtime to resolve it?
- Isn't it easier to buy a hot space raid card and replace it in case of a raid card failing?
SAN Typologies - Interfaces
- Discussion of issues and differences between iSCSI & Fibre Channel
General Questions
San Cards
- Have you planned on multipathing?
- You do have more than one port/card, right?
- If not (because of cost), you do have an open slot on motherboard to allow another one?
- Remember to consider this with your SAN switch purchase
- Confirm that your HBA's, HBA drivers, SAN switches allow for the options you want?
- multipathing can be for failover
- multipathing can increase throughput for same partition mount
- You do have more than one port/card, right?
iSCSI Interface
So far, physical devices have not featured native iSCSI interfaces on a component level. Instead, devices with SCSI Parallel Interface or Fibre Channel interfaces are bridged by using iSCSI target software, external bridges, or controllers internal to the device enclosure. Source
- iSCSI over Fast Ethernet 100 Mbit/s 12.5 MB/s
- iSCSI over Gigabit Ethernet 1,000 Mbit/s 125 MB/s
- iSCSI over 10G Ethernet (Very few products exist) 10,000 Mbit/s 1,250 MB/s
- iSCSI over 100G Ethernet (Planned) 100,000 Mbit/s 12,500 MB/s
iSCSI Performance Topics
One will find the following recommendations in regards to iscsi:
- Private network for iscsi traffic
- Jumbo frames enabled with MTU size of 9000
- All equipment must support and be enabled for this from point to point.
- Remember, point to point meaning the "server" to the host serving the iscsi disk array.
- All equipment must support and be enabled for this from point to point.
iSCSI References
- Red Hat Enterprise Linux 4 & NetApp iSCSI (How-To)
- Question about bonding nics and mtu configuration
- Jumbo Packets with netapp FAS3050
Serial Attached SCSI (SAS) Interface
- Notable Items
- Benefits
- SAS interface protocol supports both SAS and SATA Hard Disk Drives allowing tiered storage management.
- SAS supports seamless scalability through port expansion enabling customers to daisy chain multiple storage enclosures.
- SAS supports port aggregation via a x4 wide-link for a full external connection bandwidth of up to 12.0 Gbps (1200MBps) on a single cable and a single connection.
- SAS is a point-to-point interface allowing each device on a connection to have the entire bandwidth available. The current bandwidth of each SAS port is 3Gb/sec. with future generations aimed at 6Gb/sec and beyond.
- Downsides
- SAS is not backwards compatible with U320 SCSI or previous SCSI generations
- Benefits
Fibre Channel - Arbitrated Loop (FC-AL)
- Notable Items
- Benefits
- FC-AL devices can be dual ported, providing two simultaneous input/output sessions that doubles maximum throughput
- FC-AL enables "hot swapping," so you can add and remove hard drives without interrupting system operation, an important option in server environments.
- Downsides
- FC-AL adapters tend to cost more than SAS adapters
- FC-AL is currently the fastest interface at 4Gb but is expected to be passed in maximum bandwidth by the next generation of SAS interface at 6Gb
- Benefits
Multipathing
General References
- Articles on Multipathing in regards to Points of Failure and Performance
Multipathing And SAN Persistent Binding
- SAN Persistent Binding and Multipathing in the 2.6 Kernel
- Redhat - How do I make device mapper multipath ignore my local disks when generating the multipath maps in Red Hat Enterprise Linux 4/5?
Multipathing And LVM
- A hands-on guide to enterprise storage features of Red Hat Enterprise Linux 4, including Fiber Channel connected volume management and multipath I/O.
- LVM2 and Device-Mapper - 6 Presentations
Multipathing - Redundant Disk Array Controller [RDAC] (LSI mpp driver)
References:
- IBM - Installing Linux distributions on a Multipathed Device using the RDAC Driver
- SUN - Sun StorageTek RDAC Failover Driver for Linux OS
Multipathing - Fibreutils - sg3_utils (QLogic)
References:
- HP - Fibre Utility for HP branded Qlogic HBAs and Qlogic based mezzanine HBAs
- Multipathing with Qlogic HBAs - failover and LUN load balancing - Working Example
Multipathing - lpfc (Emulex)
References:
Multipathing - device-mapper [dm] & multipath-tools (Linux OSS driver & tools)
DM is the open source multipath driver.
References:
- Multipath-tools Project Homepage
- LWN - Multipath support in the device mapper
- Redhat - DM Multipath : Device-Mapper Multipath Configuration and Administration
Multipathing - Veritas (Symantec)
References:
Multipathing - HP & Linux
References:
- HP - Multi-path Device Mapper for Linux Software
- Installation and Reference Guide Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays - Linux (PDF)
Multipathing - NetApp
References:
- Linux: Snapshot Records and LUN Resizing in a SAN Environment (PDF Warning)
- "Linux® (RHEL 4) 64-Bit Performance with NFS, iSCSI, and FCP Using an Oracle® Database on NetApp Storage (PDF Warning)
- Technical Case Study: Red Hat Deploys Network Appliance™ Storage to Support a Rapidly Expanding All-Linux® Environment (PDF Warning)
Multipathing - EMC (PowerPath)
References:
- EMC PowerPath Family Product Page
- Setting up device-mapper-multipath on RHEL 4/5 with QLogic cards to a EMC Clariion
Raid Levels For Disks
General References
References:
- Raid On Wikipedia
- Starndard Raid Levels On Wikipedia
- Raid 0 , Raid 1 , Raid 2 , Raid 3 , Raid 4 , Raid 5 , Raid 6
- Nested Raid Levels On Wikipedia
- Raid 0+1, Raid 1+0 (Raid 10), Raid 0+3 and 3+0 , Raid 0+3 , Raid 30 , Raid 100 (Raid 10+0) , Raid 50 (Raid 5+0) , Raid 60 (Raid 6+0)
- Non-Standard Raid Levels On Wikipedia
Raid 10
RAID 10 (or 1+0) uses both striping and mirroring.
Reference - http://bugzilla.zimbra.com/show_bug.cgi?id=10700 - I fixed statement based upon a private comment in this RFE.
- Zimbra recommends Raid 10 for the :
- Mailstore and Logger [MySQL] databases - /opt/zimbra/db & /opt/zimbra/logger
- Indexing Volume [Lucene] database - /opt/zimbra/index
- Raid 10 is NOT Raid 0+1
- RAID 0+1 takes 2 RAID0 volumes and mirrors them. If one drive in one of the underlying RAID0 groups dies, that RAID0 group is unreadable. If a single drive simultaneously dies in the other RAID0 group, the whole RAID 0+1 volume is unreadable.
- RAID 1+0 takes "mirrored slices" and build a RAID0 on top of them. A RAID10 volume can lose a maximum of half of its underlying disks at once without data loss (as long as only one disk from each "slice pair" is lost.)
Raid 5
RAID 5 (striped disks with parity) combines three or more disks in a way that protects data against loss of any one disk; the storage capacity of the array is reduced by one disk.
Reference - http://bugzilla.zimbra.com/show_bug.cgi?id=10700
- Zimbra does NOT recommend RAID 5 for the MySQL database and Lucene volumes (RAID 5 has poor write performance, and as such is generally not recommended by MySQL, Oracle, and other database vendors for anything but read-only datastores. We have seen order of magnitude performance degradation for the embedded Zimbra database running on RAID 5!).
- Mailstore and Logger [MySQL] databases - /opt/zimbra/db & /opt/zimbra/logger
- Indexing Volume [Lucene] database - /opt/zimbra/index
Raid 6
RAID 6 (less common) can recover from the loss of two disks. Basically an extension of Raid 5.
Filesystem Choices
Other references:
- Performance_Tuning_Guidelines_for_Large_Deployments#File_System
- Says, "We recommend the ext3 file system for Linux deployments (tried and true, performance for random IO is a wash, gains only in blob store for other file systems)."
- Comparison of file systems
Bugs/RFE's that reference filesystems:
- "disaster recovery through server to server sync (beta)"
- "Installation guidelines cover basic do's and don'ts (RAID 5, GFS, XSAN, etc."
Most configuration go with ext3 because of it's implied default. My own experience has me using xfs always except for the root ( / ) partition.
Filesystem Feature And Option Descriptions
Journaling
Inodes
To see existing inode use:
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/VolGroup00-LogVol00 19005440 158715 18846725 1% / /dev/sda1 26104 39 26065 1% /boot
Block Size
Volume Managers
LVM / LVM2 (Linux)
- 1.
- A.
- B.
- 2.
- A.
- B.
- Resources:
EVMS (IBM's Enterprise Volume Management System)
- 1.
- A.
- B.
- 2.
- A.
- B.
- Resources:
Veritas Volume Manager
- 1.
- A.
- B.
- 2.
- A.
- B.
- Resources:
Regular File Systems
XFS
Supported
- 1. Bugs and RFE's
- A. RHEL7 will use xfs as default filesystem
- Add RHEL7 64-bit support
- B.
- A. RHEL7 will use xfs as default filesystem
- 2. Journaling
- A. "XFS uses what is called a metadata journal. Basically, this means that every disc transaction is written in a journal before it is written to the disc and then marked as "done" in the journal when it finishes. If the system crashes during the writing of the journal entry, that incomplete entry can be ignored since the data on the disc has not been touched yet and if the journal entry is not marked done, then that operation can be rolled back to preserve disc integrity. Its a very nice system. As stated above, XFS practices a type of journaling called "metadata journaling." This means only the inodes are journaled, not the actual data. This will preserve the integrity of the file system, but does not preserve the integrity of the data." reference: Filesystem Design Part 1 : XFS
- B.
- 3. Inodes
- A. "FS considers dynamic allocation of inodes and keeps track of such inodes using B+Trees. Each allocation group uses a B+Tree to index the locations of inodes in it. This allows to create millions of inodes in each allocation group and thus supporting large number of files" reference: PDF Warning - Failure Analysis of SGI XFS File System
- B.
- Resources:
EXT3 - EXT4
Supported
- 1.
- A.
- B.
- 2.
- A.
- B.
- Resources:
Reiser
- 1.
- A.
- B.
- 2.
- A.
- B.
- Resources:
vxfs - Veritas
- 1.
- A.
- B.
- 2.
- A.
- B.
- 3. Dynamic inode allocation
- A. A description of sorts of what this means, "Vxfs is dynamic, so the inodes are basically created on the fly. However, it's a bit inaccurate to actually state that inodes are dynamic in VxFS and leave it at that (somewhat confusing I know). Vxfs really doesn't use them per se. It creates them to be compatible with UFS. VxFS uses extent-based allocation rather than the block-based UFS controlled by inodes. So, the question of how many inodes for vxfs, is as many as it needs."
- B. http://www.docs.hp.com/en/B3929-90011/ch02s04.html
- Resources:
BTRFS - Oracle's Better File System for Linux (with Redhat, Intel, HP)
- 1.
- A.
- B.
- 2.
- A.
- B.
- Resources:
ZFS
- 1.
- A.
- B.
- 2.
- A.
- B.
- Resources:
Network File Systems
NFS
- 1. Only supported for the Zimbra backup directory or when used internally to Vmware vSphere at this time. Please see Bug/RFE and comments below.
- 2.
- A.
- B.
- Resources:
- Bugs/RFE's:
- "Need clarity on supporting nfs mounted zimbra directories - report error/msg if nfs mount is present"
- "Zimbra on NFS Storage through VMware ESX"
- http://bugzilla.zimbra.com/show_bug.cgi?id=50635
- Note - I asked this rfe to become private since it is really an internal request to do testing/qa'ing of nfs with vSphere. I asked another rfe for public viewing to be made that will let customers know when that can deploy with it under production use.
Purposed inclusion to release notes from Bug/RFE above:
- ZCS & NFS:
- Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
- Zimbra continues to view NFS storage of other parts of the system as unsupported. Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
- When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.
Samba - SMB/CIFS
- 1. Unsupported at this time for use with Zimbra.
- 2.
- A.
- B.
- Bugs/RFE's:
- "store to support a link yes/no setting"
- http://bugzilla.zimbra.com/show_bug.cgi?id=43019
- Issue tied to sym/hard link usage.
- http://bugzilla.zimbra.com/show_bug.cgi?id=43019
- "store to support a link yes/no setting"
- Resources:
Global , Distributed , Cloud, Cluster-Type Filesystems - Unsupported
Currently, Zimbra does not support or recommend the use of the various filesystems listed under this section. Please see the specific section to see if I've identified any existing bugs/RFE's against them. One general RFE for this topic is:
- "Installation guidelines cover basic do's and don'ts (RAID 5, GFS, XSAN, etc.)"
GFS
- References:
- Red Hat Global File System Homepage
- Note - GFS is not supported, but some customers have tried using it under the radar. They could hit this issue:
Lustre [Acquired by Sun]
- References:
Hadoop Distributed File System (HDFS) - Cluster Filesystem Project From Apache & Yahoo!
- References:
SGI CXFS
- References:
IBM GPFS
- References:
Veritas Storage Foundation Cluster File System
- Resources:
Google Computing
Unsupported at this time by Zimbra Support.
References:
- Google Cloud Platform Homepage
- https://cloud.google.com/compute/
- Support, QA, testing for Google Compute Engine
- https://cloud.google.com/compute/
Amazon S3 - And Amazon EC2 Information
Unsupported at this time by Zimbra Support.
References:
- Amazon Web Services Homepage - There's various "services" available there
- http://aws.amazon.com/
- Support, QA, testing for Amazon Web Services - AWS
- Amazon Simple Storage Service (Amazon S3)
- http://aws.amazon.com/s3/
- Zimbra RFE "Support, QA, testing for Amazon S3"
- Amazon Elastic Compute Cloud (Amazon EC2)
- http://aws.amazon.com/ec2/
- Zimbra RFE "Support, QA, testing for Amazon EC2"
- http://aws.amazon.com/
- Zimbra RFE "store to support a link yes/no setting"
- For more details and specifics for use with backups, see:
- http://www.zimbra.com/forums/installation/25306-remote-backup-cloud-storage-amazon-s3.html
Still going to try this when it's not supported? See the following if you run into troubles and if your doing a new setup:
- http://www.zimbra.com/forums/installation/47123-installing-zimbra-ubuntu-10-04-amazon-ec2.html
- Use this url instead, the one in the forum is bad.
Other Products - Applications (DRBD, Veritas Volume Replicator, etc.)
Unsupported at this time by Zimbra Support.
- DRBD
- A. Unsupport at this time.
- 1.
- B. Related bugs & rfe's that apply to this.
- 1. "disaster recovery through server to server sync (beta)"
- 2. "add active-active support to zcs" , marked as a dup of the above.
- References:
- A. Unsupport at this time.
- Veritas Volume Replicator
- A.
- 1.
- B.
- 1.
- References:
- A.
Background Items To Review Before Drawing Up A Plan
When you start putting all this together you'll find there's allot of "exceptions" to work out based upon your needs, equipment, kernel version, distro choice, SAN equipment, HBA's, OSS drivers vs. vendor ones, and so forth. You must TEST all your assumptions when your planning on using methods that will provide online filesystem growth. Don't deploy just on an assumption. Some situations will not allow a true online filesystem growth.
References to review:
- kernel mailing list thread on support to "detect online disk resize"
- http://thread.gmane.org/gmane.linux.scsi/44335
- Some SAN situations will not be able to detect a LUN increase without a server reboot.
- http://thread.gmane.org/gmane.linux.scsi/44335
- "SCSI - Hot add, remove, rescan of SCSI devices"
- Suse/Novell - SLES 10 Documentation
- http://www.novell.com/documentation/sles10/index.html
- Items to review:
- SLES 10 Storage Administration Guide
- SLES 10 Storage Administration Guide for EVMS
- Items to review:
- http://www.novell.com/documentation/sles10/index.html
- Suse/Novell Cool Solutions - Linux Data Management
- Suse/Novell Cool Solutions - Building a SAN Storage Device Using iSCSI on SUSE Linux
- Redhat - Online Storage Reconfiguration Guide (BETA)
- Redhat - Creating A Scalable OS Storage Infrastructure
- Redhat - LVM Administrator's Guide For RHEL 4.5
- Redhat - Cluster Logical Volume Manager (RHEL 5.x)
Online Resizing Of LUN Example 1 - Multipathing Required
- Warning, this process reminds me of what I was doing over a year ago [before zimbra]. I don't have the necessary hardware or my old notes to go through the steps and confirm.
- Resize the SAN volume.
- Reinitialize the HBA, e.g. using sg_reset or some module specific method.
- Rescan the SCSI device:
echo 1 > /sys/block/[sdx]/device/rescan.
- Now confirm that /proc/partitions should contain updated values. The fdisk and sfdisk commands may still (most likely) see the old values.
- Remove and readd the SCSI Devices.
- Warning: Without multipathing you are going to loose access to your disk!! Multipathing gives you several paths to the volume, so you will not loose access. Make sure multipathd is up and running.
echo "scsi remove-single-device <Host> <Channel> <SCSI ID> <LUN>" > /proc/scsi/scsi
echo "scsi add-single-device <Host> <Channel> <SCSI ID> <LUN>" > /proc/scsi/scsi
- This will also issue new device names!
- Multipathing updated it's device size automatically after I reloaded the last path-device.
- Warning: Without multipathing you are going to loose access to your disk!! Multipathing gives you several paths to the volume, so you will not loose access. Make sure multipathd is up and running.
- Run:
- If you have LVM: vgresize, lvresize if you have a lvm setup.
- Need to double check for the need of pvresize & vgextend use first.
- Your filesystem online grow commands if you have just a filesystem.
- If you have LVM: vgresize, lvresize if you have a lvm setup.
Q&A
- Q. The use of multipathing failover has the block device being changed in size, potentially during a write (this was, after all, online). What happens if data is being written to the device while you're doing the disconnect/reconnect operations? Do you have two conflicting pieces of information about the device?
- A. The write is being done at the "filesystem" level, which is unaware of the block device size change.
Example Filesystem Layouts
Local (Server) Disk Configuration
I would have 3 disk subsystems at a minimum. Meaning, three distinction group of disks on their own disk i/o backplane, if possible. For example:
- Two disks using the motherboard provided SATA/ATA ports. (OS)
- Multiple disks accessed through a dedicated SAN or SCSI card. Maybe these are the hot swap disk array available on the front of your server.(Backup)
- Multiple disks accessed through another dedicated SAN or SCSI card. Maybe these are the disks available from an external disk array on your server or to external disks via SAN/iSCSI. (zimbra)
OS - /boot , swap , /tmp , /
- Your first disk is referred to as disk1 or sda. Your second disk is referred to as disk2 or sdb. Partitions will start at 1 , rather than 0, throughout my examples below.
- For the OS partitions, there's no reason to use anything but "default" for the mounting options in the /etc/fstab. If you reviewed the Performance_Tuning_Guidelines_for_Large_Deployments#File_System wiki page, those settings would apply to a dedicated partition(s) used for zimbra - not the root OS level partitions.
- For OS ( /boot , swap, /tmp , / )
- A. This could be whatever for disks (SATA, SCSI, etc.).
- 1. Two Disk minimum for mirroring. Four Disks would allow raid 10.
- A. I prefer to use software raid, because then the "raid" will move with the disks with less complications if my server dies. I can simply move the disks into another server chassis.
- B. Most motherboards will have at least two SATA ports and drive slots. Make sure you put your drive on different channels if you have the options rather than being on the same.
- 1. Two Disk minimum for mirroring. Four Disks would allow raid 10.
- B. /boot partition
- 1. I would setup a /boot partition (128MB or 256MB) on drive. This would be my first partition on each drive.
- A. Disk 1 > partition type linux > partition 1 for 128MB or 256MB > filesystem ext3 > no "raiding" > NO LVM > mount point /boot
- B. Disk 2 > partition type linux > partition 1 for 128MB or 256MB > filesystem ext3 > no "raiding" > NO LVM > mount point /boot2
- 2. After OS install, I would then "rsync/copy" the contents of /boot into /boot2
- 3. Configure grub or your bootloader to have the option to boot from /boot2 (disk1/partition0)
- A. This will give you a failover in case something goes wrong with /boot (disk0/partition0). Bad kernel upgrade or a wacked partition maybe.
- 4. After any changes to /boot (disk1/partition1) confirm everything works right (confirm reboots fine), you would then do another manual rsync/copy of /boot to /boot2 (disk2/partition1).
- 1. I would setup a /boot partition (128MB or 256MB) on drive. This would be my first partition on each drive.
- C. swap
- 1. Determine how much swap you "need". I will use an example of 2GB's below. Notice both "swap" partitions will get 2 GB's.
- 2. Setup a swap partition on disk1/partition2 and disk2/partition2 for 2 GB's.
- A. Disk 1 > partition type - swap > partition 2 for 2GB > filesystem swap > no "raiding" > NO LVM > mount point swap
- B. Disk 2 > partition type - swap > partition 2 for 2GB > filesystem swap > no "raiding" > NO LVM > not set to mount
- 3. *Default Suggestion* Configure the OS to only use/mount swap on disk1/partition2. You can configure /etc/fstab for the other partition but just comment out the line for now.
- 4. Reasoning for this.
- A. swap drives that are "mirror" can cause undue complications.
- B. I configured a "same" sized swap on the other drive really for the third partition - that for /. This way the blocking/sizes are as close as possible for the / mirror.
- C. It isn't really a "lose" of space, but rather for adjustments you might need later.
- 1. It can server as a "failure" over swap partition incase things go bad on disk0/partition1.
- 2. It can server for more "production" swap if you find you need it.
- 3. It allows for a complete disk failover or simplicity in case you need to move the "one" disk to another box.
- 5. You could also configure the two swap partitions into a raid0 if you would like. Swap partitions can be turned on/off (swapon , swapoff). It's easy enough to reformat them as well, if something goes wrong, after a swapoff.
- D. /tmp
- 1. I generally never setup a partition for /tmp , but if you did decide to do this make sure it's following the swap partitions. ( IMO )
- A. If you do setup /tmp , I would go with ext3 or xfs and put it within LVM. See notes below about LVM use.
- 1. I generally never setup a partition for /tmp , but if you did decide to do this make sure it's following the swap partitions. ( IMO )
- E. / - Introduction of LVM
- 1. You'll now place the rest of the "free" disks under a "software" raid1 / mirror partition.
- A. You should then see this "mirror" as a new disk/partition to be used within the OS installer.
- 2. Place the "mirror" under LVM - this will be a "partition" type. Let's assume this new mirror is now /dev/md0.
- 3. Now configure the LVM partition for LVM. I'm assuming that a partition wasn't made for /tmp
- A. General concepts of what happens with the LVM setup and recommendations on the naming scheme. Sda3 is disk1/partion3 and sdb3 is disk1/partition3
- 1. pvcreate md0
- 2. vgcreate -s 128M rootvg md0
- A. vgchange -a y rootvg
- 3. lvcreate -L 150GB -n rootlv rootvg
- A. I put in 150GB above as an example. I would normally put in about 90% of the available space left and leave me some room to create a new partition/mount point if it becomes necessary. For example, let's say / kept filling up because of lazy admins not cleaning up in /root. I could create a new vg group called roothomevg and mount it as /root , restricting the messy admins from effecting / .
- 4. Now you would make the filesystem.
- A. Example of setting up ext3 filesystem using defaults.
- 1. mkfs.ext3 /dev/rootvg/rootlv
- A. Example of setting up ext3 filesystem using defaults.
- 5. Now you need to setup the new filesystem in /etc/fstab to mount. I'll use /root as an example, considering the / would of been done through the installer and you would of most likely used the GUI tools to have done all of this.
- A. mkdir /root (if it didn't exist - this example would of of course)
- B. vi /etc/fstab
- 1. /dev/rootvg/rootlv /root defaults 1 2
- A. Adjust for your system defaults in regards to the device naming convention that's being used.
- 1. /dev/rootvg/rootlv /root defaults 1 2
- A. General concepts of what happens with the LVM setup and recommendations on the naming scheme. Sda3 is disk1/partion3 and sdb3 is disk1/partition3
- 4. So why did I bother with LVM if I used all the disk space anyways?
- 1. If you setup the / with LVM at the start, even if you use all the disk space, it allows you in the future to add more disk to the underlying LVM setup to grow the / filesystem - online.
- A. For example, let's say I have 2 open drive bays that weren't initially used when I setup my server. And two years later I find my / becoming very close to 100%. I can throw in two new drives into those bays (assuming hot-swap drives). Setup a mirror (mdadm) between the two drives. Set the new mirror partition type to LVM. Then run through the pvcreate , vgextend , lvresize , and then online grow the filesystem (ext2online/resize2fs , xfs_growfs , etc.)
- ext2online - http://ext2resize.sourceforge.net/online.html
- resize2fs (umount fs) - http://linux.die.net/man/8/resize2fs
- xfs_growfs - http://linux.die.net/man/8/xfs_growfs
- A. For example, let's say I have 2 open drive bays that weren't initially used when I setup my server. And two years later I find my / becoming very close to 100%. I can throw in two new drives into those bays (assuming hot-swap drives). Setup a mirror (mdadm) between the two drives. Set the new mirror partition type to LVM. Then run through the pvcreate , vgextend , lvresize , and then online grow the filesystem (ext2online/resize2fs , xfs_growfs , etc.)
- 1. If you setup the / with LVM at the start, even if you use all the disk space, it allows you in the future to add more disk to the underlying LVM setup to grow the / filesystem - online.
- 1. You'll now place the rest of the "free" disks under a "software" raid1 / mirror partition.
- A. This could be whatever for disks (SATA, SCSI, etc.).
Backup
- /opt/zimbra/backup
- A. I would make sure the disk I/O is separate from /opt/zimbra . This way you minimize performance hits to your end-users. Do a review of the disk i/o bus path is clean as possible to the cpu/memory. Motherboard spec's should tell you what "slots" are on shared buses. Make sure your maximizing your raid/san cards performance to the bus path to the cpu.
- B. I would make this a multiple of some degree of the /opt/zimbra space that your spec'ing. This default backup schedule is to purge sessions older than one month. This means you'll have 4 fulls and 24 incremental sessions.
- 1. Make sure you investigate the auto-group backup method as well and the zip option. This could have a huge effect on your disk space requirements for the backup partition.
- C. If disks are local
- 1. Raid Choices
- A. If you'll be sending this to tape as well, you could go with a raid0 for performance.
- B. If you don't plan on sending to tape or another remote storage system, maybe raid5.
- 2. LVM
- A. Please encapsulate the backup partition under LVM. This will give you some flexibility later in case you need more space.
- 2. Other Topics
- 1. Raid Choices
- D. If disks are on SAN
- 1. Raid Choices
- A. If you'll be sending this to tape as well, you could go with a raid0 for performance.
- B. If you don't plan on sending to tape or another remote storage system, maybe raid5.
- C. Note, if you have allot of disks to construct your raid with you still can achieve good performance with raid5. This way you don't have to lose all the "disks" when doing raid10. It would be worth bench marking your SAN using x number of disks configured as raid10 vs. the same x number of disks are raid5. Remember to consider the i/o backplanes involved in what "disks" you choose to use throughout your different disk chassis. Going up and down your disk rack vs left to right.
- 2. LVM
- A. Please encapsulate the backup partition under LVM. This will give you some flexibility later in case you need more space.
- B. If your NAS/SAN system is going to do block level snap-shots, the choice to use LVM or not becomes more complicated. A block level snap-shot across multiple LUN's will generally not work when the top level filesystem is using LVM. If you plan on only using one LUN and growing that same LUN as needed, then LVM will still prove useful if your also using the SAN/NAS block level snap-shots.
- 2. Other Topics
- 1. Raid Choices
Zimbra
- /opt/zimbra
- A. Remember you having logging data in here as well. If this partition becomes full, zimbra will hang and could cause database corruption as well.
- C. If disks are local
- 1. Raid Choices
- A. Raid 0 or Raid 10
- B.
- 2. LVM
- A. Please encapsulate the backup partition under LVM. This will give you some flexibility later in case you need more space.
- B.
- 1. Raid Choices
- D. If disks are on SAN / NAS
- 1. Raid Choices
- A. Raid 0 or Raid 10
- B. Note, if you have allot of disks to construct your raid with you still can achieve good performance with raid5. This way you don't have to lose all the "disks" when doing raid10. It would be worth bench marking your SAN using x number of disks configured as raid10 vs. the same x number of disks are raid5. Remember to consider the i/o backplanes involved in what "disks" you choose to use throughout your different disk chassis. Going up and down your disk rack vs left to right.
- 2. LVM
- A. Please encapsulate the partitions under LVM. This will give you some flexibility later in case you need more space.
- B. If your NAS/SAN system is going to do block level snap-shots, the choice to use LVM or not becomes more complicated. A block level snap-shot across multiple LUN's will generally not work when the top level filesystem is using LVM. If you plan on only using one LUN and growing that same LUN as needed, then LVM will still prove useful if your also using the SAN/NAS block level snap-shots.
- 2. Other Topics
- 1. Raid Choices
More Details About LVM Use
The notes below were gathered from the "Zimbra Admins in Universities" mailing list.
Contents of Post by Matt on Date: Mon, 27 Oct 2008 14:24:50 -0500
We've found however that we are able to grow storage on the fly with LVM. It basically works like this for us... * Grow the LUN on the SAN o wait for the LUN to finish growing by checking the 'Jobs / Current Jobs' display until the "Volume Initialization" job is finished. * reboot the host to see the new partition size. o (partprobe -s is supposed to do this, but it doesn't) * find the device name: o pvs | grep VolumeName? * grow the Volume Group: o pvresize -v /dev/XXX * Verify the new size: o pvs | grep VolumeName? * grow the logical volume: o Grow by a specific size: lvextend --size +NNg /dev/VgName/LvName o Grow to use all free space: lvextend -l +100%FREE /dev/VgName/LvName * grow the file system: o Online method (dangerous?) + ext2online /dev/VgName/LvName o Offline method (safer?) + umount /mountpoint + e2fsck -f /dev/Vgname/Lvname + resize2fs /dev/Vgname/LvName + mount /dev/VgName/LvName /mountpoint * Verify new filesystem size: o df -h /mountpoint I've always used the online method (marked "dangerous?" by one of my cohorts) and never had a problem. One other thing we've been able to do with LVM that has been a benefit is migrating data to a new LUN... 1. Find the new physical volume that is associated with the correct LUN#. On the Zimbra servers you can use this MPP (linuxrdac) tool. # /opt/mpp/lsvdev 2. Prepare the physical volume with PVCREATE. # pvcreate /dev/sdX 3. Extend the logical volume to the new physical volume with VGEXTEND. # vgextend /dev/VolGroupName /dev/sdX 4. Use LVDISPLAY to make sure you are moving from the right physical volume. # lvdisplay /dev/VolGroupName/LogVolName -m Example Results =========== --- Logical volume --- LV Name /dev/VgMbs03Backup/LvMbs03Backup VG Name VgMbs03Backup LV UUID 0vZQx3-5A22-a4ZO-4VmV-2naM-jwoi-yc6r6k LV Write Access read/write LV Status available # open 0 LV Size 580.00 GB Current LE 148479 Segments 1 Allocation inherit Read ahead sectors 0 Block device 253:6 --- Segments --- Logical extent 0 to 148478: Type linear Physical volume /dev/sdab Physical extents 0 to 148478 5. Move the Volume Group to the new physical volume using PVMOVE. # pvmove -i 60 -v /dev/sdZ /dev/sdX -i 60 : Show progress every 60 seconds -v : Verbose /dev/sdZ : Physical volume we are moving from /dev/sdX : Physical volume we are moving to 6. When the move is completed use VGREDUCE to reduce the volume group down to the new physical volume. # vgreduce /dev/VolGroupName /dev/sdZ
A reply was given to the post above, noting the issue of the reboot in the steps above. Rich, Mon, 27 Oct 2008 15:24:06 -0500, write:
Any process including the term "reboot" isn't "on the fly." :-) Current proprietary OSes can rescan and use expanded LUNs on the fly while filesystems are mounted. Apparently, so can the latest development Linux kernel, but stacked device-mapper and LVM layers will need major changes, so don't expect to see this capability in enterprise linices for 2 years. You can save some time, though, by replacing "reboot" with the minimum steps required to clear all holders of the device: umount /file/system vgchange -an VG service multipathd stop multipath -ll # note the physical devices involved, here assuming sd[fg] multipath -f mpathVG echo 1 > /sys/block/sdf/device/rescan # partprobe -s might also do this echo 1 > /sys/block/sdg/device/rescan multipath -v2 service multipathd start ...and continue with the pvresize. But simply adding a new LUN and marking it active with the admin console (or zmvolume) can be done with zero downtime, so that's my new model.
SAN Layout As Recommend For Clusters
This is from Multi Node Cluster Installation Guide - PDF
Preparing the SAN
You can place all service data on a single volume or choose to place the service data in multiple volumes. Configure the SAN device and create the partitions for the volumes.
- Single Volume SAN Mount Point - /opt or /opt/zimbra
- If you select to configure the SAN in one volume with subdirectories, all service data goes under a single SAN volume.
- Multiple Volumes For SAN Mount Points
- If you select to partition the SAN into multiple volumes, the SAN device is partitioned to provide the multiple volumes for each Zimbra mailbox server in the cluster. Example of the type of volumes that can be created follows.
- /opt Volume for ZCS software (or really, /opt/zimbra/ )
- Directories under /opt/zimbra/
- conf Volume for the service-specific configuration files
- log Volume for the local logs for Zimbra mailbox server
- redolog Volume for the redo logs for the Zimbra mailbox server
- db/data Volume for the MySQL data files for the data store
- store Volume for the message files
- index Volume for the search index files
- backup Volume for the backup files
- logger/db/data Volume for the MySQL data files for logger service’s MySQL instance
- openldap-data Volume for OpenLDAP data
- Directories under /opt/zimbra/
- /opt Volume for ZCS software (or really, /opt/zimbra/ )
- If you select to partition the SAN into multiple volumes, the SAN device is partitioned to provide the multiple volumes for each Zimbra mailbox server in the cluster. Example of the type of volumes that can be created follows.
- Note, for a multi-volume SAN Cluster, you'll actually create the directory path differently. [ /opt/zimbra-cluster/mountpoints <clusterservicename.com> ] Please see the Cluster Installation Guides for the full planning recommendations and steps if this is what your going to do.
Zimbra Directory Layout & FHS (Filesystem Hierarchy Standard)
FHS
References:
- Filesystem Hierarchy Standard Group
Bugs/RFE's
- "move zcs data from /opt/zimbra/ to /var/opt/zimbra/ to conform to Filesystem Hierarchy Standard"
- "installer to allow something like --fhs-strict (for FHS layout requirements)"
Community Feedback
- Feedback Reference One
We are following FHS[1] standard for our deployments (or at least trying our best to follow it). It would be nice to reflect on the possibilities of mostly FHS-compliant Zimbra deploy. Here's what we've came up with so far: /etc/opt/zimbra for configs /opt/zimbra - binaries /var/opt/zimbra - message store, OpenLDAP db, MySQL db's etc. /var/log/zimbra - logs Going by the FHS standards (in our case) means deploying a well-documented system and that its layout is consistent across the board. Benefits: * A paranoid type setup like mounting /opt read-only, and /var as no-exec. * For the uber-paranoia, including /etc as read-only. * You could tune each FS for specific needs which are consistent across the board. ** Different FS / or differently tuned FS/ used for each generic case. * Migrations would be fairly simple, as it's easy to rip out configs (/etc) or data (/var) or logs (/var/log) and copy/move it someplace else. * It opens the door to possibility of mounting volume with binaries on multiple machines that only have local configs and data (not that we plan on it at the moment).
Disk Or Disk Layout Performance Testing
hdparm - Read or set the hard drive parameters
References:
- hdparm MAN page.
- "Speeding up Linux Using hdparm"
Using DD
1GB file test
sync ; time bash -c "(dd if=/dev/zero of=largefile bs=1024 count=1000000; sync)"
Now time the removal of the large file.
sync ; time bash -c "(rm largefile; sync)"
Bonnie++
References:
- Bonnie++ Homepage
- "Using Bonnie++ for filesystem performance benchmarking"
dbench - generate load patterns
References:
- dbench - Freshmeat.net entry.
IOzone
References:
- IOZone Homepage
Stress
References:
- Stress Project Homepage
Postmark (Netapp)
References:
- "Filesystem Benchmarking with PostMark from Network Appliance"
LTP - Linux Test Project
This suite of tools has filesystem performance tests.
References:
- "LTP Project Homepage"
Clustering Software
Please see Ajcody-Clustering
Virtualization
Please see Ajcody-Virtualization
What About Backups? I Need A Plan
Please Review Other Backup-Restore As Well
Please review the Ajcody-Backup-Restore-Issues section as well.
What Might Be Wrong?
First thing to check is the log file and see if anything notable stands out.
grep -i backup /opt/zimbra/log/mailbox.log
Use Auto-Group Backups Rather Than Default Style Backup
Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.
Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
Please see the following for more detailed information:
Need To Write Fewer Files - Add The Zip Option To Your Backup Commands
Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :
- Using nfs for the backup directory
- Copying/rsyncing backups to a remote server
- Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.
Please see the following for more information about using the Zip option:
SAN Snapshots For Backups
Please see:
Cloud Backups
Please see:
Tape Backups
I would then use the non-rsync network ports for your traditional network backup software to run over to dump the data to tape. This way that activity doesn't effect prod performance at all. All full DR would use the backup/ data anyways (offsite DR). I've created another section that will deal with this in more details - specifically handling the hard links that are used by Zimbra.
Please see:
Test Environments And Managing Customizations
I have some suggestions on this in the RFE below. The first comment has a recommended layout for your test/qa/dev environments:
- "Improve processes for upgrades & "support" when customer has 'customizations'"
Using Vmware ESX For A DEV/QA/Prod Test Environment
Please see Ajcody-Virtualization#Using_VMWare_ESX_For_ZCS_Test_Servers_-_How-To
Creating A Hot-spare Server
Setup along the same lines... though you could cut out some of the HA/performance items if you only see this box being used for "short-term" use. Rsync's will occur over backup network port.
Need to do a sanity check in regards to ldap data. With a normal DR restore, one would do a zmrestoreldap. zmrestoreldap looks to a backup session, there is no real option in regards to a "redolog" directory. Things that are "ldap" only are COS's, DL's, etc..
- Setup the hot-spare according to http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove
- Basically install the zimbra packages. You'll want to do this with upgrades to production as the system evolves.
- I would do an initial rsync of /opt/zimbra (remember to use nice flag to not effect prod to much)
- I would then setup 2 daily rsync jobs (following the same wiki instructions)
- rsync /opt/zimbra/backup
- This could be intergrated within the backup cron job so it kicks off after the backup is done. You'll need to monitor the times of backups and also the time for sync's so you can make sure you make it within the window of rotation - backup , rsync , next backup. Times will be different on diff and full backups.
- rsync other necessary files:
- /opt/zimbra/conf
- /opt/zimbra/redolog
- /opt/zimbra/log
- This will give some "sanity" in case issues arise with the full restore. This part could use some better feedback from more experience Zimbra staff. I can think of some circumstances where this effort would prove useful.
- rsync /opt/zimbra/backup
A Real Hot-spare DR RFE
Note: Please add your votes to this RFE!
- "Disaster recovery through server to server sync (beta)"
If It All Blows Up, Now What?
References:
http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove
http://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery
My Disaster Recovery Failed
Reference: Network_Edition_Disaster_Recovery#Restoring_to_the_new_server
First, make sure your TIME is set right! See Time_Zones_in_ZCS#The_server_OS
Though I'm still investigating why this is happening for our customers, the root issues seem to be resolved by the following. Some of the auth errors will be logged to /var/log/zimbra.log and /opt/zimbra/log/mailbox.log
Put the output of this command in a text file:
zmlocalconfig -s | grep password
These should match what's in /opt/zimbra/conf/localconfig.xml
Now, compare the passwords with what is in your restore. Put in your path of the restore specific directory.
vi /opt/zimbra/backup/sessions/YOUR_DIR/sys/localconfig.xml
Do a /password or the full variable in vi to see what the old passwords are. You'll need to adjust the following below. Remember to copy a backup.
cp /opt/zimbra/conf/localconfig.xml /opt/zimbra/conf/localconfig.xml.DR
- ldap_postfix_password
- vi /opt/zimbra/conf/localconfig.xml and put in the needed password
- ldap_amavis_password
- vi /opt/zimbra/conf/localconfig.xml and put in the needed password
- ldap_root_password
- http://wiki.zimbra.com/index.php?title=Resetting_LDAP_%26_MySQL_Passwords Note difference between 4.x and 5.x
- zimbra_ldap_password
- http://wiki.zimbra.com/index.php?title=Resetting_LDAP_%26_MySQL_Passwords Note difference between 4.x and 5.x
Hostname resolution issues and testing commands
Allot of failed installs are because administrators are skipping the steps to make sure resolution is working.
Check your /etc/hosts file. You should have the localhost similar to the one below. Your ZCS server name should be replaced with the zimbra.test.com and zimbra fields. The IP address should be the IP address bound to the network interface [ip addr or ifconfig -a to confirm].
127.0.0.1 localhost.localdomain localhost 192.168.0.1 zimbra.test.com zimbra
If your ZCS server is behind a firewall or is being NAT'd. Make sure that resolution for the hostname on the ZCS server returns the internal IP information rather than the external IP. To test:
hostname -f zimbra.test.com
or
domainname -f zimbra.test.com
As long as one of the above returns the full hostname, use the command for the following. The MX lookups depend on how you are doing your mail domain and server hostname.
host `hostname -f` zimbra.test.com has address 192.168.0.1 host -t MX `domainname -f` zimbra.test.com mail is handled by 10 zimbra.test.com. host -t MX `domainname -d` test.com mail is handled by 10 zimbra.test.com.
You should also have reverse records (PTR) as well. Replace 192.168.0.1 using your internal ip address, it should return something like:
host 192.168.0.1 1.0.168.192.in-addr.arpa domain name pointer zimbra.test.com
Please review wiki page on split-DNS :
http://wiki.zimbra.com/index.php?title=Split_DNS
and also the variable use of lmtp_host_lookup .
http://bugzilla.zimbra.com/show_bug.cgi?id=27988
What's my time and timezone?
I wrote up the steps to make sure the ZCS server is using the correct time and timezone here:
http://wiki.zimbra.com/index.php?title=Time_Zones_in_ZCS#The_server_OS
Disable LDAP Replica
This is a work in progress, please don't use for production servers at this time.
References:
http://wiki.zimbra.com/index.php?title=Promoting_Replica_to_LDAP_Master
http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
For each server that you want to change:
Stop the Zimbra services on the server, zmcontrol stop.
Check the existing ldap_url value.
zmlocalconfig | grep ldap_url
Update the ldap_url value. Remove the replica LDAP server URL, below assumes you only had one replica.
zmlocalconfig -e ldap_url=”ldap:// ”
If other replica's exist, then the list typed is like:
zmlocalconfig -e ldap_url=”ldap:// ldap:// ldap:// ”
The hosts are tried in the order listed. The master URL must always be included and is listed last.
Additional Steps for MTA hosts.
After updating the ldap_url, rerun /opt/zimbra/libexe/zmmtainit. This rewrites the Postfix configuration with the updated ldap_url.
To stop the ldap service running on the ldap replica server.
su - zimbra ldap stop
To now disable ldap from running on the old ldap replica. The - in front of zimbraServiceEnabled is [off], rather than a + for [on].
zmprov ms -- -zimbraServiceEnabled ldap
A Proxy On You!
Proxy References
From Install Guide:
- Configuring IMAP and POP Proxy Server
- Configuring ZCS HTTP Proxy (Beta 5.0.6)
- Configuring for Virtual Hosting
Notable Proxy Bugs
- "mailbox does not load with zimbraPublicServiceHostname and reverse proxy enabled"
- "WRONG_HOST error while viewing a public shared doc using http proxy"
- "In reverse-proxied mode, when redirecting with a 302, web client should set Location to proxy provided Host header"
- "ZCS Mailbox Server Proxy"
- "need to support multi-server (proxy) for mobile clients"
Public Service Hostname
variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema
Bugs about the url's for shares that use spaces (%20), the redirection drops the %20
http://bugzilla.zimbra.com/show_bug.cgi?id=27788
http://bugzilla.zimbra.com/show_bug.cgi?id=27477
Domain Rename Issues
We have a command to rename a "domain" and the resources within it - unfortunately it doesn't handle ALL of the different components for various reasons. Specifically Documents and Calendars [more down below].
Review this RFE that was done when they implemented the renameDomain command.
http://bugzilla.zimbra.com/show_bug.cgi?id=7655
Note comment 21 & 22 (confirming what QA'd). The syntax is
zmprov -l rd testold.com testnew.com
Problems that arise with Documents are explained here, comment #1 has workaround.
http://bugzilla.zimbra.com/show_bug.cgi?id=25873
Now, the very unfortunate part about calendars with a domainname change/move.
"Ability to change a username globally on all appointments"
http://bugzilla.zimbra.com/show_bug.cgi?id=26736
"Cant edit calendar entries after renameDomain"
http://bugzilla.zimbra.com/show_bug.cgi?id=27707
The root issue here's seems to be more about the calendar standards and practices with the use of the "Organizer" field and notifications. You'll see 3 "work arounds" in comment 2 of bug 26736.
Export/Importing of the calendar data is shown here:
http://wiki.zimbra.com/index.php?title=User_Migration
This part specifically :
http://wiki.zimbra.com/index.php?title=User_Migration#Copy_Calendar_From_One_Zimbra_User_to_Another
My Gal & LDAP Settings For A Domain
To see your setting, do the following - replacing with domainname with the domain in question.
zmprov gd domainname | egrep -i 'ldap|gal'
Zimlets
Location Of Zimlets
Zimlets should be already located on the zimbra server in one of these directories:
/opt/zimbra/zimlets/ /opt/zimbra/zimlets-admin-extra/ /opt/zimbra/zimlets-extra/ /opt/zimbra/zimlets-experimental/ /opt/zimbra/zimlets-network/
How To Deploy Zimlets
To deploy a zimlet, simply cd to the directory where the zimlet is located and issue this command:
/opt/zimbra/bin/zmzimletctl deploy <zimlet_name>
Archive & Discovery
References for A&D
The two main A&D references are :
http://wiki.zimbra.com/index.php?title=ZAD
http://www.zimbra.com/pdf/Zimbra%20Archiving%20and%20Discovery%20Release%20Notes.pdf
Multi-Server & New Mailstore A&D setup
See below for a very rough draft document I made for multi-server / new mailstore A&D setups.
http://wiki.zimbra.com/index.php?title=Ajcody-Notes-AD-mailstore
I also created a RFE for documentation on this.
http://bugzilla.zimbra.com/show_bug.cgi?id=25135
NO_SUCH_BLOB Errors
Actual No Such Blob Homepage
Please see Ajcody-Notes-No-Such-Blob
Other Resource Overviews
It would be good to review this wiki page before you start any "changes":
http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure
And great forum posting:
http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html
Check Permissions First
If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):
/opt/zimbra/libexec/zmfixperms --verbose --extended
Perl To Find MailboxID's That Have Problem
And down and dirty way to see what mailboxes have this error:
grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u
Mapping MailboxId to User Account
Remember, mailboxId is unique to the mailstore - it's not a global user variable that's unique.
Stolen from forum post:
http://www.zimbra.com/forums/administrators/16885-mailbox_id-email-address.html
su - zimbra mysql use zimbra; select comment from mailbox where id=257; And you'll get back: +----------------------+ | comment | +----------------------+ | user@domain.com | +----------------------+ 1 row in set (0.00 sec) If you wanted to see all the info for that account: select * from mailbox where id=257; And you'll get back: +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | id | group_id | account_id | index_volume_id | item_id_checkpoint | contact_count | size_checkpoint | change_checkpoint | tracking_sync | tracking_imap | last_backup_at | comment | last_soap_access | new_messages | idx_deferred_count | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | 257 | 71 | xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx | 2 | 257 | 0 | 1832 | 100 | 0 | 0 | NULL | user@domain.com | 1209652908 | 0 | 0 | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ 1 row in set (0.00 sec)
Zmblobchk for 5.0.6+ Systems
On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors and other inconsistencies.
zmblobchk - RFE's
zmblobchk currently just lists "issues" - for the most part. Please see the following RFE's, some will give details about resolutions based upon what zmblobchk reported.
- "blobchk: Add support for 'repair-mode' to the zmblobchk tool" Resolved for ZCS 6.0.6
- The initial RFE for the zmblobchk tool
- "tool to do consistency checks and repair for missing blob for ID x" Resolved for ZCS 5.0.6, "repair" part done in bug 27958.
Example Output On A System With No Issues
su - zimbra ls /opt/zimbra/zmblobchk.jar zmblobchk.jar java -jar zmblobchk.jar Retrieving volume information Retrieving mboxgroup list Spooling item list to /tmp/mailitems53650.lst Retrieving items from mboxgroup1 Retrieving items from mboxgroup2 Retrieving items from mboxgroup3 Retrieving items from mboxgroup4 Retrieving items from mboxgroup5 Retrieving items from mboxgroup6 Retrieving items from mboxgroup7 Retrieving items from mboxgroup8 Retrieving items from mboxgroup9 Retrieving items from mboxgroup10 / Processing BLOB store - /tmp/mailitems53650.lst: size 28141 Processed 329 items in 1223ms No inconsistencies found
Example Output On A System With Issues
Last login: Thu Oct 30 12:03:12 on ttys000 COLDMILE-LM-3:~ ajcody$ ssh root@mail3.internal.homeunix.com root@mail3.internal.homeunix.com's password: Last login: Sat Nov 1 14:25:50 2008 from 192.168.0.13 [root@mail3 ~]# su - zimbra [zimbra@mail3 ~]$ zmblobchk Retrieving volume information Retrieving mboxgroup list Spooling item list to /tmp/mailitems39690.lst Retrieving items from mboxgroup1 Retrieving items from mboxgroup2 Retrieving items from mboxgroup3 Retrieving items from mboxgroup4 Retrieving items from mboxgroup5 Retrieving items from mboxgroup6 Retrieving items from mboxgroup7 Retrieving items from mboxgroup8 Retrieving items from mboxgroup9 Retrieving items from mboxgroup10 Retrieving items from mboxgroup11 Retrieving items from mboxgroup12 Retrieving items from mboxgroup13 Retrieving items from mboxgroup14 Retrieving items from mboxgroup15 Retrieving items from mboxgroup16 Retrieving items from mboxgroup17 Retrieving items from mboxgroup18 Retrieving items from mboxgroup19 Retrieving items from mboxgroup20 - Processing BLOB store / /tmp/mailitems39690.lst: size 470853 Processed 5873 items in 18462ms 11844 inconsistencies found * MailboxGroup=1, mailbox=1, item=303: wrong volume, expected at /vol1/0/1/msg/0/303-2062.msg, found at /opt/zimbra/store/0/1/msg/0/303-2062.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=304: wrong volume, expected at /vol1/0/1/msg/0/304-2064.msg, found at /opt/zimbra/store/0/1/msg/0/304-2064.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=305: wrong volume, expected at /vol1/0/1/msg/0/305-2077.msg, found at /opt/zimbra/store/0/1/msg/0/305-2077.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=306: wrong volume, expected at /vol1/0/1/msg/0/306-2091.msg, found at /opt/zimbra/store/0/1/msg/0/306-2091.msg (move to correct volume) ## CUT OF 1000's OF SIMILAR LINES ## * MailboxGroup=1, mailbox=1, item=6480: file not found: /vol1/0/1/msg/1/6480-20100.msg (delete associated metadata) * MailboxGroup=3, mailbox=3, item=400: wrong volume, expected at /vol1/0/3/msg/0/400-11287.msg, found at /opt/zimbra/store/0/3/msg/0/400-11287.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=425: wrong volume, expected at /vol1/0/3/msg/0/425-12939.msg, found at /opt/zimbra/store/0/3/msg/0/425-12939.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=427: wrong volume, expected at /vol1/0/3/msg/0/427-12943.msg, found at /opt/zimbra/store/0/3/msg/0/427-12943.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=440: file not found: /vol1/0/3/msg/0/440-13400.msg (delete associated metadata) * /opt/zimbra/store/0/3/msg/0/363-6194.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/360-5000.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/350-3129.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/320-2272.msg: no associated metadata (delete blob) ## CUT OF 1000's OF SIMILAR LINES ## * /opt/zimbra/store/0/2/msg/0/304-6907.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/315-6918.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/312-6915.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/308-6911.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/303-6906.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/301-6904.msg: no associated metadata (delete blob) *** 11844 faults found Report saved to: /tmp/zmblobc39689.rpt [zimbra@mail3 ~]$
To see the reported output again:
zmblobchk -l /tmp/zmblobc39689.rpt
Error - Not Enough Memory To Run zmblobchk
If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.
# ZCS 5.0 and later $ zmlocalconfig -e mailboxd_java_heap_memory_percent=40
And zmmailboxdctl restart when it's appropriate.
BLOB Issue Script
I've commented out the two delete lines and also the "$msghash &= $fmask;" one.
If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store because it doesn't see them in the primary store. You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.
Comment from other support staff member on a case that used this.
Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.
#!/usr/bin/perl # On 5.0.6+ there's a script, /opt/zimbra/bin/zmblobchk , that you can use instead of this. # This script is compatible with Zimbra version 4.5.x and 5.x . # OK, there's 2 MAILBOX_*_BITS values in the VOLUME table. # Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result. # That's your mailbox hash. # Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result. That's your msgid hash. # I think. # <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`); my $ARGV = shift; chomp $basepath; my $mbmask = "1" x $mgbits; my $fmask = "1" x $fgbits; foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) { chomp; my $path = "$basepath/"; my ($id, $grid, $aid, $nm) = split; my $mbhash = $id >> $mbits; $mbhash &= $mbmask; $path .= $mbhash."/".$id."/msg/"; foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) { chomp $msgstuff; next if $msgstuff eq ""; my ($msgid, $modContent, $type) = split (' ',$msgstuff); my $msghash = $msgid >> $fbits; # $msghash &= $fmask; my $nm = $msgid; if ($modContent) {$nm .= "-$modContent";} my $npath = $path.$msghash."/".$nm.".msg"; if (-e $npath) {print $npath."\t\tOK\n";} else { print $npath."\t\tNOT OK\n"; # not ok, remove the entry from the database so it is not a nuisance print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; if ($type eq "11") { print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; } } # print $npath."\n"; } }
If this doesn't run, make sure perl is installed and in the correct path.
You run this as the zimbra user.
What I did was the following.
Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl
su - zimbra mkdir /tmp/BLOB-CHECKS for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done ls -la /tmp/BLOB-CHECKS
The ls will help identify the more troubled accounts. You'll end up with something like this:
[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/ total 16 drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 . drwxrwxrwt 8 root root 4096 Jul 10 23:46 .. -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt
And the output of one that shows some size:
$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt /opt/zimbra/store/0/3/msg/0/360-6070.msg NOT OK
Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."
One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.
To re-index the users mailbox [this can take start, cancel, status]:
zmprov rim user@domainname start
Running the script again shouldn't show "NOT OK" lines.
FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.
BLOB Script To Copy From Restore
You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:
-restoreToTime <arg> - Replay the redo logs until the time specified.
-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.
-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence
-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.
-rf - Restores to the full backup only, does not include any incremental backups since that backup
This is a walk through to test the proceedure.
$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- $ zmprov gmi restored-ajcody@mail3.internal.homeunix.com mailboxId: 12 quotaUsed: 251513 $ zmprov gmi ajcody@mail3.internal.homeunix.com mailboxId: 3 quotaUsed: 251512 $ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ rm /opt/zimbra/store/0/3/msg/0/257-25.msg rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y $ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg Log into main account and confirm I can see/use that mail.
The script to look at the blob-check.pl output and do the copies after a redirected restore is here:
#!/bin/bash #START OF VARIABLES# #CHANGEME to existing user account USER=ajcody@mail3.internal.homeunix.com #CHANGEME to the restored account name # zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com # To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt # blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME #CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default STOREDIR=/opt/zimbra/store #END VARIABLES# USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10` USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'` USERPATH=`echo $USERGID/\$USERUID` RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10` RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'` RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID` echo USERUID $USERUID echo USERGID $USERGID echo USERPATH $USERPATH echo RESTOREDUID $RESTOREDUID echo RESTOREDGID $RESTOREDGID echo RESTOREDPATH $RESTOREDPATH cd $STOREDIR echo "Run ECHO first to confirm copies look right and then uncomment COPY in script" for ROOTBLOBPATH in `grep "NOT OK" $FILE | cut -f7-9 -d/ | cut -f1` do # Comment out echo and uncomment copy after dry run echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH #cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH done
BLOB Script To Copy From Restore (HSM Issue)
One customer ran into an issue that the blob references were really expecting them in the HSM mailstore path.
Example:
Default mailstore for PROD : /opt/zimbra/mail/
HSM mailstore path : /opt/zimbra/mail0 which was another partition mount, compared to /opt/zimbra/mail
So, what this script did was used the output from the blob-check.pl script which has the missing blobs formatted for the PROD path rather than the HSM path. The blob-check.pl script doesn't handle HSM data. The script then finds the blobs in the full backup directory of the user and copies them into the HSM path that the WEBCLIENT is expecting. This whole situation with this customer might of been a fluke. So don't consider these steps as applying to your situation unless you do your research. This case took 9 days to resolve and figure out.
Notice, if you use this script below you really need to review it in details and adjust it for your situation. There's more to adjust beyond the base variables.
#!/bin/bash # Steps # Make sure directory path exists for the copy command # Get mailboxId - zmprov ga USER@DOMAIN | grep zimbraId: # mkdir /opt/zimbra/mail0/0/[mailboxId #]/msg/{1..15} or how what ever # you need. # Change USEREMAIL # Get path for BACKUPFILES and uncomment for use # Run find command with echo and confirm. Then rerun using the find with cp rather than echo # Put in user's email for USEREMAIL USEREMAIL="USER@DOMAIN" ZIMBRAID=`zmprov ga $USEREMAIL | grep zimbraId: | awk '{print $2}'` # See below # ROOTBACKUPPATH="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789" #The line below takes to long, see instructions underneath #BACKUPFILES=`find $ROOTBACKUPPATH -name *$ZIMBRAID*` # To run manually, which will increase speed of script if you then paste full path in for BACKUPFILES var # zmprov ga USER@DOMAIN | grep zimbraId: | awk '{print $2}' # Then place the out in this find command, leave the wildcards on each side of it # find /opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789 -name *OUTPUTHERE* # This should give you path like the exampe below, paste it between the quotes and uncomment the variable BACKUPFILES #EXAMPLE FOR BACKUPFILES="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789/accounts/a07/1b5/a071b5ad-b341-4de4-b6ee-4463f322873b" # BACKUPFILES="PASTE_IN_PATH_HER" for BLOBPATH in `blob-check.pl $USEREMAIL | grep "NOT OK" | awk '{print $1}' | sed 's/store/mail0/'` do MSGID=`echo $BLOBPATH | awk -F/ '{print $NF}'` # Run echo first to make sure copy path seems correct, ctrl-c to cancel. And then uncomment the other find command with the cp. Comment the echo one. find $BACKUPFILES -name *$MSGID* -exec echo {} $BLOBPATH \; #find $BACKUPFILES -name *$MSGID* -exec cp -uv {} $BLOBPATH \; done
The Really Bad Situation, Blob References Gone & Mail Still Missing
For now, this is all I have to offer.
Method One - zmlmtpinject
See forum post:
http://www.zimbra.com/forums/installation/12617-recover-data-mbox-folders.html#post64962
Method Two - Use IMAPSYNC
I haven't tested this and would just be guessing, but I believe IMAPSYNC could be used.
From the IMAPSYNC notes, "imapsync is the adequate tool because it reduces the amount of data transferred by not transferring a given message if it is already on both sides. Same headers, same message size and the transfer is done only once. All flags are preserved, unread will stay unread, read will stay read, deleted will stay deleted."
This would involve having imap setup on ZCS and then doing a redirected user restore (which would consume a license).
See the following on imapsync details:
- Ajcody-Server-Plan-Move-Migration-Upgrade-DR#Non-Zimbra_IMAP_Accounts_To_Zimbra
- Mail_Migration#using_imapsync_.28Recommended_Method.29
Imapsync During A Certain Time Frame
You might want to check out the imapsync options of --maxage and --minage
Proposed Steps
- Confirm these steps are necessary
- Install imapsync
- Do a redirected restore of the account
- This is an example format of the redirected restore:
zmrestore -ca -a prod-user@servername.com -pre restored- -rf -lb full-'date of full'
- The "new" account would be restored-userPROD@servername.com . The -rf flags should look only at the backup data in regards to the restore and not catch the "production" references to missing blobs and what not.
- This is an example format of the redirected restore:
- Confirm in webclient that msg's are there
- Do a IMAPSYNC
- Two ways, unsure what is the best method
- From PROD to NEW
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 prod-user --password1 test123 -user2 restored-prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
- From NEW to PROD
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 restored-prod-user --password1 test123 -user2 prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
- From PROD to NEW
- Two ways, unsure what is the best method
- Reindex restored-prod-user
- Rename prod-user to prod-user-old
zmprov ra prod-user@servername.com prod-user-old@servername.com
- Rename restored-prod-user to prod-user
zmprov ra restored-prod-user@servername.com prod-user@servername.com
- Do a full backup of each account
zmbackup -f - <servername.com> -a <newaccountname@servername.com>
Notes On - No Associated Metadata
Tentative Research At This Point - Haven't Proven Out Yet
This is for the errors you see from the blobchk like this:
"* /opt/zimbra/store/0/1303/msg/0/1059-1726.msg: no associated metadata (delete blob) "
You could use the following command to bring the blob back in and regenerate the metadata -- zmmailbox am along -d . The am flag details are : addMessage(am) [opts] {dest-folder-path} {filename-or-dir} [{filename-or-dir} ...] . But the -d flag states : -d/--date <arg> received date (msecs since epoch) . To get the epoch time in seconds using the same receive data, for example :
ls -l --time-style=+%s /opt/zimbra/store/0/1303/msg/0/1059-1726.msg
And then remember the -d requires milliseconds [a thousandth (1/1,000) of a second] so you'll need to convert it. Or, you could just use something like this below. This with the cut would work well for a for loop script against a number of msg files.
ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg | cut -d" " -f6 1272642001000 ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg -rw-r--r-- 1 zimbra zimbra 0 1272642001000
Another suggestion for scripting was:
date=$(stat -c +%Y /opt/zimbra/store/0/1303/msg/0/1059-1726.msg); echo $(($date*1000));
Mysql Queries To See What Is Shown For Missing Message ID
References
See also :
Getting User Variables we need to query MySQL
Please see Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL
Mysql Query For A Users Specific Blob-Data ID
Please see Useful Mysql Query Examples
Migration Issues
IMAPSYNC with admin login
Reference - http://wiki.zimbra.com/index.php?title=User_Migration
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \ --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \ --user2 yourZimbraAccount --authuser2 admin --password2 adminZimbraPassword --authmech2 LOGIN
I found this description in one of the imapsync files:
"You may authenticate as one user (typically an admin user), but be authorized as someone else, which means you don't need to know every user's personal password. Specify --authuser1 "adminuser" to enable this on host1. In this case, --authmech1 PLAIN will be used, but otherwise, --authmech1 CRAM-MD5 is the default. Same behavior with the --authuser2 option."
Mailing Lists And Mailman
If you are planning to use a distro list of over 60K members, you may be inclined to use a dedicated machine for this purpose. According to bug 19153, you have 2 options: "set up mailing list manager or change the value of virtual_alias_expansion_limit as a customization. We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients." Mailman integration is an option. There is a forum thread on how to integrate zcs and mailman. This would be the preferred fix to your issue, also noted in the bug report. We probably won't see Mailman integration in ZCS at least for another year or so, not until 6.0
Restricting Who Can Send To Mailing List
http://wiki.zimbra.com/index.php?title=RestrictPostfixRecipients
Problems Resolving Virtual Aliases For Members Of Large Distribution Lists
http://bugzilla.zimbra.com/show_bug.cgi?id=19153
Mailman Configuration
http://wiki.zimbra.com/index.php?title=ZCS_Mailman_Integration
http://bugzilla.zimbra.com/show_bug.cgi?id=8968
http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html