Ajcody-Notes: Difference between revisions
mNo edit summary |
mNo edit summary |
||
Line 10: | Line 10: | ||
{{:Ajcody-New-Customers-Users}} | {{:Ajcody-New-Customers-Users}} | ||
{{:Ajcody-General-Notes}} | |||
{{:Ajcody-Logging}} | |||
[[Category: Community Sandbox]] | [[Category: Community Sandbox]] |
Revision as of 03:44, 17 November 2008
This page holds a Table of Contents of all my pages but each main Subject is an actual link to another page. I'm hoping this gives the best of both worlds. Where there is one long page with all topics as well as pages that contain the details to a specific subject. I did this by the following wiki code.
Enjoy, Adam
Information For New Support Customers
Details On Response Times and What's Supported
Please review the Support Overview and Support Program, specifically the "Exclusions" and "Reasonable Use Policy" sections. You can also find a link to the Support Plan document when you log in to your Zimbra Support Portal account (Network Edition Customers). As of 12 August 2008, it includes the following statement:
Requests that are excluded from support include but are not limited to: performing initial installs of customer systems, performing migrations, performing upgrades of customer systems, performing configuration changes to add or remove servers or services, and testing a customer’s system. Zimbra Professional Services may be available to assist with these projects.
New Network Edition Customers
Your sales contact should already have made a support record for your company and added one or more user accounts for our primary support tool, the Support Portal. Please Contact Sales if you are unsure whether you and/or your company have been set up for access to the Support Portal. Additional user accounts can be associated with your support record if necessary by submitting a case to support with the following information:
- Full name of new contact
- Title of new contact
- Email address of new contact
- Best phone number to reach new contact (this will potentially be used in the event of an email outage)
Understanding The Support Process For Urgent Issues
In order to receive the most efficient and timely response, please follow this procedure.
- Create a new case via the Support Portal
- In the lower section of the navigation menu on the left, choose Case Menu > Submit A New Case
- When you need urgent help, please open a case through the Support Portal instead of sending email to support@zimbra.com. Cases submitted through email will not be assigned the appropriate urgency for an outage or otherwise urgent situation, and they will appear lower in priority in our incoming case queue. When submitting a case through the Support Portal you may assign the appropriate urgency level according to scope, impact, and urgency of issue.
- Please include as much detail as possible to allow us to effectively prepare prior to calling you. Having details of the situation allows us to reach out to any resources (people) we may feel are needed prior to calling you back.
- Include contact information in the new case record. Desk phone, cell phone, alternate email address, and YahooIM handle are all useful. This allows us to contact you for clarification as needed.
- Call the Support Phone Line
- If you get the voice mail - don't panic or hang up without leaving a voice mail.
- Give your name and company first
- Provide the number of the case that you created in the Support Portal
- Summarize the outage situation
- Mention your contact information
- If you get the voice mail - don't panic or hang up without leaving a voice mail.
- Continue to work on the issue yourself.
- Promptly update the case with all relevant information you discover
- Left-hand lower section, Case Menu > Case Management
- If you can't "email", use the Comments section with your case record to input the information.
- Promptly update the case with all relevant information you discover
- If you have not received a response from us, please try updating the case and calling into the Support Phone Line again
- Note that depending on the day and hour of your outage, time might be needed for the information to route to the on-call staff and for them to get into a position to respond.
How To Communicate Support Need Clearly
Over the years, open source luminary Eric Raymond has assembled a guide to getting help with technical issues online. Bear in mind that the guide has been written for those seeking free help, so the overall guide taken at face value is most applicable for posting to the Zimbra Forums. There is, however, plenty of useful guidance for those submitting support cases. The complete guide is available online as How To Ask Questions The Smart Way. Here are a few examples:
- Use meaningful, specific subject headers
- Write in clear, grammatical, correctly-spelled language
- Be precise and informative about your problem
- Describe your problem's symptoms in chronological order
Simon Tatham, author of the PuTTY software, wrote a similar guide for How to Report Bugs Effectively.
Support Tools
Case Management
Subscribe To RSS Feeds
Our sites have unrestricted (anonymous) access for RSS feeds. The new RSS feeds are:
- https://support.zimbra.com/customer/rss.xml
- https://support.zimbra.com/hspfront/rss.xml
- https://support.zimbra.com/downloads/rss.xml
Clicking on an article from the RSS feed will direct the user to the login page with an "Authorization Required" message.
Email Communication
When you send emails to support@zimbra.com, it will create a case for us and show in our new case tool. What is important to note though, is our tools first check to see if the subject line of the email has SF: Case ########: in it. If so, it redirects that email to update an existing case. If those values aren't in the subject line, it creates a new case. So, if you wanted to update an existing case - let's say case number 00000001 - you would have your subject line as "SF: Case 00000001: some text of the topic" .
New Cases Created By Email Are Always Sev-4
If you send your initial support request via email to support@zimbra.com it will automatically be set at a "Severity Level 4", which is the lowest priority. If you need your case set to a higher severity level, please create your case via the support portal page where you can set the severity level. At this time, you can not change a case severity level once it's made via the support portal case management tools.
Always Include support@zimbra.com In Case Email Threads
If emails for your case do not include the support@zimbra.com address, there's a good chance the support member your working with will not be updated about your response correctly.
Sales Force Case Comments
Sales Force [zimbra support portal case management tool] has the option to do "Case Comments". Please use this only when necessary - for example, your email isn't working. Currently, case comments from sales force don't work appropriately with our other tools because it doesn't create a email subject to match the "SF: Case ########:" pattern and it also doesn't change the status of the case to reflect "customer replied". The email subject line is important because most of us here at support use the Zimbra's email conversation view and also "tagging" of our case threads. The case comments from Sales Force doesn't work against this. And of course, the case not being updated as "customer replied" causes issues as it most likely leaves it in the state of "Awaiting Customer Response".
Zimbra Forums
The forums provide a place where the greater Zimbra community (server administrators, users, developers, commentators, etc) can work together on solving problems. There is a wide range of experience among the community, and a long history of issues available for searching.
Create A Forums Account
This is open to anyone. It is not tied to your Network Edition support id.
Zimbra Wiki
The wiki also centers around the community, providing a site where anyone with an account can contribute documentation, especially how-to guides, non-standard configurations, and topics related to but external to ZCS.
Create A Wiki Account
This is open to anyone. It is not tied to your Network Edition support id.
Why is this important?
- It allows you to participate in improving the wiki articles and content. Note that you need to be logged in to make edits.
- Create your own articles if you find something missing that might prove useful to others.
- Improve existing articles.
- Add to the "discussion" page for an article. This is a good place to add comments for those less inclined to be bold and modify the main article.
- You can propose questions and other things here as well.
- You can add articles to your "watchlist" in order to monitor your favorite articles for changes.
- In your wiki preferences you can specify whether to be notified by email when different events occur.
- You can create your own personal section on the wiki.
- Use the category of "community sandbox" at the bottom of them.
- [[Category: Community Sandbox]]
- If you decide to create sandbox articles, the wiki convention is to prefix the topic with your wiki username: Username-Topic
- For example, the original staging page for this article was written by Ajcody, so the sandbox article was named Ajcody-New-Customers-Users
- Use {{:Special:PrefixIndex/YourUsername}} to automatically generate a list of these articles in a wiki document, on your user profile page for example
- This wiki tag on Ajcody's profile page causes a list of all of his notes articles to be displayed: {{:Special:PrefixIndex/Ajcody}}
- Use the category of "community sandbox" at the bottom of them.
Remote Server Access
Conditions For Remote Access
In some cases, a Zimbra TSE may request access to the system in question. Direct access is not generally the preferred method of investigation on cases, and we typically request that the customer please perform the hands-on administration of your system, with which you would be far more familiar than Zimbra; in some cases however, direct access may be considered an option for investigating the situation.
Zimbra TSE's will need some way to communicate with someone on your IT staff when they are going to log in to your servers. Zimbra Support helps and facilitates customers with their problems but we are not the "administrators" of customers servers and can't make independent decisions on what to do or not on their servers. We need to have access to immediate forms of communication [phone, IM, etc.] with someone at the company with the authority to make those decisions.
Remote Access Starts With Webex
Support agents can setup Webex sessions with customers when the situation calls for it. The standard policy is to use Webex for various legal & other policy justifications. Support staff will give guidance to the customer over the Webex session for the particular tasks in question. Customer participation is required throughout the Webex session. Other forms of access, ssh for example, should occur along side the Webex session. Ssh would use the screen utility mentioned below to accomplish this.
Your Webex meeting request will most likely also include a telephone conference number and instructions on how to file transfer log files as necessary.
Secondary Remote Access Options
Support SSH Keys
- Note that it is not required that customers allow remote server access.
- Updated: Switched to rsa rather than dsa. [2010/02/23]
In the Zimbra Support Portal, a public key for ssh is available for situations when Zimbra support will log in to a customer server using ssh. You can allow remote connections for a user (typically zimbra or root) by logging in to the server as that user and following these steps.
Running the script command within the ssh/shell session is a good idea for both the support staff and customer. See the following script man page
1. Save the key file on the server as /opt/zimbra/.ssh/zsupport_rsa.key.pub
2. Add the key to the user's authorized_keys file; command #1 outputs the current list of keys; command #2 appends the support key
1% cat ~/.ssh/authorized_keys 2% cat /opt/zimbra/.ssh/zsupport_rsa.key.pub >> ~/.ssh/authorized_keys
- Note, the ~/ above will be setting up the ssh access for the same username that is running the commands. This is the "user" that you'll tell Zimbra is the login account for ssh to the server.
3. Verify that public key access is allowed by sshd
- In the sshd config file (typically /etc/ssh/sshd_config), the value of the PubkeyAuthentication keyword should not be no. It's OK if it is not listed or if it is commented out; the default is yes.
4. Confirm, if appropriate, that the user you setup for ssh access will be able to su and/or sudo to gain zimbra and/or root access.
5. You'll then need to communicate to support that the key is setup and provide the necessary information:
- username to login with
- password if required for ssh access
- the initial server [hostname/ipaddr] to ssh to and then how to access other servers inside if needed
- way to gain zimbra or root access [su - [username] or via sudo]
- It is up to you, how you would like to communicate the login information to support - email, IM, phone, etc..
The screen command line utility allows multiple users to attach to the same terminal session. To start a new screen session, run screen -m or screen -R. To attach to an existing screen session, run screen -x. Additional documentation for screen is available online.
Sun Microsystems offers a java application to allow shared access to a terminal session. More information is available at their site.
DimDim
This might interest some customers as well DimDim Web Conferencing . They have an OSS edition - DimDim OSS Edition
Installation and Migration Planning
Expectations
Support doesn't do migration for customers, nor do we do migration planning. We can help if you run into issues during migration. Are there gray areas involved in this? Yes, of course. Some guidelines to follow:
- Engage your sales contact about your needs.
- Sales has access to pre-sales engineers and tools that can help plan your Zimbra deployment and migrations.
- If your needs exceed the above, they can determine if our Professional Services group would be a good option for you.
- When you start to engage the Support Team
- Try to have specific questions or issues rather than large open-ended ones - those that imply support is doing the planning or migration for you.
- It would be helpful if you documented your "plan" to the best of your ability and to share that with support as issues or questions come up.
- This gives us more "concrete" information to work against.
- Be prepared to know and investigate the technical requirements for the "other" mail server. Support will offer what we can but there is no obligation for us to know all the in's and out's of another mail server product. The exception to this would be when we are working with our tools that we developed for migration. And even then, there is a certain expectation of knowledge we require of the administrator in regards to the other mail system.
- Remember, anyone can use the Zimbra Forums! Advise and help is free and open to everyone on the forums.
ZCS Upgrade And Install Specific Expectations
Zimbra support works on a break-fix model, we do not provide administrative services.
For upgrades and installs, we generally offer :
- Provide input on upgrade and install plans or the remaining questions that aren't addressed documentation resources listed below:
- Latest Releases are at:
- http://www.zimbra.com/downloads/ne-downloads.html
- See the the following for older releases:
- http://www.zimbra.com/downloads/ne-downloads.html
- Documentation Resources are at:
- http://www.zimbra.com/support/documentation/
- click on your specific version and you'll see the various documentation references.
- http://www.zimbra.com/support/documentation/
- Please see "System Requirements" under the "Documentation Resources" section for system requirements - server and client.
- The "Installation Resources" section addresses:
- Steps to install ZCS - see either the Single-Server Installation Guide or Multi-Server Installation Guide.
- Checks, prerequisites, and other items you might need to address prior to installing.
- The Release Notes & Upgrade Instructions addresses:
- Steps to upgrade your server/s.
- Checks, prerequisites, and other items you might need to address prior to upgrading.
- List of notable bugs and RFE's addressed with this release. It will have a history going back through other older versions also.
- After reviewing the above resources for your specific version, you should also login to the support portal for any other critical information that might not be listed in the Release Notes & Upgrade Instructions.
- Latest Releases are at:
- We can add a customer's intentions about upgrading or installing ZCS on our team calendar. This allows our staff during our 'after hours' to be aware of customers that are intending to upgrade/install ZCS and that they might be submitting sev1 support cases if they run into issues.
- If the customer runs into a problem during an upgrade/install, they should submit a support case [setting the right severity level] for the specific issue at hand. The resolution of the support case is specific to the issue that was encountered. The support case is closed and the customer continues to do upgrade/install or other remaining steps they have independently.
First
Please take time to review the Administrators Guide. Noting what features you might want or need for the installation. It would be wise to make notes while your skimming the guide of features your wanting. Depending on the choices you have, it might drastically effect your installation steps. Do you want HSM, Archive & Discovery, Proxies - what type, and so forth.
- Official Administration Guide
The Basics
Starting Official Source Guides
- The Main Page
- The Release Notes
- System Requirements
- Quick Start Installation Guide
Other Resources To Review
General Ways To Find More Specific Information:
- Search Zimbra Site
- http://www.zimbra.com/search/
- Notice that you can refine where your search looks, click on the "Advanced Search" link next to the search button.
- http://www.zimbra.com/search/
- Looking at "Categories" within the Wiki pages. This might be more helpful compared to a general search.
- My wiki notes are actually spread out among multiple pages but I also have them displayed into one page. This makes it easier to search the table of contents or the complete listing using your web browser (usually ctrl-f).
- Ajcody-Notes
- I had to separate my Server related notes from this table of contents, please search this one as well:
- Still confused by HSM, see the following:
- GAL - Global Address Book Items To Consider
- SSL Certificates
- Proxy Items
- What about Mobility Options?
Please Consider Your Backup Issues Before Deploying
Assuming you've consulted the other above resources as well.
More items to consider:
- Backup Plans
- General Backup / Restore Issues
Be Aware Of DR Processes
Multi-Server Installations
Assuming you read or skimmed the above.
Starting Official Source Guides
- Main Page For Them
- Multi-Server Installation Guide
- Migration Wizards, Connectors, and Clusters Guides
- http://www.zimbra.com/products/documentation_additional.html
- Cluster Guides
- Single Node Cluster Installation Guide
- Multi Node Cluster Installation Guide
- Cluster Guides
- http://www.zimbra.com/products/documentation_additional.html
- Archiving and Discovery
Other Resources To Review
- Understanding your hardware issues - work in progress still.
- Archiving and Discovery For Multi-servers
- Issues Around LDAP Master And Replica Install/Setups
Migration Issues
- My Migration Issue Notes
- Ajcody-Migration-Notes
- This will also reference other necessary documentation.
- Ajcody-Migration-Notes
- Understanding Mailing List options
- MTA/Postfix differences you might need to be aware of
This will pull all of them regardless of their "status".
Quick Tips
License issues (Dos/Unix New Lines)
Sometimes when customers get their license file on a Windows machine and then go about transferring it to the zimbra server they accidentally convert the license file to have windows new line characters that will not work when Zimbra goes to read the file. To see if this happen, to the following on the zimbra server:
vi file and then
:set list
Confirm you're not getting the ^M characters at the end. The ^M is the windows new line character, they'll need to be removed. There is a script or even rpm package you can get, usually called dos2unix, that will do this for you.
General Notes
Actual General Notes Homepage
Please see Ajcody-General-Notes
Zimbra Variables
Ways to check on different zimbra variables.
su - zimbra zmlocalconfig -i zmlocalconfig -d cd /opt/zimbra/conf/attrs more zimbra-attrs.xml cd /opt/zimbra/openldap/etc/openldap/schema/ more zimbra.schema
Made the following concerning zmlocalconfig's behavior:
- "RFE: zmlocalconfig sanity check and clean up of -i output"
Also, with newer versions of ZCS, you can get it via zmprov - for example:
#zmprov desc -a zimbraFreeBusyExchangeAuthScheme zimbraFreebusyExchangeAuthScheme auth scheme to use type : enum value : basic,form callback : immutable : false cardinality : single requiredIn : optionalIn : cos,domain,globalConfig,account flags : accountInherited,domainInherited defaults : min : max : id : 611 requiresRestart : since : 5.0.3 deprecatedSince :
Addition note, example, for those checking source : src/6.0.7/com/zimbra/cs/account/ProvUtil.java.html
Zimbra Contact Fields For CSV Import Mapping
Please see /opt/zimbra/conf/zimbra-contact-fields.xml
This will show you what "fields" Zimbra is looking for and mapping to in regards to importing from another applications contact csv files.
One easy way to do this is the following:
- From zimbra ajax client as a test user (or any old user) do an export Addressbook.
- This is under the preference tab > Address Book > Import / Export > Export - select Addressbook and click Export button
- Then open file file something like Excel or a Text Editor
- Your only interested in the Top line - that gives the default fields for a Zimbra Addressbook.
- You'll get something like this:
"company","companyPhone","email2","fileAs","firstName","homepageURL","imAddress3","lastName","otherIMLabel","workCity","workCountry","workEmail2","workEmail3","workEmail4","workEmail5","workPostalCode","workState","workStreet"
- Note, ordering [left to right] isn't important. What is important, is the data lines up under the right column header.
- With your other Mail system, export the addressbook to a csv file.
- Open this file in another Excel window.
- Copy the column data from the one Excel sheet and paste it into the appropriate column within the Zimbra Excel sheet.
- Move as much as you can and then save.
- For example:
"company","companyPhone","email2","fileAs","firstName","homepageURL","imAddress3","lastName","otherIMLabel","workCity","workCountry","workEmail2","workEmail3","workEmail4","workEmail5","workPostalCode","workState","workStreet" "Apple Computer Inc.","1-800-MY-APPLE","","3","","http://www.apple.com","","","","Cupertino","United States","","","","","95014","CA","1 Infinite Loop"
- Now you can try importing this csv file back into Zimbra.
- Either with the webclient or you can use some on the CLI commands
- See User_Migration for variations of CLI commands that could be used.
- Here's the documented example:
curl -u schemers:password --data-binary @/tmp/new.csv http://server/service/home/schemers/contacts?fmt=csv
Example Header For Horde Contacts
I have not used this myself, I found it on the forums.
"assistantPhone","birthday","callbackPhone","carPhone","company","companyPhone","email","email2","email3","firstName","fullName","homeCity","homeCountry","homeFax","homePhone","homePhone2","homePostalCode","homeState","homeStreet","homeURL","imAddress1","imAddress2","imAddress3","jobTitle","lastName","middleName","mobilePhone","notes","otherCity","otherCountry","otherFax","otherPhone","otherPostalCode","otherState","otherStreet","otherURL","pager","workCity","work"
I'm hoping the forum user simply renamed the top header column to match description fields that Zimbra knows and that it's in order of the data (left to right) that Horde normally exports in. You'll need to review it and make sure it fits with your exported data columns from Horde.
Admin With Curl And Wget
You will most likely need to include :7071 in the url string to get around the pop-up issue that you would get if you used the same string in a web browser, without :7071 you'll see a permission error when trying to see/get data from a users account via the admin account.
curl -u admin:pass https://hostname:7071/home/user/Contacts wget https://admin:pass@hostname:7071/home/user/Contacts
What's My Server Like
To dump out your server details, these commands are useful. Login to your zimbra server and as the zimbra user do:
zmprov gs `zmhostname` zmprov gacf zmlocalconfig zmdumpenv
Another good document to review for "odd performance" issues is the large server wiki.
http://wiki.zimbra.com/index.php?title=Performance_Tuning_Guidelines_for_Large_Deployments
Restarting Jetty - ZCS 5+
As zimbra
zmmailboxdctl restart
How Do I Find Out What Version Of Zimbra I'm Using
See:
Get Version From Ajax Client
In newer versions of ZWC, the Help > About will also state the ZCS server version being used.
For older version, put the below in the search field of the Ajax client and hit enter:
$set:get version
Also:
https://SERVERNAME/js/zimbraMail/share/model/ZmSettings.js
Search for the CLIENT_VERSION line.
Other Ways To Get Version Of Server
See this forum post:
What's Compiled With Postfix
Moved to What's_Complies_With_Zimbra's_Postfix
REST Information
Resources to review
- Rest Overview
- For queries:
- Working Examples
Format to get around permission denied on multi-domain hosting servers
If you getting permission denied rather than resource not found...try this format example:
http://MAILSTOREofUSER.DOMAIN.com/zimbra/user/FIRST.LAST@USERS_DOMAIN.com/inbox.zip
or
http://MAILSTOREofUSER.DOMAIN.com/zimbra/user/USERNAME@USERS_DOMAIN.com/inbox.zip
I but both in case one issue is where the username has special characters, like the period.
Server Source Doc's
Please see:
http://svn.sourceforge.net/viewvc/zimbra/trunk/ZimbraServer/docs/
Logging
Actual Logging Homepage
Please see Ajcody-Logging
Server
RFE's Related To Better Logging And Historical Data Of Systems
See : Ajcody-Testing-Debugging#RFE.27s_Related_To_Better_Logging_And_Historical_Data_Of_Systems
Debugging
See : Ajcody-Testing-Debugging for more complete debugging information.
When Was A ZCS Service Enabled Or Disabled
See : Ajcody-Notes-Archive-Discovery#When_Was_A_ZCS_Service_Enabled_Or_Disabled
Syslog Items
Single Server Setup
/etc/syslog.conf should have lines similiar to:
[towards bottom of conf file] local0.* -/var/log/zimbra.log auth.* -/var/log/zimbra.log mail.* -/var/log/zimbra.log
Make sure syslog allows messages from log4j to be written, log4j doesn't do unix pipes. Specifically, it uses internet domain sockets (514/upd) instead of unix domain sockets (/dev/log). On a single server setup, this means log4j talking to localhost via 514/udp. Again, /dev/log is never used by log4j (AFAIK, this could change in the future). Adding the "-r" applies for the centralized syslog server as usual, but it also applies for the case where you want log4j data to be logged via syslog locally [single server setup]:
[root@zimbra sysconfig]# diff -u /etc/sysconfig/syslog.ORIG /etc/sysconfig/syslog --- /etc/sysconfig/syslog.ORIG 2008-03-25 09:14:28.000000000 -0400 +++ /etc/sysconfig/syslog 2009-06-09 16:01:13.000000000 -0400 @@ -3,7 +3,7 @@ # -r enables logging from remote machines # -x disables DNS lookups on messages recieved with -r # See syslogd(8) for more details -SYSLOGD_OPTIONS="-m 0" +SYSLOGD_OPTIONS="-r -m 0" # Options to klogd # -2 prints all kernel oops messages twice; once for klogd to decode, and # once for processing with 'ksymoops'
Restart syslog:
/etc/init.d/syslog restart
Setup ZCS to be aware of syslog [we'll restart Zimbra later once all the changes are done]:
zmprov mcf zimbraLogToSyslog TRUE
This will set the appropriate ldap values (after a zimbra restart) to make log4j.properties be written correctly:
/opt/zimbra/conf/log4j.properties will now show some modifications, for example:
[snips - DON'T cut cut/paste this info below for your server] #log4j.rootLogger=INFO,LOGFILE log4j.rootLogger=INFO,LOGFILE,SYSLOG ... # Syslog appender log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.SyslogHost=localhost log4j.appender.SYSLOG.Facility=LOCAL0 log4j.appender.SYSLOG.layout=com.zimbra.common.util.ZimbraPatternLayout log4j.appender.SYSLOG.layout.ConversionPattern=mailboxd: %-5p [%t] [%z] %c{1} - %m
This will actually cause very little to goto syslog though, it's a very minimalist setup. Now adjust /opt/zimbra/conf/log4j.properties.in to log more items to syslog.
Note: You may or may not want all these things going to syslog and you may be able to send more/other things to syslog as well. I've not spend lots of time with log4j and I'm not familiar enough with all the code to know which pieces are using which logger/settings.
WARNING: the AUDIT logs may have sensitive data so be careful to protect the logs so that sensitive data is not leaked accidentally.
[zimbra@zimbra conf]$ diff -u log4j.properties.in.ORIG log4j.properties.in --- log4j.properties.in.ORIG 2009-06-05 15:31:20.000000000 -0400 +++ log4j.properties.in 2009-06-09 15:54:07.000000000 -0400 @@ -35,7 +35,7 @@ # Save zimbra.security to AUDIT appender log4j.additivity.zimbra.security=false -log4j.logger.zimbra.security=INFO,AUDIT +log4j.logger.zimbra.security=INFO,AUDIT,SYSLOG # Syslog appender log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender @@ -88,7 +88,7 @@ log4j.additivity.zimbra.wbxml=false log4j.logger.zimbra.wbxml=DEBUG,WBXML -log4j.logger.zimbra=INFO +log4j.logger.zimbra=INFO,SYSLOG log4j.logger.zimbra.op=WARN -log4j.logger.com.zimbra=INFO +log4j.logger.com.zimbra=INFO,SYSLOG
At this point, you can restart zimbra for all the changes to be in effect:
zmcontrol stop ; zmcontrol start
Centralized Syslog Server
Please see:
This will show you how to setup a syslog server and then to configure the "clients" to log to the local filesystem as well as the syslog server.
You'll need to then take into account the information in Ajcody-Logging#Single_Server_Setup to handle the "Zimbra" part as well as dealing with log4j.
Centralized Log Server Project Ideas
Problems:
- Logs are to large to manage on each server.
- Want to build a central logging server for all logging events.
- The syslog events are very minimal with Zimbra (/var/log/zimbra.log). Most of the logging is control by Log4j (tomcat/jetty). Output from log4j generally goes into the /opt/zimbra/log/ directory. There's some conf files in /opt/zimbra/conf/log4j.properties* for log4j. You can adjust this file to also output into the syslog environment but syslog has very limited choices for "facility". See Ajcody-Logging#Single_Server_Setup and Log4j & Syslog facility for details. You'll end up getting huge files and god knows how many log events per second if you adjusted log4j to dump all zimbra logging into syslog control.
General Thought On This:
Use syslog-ng with mysql for the central log server. Add on php-syslog-ng for comfort.
Leave the log4j stuff alone and configure syslog-ng to monitor the actual log files as they are generated by log4j in /opt/zimbra/log/* .
Then decrease the log sizes and archiving done on the local servers, i.e. zimbraLogRawLifetime & zimbraLogSummaryLifetime .
References:
- Syslog-ng Homepage
- "O'Reilly - Building Secure Servers With Linux - Chapter On Syslog-ng"
- http://eduunix.ccut.edu.cn/index2/html/linux/O'Reilly%20-%20Building.Secure%20Servers%20with%20Linux/bssrvrlnx-CHP-10-SECT-2.html
- Most notably, the section on "sources"
- Php-syslog-ng
- http://code.google.com/p/php-syslog-ng/
- How-To Setup and Install Syslog-ng, mysql, and php-syslog-ng
Log Rotation and Removal
Logs In /opt/zimbra/log Directory
Please look at the output of your crontab - AS ZIMBRA, look for the "Log pruning" line.
crontab -l
And also the config files for logrotate:
/etc/logrotate.conf
/etc/logrotate.d/zimbra
Example From 5.0.11 - Complete ZCS Install Single Server
crontab for zimbra:
# Log pruning # 30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1 35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1 # Log pruning # 30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
And the logrotate files:
$ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp -- we'll rotate them here /var/log/wtmp { monthly minsize 1M create 0664 root utmp rotate 1 } # system-specific logs may be also be configured here.
And also:
$ cat /etc/logrotate.d/zimbra /var/log/zimbra.log { daily missingok notifempty create 0644 zimbra zimbra postrotate killall -HUP syslogd 2> /dev/null || true kill -HUP `cat /opt/zimbra/log/swatch.pid 2> /dev/null` 2> /dev/null || true kill -HUP `cat /opt/zimbra/log/logswatch.pid 2> /dev/null` 2> /dev/null || true endscript compress } /opt/zimbra/log/myslow.log { daily missingok copytruncate rotate 30 notifempty create 0644 zimbra zimbra compress } /opt/zimbra/log/logger_myslow.log { daily missingok copytruncate notifempty create 0660 zimbra zimbra compress size 5000k rotate 7 } /opt/zimbra/log/clamd.log { daily missingok copytruncate notifempty create 0660 zimbra zimbra postrotate kill -HUP `cat /opt/zimbra/log/clamd.pid 2> /dev/null` 2> /dev/null || true endscript compress size 5000k rotate 7 } /opt/zimbra/log/zmlogswatch.out { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl stop" su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl start" endscript rotate 5 compress } /opt/zimbra/log/zmswatch.out { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmswatchctl stop" su - zimbra -c "/opt/zimbra/bin/zmswatchctl start" endscript rotate 5 compress } /opt/zimbra/log/zmmtaconfig.log { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmmtaconfigctl restart" endscript rotate 5 compress } /opt/zimbra/log/nginx.log { daily missingok notifempty create 0644 zimbra zimbra postrotate kill -USR1 `cat /opt/zimbra/log/nginx.pid 2> /dev/null` 2> /dev/null || true endscript rotate 7 compress } /opt/zimbra/log/zmconvertd.log { daily missingok copytruncate notifempty create 0644 zimbra zimbra rotate 7 } /opt/zimbra/zmstat/zmstat.out { daily missingok copytruncate rotate 7 notifempty compress }
Openldap Logs - In /opt/zimbra/openldap-data/
Ldap logs files in /opt/zimbra/openldap-data/ and in /opt/zimbra/data/ldap/* subdirectories [ZCS 6+] that are like log.0000000001 , log.0000000002 , etc.
This logpurge is controlled by:
$ cd /opt/zimbra/openldap/etc/openldap/ [zimbra@mail3 openldap]$ grep logpurge * master-accesslog-overlay.conf:logpurge 07+00:00 01+00:00
Logpurge Directive Description
From Accesslog Chapter
logpurge Directive
logpurge age interval
Defines both the maximum age for log entries to be retained in the database and how often to scan the database for old entries. Bothage and interval are specified as a time span in days, hours, minutes, and seconds. The time format is [ddd+]hh:mm[:ss], for example, the days and seconds components are optional but hours and minutes are required. Except for days, which can be up to 5 digits, each numeric field must be exactly two digits. Example:
- the log database will be scanned every day
- entries older than two days will be deleted.
logpurge 2+00:00 1+00:00
When using a log database that supports ordered indexing on generalizedTime attributes, specifying an eq index on the reqStart attribute will increase the performance of purge operations.
What's up with all the logs?
Other Logging Page Resources
http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.14.1.html
http://wiki.zimbra.com/index.php?title=Server_Monitoring
http://wiki.zimbra.com/index.php?title=Log_Files
http://wiki.zimbra.com/index.php?title=Unresponsive_Server_Troubleshooting
The Bread And Butter Logs
/opt/zimbra/log/mailbox.log - where most of your mailbox store activity is logged
- This log is the mailboxd log4j server log containing the logs from the mailbox server. This includes activity from the mailbox store, LMTP server, IMAP and POP servers, and Index server.
- Location: /opt/zimbra/log/mailbox.log
- This log is the mailboxd log4j server log containing the logs from the mailbox server. This includes activity from the mailbox store, LMTP server, IMAP and POP servers, and Index server.
/opt/zimbra/log/zmmailboxd.out - mailboxd/jvm output log
- Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
- Location: /opt/zimbra/log/zmmailboxd.out
- Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
/opt/zimbra/log/stacktrace.<pid> - stacktrace logs
- stacktraces related to mailboxd
- Location: /opt/zimbra/log/stacktrace.<pid>
- stacktraces related to mailboxd
/opt/zimbra/db/data/YOURHOSTNAME.err - errors for MySQL - ZCS 4.x & ZCS 5.x
- This is the message store database error log.
- Location: /opt/zimbra/db/data/YOURHOSTNAME.err
- This is the message store database error log.
/var/log/zimbra.log - mta and system status log, postfix, amavisd
- The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
- Location: /var/log/zimbra.log
- The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
- Probably has mta logging events [which also show in /var/log/zimbra.log] and also log events related to your OS
- Location: /var/log/messages
- Probably has mta logging events [which also show in /var/log/zimbra.log] and also log events related to your OS
/opt/zimbra/log/mysql_error.log - problems with MySQL
- If there is data corruption or another problem causing direct mysql errors, events will be logged here.
- /opt/zimbra/log/mysql_error.log
- If there is data corruption or another problem causing direct mysql errors, events will be logged here.
/opt/zimbra/log/myslow.log - slow db/MySQL queries
- If certain search requests are taking longer to complete than others, they will be logged here.
- /opt/zimbra/log/myslow.log
- If certain search requests are taking longer to complete than others, they will be logged here.
Other Logs
/opt/zimbra/log/audit.log - authentication events
- A log of all admin actions taken as well as logins to the server.
- /opt/zimbra/log/audit.log
- A log of all admin actions taken as well as logins to the server.
- Want to see all uses of the "View Mail" button from the admin console? Will have the IP address, User Agent of the browser, user that the admin is logged in as, and the account that they are viewing.
grep DelegateAuth /opt/zimbra/log/audit.log
/opt/zimbra/log/clamd.log - antivrius db
- Status and checks for clamav
- /opt/zimbra/log/clamd.log
- Status and checks for clamav
- If your clamav db is out of date, you'll see a log event here like this:
- LibClamAV Warning: **************************************************
- LibClamAV Warning: *** The virus database is older than 7 days! ***
- LibClamAV Warning: *** Please update it as soon as possible. ***
- LibClamAV Warning: **************************************************
/opt/zimbra/log/convertd.log - attachment conversion
- If your having problems with your attachments and the ability to view them in ZWC, check here for errors.
- /opt/zimbra/log/convertd.log
- If your having problems with your attachments and the ability to view them in ZWC, check here for errors.
/opt/zimbra/log/freshclam.log - clam antivirus updates
- Shows the actual attempt to update the clamav updates.
- /opt/zimbra/log/freshclam.log
- Shows the actual attempt to update the clamav updates.
/opt/zimbra/log/spamtrain.log - spam/ham training details
- Spam/Ham training details are logged here. Errors are also logged here related to this service.
- From a default single ZCS configurations crontab [zimbra]:
- # Spam training
- #:
- 0 :23 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
- #
- # Spam training cleanup
- #
- 45 23 * * * /opt/zimbra/bin/zmtrainsa --cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
/opt/zimbra/log/sync.log - zimbra mobile and activesync
- Your mobile devices will basically log here.
- /opt/zimbra/log/sync.log
- Your mobile devices will basically log here.
/opt/zimbra/log/synctrace.log - zimbra mobile and activesync
- Still looking for description that describes difference with sync.log
- /opt/zimbra/log/synctrace.log
- Still looking for description that describes difference with sync.log
/opt/zimbra/log/syncstate.log - no description found
- No description found.
- /opt/zimbra/log/syncstate.log
- No description found.
/opt/zimbra/log/wbxml.log - no description found
- No description found.
- /opt/zimbra/log/wbxml.log
- No description found.
/opt/zimbra/log/zmlogswatch.out -
/opt/zimbra/log/zmmtaconfig.log -
/opt/zimbra/log/zmmyinit.log -
/opt/zimbra/log/zmsetup.`date`-'pid'.log -
/tmp/install.log -
After install, gets moved to : /opt/zimbra/.install/
/tmp/zmsetup.log -
After install, gets moved to : /opt/zimbra/log/zmsetup.log.[some numbers]
/opt/zimbra/httpd_access.log -
/opt/zimbra/httpd_error.log -
New Data Directory In ZCS6+
In ZCS 6+ , there's a directory to organize 'data' from our various components.
[root@mail3 data]# pwd /opt/zimbra/data [root@mail3 data]# ls -F altermime/ amavisd/ clamav/ dspam/ ldap/ postfix/ tmp/
This information could prove useful as well with trouble-shooting and debugging problems.
Increase Logging
Increase Logging Per User
Depending on the ZCS version the below will clear all the per account loggers on zimbra reboot. IMAP is used for these examples.
- addAccoutLogger
zmprov aal user@domain.com zimbra.imap debug
- removeAccountLogger
zmprov ral user@domain.com zimbra.imap
Example usuage:
zmprov aal user@domain.com zimbra.soap debug
And tail the log file, while attempting to login to the AJAX client.:
tail -f /opt/zimbra/log/mailbox.log
See Ajcody-Logging#Log4J_Variables for variables that can be used.
To See What Accounts Have Extended Logging Enabled
There's the gaal option for zmprov . It list the accounts on a per mailstore basis :
getAllAccountLoggers(gaal) [-s/--server hostname]
[zimbra@zcs806 ~]$ zmprov gaal `zmhostname` # name test01@zcs806.domain.com zimbra.imap=debug
Bugs And RFE's
I filed the following RFE:
- "Allow "secondary" logging file for zmprov addAccountLogger"
Increase Logging Per Server
If you want to do it globally edit this file:
vi /opt/zimbra/conf/log4j.properties
Add a line at the end that's similar to something like this: log4j.logger.zimbra.imap=DEBUG.
No restart of any service is needed, BUT if something happens that causes a regeneration of the log4j.properties file then your changes will be overwritten. A regeneration of this file takes the contents of /opt/zimbra/conf/log4.properties.in .
To permanently make a change, modify the /opt/zimbra/conf/log4.properties.in file, then a restart would be necessary.
$ zmcontrol stop $ zmcontrol start
or you could run this on the server:
$ zmmtaconfig mailboxd
Wait for a minute for the server to pick up your change.
See Ajcody-Logging#Log4J_Variables for variables that can be used.
Log4J Variables
An Official page has been made for this topic now per my RFE request, bug 41894
Please see:
OpenLDAP Logging Levels
Please see King0770-Notes-Change-LDAP-Log-Levels
Want To See What's In Redolog Files
If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:
$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.
See Ajcody-Backup-Restore-Issues#Redolog_Files for more details.
Internal Zimbra Charting - zmstat-chart
RFE/Bugs You Might Be Interested In
- "support for real time counters (snmp, jmx, etc)"
- "zmstat - expand documentation"
- "zmstat-chart - graphing besides just one day"
- "zmstat - integrate webpage setup and allow configuration of chart run"
- "zmstat - should have some trending data on ALL client connections (web/soap)"
zmstat IS NOT Logger (graphs in admin console)
Please note, zmstat is different than the processes and graphs involved with Logger. Logger is behind the graphs that show up in the admin web console.
Please see Logger for details and trouble-shooting steps with Logger.
Rick has some other steps for King0770-Notes#Reinitialize_the_Logger_DB
Main References For zmstat-chart Command
Please see the following:
zmstat-chart CLI Administration Guide
zmstat-chart-config CLI Administration Guide
zmstatctl CLI Administration Guide
Monitoring Administration Guide
JFreeChart is the project/binary behind the charting tool. You'll find the jar files here [ZCS 5.0.9 example]:
- /opt/zimbra/lib/jars/jfreechart-1.0.1.jar
- /opt/zimbra/jetty-6.1.5/common/lib/jfreechart-1.0.1.jar
Issues Being Investigated With zmstat And Other "Charting" Items
I currently have an internal thread going with the dev's about some issues with zmstat and also with attempts of getting the active sessions data from the admin web console to be seen in trends.
I'll just be dumping my notes here and then cleaning them up once I'm done with the internal conversation.
- Question: (mailboxd: active connections by client protocol) What exactly is this tracking? I'm on a test box with no connections and it has the different threads in the 100's/1000's. Customer was hoping (as was I) that this would be giving data much like is shown in the admin console about user connections. Also, there doesn't seem to be a comparable "active connections" for the web client (soap)?
- Answer: For the connections issue, it's the number of clients that are currently connected. There's no analog for SOAP because SOAP doesn't have persistent connections. We may be able to log stats for number of SOAP sessions, but that would include noise for cases where the user closes the browser window and the session hasn't timed out yet. If you think this is useful, please file an RFE.
- Question: (graph time plot) If I run zmstat-chart with the --aggregate-start-at option with something like "08/07/2008 01:00:00" I notice that the graphs don't actually reference "days" at the bottom but still use what looks to be "hours". Is this a bug where the bottom reference of the graph doesn't get adjusted or is the syntax to run zmstat-chart to do this more tricky that I think.
- Answer: zmstat-chart doesn't support multi-day ranges, it really only supports a single directory with a single day of charts, the aggregate options are for specifying ranges at which to calculate the max(col)/min(col)/avg(col) values. Our usual practice is to generate one set of charts for each day we want to look at; I think it gives us a better look at when peaks occur and how they compare day-to-day (it's much easier to look at them in an overlay fashion).
- Answer: Seems like a reasonable request to be able to generate charts for multiple days. Please file an RFE if you think this would be useful.
- Question: How reliable are these numbers are in admin console....
- Answer: I imagine they should be quite reliable; but HTTP (Web) is stateless, so you can't really determine whether a session is really active or not. I believe multiple sessions appearing for a single user would be the result of reloading or closing and then reopening the browser to log in again.
- Perhaps it is best to ask the users in question what their usage is like? Is there any concern over the high number of sessions? I don't think it should affect server performance.
- Answer: I imagine they should be quite reliable; but HTTP (Web) is stateless, so you can't really determine whether a session is really active or not. I believe multiple sessions appearing for a single user would be the result of reloading or closing and then reopening the browser to log in again.
- Question: Could you please explain the following:
- Mailboxd Mailbox Add Rate (Delivery Rate)
- Answer: The rate at which messages are being added to the mailbox server, e.g. 1 message being delivered per second is... 1 m/s
- Mailboxd Mailbox Add Latency (Delivery Speed)
- Answer: How long it takes on average to add a message to a mailbox (averaged over the period of a minute)
- Mailboxd Active Connections by Client Protocol- This one does NOT give us counts for web client. Host zcs2.mail.XXXX.xxx: Mailboxd: Active Connections by Client Protocol avg(IMAP) = 169.86 avg(IMAP SSL) = 0.00 avg(POP) = 0.21 avg(POP SSL) = 0.00
- Answer: It doesn't give counts for web client because the web client does not have persistent connections (being HTTP-based). SOAP requests (used by the Web client) use transient connections that often close after they are used; i.e. if you took the number of "SOAP" connections and compared it to the number of "active" sessions, it'd likely be something like 0-1% of the latter figure.
- Mailboxd Mailbox Get Count
- Answer: The number of mailboxes that got opened in the last minute (I don't recall if this counting cache hits or not?)
- Mailboxd Mailbox Get Latency
- Answer: How long it takes to open a mailbox, on average (over a minute)
- Also, we don't get stats for any of the soap parameters:
- SOAP Invocation Count Summary (Top 10 max)
- SOAP Average Call Duration AuthRequest
- Answer: I don't know why this could be; perhaps zmstat-chart-config.xml needs to be regenerated, is /opt/zimbra/zmstat/soap.csv available and does it contain data?
- Mailboxd Mailbox Add Rate (Delivery Rate)
Running zmstat-chart
Confirm it's running:
zmprov gs [mailserver hostname] | grep -i stats zimbraServiceEnabled: stats zimbraServiceinstalled: stats
Default location of stat files is:
/opt/zimbra/zmstat/
Example command to run:
mkdir /tmp/stats zmstat-chart -s /opt/zimbra/zmstat -d /tmp/stats/
Like To Have zmstat-chart Data Integrated With Zimbra
I've made an RFE for this integration:
- "zmstat - integrate webpage setup and allow configuration of chart run"
With Zimbra Jetty
- Note:
- The default Jetty behavior will NOT display directory listing. You'll need a valid html type file in the directory target.
- I would recommend the apache solution over this one as you can avoid hitting performance issues this might cause.
Dump some graphing directories in there:
zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/jetty/webapps/zimbra/downloads/zmstat-chart/`date +%F-%H-%M`
You'll see there's the directory and when you go into it - there's your charts.
You can now place this command in your crontab to run on a schedule basis.
Things you'll want to check before deploying:
- Place charting directory in a location where space is available and will not create a risk to the mail services if it's full.
- Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
- Make sure your following and security guidelines for your environment.
You'll be able to view the data with the following url:
http://hostname/zimbra/downloads/zmstat-chart/
With Zimbra Apache Or Non-Zimbra Server With Apache
Before You Proceed
Things to check or confirm first before deploying:
- Place your directory root [/opt/zimbra/support in this example] in a location where space is available and will not create a risk to the mail services if it's full.
- If you'll be using NFS to have a centralized storage point:
- Your NFS mount point on the Zimbra servers would be /opt/zimbra/support in this example.
- You'll be exporting /opt/zimbra/support in this example from the NFS server.
- If this is a non-zimbra server, copy the zimbra entry from a zimbra server's /etc/passwd and /etc/group into the non-zimbra servers passwd and group file. This way, if NFS is also used the UID/GID matches.
- Apache Directory variable will be /opt/zimbra/support/data in this example.
- Apache .htpasswd location will be /opt/zimbra/support/.htpasswd in this example, putting it above of the apache Directory variable.
- Make sure your following the security guidelines for your environment.
- Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
Directory Layout
As root:
mdir -p /opt/zimbra/support/data cd /opt/zimbra/support/data for i in `/opt/zimbra/bin/zmprov gas` do mkdir $i done
If your not on a ZCS server, you'll be manually creating directories named after the output of zmhostname from each of your ZCS servers.
Now, create subdirectory paths under each zmhsotname directory.
cd /opt/zimbra/support/data/ for i in `ls` do mkdir $i/debug $i/logs $i/zmstat $i/tops done
Set permissions on the directories we made:
chown -R zimbra:zimbra /opt/zimbra/support chmod -R 755 /opt/zimbra/support
With this structure and with NFS setup, you'll have a common repository that will have a standard path to use on a per server basis. Example :
script-command -PathOption /opt/zimbra/support/data/`zmhostname`/TYPE_OF_DATA/`date +%F-%H-%M`
Apache Configuration
Let's configure apache to show this directory.
- For ZCS Apache Setup:
vi /opt/zimbra/conf/httpd.conf ** Towards the end, add the following** # Include zmstat-chart directory Include /opt/zimbra/httpd/conf/extra/zimbra-support.conf
- Non-ZCS Server With Apache. Check that there's an existing Include directive that will see our zimbra-support.conf:
vi /etc/httpd/conf/httpd.conf ** Look for something that will allow the following** ** /etc/httpd/conf.d/zimbra-support.conf to be used** ** Your distro might differ on apache paths as well** ** as Include statement for other conf files** Include conf.d/*.conf
Let's now make the apache conf file for zimbra-support.conf
- For ZCS Apache Setup:
vi /opt/zimbra/httpd/conf/extra/zimbra-support.conf Alias /support "/opt/zimbra/support/data" <Directory "/opt/zimbra/support/data"> AuthName "Secure Area For Zimbra Support" AuthType Basic AuthUserFile /opt/zimbra/support/.htpasswd require valid-user Options Indexes IndexOptions FancyIndexing VersionSort AllowOverride None Order allow,deny Allow from all </Directory>
- Non-ZCS Server With Apache:
vi /etc/httpd/conf.d/zimbra-support.conf Alias /support "/opt/zimbra/support/data" <Directory "/opt/zimbra/support/data"> AuthName "Secure Area For Zimbra Support" AuthType Basic AuthUserFile /opt/zimbra/support/.htpasswd require valid-user Options Indexes IndexOptions FancyIndexing VersionSort AllowOverride None Order allow,deny Allow from all </Directory>
Make sure permissions are set correctly on zimbra-support.conf so apache will use it.
Now let's setup a username and password to use with the .htpasswd file.
cd /opt/zimbra/support/ htpasswd -c .htpasswd zimbrasupport chmod 644 .htpasswd
Restart apache so the changes take effect:
- For ZCS Apache Setup:
zmapachectl stop zmapachectl start
- Non-ZCS Server With Apache:
httpd restart
Now, let's dump some graphing directories in there:
zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/support/data/`zmhostname`/zmstat/`date +%F-%H-%M`
To now view the charts. Adjust, SERVERHOSTNAME, for your servername.
- For ZCS Apache Setup:
http://servername:7780/support/data/SERVERHOSTNAME/zmstat
- Non-ZCS Server With Apache:
http://servername/support/data/SERVERHOSTNAME/zmstat
You'll see there's the directory and when you go into it - there's your charts.
You can now place this command in your crontab to run on a schedule basis.
Tweaking zmstat-chart
zmstat-chart also has the option to use a configuration file. The default configuration file is:
/opt/zimbra/conf/zmstat-chart.xml
Copy this to another location and edit away. Then add the -c /location/to/config/file to use your new configuration.
Default Items Charted
Here's a list of the default items that will be charted with zmstat-chart:
- Postfix Queue Size
- convertd CPU time used
- convertd Resident Memory
- convertd Processes and Threads
- Total CPU
- Process CPU
- Process Total Memory
- Process Resident Memory
- Virtual Memory
- Context Switches
- Run/Blocked Process Queue Size
- Disk Partition Throughput
- Disk Utilization
- Disk Throughput
- Disk IOPs
- Swap Activity
- Total file descriptors open
- SOAP Invocation Count Summary (Top 10 max)
- SOAP Average Call Duration Summary (Top 10 avg)
- SOAP Invocation Count GetServerRequest
- SOAP Invocation Count BackupRequest
- SOAP Invocation Count GetAllConfigRequest
- SOAP Invocation Count GetDomainRequest
- SOAP Invocation Count GetAllDomainsRequest
- SOAP Invocation Count AuthRequest
- SOAP Average Call Duration GetServerRequest
- SOAP Average Call Duration BackupRequest
- SOAP Average Call Duration GetAllConfigRequest
- SOAP Average Call Duration GetDomainRequest
- SOAP Average Call Duration GetAllDomainsRequest
- SOAP Average Call Duration AuthRequest
- MySQL Database Connections
- MySQL InnoDB Buffer Pool Pages
- MySQL InnoDB Buffer Pool Hit Rate
- MySQL Tables Open/Opened
- MySQL Total Slow Queries Count
- Mailboxd Connection Pool Get Latency
- Mailboxd Dirty Lucene Index Writers
- Mailboxd Lucene IndexWriterCache Hitrate
- Mailboxd Lucene IO
- Mailboxd LMTP Delivery Throughput
- Mailboxd LMTP Delivery Rate
- Mailboxd Mailbox Add Rate (Delivery Rate)
- Mailboxd Mailbox Add Latency (Delivery Speed)
- Mailboxd Request Rate by Client Protocol
- Mailboxd Response Time by Client Protocol
- Mailboxd Active Connections by Client Protocol
- Mailboxd Mailbox Get Count
- Mailboxd Mailbox Get Latency
- Mailboxd Mailbox Cache Hit Rate
- Mailboxd Mailbox Item/Blob Cache Hit Rate
- Mailboxd Garbage Collection Time
- Mailboxd Garbage Collection Count
- Mailboxd JVM Heap Used
- Mailboxd JVM Heap Free
- Mailboxd JVM Permanent Generation and Code Cache
Moved to Getting_All_Users_Quota_Data
Moved to Getting_All_Users_Quota_Data
Third Party Tools (Monitoring & Logging)
FYI - Support For Real Time Counters (snmp,etc.)
This is for those that need to go beyond what zmstat-chart is doing or need to intergrate within a third party monitoring systems.
Please see RFE:
- "Support for real time counters (snmp, jmx, etc)"
Currently, this RFE has only resolved support for JMX, not SNMP. Customers who need SNMP can use a product like jManage to do the translation from JMX to SNMP.
SNMP And Zimbra
SNMP Related Bugs And RFEs
Please read the following below as they'll have little bits of information that you might need for your customizations / external setup for snmp use against Zimbra. :
- "support for real time counters (snmp, jmx, etc)"
- "zimbra snmp is broken"
- "snmp disk checks report incorrect usage."
- "replace zmdisklog / zimbra-snmp integration"
SNMP Setup On Zimbra To Notify A Remote Host
Our SNMP support is pretty basic; currently, we only send traps when a service (mta, mailbox, ldap) changes state (stop/start).
First, you'll need net-snmp package install on the zimbra host for the notifications to be sent to a remote host.
- When you install the net-snmp package, it creates /etc/snmp/snmpd.conf. This file needs to be edited with the correct community string to allow for snmp mibs to be read from remote machines The default value is "public" with the net-snmp package.
- /opt/zimbra/conf/swatchrc.in is set to use [ perlcode 0 my $snmpargs="-v 2c -c zimbra localhost "; ] for the community string. Adjust this if needed for your corporate snmp environment.
- Your security policy for your company might require you to review the other options there as well - i.e. limit to read-only.
- Also, if there is a firewall between the zimbra server and the snmp server host you'll need to open up port 161/UDP.
- When you install the net-snmp package, it creates /etc/snmp/snmpd.conf. This file needs to be edited with the correct community string to allow for snmp mibs to be read from remote machines The default value is "public" with the net-snmp package.
Then your zimbra host with the zimbra snmp service installed, do the following for a basic default setup:
zmlocalconfig -e snmp_notify=1 zmlocalconfig -e snmp_trap_host=your.host.name
/opt/zimbra/libexec/zmsnmpinit
zmswatchctl stop zmswatchctl start
We watch for something matching /err: Service status change/ and send the trap with:
/opt/zimbra/snmp/bin/snmptrap
See other topics below for customizations that might be needed for your snmp environment.
Other reference for Zimbra and SNMP:
Files To Review For SNMP
Look at the contents of the following files:
- /opt/zimbra/libexec/zmsnmpinit
- /opt/zimbra/conf/swatchrc.in
- zmsnmpinit reads swatchrc.in and writes out the file swatchrc for the running configuration
- /opt/zimbra/conf/swatchrc
- /opt/zimbra/snmp/share/snmp/snmpd.conf.in which is the SOURCE file to
- /opt/zimbra/conf/snmpd.conf [see /opt/zimbra/libexec/zmsnmpinit ]
- The two above are used by zmsnmpinit to generate the /opt/zimbra/conf/swatchrc
- /opt/zimbra/conf/snmpd.conf [see /opt/zimbra/libexec/zmsnmpinit ]
- /opt/zimbra/conf/swatchrc.in
- /opt/zimbra/net-snmp/share/snmp/mibs/zimbra.mib
- /opt/zimbra/net-snmp/share/snmp/mibs/zimbra_traps.mib
- /opt/zimbra/log/zmswatch.out
- Monitor this to see the services when they go up and down successfully send out the email notification for your snmp configuration
Zimbra MIBS
You'll find zimbra.mib and zimbra_traps.mib in the following directory. This listing is mine under 5.0.19 :
[root@mail3 ~]# cd /opt/zimbra/net-snmp/share/snmp/mibs/ [root@mail3 mibs]# ls AGENTX-MIB.txt IPV6-TC.txt SNMP-USER-BASED-SM-MIB.txt DISMAN-EVENT-MIB.txt IPV6-UDP-MIB.txt SNMP-USM-AES-MIB.txt DISMAN-SCHEDULE-MIB.txt NET-SNMP-AGENT-MIB.txt SNMP-USM-DH-OBJECTS-MIB.txt DISMAN-SCRIPT-MIB.txt NET-SNMP-EXAMPLES-MIB.txt SNMPv2-CONF.txt EtherLike-MIB.txt NET-SNMP-EXTEND-MIB.txt SNMPv2-MIB.txt HCNUM-TC.txt NET-SNMP-MIB.txt SNMPv2-SMI.txt HOST-RESOURCES-MIB.txt NET-SNMP-TC.txt SNMPv2-TC.txt HOST-RESOURCES-TYPES.txt NET-SNMP-VACM-MIB.txt SNMPv2-TM.txt IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt NOTIFICATION-LOG-MIB.txt SNMP-VIEW-BASED-ACM-MIB.txt IANAifType-MIB.txt RFC1155-SMI.txt TCP-MIB.txt IANA-LANGUAGE-MIB.txt RFC1213-MIB.txt TRANSPORT-ADDRESS-MIB.txt IANA-RTPROTO-MIB.txt RFC-1215.txt UCD-DEMO-MIB.txt IF-INVERTED-STACK-MIB.txt RMON-MIB.txt UCD-DISKIO-MIB.txt IF-MIB.txt SMUX-MIB.txt UCD-DLMOD-MIB.txt INET-ADDRESS-MIB.txt SNMP-COMMUNITY-MIB.txt UCD-IPFWACC-MIB.txt IP-FORWARD-MIB.txt SNMP-FRAMEWORK-MIB.txt UCD-SNMP-MIB.txt IP-MIB.txt SNMP-MPD-MIB.txt UDP-MIB.txt IPV6-ICMP-MIB.txt SNMP-NOTIFICATION-MIB.txt zimbra.mib IPV6-MIB.txt SNMP-PROXY-MIB.txt zimbra_traps.mib IPV6-TCP-MIB.txt SNMP-TARGET-MIB.txt
What Is Looked For
Take a look at your /opt/zimbra/conf/swatchrc - this is mine under 5.0.19
perlcode 0 my %notifications=(); perlcode 0 $notifications{smtp}="yes"; perlcode 0 $notifications{snmp}="yes"; perlcode 0 my $fr='admin@mail3.zimbra.REMOVED.com'; perlcode 0 my $pwc='admin@mail3.zimbra.REMOVED.com'; perlcode 0 my $snmpargs="-v 2c -c zimbra localhost ''"; perlcode 0 my $snmptrap="/opt/zimbra/snmp/bin/snmptrap $snmpargs"; perlcode 0 my $snmpsvctrap="ZIMBRA-TRAP-MIB::zmServiceStatusTrap"; perlcode 0 my $snmpsvcname="ZIMBRA-MIB::zmServiceName"; perlcode 0 my $snmpsvcstatus="ZIMBRA-MIB::zmServiceStatus"; perlcode 0 my %statuses=('started'=>1,'stopped'=>0); perlcode 0 my $hostname="mail3.zimbra.homeunix.com"; perlcode 0 sub donotify { my %args = (@_); if ($args{HOST} eq "localhost") {$args{HOST}=$hostname;}; if ($notifications{smtp}) { dosmtp(%args) if $args{SERVICE}; dodisksmtp(%args) if $args{DISK};}; if ($notifications{snmp}) {dosnmp(%args);}; } perlcode 0 sub dosmtp { my %args = (@_); print "SMTP notification: $args{MESSAGE}\n"; open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t"); print FOO "To: $pwc\nFrom: $fr\nSubject: Service $args{SERVICE} $args{STATUS} on $args{HOST}\n\n$args{MESSAGE}\n"; close FOO; } perlcode 0 sub dodisksmtp { my %args = (@_); print "SMTP notification: $args{MESSAGE}\n"; open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t"); print FOO "To: $pwc\nFrom: $fr\nSubject: Disk $args{DISK} at $args{UTIL}\% on $args{HOST}\n\n$args{MESSAGE}\n"; close FOO; } perlcode 0 sub dosnmp { my %args = (@_); print "SNMP notification: $args{MESSAGE}\n"; `$snmptrap $snmpsvctrap $snmpsvcname s $args{SERVICE} $snmpsvcstatus i $statuses{$args{STATUS}}`; } ignore /DEBUG/ watchfor /err: Service status change: (\S+) (.*) changed from stopped to running/ donotify SERVICE=$2,STATUS=started,HOST=$1 watchfor /err: Service status change: (\S+) (.*) changed from running to stopped/ donotify SERVICE=$2,STATUS=stopped,HOST=$1 watchfor /err: Disk warning: (\S+) (\S+) at (\d+)/ donotify DISK=$2,UTIL=$3,HOST=$1 watchfor /crit: Disk warning: (\S+) (\S+) at (\d+)/ donotify DISK=$2,UTIL=$3,HOST=$1
Enhanced MIB Files For HP OpenView
I've created an RFE for this:
- "enchanced MIB and OID information to work with HP Openview"
Zimbra does not provide "enhanced" mib files" at this time. Thresholds can be set by the customer within their individual monitoring system. Zimbra is alerting on is service up or service down, see the other information above in Ajcody-Logging#SNMP_And_Zimbra .
Some Choices
Charting & Graphing The Data
Montoring Software
- Nagios
- Centreon - based upon Nagios
- Splunk
- You could also look at using Splunk and Nagios together.
- BigSister - take off of BigBrother
- Zenoss
- Cacti
- OpenNMS
- Munin
Nagios On Zimbra
This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.
Configure Nagios to run on single server Zimbra box - Centos 5.x
Configure yum with repo and install nagios
vi /etc/yum.repos.d/Dag.repo yum update yum install nagios nagios-plugins nagios-devel nagios-plugins-nrpe
Move nagios.conf http file into main zimbra directory.
cp /etc/httpd/conf.d/nagios.conf /opt/zimbra/httpd/conf/extra/
Setup nagios to run as zimbra
vi /etc/nagios/nagios.cfg **Change nagios user to zimbra** nagios_user=zimbra nagios_group=zimbra
Change ownership of directories from nagios to zimbra.
chown -R 500:500 /var/log/nagios/ chown -R 500:500 /etc/nagios/ chown -R 500:500 /usr/share/nagios/
Configure authentication within Nagios
vi /etc/nagios/nagios.cfg # AUTHENTICATION USAGE use_authentication=1 # SYSTEM/PROCESS INFORMATION ACCESS authorized_for_system_information=nagiosadmin # CONFIGURATION INFORMATION ACCESS authorized_for_configuration_information=nagiosadmin # SYSTEM/PROCESS COMMAND ACCESS authorized_for_system_commands=nagiosadmin # GLOBAL HOST/SERVICE VIEW ACCESS authorized_for_all_services=nagiosadmin authorized_for_all_hosts=nagiosadmin # GLOBAL HOST/SERVICE COMMAND ACCESS authorized_for_all_service_commands=nagiosadmin authorized_for_all_host_commands=nagiosadmin
Set up httpasswd's for the accounts for Nagios
htpasswd -c /etc/nagios/htpasswd.users nagiosadmin htpasswd /etc/nagios/htpasswd.users guest
Configure Zimbra's http/apache to use nagios http config file
vi /opt/zimbra/conf/httpd.conf **Add the following towards bottom** # Include Nagios Include /opt/zimbra/httpd/conf/extra/nagios.conf
Starting nagios is done as root
/etc/init.d/nagios start
Restarting apache for nagios issues would be done with (as zimbra)
zmapachectl stop zmapachectl start
The webpage address to view Nagios will be like this:
http://IP_OF_SERVER:7780/nagios/
Use the rest of this how-to to configure it now: http://wiki.centos.org/HowTos/Nagios
MRTG - SNMP On Zimbra
This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.
Configure yum with repo and install mrtg, net-snmp, net-snmp-utils
vi /etc/yum.repos.d/Dag.repo yum update yum instal mrtg net-snmp net-snmp-utils
Follow some how-to on setting up the basics.
Create a http config:
vi /opt/zimbra/httpd/conf/extra/mrtg.conf Alias /mrtg "/opt/zimbra/mrtg" <Directory "/opt/zimbra/mrtg"> # SSLRequireSSL Options None AllowOverride None Order allow,deny Allow from all # Order deny,allow # Deny from all # Allow from 127.0.0.1 </Directory>
Add mrtg to http configuration within zimbra:
vi /opt/zimbra/conf/httpd.conf # Include Mrtg Include /opt/zimbra/httpd/conf/extra/mrtg.conf
Restart apache:
zmapachectl stop zmapachectl start
Create directory to hold mrtg data:
mkdir /opt/zimbra/mrtg
Address will be something like:
http://IP_OF_SERVER:7780/mrtg/index.html
Mailq Pointing To Right Binary
# ls -la /usr/bin/mailq lrwxrwxrwx 1 root root 27 Sep 3 17:00 /usr/bin/mailq -> /etc/alternatives/mta-mailq # ls -la /etc/alternatives/mta-mailq lrwxrwxrwx 1 root root 23 Apr 1 10:17 /etc/alternatives/mta-mailq -> /usr/bin/mailq.sendmail # rm /usr/bin/mailq # ln -s /opt/zimbra/postfix/sbin/mailq /usr/bin/mailq # mailq Mail queue is empty
Web Client Logging
Active Sessions
Please see Zmsoap#Active_Server_Sessions_With_DumpSessionsRequest
Debug (SOAP) via Browser
See http://wiki.zimbra.com/index.php?title=Web_Client_URL_Tricks&redirect=no
Admins To View Client Issues
Within the admin console, you can view users mail.
- Goto accounts and highlight the user having the problem.
- Click on the View Mail button above that frame.
- Then goto the url field of that new window and modify it to look like this [replace mailserver with yours]:
- Hit your return key to cause the browser to reload.
- If you get warning about pop-up, accept it.
- If the debug window doesn't show, just mouse in the url field and hit the return key again. It should now pop up.