Centralized Logs - Elasticsearch, Logstash and Kibana

Revision as of 11:22, 8 September 2017 by Jorge de la Cruz (talk | contribs)

Centralized Logs - Elasticsearch, Logstash and Kibana

   KB 21551        Last updated on 2017-09-8  




0.00
(0 votes)

The goal is install in a dedicated server or VM, all the components to have a Centralized Log Server, and also a powerfull Dashboard to configure all the reports.

Zimbra-kibana-logstash-Diagram.png

The Logstash, Elasticsearch and Kibana will be installed on this dedicated VM, in the Zimbra Server, or servers, will be installed the Agent.


Hardware and Software requisites

In the Server, or VM, we will install a fresh Ubuntu Server 14.04LTS. For the Hardware part, depends on how many Zimbra Servers, and how detailed are the Logs. For a regular environment, with the next resources is enough:

  • OS: Ubuntu 14.04 LTS
  • vRAM: 4GB
  • vCPU: 2
  • vDisk: 100GB (SAS 10K or even better 15K)

Install the Centralized Log Server

Installing Java

Elasticsearch and Logstash needs Java 7 to work, to install it, we need to add the PPA from Oracle to our apt:

root@logstashkibana01:/home/oper# sudo add-apt-repository -y ppa:webupd8team/java
gpg: keyring `/tmp/tmptjs1zwc5/secring.gpg' created
gpg: keyring `/tmp/tmptjs1zwc5/pubring.gpg' created
gpg: requesting key EEA14886 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmptjs1zwc5/trustdb.gpg: trustdb created
gpg: key EEA14886: public key "Launchpad VLC" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

Once the Oracle repository is added, is time to do an apt-get update to refresh the packages list:

root@logstashkibana01:/home/oper# apt-get update

Great! Now, install the last stable Java 7 version:

root@logstashkibana01:/home/oper# sudo apt-get -y install oracle-java7-installer

Installing Elasticsearch

To install Elasticsearch, we need to add the public GPG key into our apt:

root@logstashkibana01:/home/oper# wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

Now, we need to add the source tree with the next command:

root@logstashkibana01:/home/oper# echo 'deb http://packages.elasticsearch.org/elasticsearch/1.1/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list

And do an apt-get update to update the packages list:

root@logstashkibana01:/home/oper#apt-get update

With all this previous steps, now is time to finally install Elasticsearch:

root@logstashkibana01:/home/oper# sudo apt-get -y install elasticsearch=1.1.1

Once installed, we need to edit a few parameters to improve the security of our Environment:

root@logstashkibana01:/home/oper# sudo vi /etc/elasticsearch/elasticsearch.yml

At the end of the file, add the next line to disable the dynamic scripts:

script.disable_dynamic: true

Also, to disable the API calls, we need to edit the network.host line:

network.host: localhost

Once we tunned our Elasticsearch, is time to restar the service:

root@logstashkibana01:/home/oper# sudo service elasticsearch restart
* Starting Elasticsearch Server
...done.

To add the Elasticsearch service into the init, run the next command:

root@logstashkibana01:/home/oper# sudo update-rc.d elasticsearch defaults 95 10
Adding system startup for /etc/init.d/elasticsearch ...
/etc/rc0.d/K10elasticsearch -> ../init.d/elasticsearch
/etc/rc1.d/K10elasticsearch -> ../init.d/elasticsearch
/etc/rc6.d/K10elasticsearch -> ../init.d/elasticsearch
/etc/rc2.d/S95elasticsearch -> ../init.d/elasticsearch
/etc/rc3.d/S95elasticsearch -> ../init.d/elasticsearch
/etc/rc4.d/S95elasticsearch -> ../init.d/elasticsearch
/etc/rc5.d/S95elasticsearch -> ../init.d/elasticsearch

Installing Kibana

At the time of this Wiki, we will install Kibana 3.1.2, please go to the Official Kibana Webiste to use the last release.

Download the Kibana release with the next command:

root@logstashkibana01:/home/oper# wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz

Extract the Kibana package:

root@logstashkibana01:/home/oper# tar xvf kibana-3.1.2.tar.gz

Move into the Kibana directory and edit the Config File:

root@logstashkibana01:/home/oper# cd kibbana-3.1.2
root@logstashkibana01:/home/oper/kibana-3.1.2# vi kibana-3.1.2/config.js

Once inside the file, search for the line elasticsearch: and change the port number (default 9200) for the port number 80, later we will connect to the Kibana Server in a easy way, trought the 80 HTTP port:

elasticsearch: "http://"+window.location.hostname+":80",

Also, we will use nginx to serve our app, Kibana, so we will create first the folder in the /var/www directory:

root@logstashkibana01:/home/oper/kibana-3.1.2# sudo mkdir -p /var/www/kibana3

Now, copy all the Kibana folder inside the new path:

root@logstashkibana01:/home/oper# sudo cp -R ~/kibana-3.1.2/* /var/www/kibana3/

Like I said, we will use Nginx to serve our Kibana app.

Installing Nginx

We will install nginx from the official apt repositories:

root@logstashkibana01:/home/oper# sudo apt-get install nginx

Kibana and Elsaticsearch works in a particular way, the user needs to access to Elasticsearch directly, so we need to configure Nginx to redirect all the packets to the 9200 port to the 80 port. But no worries, Kibana have and example that we can use for this. .

We will download the Nginx configuration from the GitHub to our folder:

cd ~; wget https://gist.githubusercontent.com/thisismitch/2205786838a6a5d61f55/raw/f91e06198a7c455925f6e3099e3ea7c186d0b263/nginx.conf

Edit the Config file:

root@logstashkibana01:~# vi nginx.conf

Find the line called server_name and add our own FQDN, or localhost if we don't use any particular fqdn. Also we need to add the path to our Kibana installation:

server_name FQDN;
root /var/www/kibana3;

Save the file and copy it inside the nginx, to make it a default config file:

root@logstashkibana01:~# sudo cp nginx.conf /etc/nginx/sites-available/default

To allow other users to acces into the Kibana, we need to install the apache2-utils:

root@logstashkibana01:~# sudo apt-get install apache2-utils

Is time to create an username for Kibana, to save the dashboards.

root@logstashkibana01:~# sudo htpasswd -c /etc/nginx/conf.d/kibana.myhost.org.htpasswd <span style="color: #ff0000;">user</span> <span style="color: #ff0000;">admin</span>
New password:
Re-type new password:
Adding password for user admin

We are almost done, just restart the nginx service:

root@logstashkibana01:~# sudo service nginx restart
* Restarting nginx nginx
...done.

Installing Logstash

This is the last package that we will install on the Server or VM. Now is time to install Logstash. We will install it from the Elasticsearch repository, that we have from before, so just launch the next commands:

root@logstashkibana01:~# echo 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list
deb http://packages.elasticsearch.org/logstash/1.4/debian stable main

Run an apt-get update to refresh the packages list:

root@logstashkibana01:~# apt-get update

Run the next command to install Logstash:

root@logstashkibana01:~# sudo apt-get install logstash=1.4.2-1-2c0f5a1

Logstash is now installed, but we need to do this step before continue.

Generate the SSL Certificates to use in the server/client connection

We will use Logstash Forwarder in the Zimbra servers to send the logs to the Centralized Log Server. We want to do it in a secure way also. We need to generate an SSL and a key pair. The SSL will be used for the Client to verify the server identity.

First step is create the path while we will save the SSL and the private key:

root@logstashkibana01:~# sudo mkdir -p /etc/pki/tls/certs
root@logstashkibana01:~# sudo mkdir /etc/pki/tls/private

Generate the SSL and the private key:

root@logstashkibana01:~# cd /etc/pki/tls; sudo openssl req -x509 -batch -nodes -days 3650 -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

Configuring the Logstash Server

All the Logstash configiration files are based in JSON format, and the path where they are located is /etc/logstash/conf.d. The configuration is based in three sections; inputs, filters y outputs.

Let's create a Configuration file called 01-lumberjack-input.conf and then we will configure oir input "lumberjack":

root@logstashkibana01:/etc/pki/tls# sudo vi /etc/logstash/conf.d/01-lumberjack-input.conf

And fill the file with the next configuration:

input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

Save the file. With this step we specified an input "lumberjack" that is listening in the TCP port 5000 and also will use the SSL and the private key.

Now, is time create the file called 10-syslog.conf, and we will add the filter to our syslog messages:

root@logstashkibana01:/etc/pki/tls# sudo vi /etc/logstash/conf.d/10-syslog.conf

We will add the next content to the file to define our filter:

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

Save it. This filter will look into the logs, and will parse them using grok, to make them easier to understand it.

The last file that we need to edit, we will call it 30-lumberjack-output.conf:

root@logstashkibana01:/etc/pki/tls# sudo vi /etc/logstash/conf.d/30-lumberjack-output.conf

And the content needs to have the next:

output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}

Bascially, this output save the logs from Logstash inside Elasticsearch.

Restart the services:

root@logstashkibana01:/etc/pki/tls# sudo service logstash restart

We have our Server or VM 100% ready.

Configuring the Zimbra Servers

Now is time to configure the Zimbra Servers to send the Logs to our Centralized Log Server.

Next steps are for Ubuntu 14.04 LTS.

Copy the SSL certificate from the Logstash Server to Zimbra Servers

On the Logstash server, launch the next command to our Zimbra Server:

root@logstashkibana01:/etc/pki/tls# scp /etc/pki/tls/certs/logstash-forwarder.crtuser@server_private_IP:/tmp:

Installing the Logstash Forwarder Package

On the Zimbra Servers, we need to create the packages list for Logstash Forwarder:

root@zimbra-sn-u14-01:/home/oper# echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list

Once we've added the repository, install the Logstash Forwarder package:

root@zimbra-sn-u14-01:/home/oper# sudo apt-get update
root@zimbra-sn-u14-01:/home/oper# sudo apt-get install logstash-forwarder

Add the Logstash Forwarder to the boot sequence:

root@zimbra-sn-u14-01:/home/oper# cd /etc/init.d/; sudo wget https://raw.github.com/elasticsearch/logstash-forwarder/master/logstash-forwarder.init -O logstash-forwarder
root@zimbra-sn-u14-01:/home/oper# sudo chmod +x logstash-forwarder
root@zimbra-sn-u14-01:/home/oper# sudo update-rc.d logstash-forwarder defaults

Copy the SSL certificate to the proper path:

root@zimbra-sn-u14-01:/home/oper# sudo mkdir -p /etc/pki/tls/certs root@zimbra-sn-u14-01:/home/oper# sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

Configuring Logstash Forwarder

We are close to finish, inside the Zimbra Server, we need to think about what Logs we need to send to the Centralized Log Server.

Create a configuration file for Logstash Forwarder in JSON format:

root@zimbra-sn-u14-01:/home/oper# sudo vi /etc/logstash-forwarder

Now, we will fill the configuration file, change the IP for your own Centralized Log Server IP. Here in this example I will send to the Centralized Log Server the next logs: syslog, auth.log, mailbox.log, nginx.access.log, nginx.log, zimbra.log y mail.log, but you can add whatever log that you want:

{
"network": {
"servers": [ "IPDEVUESTROLOGSTASH:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/syslog",
"/var/log/auth.log",
"/opt/zimbra/log/mailbox.log",
"/opt/zimbra/log/nginx.access.log",
"/opt/zimbra/log/nginx.log",
"/var/log/zimbra.log",
"/var/log/mail.log"
],
"fields": { "type": "syslog" }
}
]
}

Save the configuration file.

Last step here is restart the Logstash Forwarder Service inside the Zimbra server:

root@zimbra-sn-u14-01:/home/oper#  sudo service logstash-forwarder restart

We need to repeat this steps in each Zimbra Server that we want to have the Logs centralized.

Connecting to Kibana

Now is time to play! and also play in HTML5. Open a Web browser and type your IP or FQDN from your Centralized Log Server. The first thing that we will see is an overview of Kibana, etc. We will select the option 11. Sample Dashboard.

Zimbra-logstashkibana-001.png

I really like it Kibana and also have a Centralized Log Server, but this is specially useful because we can search inside the Logs using checkbox, to filter and have the answer easier. Also we can mix the search and order for type of field, awesome!

Zimbra-logstashkibana-002.png

Also, we can play with the Dashboard as we want, share a Public URL with some Customers, or between the IT Department and other Departments, etc.

Zimbra-logstashkibana-003.png

This is a real overview, and I can see the total Logs received during a period of time.

Zimbra-logstashkibana-004.png

Here we can see an example of how we can see a Log file, perfectly parsed to consume the information in a easier, and human, way.

Zimbra-logstashkibana-005.png

That's it folks!

This Wiki is based on DigitalOcean ELK Tutorial

CentOS 6 Installation of ELK

YUM Configuration

Install the ElasticSearch Signing Key

rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch

Configure the repos

Create /etc/yum.repos.d/elasticsearch.repo (for the server)
/etc/yum.repos.d/elasticsearch.repo:
[elasticsearch-1.4]
name=Elasticsearch repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
Create /etc/yum.repos.d/logstash.repo (for the server)
/etc/yum.repos.d/logstash.repo
[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
Create /etc/yum.repos.d/logstash.repo (for the client)
/etc/yum.repos.d/logstash-forwarder.repo
[logstash-forwarder]
name=logstash repository packages
baseurl=http://packages.elasticsearch.org/logstashforwarder/centos/
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Installation

ElasticSearch

yum install elasticsearch
chkconfig --add elasticsearch

Logstash

yum install logstash
chkconfig --add logstash

Kibana

mkdir /data/kibana4    # locate in your data location of choice
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz
tar xvfz kibana-4.0.1-linux-x64.tar.gz
cd kibana-4.0.1-linux-x64
cp -R * /data/kibana4/

Kibana init.d start/stop script

Note: Kibana4 has an integrated webserver and does not require a separate webserver. The following script is for the integrated web server startup.

Create /etc/init.d/kibana
#!/bin/sh
#
# /etc/init.d/kibana -- startup script for kibana
# Wolfyxvf 2015-04-16; used httpd init script as template
#
### BEGIN INIT INFO
# Provides:          kibana
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts kibana
# Description:       Starts kibana using daemon
### END INIT INFO

#configure this with wherever you unpacked kibana:
KIBANA_BIN=/data/kibana4/bin
KIBANA_LOG="/var/log/kibana.log"

NAME=kibana
DESC="Kibana"
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/kibana
RETVAL=0

if [ `id -u` -ne 0 ]; then
        echo "You need root privileges to run this script"
        exit 1
fi

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/kibana ]; then
        . /etc/sysconfig/kibana
fi

start() {
        echo "Starting $DESC : "

        pid=`pidofproc -p $PID_FILE kibana`
        if [ -n "$pid" ] ; then
                echo "Already running."
                exit 0
        else
        # Start Daemon
                if [ ! -d "$PID_FOLDER" ] ; then
                        mkdir $PID_FOLDER
                fi

                daemon $DAEMON >> $KIBANA_LOG 2>&1 &
                sleep 2
                pidofproc node > $PID_FILE
                echo
                RETVAL=$?
                [ $RETVAL = 0 ] && touch $LOCK_FILE
                return $RETVAL
        fi
}

stop() {
        echo -n $"Stopping $DESC : "
        killproc -p $PID_FILE $DAEMON
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}


# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status -p $PID_FILE $DAEMON
        RETVAL=$?
        ;;
  restart)
        stop
        start
        ;;
  *)
        echo $"Usage: $prog {start|stop|restart|status}"
        RETVAL=2
esac

exit $RETVAL
make the init file executable
chmod +x /etc/init.d/kibana
Configure to start at boot
chkconfig --add kibana
Create Certs files for putting the Kibana Web interface in HTTPS
cd /etc/pki/tls;  openssl req -subj '/CN=FQDN_OF_KIBANA_SERVER/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/kibana.key -out certs/kibana.crt
Configure the kibana configuration file take our modification

/data/kibana4/config/kibana.yml

change to :
...
port 443
...
# SSL for outgoing requests from the Kibana Server (PEM formatted)
ssl_key_file: /etc/pki/tls/private/kibana.key
ssl_cert_file: /etc/pki/tls/certs/kibana.crt


Verified Against: Zimbra Collaboration Suite 8.6, 8.5 Date Created: 02/11/2015
Article ID: https://wiki.zimbra.com/index.php?title=Centralized_Logs_-_Elasticsearch,_Logstash_and_Kibana Date Modified: 2017-09-08



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search