Centralized Logs - Elasticsearch, Logstash and Kibana
Contents
- 1 Centralized Logs - Elasticsearch, Logstash and Kibana
- 2 CentOS 6 Installation of ELK
Centralized Logs - Elasticsearch, Logstash and Kibana
The goal is install in a dedicated server or VM, all the components to have a Centralized Log Server, and also a powerfull Dashboard to configure all the reports.
The Logstash, Elasticsearch and Kibana will be installed on this dedicated VM, in the Zimbra Server, or servers, will be installed the Agent.
Hardware and Software requisites
In the Server, or VM, we will install a fresh Ubuntu Server 14.04LTS. For the Hardware part, depends on how many Zimbra Servers, and how detailed are the Logs. For a regular environment, with the next resources is enough:
- OS: Ubuntu 14.04 LTS
- vRAM: 4GB
- vCPU: 2
- vDisk: 100GB (SAS 10K or even better 15K)
Install the Centralized Log Server
Installing Java
Elasticsearch and Logstash needs Java 7 to work, to install it, we need to add the PPA from Oracle to our apt:
root@logstashkibana01:/home/oper# sudo add-apt-repository -y ppa:webupd8team/java gpg: keyring `/tmp/tmptjs1zwc5/secring.gpg' created gpg: keyring `/tmp/tmptjs1zwc5/pubring.gpg' created gpg: requesting key EEA14886 from hkp server keyserver.ubuntu.com gpg: /tmp/tmptjs1zwc5/trustdb.gpg: trustdb created gpg: key EEA14886: public key "Launchpad VLC" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) OK
Once the Oracle repository is added, is time to do an apt-get update to refresh the packages list:
root@logstashkibana01:/home/oper# apt-get update
Great! Now, install the last stable Java 7 version:
root@logstashkibana01:/home/oper# sudo apt-get -y install oracle-java7-installer
Installing Elasticsearch
To install Elasticsearch, we need to add the public GPG key into our apt:
root@logstashkibana01:/home/oper# wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
Now, we need to add the source tree with the next command:
root@logstashkibana01:/home/oper# echo 'deb http://packages.elasticsearch.org/elasticsearch/1.1/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list
And do an apt-get update to update the packages list:
root@logstashkibana01:/home/oper#apt-get update
With all this previous steps, now is time to finally install Elasticsearch:
root@logstashkibana01:/home/oper# sudo apt-get -y install elasticsearch=1.1.1
Once installed, we need to edit a few parameters to improve the security of our Environment:
root@logstashkibana01:/home/oper# sudo vi /etc/elasticsearch/elasticsearch.yml
At the end of the file, add the next line to disable the dynamic scripts:
script.disable_dynamic: true
Also, to disable the API calls, we need to edit the network.host line:
network.host: localhost
Once we tunned our Elasticsearch, is time to restar the service:
root@logstashkibana01:/home/oper# sudo service elasticsearch restart * Starting Elasticsearch Server ...done.
To add the Elasticsearch service into the init, run the next command:
root@logstashkibana01:/home/oper# sudo update-rc.d elasticsearch defaults 95 10 Adding system startup for /etc/init.d/elasticsearch ... /etc/rc0.d/K10elasticsearch -> ../init.d/elasticsearch /etc/rc1.d/K10elasticsearch -> ../init.d/elasticsearch /etc/rc6.d/K10elasticsearch -> ../init.d/elasticsearch /etc/rc2.d/S95elasticsearch -> ../init.d/elasticsearch /etc/rc3.d/S95elasticsearch -> ../init.d/elasticsearch /etc/rc4.d/S95elasticsearch -> ../init.d/elasticsearch /etc/rc5.d/S95elasticsearch -> ../init.d/elasticsearch
Installing Kibana
At the time of this Wiki, we will install Kibana 3.1.2, please go to the Official Kibana Webiste to use the last release.
Download the Kibana release with the next command:
root@logstashkibana01:/home/oper# wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz
Extract the Kibana package:
root@logstashkibana01:/home/oper# tar xvf kibana-3.1.2.tar.gz
Move into the Kibana directory and edit the Config File:
root@logstashkibana01:/home/oper# cd kibbana-3.1.2 root@logstashkibana01:/home/oper/kibana-3.1.2# vi kibana-3.1.2/config.js
Once inside the file, search for the line elasticsearch: and change the port number (default 9200) for the port number 80, later we will connect to the Kibana Server in a easy way, trought the 80 HTTP port:
elasticsearch: "http://"+window.location.hostname+":80",
Also, we will use nginx to serve our app, Kibana, so we will create first the folder in the /var/www directory:
root@logstashkibana01:/home/oper/kibana-3.1.2# sudo mkdir -p /var/www/kibana3
Now, copy all the Kibana folder inside the new path:
root@logstashkibana01:/home/oper# sudo cp -R ~/kibana-3.1.2/* /var/www/kibana3/
Like I said, we will use Nginx to serve our Kibana app.
Installing Nginx
We will install nginx from the official apt repositories:
root@logstashkibana01:/home/oper# sudo apt-get install nginx
Kibana and Elsaticsearch works in a particular way, the user needs to access to Elasticsearch directly, so we need to configure Nginx to redirect all the packets to the 9200 port to the 80 port. But no worries, Kibana have and example that we can use for this. .
We will download the Nginx configuration from the GitHub to our folder:
cd ~; wget https://gist.githubusercontent.com/thisismitch/2205786838a6a5d61f55/raw/f91e06198a7c455925f6e3099e3ea7c186d0b263/nginx.conf
Edit the Config file:
root@logstashkibana01:~# vi nginx.conf
Find the line called server_name and add our own FQDN, or localhost if we don't use any particular fqdn. Also we need to add the path to our Kibana installation:
server_name FQDN; root /var/www/kibana3;
Save the file and copy it inside the nginx, to make it a default config file:
root@logstashkibana01:~# sudo cp nginx.conf /etc/nginx/sites-available/default
To allow other users to acces into the Kibana, we need to install the apache2-utils: </pre>root@logstashkibana01:~# sudo apt-get install apache2-utils</pre>
Is time to create an username for Kibana, to save the dashboards.
root@logstashkibana01:~# sudo htpasswd -c /etc/nginx/conf.d/kibana.myhost.org.htpasswd <span style="color: #ff0000;">user</span> <span style="color: #ff0000;">admin</span> New password: Re-type new password: Adding password for user admin
We are almost done, just restart the nginx service:
root@logstashkibana01:~# sudo service nginx restart * Restarting nginx nginx ...done.
Installing Logstash
This is the last package that we will install on the Server or VM. Now is time to install Logstash. We will install it from the Elasticsearch repository, that we have from before, so just launch the next commands:
root@logstashkibana01:~# echo 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list deb http://packages.elasticsearch.org/logstash/1.4/debian stable main
Run an apt-get update to refresh the packages list:
root@logstashkibana01:~# apt-get update
Run the next command to install Logstash:
root@logstashkibana01:~# sudo apt-get install logstash=1.4.2-1-2c0f5a1
Logstash is now installed, but we need to do this step before continue.
Generate the SSL Certificates to use in the server/client connection
We will use Logstash Forwarder in the Zimbra servers to send the logs to the Centralized Log Server. We want to do it in a secure way also. We need to generate an SSL and a key pair. The SSL will be used for the Client to verify the server identity.
First step is create the path while we will save the SSL and the private key:
root@logstashkibana01:~# sudo mkdir -p /etc/pki/tls/certs root@logstashkibana01:~# sudo mkdir /etc/pki/tls/private
Generate the SSL and the private key:
root@logstashkibana01:~# cd /etc/pki/tls; sudo openssl req -x509 -batch -nodes -days 3650 -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
Configuring the Logstash Server
All the Logstash configiration files are based in JSON format, and the path where they are located is /etc/logstash/conf.d. The configuration is based in three sections; inputs, filters y outputs.
Let's create a Configuration file called 01-lumberjack-input.conf and then we will configure oir input "lumberjack":
root@logstashkibana01:/etc/pki/tls# sudo vi /etc/logstash/conf.d/01-lumberjack-input.conf
And fill the file with the next configuration:
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
Save the file. With this step we specified an input "lumberjack" that is listening in the TCP port 5000 and also will use the SSL and the private key.
Now, is time create the file called 10-syslog.conf, and we will add the filter to our syslog messages:
root@logstashkibana01:/etc/pki/tls# sudo vi /etc/logstash/conf.d/10-syslog.conf
We will add the next content to the file to define our filter:
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
Save it. This filter will look into the logs, and will parse them using grok, to make them easier to understand it.
The last file that we need to edit, we will call it 30-lumberjack-output.conf:
root@logstashkibana01:/etc/pki/tls# sudo vi /etc/logstash/conf.d/30-lumberjack-output.conf
And the content needs to have the next:
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
Bascially, this output save the logs from Logstash inside Elasticsearch.
Restart the services:
root@logstashkibana01:/etc/pki/tls# sudo service logstash restart
We have our Server or VM 100% ready.
Configuring the Zimbra Servers
Now is time to configure the Zimbra Servers to send the Logs to our Centralized Log Server.
Next steps are for Ubuntu 14.04 LTS.
Copy the SSL certificate from the Logstash Server to Zimbra Servers
On the Logstash server, launch the next command to our Zimbra Server:
root@logstashkibana01:/etc/pki/tls# scp /etc/pki/tls/certs/logstash-forwarder.crtuser@server_private_IP:/tmp:
Installing the Logstash Forwarder Package
On the Zimbra Servers, we need to create the packages list for Logstash Forwarder:
root@zimbra-sn-u14-01:/home/oper# echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list
Once we've added the repository, install the Logstash Forwarder package:
root@zimbra-sn-u14-01:/home/oper# sudo apt-get update root@zimbra-sn-u14-01:/home/oper# sudo apt-get install logstash-forwarder
Add the Logstash Forwarder to the boot sequence:
root@zimbra-sn-u14-01:/home/oper# cd /etc/init.d/; sudo wget https://raw.github.com/elasticsearch/logstash-forwarder/master/logstash-forwarder.init -O logstash-forwarder root@zimbra-sn-u14-01:/home/oper# sudo chmod +x logstash-forwarder root@zimbra-sn-u14-01:/home/oper# sudo update-rc.d logstash-forwarder defaults
Copy the SSL certificate to the proper path:
root@zimbra-sn-u14-01:/home/oper# sudo mkdir -p /etc/pki/tls/certs root@zimbra-sn-u14-01:/home/oper# sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/
Configuring Logstash Forwarder
We are close to finish, inside the Zimbra Server, we need to think about what Logs we need to send to the Centralized Log Server.
Create a configuration file for Logstash Forwarder in JSON format:
root@zimbra-sn-u14-01:/home/oper# sudo vi /etc/logstash-forwarder
Now, we will fill the configuration file, change the IP for your own Centralized Log Server IP. Here in this example I will send to the Centralized Log Server the next logs: syslog, auth.log, mailbox.log, nginx.access.log, nginx.log, zimbra.log y mail.log, but you can add whatever log that you want:
{
"network": {
"servers": [ "IPDEVUESTROLOGSTASH:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/syslog",
"/var/log/auth.log",
"/opt/zimbra/log/mailbox.log",
"/opt/zimbra/log/nginx.access.log",
"/opt/zimbra/log/nginx.log",
"/var/log/zimbra.log",
"/var/log/mail.log"
],
"fields": { "type": "syslog" }
}
]
}
Save the configuration file.
Last step here is restart the Logstash Forwarder Service inside the Zimbra server:
root@zimbra-sn-u14-01:/home/oper# sudo service logstash-forwarder restart
We need to repeat this steps in each Zimbra Server that we want to have the Logs centralized.
Connecting to Kibana
Now is time to play! and also play in HTML5. Open a Web browser and type your IP or FQDN from your Centralized Log Server. The first thing that we will see is an overview of Kibana, etc. We will select the option 11. Sample Dashboard.
I really like it Kibana and also have a Centralized Log Server, but this is specially useful because we can search inside the Logs using checkbox, to filter and have the answer easier. Also we can mix the search and order for type of field, awesome!
Also, we can play with the Dashboard as we want, share a Public URL with some Customers, or between the IT Department and other Departments, etc.
This is a real overview, and I can see the total Logs received during a period of time.
Here we can see an example of how we can see a Log file, perfectly parsed to consume the information in a easier, and human, way.
That's it folks!
This Wiki is based on DigitalOcean ELK Tutorial
CentOS 6 Installation of ELK
YUM Configuration
Install the ElasticSearch Signing Key
rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch
Configure the repos
- Create /etc/yum.repos.d/elasticsearch.repo (for the server)
/etc/yum.repos.d/elasticsearch.repo: [elasticsearch-1.4] name=Elasticsearch repository for 1.4.x packages baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
- Create /etc/yum.repos.d/logstash.repo (for the server)
/etc/yum.repos.d/logstash.repo [logstash-1.4] name=logstash repository for 1.4.x packages baseurl=http://packages.elasticsearch.org/logstash/1.4/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
- Create /etc/yum.repos.d/logstash.repo (for the client)
/etc/yum.repos.d/logstash-forwarder.repo [logstash-forwarder] name=logstash repository packages baseurl=http://packages.elasticsearch.org/logstashforwarder/centos/ gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
Installation
ElasticSearch
yum install elasticsearch chkconfig --add elasticsearch
Logstash
yum install logstash chkconfig --add logstash
Kibana
mkdir /data/kibana4 # locate in your data location of choice wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz tar xvfz kibana-4.0.1-linux-x64.tar.gz cd kibana-4.0.1-linux-x64 cp -R * /data/kibana4/
Kibana init.d start/stop script
Note: Kibana4 has an integrated webserver and does not require a separate webserver. The following script is for the integrated web server startup.
- Create /etc/init.d/kibana
#!/bin/sh
#
# /etc/init.d/kibana -- startup script for kibana
# Wolfyxvf 2015-04-16; used httpd init script as template
#
### BEGIN INIT INFO
# Provides: kibana
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Starts kibana
# Description: Starts kibana using daemon
### END INIT INFO
#configure this with wherever you unpacked kibana:
KIBANA_BIN=/data/kibana4/bin
KIBANA_LOG="/var/log/kibana.log"
NAME=kibana
DESC="Kibana"
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/kibana
RETVAL=0
if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi
# Source function library.
. /etc/rc.d/init.d/functions
if [ -f /etc/sysconfig/kibana ]; then
. /etc/sysconfig/kibana
fi
start() {
echo "Starting $DESC : "
pid=`pidofproc -p $PID_FILE kibana`
if [ -n "$pid" ] ; then
echo "Already running."
exit 0
else
# Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
mkdir $PID_FOLDER
fi
daemon $DAEMON >> $KIBANA_LOG 2>&1 &
sleep 2
pidofproc node > $PID_FILE
echo
RETVAL=$?
[ $RETVAL = 0 ] && touch $LOCK_FILE
return $RETVAL
fi
}
stop() {
echo -n $"Stopping $DESC : "
killproc -p $PID_FILE $DAEMON
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p $PID_FILE $DAEMON
RETVAL=$?
;;
restart)
stop
start
;;
*)
echo $"Usage: $prog {start|stop|restart|status}"
RETVAL=2
esac
exit $RETVAL
- make the init file executable
chmod +x /etc/init.d/kibana
- Configure to start at boot
chkconfig --add kibana
- Create Certs files for putting the Kibana Web interface in HTTPS
cd /etc/pki/tls; openssl req -subj '/CN=FQDN_OF_KIBANA_SERVER/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/kibana.key -out certs/kibana.crt
- Configure the kibana configuration file take our modification
/data/kibana4/config/kibana.yml
change to : ... port 443 ... # SSL for outgoing requests from the Kibana Server (PEM formatted) ssl_key_file: /etc/pki/tls/private/kibana.key ssl_cert_file: /etc/pki/tls/certs/kibana.crt
