Logstash for Weblogic - Part IV - Working across clustered environments

This is where logstash act as a real Champ..!!!

Logstash can be installed in any servers where it act as shipper and all these logs are accumulated in single place and can be used for real time monitoring. 

For this, additional components like redis, elsatic search are used. 

Logstash acts as a log collector, processor, shippers and indexer. 

It needs to be installed and running in servers where it will read the logs as events and as per the conf file configurations, here logstash acts as log collector and shipper. 

And in main server, it will act as indexer and processor. 

Refer previous post for more details. Post1, Post2, Post3;

Logstash has Kibana which is a dashboard for data visualization. 

Redis acts as a broker and receives the captured events from other servers ( Logstash that is acting as shipper) to main server. 

Elastic search acts as a persistent storage & search medium for all captured events

Below flow chart is self-explanatory. 

Now I shall explain steps to configure it across the servers.

For easy understanding, we have SOA server1 and SOA server2 and SOA Server1 will act as main server here and logs will be captured as event and shipped to SOA Server1 from SOA server2. So we can have as many as servers. 

Redis and Elastic search needs to be downloaded and running before any shipper starts shipping the events.  So as per our work case, we need to install it in SOA Server1.

Redis: 

•       Download redis and extract the source.
•       Execute the redis : src/redis-server --loglevel verbose 

You will get output as mentioned in below picture.

Elastic Search:

• Download the latest version of elastic search.
• Execute the command : bin/elasticsearch –f

Logstash as Shipper:

Run the logstash as shipper now in SOA Server2.

In config, make sure that output is directed to redis that is running in Server 1.

Like this,

Save this as say, shipper_server2.conf and run the logstash using below command.

java -jar logstash-1.2.0-flatjar.jar agent -f shipper_server2.conf

Now check your redis output, clients connected value will be 2 which indicated that shipper is successfully connected to Broker.

Now all the events captured will be stored in elastic search.

Logstash as Indexer:

To run logstash as indexer, make sure that input is read from redis.

List details mentioned in shipper output will be used here in input. 

Conf file will look like below,

  

And save this as indexer.conf and run using below command,

java -jar logstash-1.2.0-flatjar.jar agent -f indexer.conf

Kibana:

Use individual terminal to run kibana using below command,

java -jar logstash-1.2.0-flatjar.jar – web

Open kibana using this URL – http://localhost:9292

Logstash is so modular that all the above components can be installed on dedicated servers to distribute the load and specify the host accordingly.

Logstash for weblogic - Part III - Using GROK patterns

This post explains the concept of GROK filter which gives more flexibility in parsing the logs and analyzing. Weblogic SOA logs will have information’s like severity, host details, composite details, timestamps etc. and this information will be more helpful when we use logstash centralized logging solution across multiple environments.

Using grok, we can parse unstructured log data into structured and also queryable. Logstash has 120 patterns which is available in this link. https://github.com/logstash/logstash/tree/v1.3.2/patterns

General information about logstash is available in this link - http://logstash.net/docs/1.3.2/filters/grok 

Below picture explains the structure of admin log in weblogic and how we can identify the pattern to parse it using grok. 

This is the format of admin logs and you can decide your own pattern using this link which has 120 patterns for Logstash.  Link is https://github.com/logstash/logstash/tree/v1.3.2/patterns

"####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> %{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"

Words that are in bold are valid patterns that can be used in logstash. When your log entries matches these patterns which are in bold, then those particular data will be indexed in the name of one that is highlighted in Yellow.

Please use below config plan to achieve grok filters and indexing, you can change accordingly to different logs. You need to include multiline filter as well to get rid of spaces in Java exception. Using multiline filter is discussed in this post.  

input {

 stdin {

    type => "stdin-type"

  }

  file {

    type => "ADMdomainlog"

    path => [ "D:/Logstash/Log/soa_domain.log"]

  }

  }

 

  filter {

  multiline {

    type => "ADMdomainlog"

    pattern => "^####"

    negate => true

    what => "previous"

  }

    grok {

    type => "ADMdomainlog"

    pattern => ["####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"]

    add_field => ["Log", "Admin Domain Log"]

  }

  }

 

output {

  elasticsearch { embedded => true }

}

Run using the logstash and open Kibana to view the logs, you can see that there are new indexes as per your grok patterns and you can even filter using those indexes as mentioned in below screen shots.

This grok adds more flexibility in analyzing the logs and we can use it more effectively when we define our own dashboard in Kibana which will be discussed in further posts...

Logstash for Weblogic - Part II - Using multiline filters

Normally our server logs are like one mentioned below and so if you use the config file mentioned in first post, then you can see that each and every line of one log event is captured as separate event which leads to confusion..!!

There is way to overcome this problem in logstash by using filter called multiline. This filter will collapse multiline messages into a single event. The multiline filter is for combining multiple events from a single source into the same event. The goal of this filter was to allow joining of multi-line messages from files into a single event. For example - joining java exception and stack trace messages into a single event.

General syntax of multiline filter is

filter {

  multiline {

    type => "type"

    pattern => "pattern, a regexp"

    negate => boolean

    what => "previous" or "next"

  }

}

Where, ‘regexp’ should match what you believe to be an indicator that the field is part of a multi-line event. Here we can match logs that start with ^#### which will be common for all weblogic logs. ^ à indicates that logs start with ####

The 'what' must be "previous" or "next" and indicates the relation to the multi-line event. Here we provide previous as we need to relate the space with previous lines.
The 'negate' can be "true" or "false" (defaults false).
  input {
Save this file as sample.conf
Run logstash and feed your logs with sample logs, now you can see that all your java exception log entry is captured as single event.  This simple multilane filter helps to solve the problem.
You can see that logs are captured as single event. 
Please contact me incase of any doubts and in next post, we I will share about GROK filters which gives more flexibility in analyzing the logs..

You need to add the filter in between input and output like mentioned in below config file. 

 stdin {

    type => "stdin-type"

  }

  file {

    type => "ADMdomainlog"

    path => [ "D:/Logstash/Log/soa_domain.log"]

  }

  }

  

  filter {

  multiline {

    type => "ADMdomainlog"

    pattern => "^####"

    negate => true

    what => "previous"

  }

  }

  

output {

  elasticsearch { embedded => true }

}

 View in Kibana,

Logstash for Weblogic and SOA Part - I

Those who wonder what is logstash, it is an open source tool for managing events and logs. This can be used to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, log stash comes with a web interface for searching and drilling into all of your logs.

All sorts of information required to deploy logstash is available in logstash official website (http://logstash.net/) but this post will share my experience and learning’s while we tried to use it in our project especially for analyzing the weblogic and SOA logs…

Step 0:

1. Download https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
2. Works well in windows as well, so no worries for POC
3. This seems to be very important one; I spend hours to get this clicked. LOGSTASH can be used only to read live logs or new logs and not existing logs from your logs folders. There is a way to read those logs which will be shared in the end of this post.
4. Logstash is shipped along with elastic search as storage engine and Kibana to display the parsed logs

Another important thing in logstash is Configuration file:

1. Configuration file is the simple .conf file that is used to read and parse the logs
2. It primarily deals with input, output and filters
1. Inputs are inputs to logstash which can be a file path to the logs
2. Outputs are section which deals with how the parsed logs needs to be viewed

All these sections let us to configure plugins and all available plugins are available in in this link http://logstash.net/docs/1.2.2/ under plugin documentations.

To configure logstash in standalone server, 

Save as sample.conf.  Now we need to run the logstash using below syntax which is very simple. Before that if you are trying in windows, set your java home in environment variables and if it is in linux, then you need to run with java path. Both the examples are provided below. 

java -jar logstash-1.2.2-flatjar.jar agent -f logstash_sample.conf  --- For windows

/u01/dev/fmwadm/Middleware/jrockit_160_29_D1.2.0-10/jre/bin/java -jar logstash-1.2.2-flatjar.jar agent -f logstash_sample.conf --- For Linux

make sure that you have placed a empty soa_domain.log file in the mentioned path. After executing this command, paste your log entries to the soa_domain.log file so that logstash parse it as new entry in log file :). 
These logs will be captured as events and stored in elastic search and to view this in UI, kibana is used. But to acheive this, stop the above command with ctrl+c and execute below command which will invoke the logstash with kibana web service. 

java -jar logstash-1.2.2-flatjar.jar agent -f logstash-sample.conf – web

Kibana Web by default use 9292 port and make sure that nothing is running in this port, open http://localhost:9292/ which will kick start kibana for you. There are lot many options in kibana which I will share in separate post. 

As of now use default dashboard and view your logs that are parsed with default indexes like  _id, _index, type,@timestamp,@version.

Now you can search the logs as you wish...

Same steps are used in linux or unix well.

Here we do face problem when your logs has space in between the log entries and that is where we need to use filter which I will share in next post. 

Purpose of this post to give basic introduction about logstash and to understand how it can be used for analyzing weblogic logs... Feel free to contact me if you face any difficulties.