Splunk: Centralized Machine Data Storage

Splunk Installation steps on Linux


Installing Splunk

Download it from the Website (create a splunk account)

sudo dpkg -i splunk-5.0-140868-linux-2.6-amd64.deb

Output –

Splunk has been installed in:

To start Splunk, run the command:
/opt/splunk/bin/splunk start
To use the Splunk Web interface, point your browser at:
Complete documentation is at http://docs.splunk.com/Documentation/Splunk
Installation DONE 🙂






Forwarding Logs to the Central Server

In terms of Splunk to get remote machine data – there are 2 machines Forwarders (Splunk client) and Receivers (Splunk server)
Configuring the Splunk Server (The Receiver) :
Use Splunk Manager to set up a receiver:

1. Log into Splunk Web as admin on the server that will be receiving data from a forwarder.

2. Click the Manager link in the upper right corner.

3. Select Forwarding and receiving in the Data area.

4. Click Add new in the Receive data section.

5. Specify which TCP port you want the receiver to listen on (the listening port, also known as the receiving port). For example, if you enter “9997,” the receiver will receive data on port 9997. By convention, receivers listen on port 9997, but you can specify any unused port. You can use a tool like netstat to determine what ports are available on your system. Make sure the port you select is not in use by splunkweb or splunkd.

6. Click Save. You must restart Splunk to complete the process.
Quick Navigation

GO TO HOME PAGE : Click App -> Search
Create Users : Click Manager -> Access Controls
Include Data Inputs : Manager -> Data inputs -> Files & directories


If any of the universal forwarders will be running on a different operating system from the receiver, install the app for the forwarder’s OS on the receiver. For example, assume the receiver in the diagram above is running on a Linux box. In that case, you’ll need to install the Windows app on the receiver. You might need to install the *nix app, as well. — However, since the receiver is on Linux, you probably have already installed that app. Details and provisos regarding this can be found here.

After you have downloaded the relevant app, remove its inputs.conf file before enabling it, to ensure that its default inputs are not added to your indexer. For the Windows app, the location is: $SPLUNK_HOME/etc/apps/windows/default/inputs.conf.
Point Splunk at a data source. Tell Splunk a bit about the source. That source then becomes a data input to Splunk. Splunk begins to index the data stream, transforming it into a series of individual events. You can view and search those events right away. If the results aren’t exactly what you want, you can tweak the indexing process until you’re satisfied.

The data can be on the same machine as the Splunk indexer (local data), or it can be on another machine altogether (remote data). You can easily get remote data into Splunk, either by using network feeds or by installing Splunk forwarders on the machines where the data originates. Forwarders are lightweight versions of Splunk that consume data and then forward it on to the main Splunk instance for indexing and searching. For more information on local vs. remote data, see “Where is my data?”.

A Splunk instance that forwards data to another Splunk instance (an indexer or another forwarder) or to a third-party system is called FORWARDER

A Splunk instance that receives data from a forwarder is called a RECEIVER.

There are three types of forwarders:

The universal forwarder is a streamlined, dedicated version of Splunk that contains only the essential components needed to forward data.
A heavy forwarder is a full Splunk instance, with some features disabled to achieve a smaller footprint.
A light forwarder is also a full Splunk instance, with most features disabled to achieve as small a footprint as possible. The universal forwarder, with its even smaller footprint yet similar functionality, supersedes the light forwarder for nearly all purposes.

In most respects, the universal forwarder represents the best tool for forwarding data to indexers. Its main limitation is that it forwards only unparsed data. Therefore, you cannot use it to route data based on event contents. For that, you must use a heavy forwarder

Universal Forwarder Vs full Splunk

The universal forwarder’s sole purpose is to forward data. Unlike a full Splunk instance, you cannot use the universal forwarder to index or search data. To achieve higher performance and a lighter footprint, it has several limitations:

The universal forwarder has no searching, indexing, or alerting capability.
The universal forwarder does not parse data.
The universal forwarder does not output data via syslog.
Unlike full Splunk, the universal forwarder does not include a bundled version of Python.

@Splunk Client

Install universal forwarders on each machine that will be generating data. These will forward the data to the receiver.
Download Link

#wget -O splunkforwarder-5.0-140868-linux-2.6-amd64.deb ‘http://www.splunk.com/page/download_track?file=5.0/universalforwarder/linux/splunkforwarder-5.0-140868-linux-2.6-amd64.deb&ac=&wget=true&name=wget&typed=releases’
#dpkg -i splunkforwarder-5.0-140868-linux-2.6-amd64.deb

Output :::

Selecting previously deselected package splunkforwarder.
(Reading database … 160868 files and directories currently installed.)
Unpacking splunkforwarder (from splunkforwarder-5.0-140868-linux-2.6-amd64.deb) …
Setting up splunkforwarder (5.0-140868) …
Splunk has been installed in:

To start Splunk, run the command:
/opt/splunkforwarder/bin/splunk start

Complete documentation is at http://docs.splunk.com/Documentation/Splunk

Configuration steps

After you start the universal forwarder and accept the license agreement, follow these steps to configure it:

1. Configure universal forwarder to auto-start:

#splunk enable boot-start

#cd /opt/splunkforwarder/bin
./splunk add forward-server <host>:<port> -auth <username>:<password>

For <host>:<port>, substitute the host and receiver port number of the receiver. For example, splunk_indexer.acme.com:9995.

Alternatively, if you have many forwarders, you can use an outputs.conf file to specify the receiver. For example:

server= splunk_indexer.acme.com:9995

You can create this file once, then distribute copies of it to each forwarder.

./splunk add forward-server -auth admin:changeme

9995 is the port i had openend on the server (receiver)
user/passwd is the forwarders username and passwd which by default is admin/changeme
IP – of the server.

Restart Splunk Forwarder
#cd /opt/splunkforwarder/bin
#./splunk stop/start

Test Forwarder connection:
/opt/splunkforwarder/bin/splunk list forward-server
Add Data:
/opt/splunkforwarder/bin/splunk add monitor /path/to/app/logs/ -index main -sourcetype %app%
Where /path/to/app/logs/ is the path to application logs on the host that you want to bring into Splunk, and %app% is the name you want to associate with that type of data

This will create a file: inputs.conf in /opt/splunk/etc/apps/search/local/ — here is some documentation on inputs.conf:

Note: System logs in /var/log/ are covered in the configuration part of Step 7. If you have application logs in /var/log/*/

./splunk add monitor /var/log/ -index main -sourcetype %VishnuMachine%
Added monitor of ‘/var/log’.
Final Setting on the Splunk server (Receiver)

Manager -> Add data -> TCP -> Add new
TCP Port : 8088 (port number on the client ie forwarder)
Set sourcetype : VishnuMachine (as mentioned in the command)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s