Setting Up the Splunk Universal Forwarder on Kali Linux for Your Cybersecurity Home Lab

11 min readNov 28, 2024

The Splunk Universal Forwarder is a lightweight tool used to collect logs and forward them to a Splunk instance for analysis. This guide walks you through the steps of setting up the Universal Forwarder on Kali Linux, connecting it to your Splunk instance for log collection and centralized analysis.

Types of Forwarders:

Heavy Forwarders

A heavy forwarder processes logs before sending them to the Splunk server.

It can filter, analyze, or change the logs at the source.

2. It is more powerful but uses more resources (CPU and memory).

Use a heavy forwarder if you need to:

Applying specific filters.

2. Processing data locally before sending it.

Universal Forwarders

A universal forwarder is a lightweight agent (small software program installed on a computer or server to perform a specific task with minimal use of system resources like CPU, memory, or storage) that:

Collects logs from a machine (like a server or computer).

2. Sends the logs as-is to the Splunk server without processing or analyzing them.

Use a Universal Forwarder when the goal is to send raw, unaltered logs to the Splunk server for centralized analysis.

It is designed for scenarios where no preprocessing or filtering of data is required at the source.
It is lightweight, consuming minimal system resources like CPU and memory, making it suitable for large-scale deployments or resource-constrained environments

Step 1: Switching to Root User

Before starting, switch to the root user

sudo su

Step 2: Downloading the Splunk Universal Forwarder

Visit the official Splunk website to download the Universal Forwarder:

Website: Splunk Universal Forwarder

Follow the instructions on the website to navigate to the correct version for Linux.

Download the .deb package suitable for your operating system (splunkforwarder-9.3.2-d8bb32809498-linux-2.6-amd64.deb).

2. Alternatively, to download the Universal Forwarder directly via the terminal, use the following wget command:

wget -O splunkforwarder-9.3.2-d8bb32809498-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/9.3.2/linux/splunkforwarder-9.3.2-d8bb32809498-linux-2.6-amd64.deb"
ls

Step 3: Installing Splunk Universal Forwarder

Install the Package: Use the dpkg command to install the forwarder:

dpkg -i splunkforwarder-9.3.2-d8bb32809498-linux-2.6-amd64.deb

2. Confirm Installation: Check if the forwarder is installed by listing its directory

ls /opt/splunkforwarder

Step 4: Starting the Splunk Forwarder

Initial Startup: Start the Splunk Universal Forwarder for the first time:

/opt/splunkforwarder/bin/splunk start --accept-license

2. Set Up an Admin Account: During the initial startup, you’ll be prompted to create an admin username and password:

Username: admin

Password: Choose a strong password and confirm it.

3. Port Configuration: By default, the forwarder runs on port 8089. If the port is unavailable, you’ll be prompted to specify a different port.

y
8090

4. Verify Splunk Forwarder Status: Check the status of the forwarder to ensure it is running correctly:

/opt/splunkforwarder/bin/splunk status

5. Change Ownership of the Splunk Forwarder Directory: To ensure proper permissions, change the ownership of the Splunk installation directory to a specific user and group (replace splunkfd with the appropriate username and group):

List All Files in the Directory: Verify the file structure and permissions:

chown -R splunkfwd:splunkfwd /opt/splunkforwa
ls -la

Notes: The port configuration ensures the forwarder does not conflict with other services on your machine.

Changing ownership is only necessary if the forwarder will run under a specific user account (splunkfd). If not, this step can be skipped.

Ensure the forwarder status displays as running before proceeding to the next steps.

Step 5: Configuring the Splunk Server

Understand the Forwarder-to-Server Communication:

After installing the Universal Forwarder, it must be configured to send data to the Splunk server.

The Splunk server must also be configured to receive data from the forwarder and store it in an appropriate index.

2. Access the Splunk Web Interface:

Log into your Splunk server via the web interface. The default URL is typically:

http://<SPLUNK_SERVER_IP>:8000

Use the Splunk administrator username and password to log in.

3. Configure Receiving Settings:

Navigate to Settings > Forwarding and Receiving.

4. Under the Receiving section, click on Configure Receiving.

Click New Receiving Port to add a port for the Universal Forwarder to send data.

5. By default, Splunk uses port 9997 for receiving data from forwarders.

You can either stick with 9997 or specify a different port if necessary.

Save the configuration to enable the selected receiving port.

6. Our listening port 9997 is now enabled and waiting for the data. If we want, we can delete this entry by clicking on the Delete option under the Actions column.

7. Configure an Index to Store Incoming Data:

The Indexes page displays all existing indexes and their metadata, such as:

Size: Total disk space used.

Event Count: Number of log events stored.

Home Path: Location of the index on disk.

Status: Active or inactive.

8. Click on the New Index button to create a dedicated index for your incoming data.

9. Create a new index (Linux_host) to store the incoming logs.

Save the new index configuration.

Step 6: Configuring Forwarding Settings

To send logs to your Splunk server

Navigate to the Splunk Forwarder directory
Add the Splunk server as a forwarding destination (replace <SPLUNK_SERVER_IP> with your server’s IP)

cd /opt/splunkforwarder/bin
./splunk add forward-server <SPLUNK_SERVER_IP>:9997

3. Specify the logs to monitor. For example, to monitor the syslog

./splunk add monitor /var/log/syslog -index Linux_host

4. Open Splunk forwarder inputs.conf configuration

nano /opt/splunkforwarder/etc/system/local/inputs.conf

Add the following configuration to monitor /var/log/syslog:

monitor:///var/log/syslog]
disabled = false
index = Linux_host
sourcetype = syslog

Save and Exit

5. Need to Navigate to the Splunk forwarder configuration directory

This is the directory where the inputs.conf file is stored. This file contains the configurations for monitoring files and sending logs to the Splunk indexer.

cd /opt/splunkforwarder/etc/apps/search/local

6. List the contents of the directory

ls

7. Need to Ensure Splunk Forwarder and Monitoring are Set Up

Navigate to /opt/splunkforwarder/bin/ and check if the binary exists

cd /opt/splunkforwarder/bin/                           
./splunk status

Step 7: Testing the Setup

Generate a test log

Logger is a built-in command line tool to create test logs added to the syslog file. As we are already monitoring the syslog file and sending all logs to the Splunk, the log we generate in the next step can be found with Splunk logs.

We will generate a Test Log Entry: Useing the logger command to create a unique test log entry that will be added to the /var/log/syslog file.

logger "This is a test log for Splunk verification."

2. Verifying the Logs in Splunk:

In Splunk Web interface need to go to the Search & Reporting app

3. Use a search query to check for our test log

index=Linux_host "This is a test log for Splunk verification."

The search result in Splunk shows that the test log was successfully ingested into the Splunk index.

The event confirms that:

The Splunk Universal Forwarder is successfully forwarding logs from the /var/log/syslog file.

The Linux_host index is correctly set up to receive the logs.

Splunk is ingesting and indexing the logs properly.

Step 8: Configuring Splunk to Listen on All Interfaces

Why Configure Splunk to Listen on All Interfaces:

By default, Splunk may only listen on localhost (127.0.0.1), meaning it can only be accessed locally.

Configuring it to listen on all interfaces (0.0.0.0) allows remote access from other devices, such as a Windows machine or another system.

Edit the web.conf File:

Use a text editor nano to edit the web.conf file, located in the Splunk configuration directory.

nano /opt/splunk/etc/system/local/web.conf

3. Modify the Configuration:

Add or modify the following lines under the [settings] section to enable Splunk to listen on all network interfaces:

[settings]
httpServer = 0.0.0.0:8000

By default, Splunk may only listen on localhost (127.0.0.1), meaning it can only be accessed locally.

Configuring it to listen on all interfaces (0.0.0.0) allows remote access from other devices, such as a Windows machine or another system.

3. Save and Exit:

Step 9: Finalizing and Restarting Splunk

To ensure everything is configured correctly, restart the Splunk Forwarder

/opt/splunkforwarder/bin/splunk restart

2. Verifying Splunk is Listening

netstat -tuln | grep 8000

5. In Windows machine Confirming Splunk is accessible on both eth0 and eth1 IPs. From another machine (Windows),

http://192.168.19.15:8000

http://192.168.1.132:8000

6. Check the logs to ensure the forwarder is sending logs to the correct indexer

tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log

Step 10: Configuring the Splunk Universal Forwarder to Monitor Windows Event Logs

On the Windows machine Navigate to the default Directory and Check if inputs.conf exists and Copy inputs.conf file from default to local

cd "C:\Program Files\SplunkUniversalForwarder\etc\system\default"
Get-ChildItem inputs.conf
Copy-Item -Path "C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf" -Destination "C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf"

Step 10: Adding Configuration

You will configure the Splunk Universal Forwarder to monitor specific Windows Event Log files (System.evtx and Security.evtx) and send the data to the Splunk indexer. Additionally, you’ll define how Splunk should parse the incoming data using a props.conf file.

Append Monitoring Configurations to inputs.conf: These commands configure the Universal Forwarder to monitor the System and Security event logs and send the data to the Splunk server.

Add-Content -Path "C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" -Value "`n[monitor://C:\Windows\System32\winevt\Logs\System.evtx]`nindex = main`nsourcetype = WinEventLog:System`n"
Add-Content -Path "C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" -Value "`n[monitor://C:\Windows\System32\winevt\Logs\Security.evtx]`nindex = main`nsourcetype = WinEventLog:Security`n"

monitor: Specifies the file to monitor.

index = main: Sends the data to the main index in Splunk.

sourcetype: Identifies the data format so Splunk can parse it correctly.

System.evtx → WinEventLog:System

Security.evtx → WinEventLog:Security

These configurations allow Splunk to efficiently process and categorize logs from the Windows System and Security event logs.

2. Add Parsing Rules Using props.conf:

Navigate to the local directory where you will create or modify the props.conf file and append parsing rules for the System Event Log in props.conf

cd "C:\Program Files\SplunkUniversalForwarder\etc"
cd "C:\Program Files\SplunkUniversalForwarder\etc\system\local"
Add-Content -Path "C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf" -Value "`n[WinEventLog:System]`nINDEXED_EXTRACTIONS = json`nLINE_BREAKER = ([\r\n]+)`nSHOULD_LINEMERGE = false`nTIME_PREFIX = ^`nTIME_FORMAT = %Y-%m-%d %H:%M:%S`n"

3. Restart the Universal Forwarder

net stop splunkforwarder
net start splunkforwarder

Step 12: Create a New Index for Windows Logs (on Kali Linux)

Create or Edit indexes.conf: Define a new index called win_log to store Windows event logs.

nano /opt/splunk/etc/system/local/indexes.conf

2. Add the Following Configuration:

[win_log]
homePath = $SPLUNK_DB/win_log/db
coldPath = $SPLUNK_DB/win_log/colddb
thawedPath = $SPLUNK_DB/win_log/thaweddb

3. Save and Exit

Step 21: Edit the inputs.conf File

Open the inputs.conf file in the Splunk Universal Forwarder configuration directory:

nano /opt/splunkforwarder/etc/system/local/inputs.conf

2. Add the following configurations for monitoring Windows Event Logs:

[monitor:///var/log/syslog]
disabled = false
index = syslog
sourcetype = syslog

[monitor:///var/log/syslog]
disabled = false
index = syslog 
sourcetype = pfsense             [monitor://C:\Windows\System32\winevt\Logs\Security.evtx]
disabled = false
index = win_log
sourcetype = WinEventLog:Security[monitor://C:\Windows\System32\winevt\Logs\Application.evtx]
disabled = false
index = win_log
sourcetype = WinEventLog:Application[monitor://C:\Windows\System32\winevt\Logs\System.evtx]
disabled = false
index = win_log
sourcetype = WinEventLog:System

monitor: Specifies the file path to monitor.

index = win_log: Logs will be sent to the win_log index on the Splunk indexer.

sourcetype: Identifies the format of the data to help Splunk parse the logs correctly:

WinEventLog:Security: Security event logs.

WinEventLog:Application: Application event logs.

WinEventLog:System: System event logs.

3. Save and exit the file

4. Restart the Splunk Forwarder to apply the changes:

/opt/splunk/bin/splunk restart

Step 13: Checking Logs on the Splunk Indexer

index=win_log sourcetype="WinEventLog:Security"

In Windows machine DNS or Hosts File: For convenience, you can map the Splunk server’s IP to a hostname in our Windows local hosts file on the accessing when machines

1. Open PowerShell as Administrator

2. Back up the Original Hosts File

Copy-Item -Path "C:\Windows\System32\drivers\etc\hosts" -Destination "C:\Windows\System32\drivers\etc\hosts.bak"

3. Add the Splunk Server IP and Hostname

cd

4. Verify Changes

Get-Content -Path "C:\Windows\System32\drivers\etc\hosts"

Congratulations!

You’ve successfully set up the Splunk Universal Forwarder on Kali Linux to collect and forward logs to your Splunk server for analysis. The process is lightweight, requiring minimal system resources, and suitable for large-scale deployments or monitoring a variety of logs in a home lab environment.

Make sure to keep your Splunk configurations organized and update them as needed to monitor new logs or configure additional sources.