Setting Up Splunk & Universal Forwarderon on Windows 10 for Your Cybersecurity Home Lab

As cybersecurity becomes an increasingly critical concern for individuals and businesses alike, the ability to monitor, analyze, and understand system logs has never been more important. One of the most powerful tools for doing this is Splunk — a platform designed to collect, index, and analyze data generated by your systems and applications. Whether you’re looking to track security threats, troubleshoot system performance, or gain deeper insights into your data, Splunk is an essential tool.

Step 1: Choosing the Right Version of Splunk Enterprise

Splunk Enterprise is the full version of the platform, designed to collect, index, and analyze logs from various sources. It’s ideal for use in a home lab to gain deeper insights into system and application data.

Why Choose Splunk Enterprise?

Real-time Analysis: Splunk allows you to search, track, and analyze logs in real-time, helping you identify issues promptly.

Scalability: It can scale from a single installation on a test machine to a fully distributed enterprise-level setup.

Security: Splunk helps you detect and respond to security anomalies by analyzing logs and identifying potential threats.

To begin the installation, we first need to download the Splunk Enterprise installer from the official website.

1. Open your web browser and visit the Splunk download page.
2. Select Splunk Enterprise and choose the Windows version (in MSI format).
3. Click on Download Now and save the installer to your computer.

This installer will set up the Splunk Enterprise instance that will allow you to collect and manage logs from different systems.

Step 2: Downloading the Installer

Once the installer is downloaded, locate the Splunk Enterprise MSI file on your computer, typically found in your Downloads folder. This file is required to install Splunk, and depending on your internet speed, the download may take a few minutes.

Step 3: Opening PowerShell as Administrator

To install Splunk, we need to run some commands that require administrative privileges.

1. In the Windows search bar, type PowerShell.
2. Right-click Windows PowerShell from the search results and select Run as Administrator.
3. This ensures we have the necessary permissions to install software on the system.

Step 4: Creating a C:\Temp Directory and Downloading the Installer

Next, we will create a temporary folder where the Splunk installer will be stored and downloaded. Open PowerShell and enter the following commands:

1. Create a temporary folder on the C: drive:

New-Item -Path "C:\Temp" -ItemType Directory

2. Download the Splunk installer into this folder:

This will place the Splunk installer in the C:\Temp folder.

The process, usually takes 10 to 15 minutes.

wget -O "C:\Temp\splunk-9.3.2-d8bb32809498-x64-release.msi" "https://download.splunk.com/products/splunk/releases/9.3.2/windows/splunk-9.3.2-d8bb32809498-x64-release.msi"

Step 5: Verifying the Download

After downloading, we want to ensure that the file is present in the correct folder:

dir C:\Temp

Step 6: Running the Installer

To start the installation process, we need to run the Splunk installer:

Start-Process -FilePath "C:\Temp\splunk-9.3.2-d8bb32809498-x64-release.msi" -Wait

Step 7: Accepting the License Agreement

During the installation, you’ll be prompted to accept the Splunk License Agreement.

1. Check the box to accept the agreement.
2. If you’d like to read the full agreement, click on View License Agreement.
3. After accepting, you can either proceed with the default installation settings or click Customize Options to adjust specific settings like the installation directory.

Step 8: Creating the Administrator Account for Splunk

At this step, you will create an administrator account for Splunk.

1. Username: By default, the username is admin, but you can choose any name you prefer.
2. Password: The password should be at least 8 characters long, containing letters, numbers, and symbols. Make sure the password is strong and memorable.
3. Confirm Password: Enter the password again to confirm.
4. Click Next once you have completed the account setup.

Step 9: Finalizing the Installation

Before the installation completes, you can choose whether or not to create a Start Menu shortcut for Splunk. If you want one, check the corresponding box.

Click Install to begin the actual installation. It will take several more minutes for Splunk to install and set up.

Step 10: Logging into Splunk

After the installation completes, Splunk will open in your web browser. Go to http://localhost:8000 to access the Splunk Web Interface.

1. Log in using the admin username and the password you created during setup.
2. Click Log in to access the Splunk dashboard.

Step 11: Exploring the Splunk Dashboard

Once logged in, you’ll be greeted with the Splunk Dashboard. This dashboard displays real-time data and insights collected by Splunk, giving you a powerful interface for monitoring logs, running searches, and analyzing data.

Security Risk Warning in Splunk

The warning you’re seeing indicates a potential security risk in your Splunk configuration related to email alerts. Specifically, the setting for allowedDomainList in the alert_actions.conf configuration file is empty. This setting controls which email domains are allowed when sending search results via email.

By default, if this setting is left empty, it could allow users to send email alerts to any domain, which may lead to data leakage, especially if sensitive information is included in the search results.

What is allowedDomainList?

The allowedDomainList is a configuration option that specifies which email domains are allowed when users configure email alerts in Splunk. If this setting is not configured (or left empty), users may potentially send email alerts to any external email address, which could be a security concern in environments where sensitive data is involved.

How to Fix This Warning

Modify Email Settings in Splunk Web Interface

1. Login to Splunk Web:

Access your Splunk web interface by navigating to http://<splunk-server>:8000.

2. Go to Email Settings:

In the top-right corner, click on the gear icon and select Settings.

Under System, choose Server Settings.

Select Email Settings.

3. Configure allowedDomainList:

In the Email Domains section, add the domains you want to allow for email alerts.

For example, you can add example.com, example.org to restrict email alerts to these domains.

4. Save the Settings.

By adding specific allowed domains, you limit the possibility of sending email alerts to unauthorized recipients, thus improving the security of your Splunk environment.

You should no longer see the security warning related to the empty allowedDomainList.

Additional Considerations

If you are using an email provider with dynamic domain lists or if your organization frequently adds new domains, you may need to update the allowedDomainList periodically.

For high-security environments, consider restricting access to alert configurations to only trusted users or roles.

Step 12: Configuring the Splunk Environment

To simplify the process of managing Splunk through the command line, we’ll set an environment variable called SPLUNK_HOME. This allows us to easily navigate to the Splunk installation folder from PowerShell.

1. Set the SPLUNK_HOME environment variable:

2. Navigate to the configuration folder:

3. Restart Splunk:

This ensures that Splunk is properly configured and ready to start collecting and analyzing data.

$env:SPLUNK_HOME = "C:\Program Files\Splunk"
cd "$env:SPLUNK_HOME\etc\system\local"
cd "$env:SPLUNK_HOME\bin"
.\splunk restart

Installing and Configuring Splunk Universal Forwarder

Now that Splunk Enterprise is installed, the next step is to configure Splunk Universal Forwarder, which will send logs from remote systems to Splunk Enterprise. This is an essential step if you want to collect log data from multiple machines.

Step 1: Configuring Receiving in Splunk

Before setting up the Universal Forwarder, we need to configure Splunk Enterprise to receive the data from the forwarders.

1. In the Splunk Web Interface, go to Settings > Forwarding and Receiving.

2. Click Add New to create a new receiving port.

3. Set the port to 9997, the default port for receiving data, and click Save.

Step 2: Downloading the Universal Forwarder

The Splunk Universal Forwarder can be downloaded from the official Splunk download page.

https://www.splunk.com/en_us/download.html

This software will send the log data from a system to your Splunk instance.

Step 3: Installing the Universal Forwarder

1. In PowerShell, download the installer:

Invoke-WebRequest -Uri "https://download.splunk.com/products/universalforwarder/releases/9.3.2/windows/splunkforwarder-9.3.2-d8bb32809498-x64-release.msi" -OutFile "splunkforwarder-9.3.2-x64.msi"

Check if the file has been successfully downloaded in the working directory

Get-ChildItem splunkforwarder-9.3.2-x64.msi

2. Run the installer:

msiexec /i "C:\Windows\System32\splunkforwarder-9.3.2-x64.msi"

Step 4: Configuring the Forwarder

During the installation, you’ll need to specify configuration details for the Universal Forwarder:

1. License Agreement: Accept the license agreement.

Crate credentials

2. Deployment Server: Set the Deployment Server to the IP address of your Splunk instance (use 127.0.0.1 if it’s on the same machine ).

Deployment Server

This is used if you have a centralized Deployment Server to manage configurations for multiple Universal Forwarders.

In most simple lab setups , you don’t need a Deployment Server. You can leave this blank or skip it by clicking Next without entering anything.

3. Listener Configuration: Ensure that the Universal Forwarder is configured to send data to port 9997.

Receiving Indexer (current screen):

This is where you specify the Splunk Enterprise instance (on Kali Linux, 192.168.19.15) that will receive the logs forwarded by this Universal Forwarder.

You must enter the IP and port here because Splunk Enterprise needs to know where to accept forwarded data.

Click Install to begin installation.

Finish

Step 5: Completing the Forwarder Installation

After completing the configuration, the Universal Forwarder will be ready to send logs to Splunk. The installation should only take a few minutes.

Step 6: Configure TCP Port 9997 to Receive Data:

1. Go to Settings > Data Inputs

2. Click on “New Local UDP” (or similar option depending on your Splunk version).

Configure the new UDP input:

UDP port: Enter 9997 (this is the port where your Universal Forwarder is sending data).

Host Restriction: Leave it as is or configure it if you want to restrict which machines can send data to this port.

Source Type: You’ve correctly selected Windows Event Logs, which is the appropriate source type for forwarding Windows event logs.

Host Field: You’ve set the Host field to <Host_IP>, which is your Splunk server’s IP address. However, it’s best to use the Windows machine’s hostname or IP address here.

This will make it easier to differentiate logs from different machines when analyzing data in Splunk.

If you’re forwarding logs from multiple sources, using the Windows machine’s hostname or IP will allow for better identification.

Source Type Description: Indicates the data source.eventlog

Ensure you are specifying an appropriate index for the logs.

Create a new index win_log

Click Review to continue.

Submit

Start Searching

Verify Splunk is Listening on Port 9997

netstat -an | findstr "9997"

After confirming that port 9997 is configured, you need to restart Splunk to ensure it begins listening on that port

cd "C:\Program Files\Splunk\bin"
.\splunk restart

Step 8: Check for Incoming Data

In the Splunk Web Interface, search for any data coming from your Windows machine by running a query in Search & Reporting:

index=_internal sourcetype=splunkd host=<Windows-Machine-Hostname>

If you don’t see data coming into Splunk, check the logs on the Splunk Universal Forwarder for any issues:

Navigate to C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log.

Look for any warnings or errors that might indicate issues with the connection.

cd "C:\Program Files\SplunkUniversalForwarder\var\log\splunk"

View the last 20 lines of the log file, which can help you see any recent errors or warnings related to the forwarding process.

Get-Content .\splunkd.log -Tail 20

You can also use the -Head parameter to view the first few lines, or search through it to find any errors:

Get-Content .\splunkd.log -Head 50

Step 7:Setting Up Splunk Universal Forwarder to Send Logs to Splunk on Kali Linux

1. Open the outputs.conf File in Notepad

The outputs.conf file controls where the Universal Forwarder sends the log data. To modify it:

Open PowerShell as an administrator.

Type the following command and press Enter to open the outputs.conf file in Notepad:

notepad "C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf"

2. Modify the outputs.conf File

In the Notepad editor, make sure the configuration looks like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = <Host_IP>:9997
[tcpout-server://<Host_IP>:9997]

3. Save the Changes

Once you have made the changes, save the file and close Notepad.

Configure Inputs in inputs.conf

To collect Windows Event Logs (Security, Application, System), you need to manually edit the inputs.conf file.

Open the inputs configuration file:

notepad "C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf"

[monitor://C:\Windows\System32\winevt\Logs\System.evtx]
index = win_log
sourcetype = WinEventLog:System

Save the Changes

Once you have made the changes, save the file and close Notepad.

Restart the Splunk Universal Forwarder

For the changes to take effect, you need to restart the Splunk Universal Forwarder.

Change the directory to the Splunk installation directory and Run Splunk

cd "C:\Program Files\Splunk\bin"
.\splunk restart

This will restart the Splunk Universal Forwarder and apply the new settings.

Optionally, Add Splunk to the PATH

If you want to be able to run the splunk command from any directory in the future without navigating to C:\Program Files\Splunk\bin, you can add the bin folder to your system’s PATH environment variable.

[System.Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Program Files\Splunk\bin", [System.EnvironmentVariableTarget]::Machine)

Open the props.conf file:

Navigated to the Splunk bin directory and Configure the forwarder to send logs to a remote server Kali_Linux (host).

cd "C:\Program Files\Splunk\bin"
.\splunk add forward-server 192.168.19.15:9997

Restart Splunk:

.\splunk restart

Check if Port 9997 is Listening

netstat -an | findstr 9997

.\splunk list forward-server

Step 8: Managing the Forwarder

Once the Universal Forwarder is installed, you can manage it through the Forwarder Management tab in Splunk. Here, you will be able to monitor the forwarders and their data.

Ensure that the Universal Forwarder was installed and configured to forward data to Host_IP:9997 (192.168.19.15:9997).

You can double-check this by running

cd "C:\Program Files\SplunkUniversalForwarder\bin"
.\splunk list forward-server

Verify Network Connectivity
Since the forwarder is correctly configured, let’s verify that there are no network issues. On your Windows machine, run this command to ensure that port 9997 is open and reachable from your Windows machine to Kali:

Test-NetConnection -ComputerName 192.168.19.15 -Port 9997

1. Go to Settings > Add management

You will see that one Universal Forwarder client has successfully connected to the Splunk deployment server or indexer. It is listed under the “Clients” tab.

Step 7: Selecting Forwarder

1. Go to Settings > Add data (It shows all the options to add data from different sources) .

Skip

You will see three options for selecting how to ingest our data.

Choose the Forward option to get the data from Splunk Forwarder:

2. In the Select Forwarders section, we need to click on the host DESKTOP-G4ANHC shown in the Available host(s) tab, and it will be moved to the Selected host(s) tab.

3. You need to defining a New Server Class (Win_Logs) for the selected forwarders. A server class groups forwarders for centralized configuration and data management.

Then, Next to proceed.

This server class will be used to apply specific configurations (which logs or data to monitor).

Then, Next to proceed.

Step 8: Creating a new index that will store the incoming Event logs.

Once created, select the Index from the list and move to the next step.

The review tab summarizes the settings we just did to configure Splunk.

Submit tab to move to the next step.

Local Event Logs Input Configuration: The Splunk setup for local event logs was successfully created. This means Splunk is set up to index logs from the local system.

Skip

Skip tour

Step 11: Verifying Log Collection

To ensure that Splunk is receiving data from the Universal Forwarder, you can run the following search query in Splunk’s Search & Reporting interface:This query will show Windows Event Logs, confirming that the Universal Forwarder is working correctly and sending logs to Splunk.

1. Open the Splunk Web Interface on your Kali machine (were Splunk was installed) by going to http://<kali_IP>:8000.
2. Log in with your Splunk admin credentials.

1. Go to Settings > Data Inputs to check if the logs from your Windows machine are being received.
2. You should see the logs coming in through the port 9997.

index=win_log sourcetype=WinEventLog

Conclusion

Congratulations! You’ve successfully installed and configured Splunk Enterprise and Splunk Universal Forwarder in your home cyber lab. Now, you can start monitoring and analyzing log data from various systems in real-time. This setup will help you improve your security posture and performance monitoring, as well as provide insights into system activity.