Log Universe — TryHackMe Walkthrough

Explore log files from various systems and learn how to carve data to adopt a course of action!

Site Link: https://tryhackme.com/r/room/loguniverse

Task 1 Introduction

Introduction: Log Universe

In this room, you will learn and experience the various log types generated from different systems and carve data from the log files.

Learning Objectives

• Learn how to identify and use different log sources.
• Understand what to expect from different log types.
• Gain hands-on experience in data carving from the log files.

Room Prerequisites

• Working knowledge of MS Windows and Linux
• Working knowledge of network and endpoint log systems
• Intro to Logs
• Log Operations
• Windows Event Logs

Answer the questions below

Read the task above.

Task 2 Treasure Among the Lines: Logs

Logs

Logs are the footprints of digital components and invaluable tools and resources in the storytelling of systems, applications and networks. However, harnessing the power of logs is a challenging task. First, you need to understand the different types, formats, offerings and importance of logs. Then, you can start to uncover the insights, patterns and “treasures” among the lines!

Logs provide visibility, context, and carry evidence value in security operations, incident response, threat hunting, compliance, cyber resilience, and cyber maturity. Most importantly, the ability to correlate simultaneous and distinct events among distributed networks and systems is also achieved with the visibility and context gained by logs. No matter what kind of security operation is being implemented, the logs are one of the essential components.

Data Sources

There are various log sources to deal with, so categorising them is a great practice to start. Data sources can be divided into two main categories.

Physical

• Facilities/areas where critical systems are hosted.
• Security camera surveillance and access logs of physical entry and exit from:
• System control rooms
• Offices

Virtual

• Network Components
• Routers and switches
• Plug-in devices
• Operating Systems
• Servers
• Security Appliances
• Firewall
• Anti-virus/malware
• IDS/IPS
• DLP
• VPN
• Applications and Frameworks
• .Net
• Java
• PHP
• Python
• Mobile Devices
• Virtualisation and Cloud Components.
• Databases

Note that each data source produces different logs and serves a different purpose. The Intro to Logs room covers common log types, formats and standards. If you are unfamiliar with the details of logs, you can find more information in this room.

Data Specifications

We have learned that multiple data sources generate different logs for different purposes. However, to avoid log noise and to enable log management, correlation, and investigation, some “minimum qualification requirements” must be present in each log. Common base qualifications are listed below:

• The system which created the log.
• The log creation time (date, time and timezone).
• The event that caused the log to be created (message).
• The severity of the log.
• The source associated with the log (IP and port, MAC address, username, or system name).

Answer the questions below

Read the task above.

Task 3 Toolset and Hints

VM and Toolset

You have a custom Ubuntu machine with an open-source log parser tool (ULogViewer) and sample logs to investigate details and sample log files to discover each covered log file and practice.

Available Interface

• In-browser access

Datasets

• Multiple log file samples are available on the Desktop.

Log Details

• Windows, Linux and Apache logs.

Let’s start the Virtual Machine by pressing the Start Machine button at the top of this task. The machine will start in split-screen view. In case the VM is not visible, use the blue Show Split View button at the top-right of the page.

There are a variety of tools used in log analysis and discovery phases. Each tool has advantages and disadvantages, so the tool preference varies between analysts and the environment. One of the most important details of the log analysis tool is flexibility, compatibility, and support for parsing multiple log formats. We will use “ULogViewer” to parse and analyse various log file types in this room. Details are listed below.

Refer to the following explanations in case you opened the application and couldn’t select a profile and ended up with an empty screen similar to the one shared above.

1. Select the log profile to be parsed.
2. Add the log file to be parsed.
3. Filter logs and perform search.
4. Delete the loaded logs from the UI.
5. Open a new tab for a different log source.

In addition to the processes discussed above, the image below shows what to expect after data parsing and viewing the log message with filter options.

Answer the questions below

Read this task and proceed to the next one.

Task 4 Zoom In: Windows Event Logs

Windows Event Log Anatomy

Windows event logs provide in-depth footprint information on the system, security, and applications installed on a Windows operating system. Windows provides a generous amount of logs, and you will need to activate them according to your visibility needs and capacity. Remember, the logging scope is fully configurable, and the default settings are not enough for the current state of the threats. Being comfortable with logs is a vital skill, but it is also important to have the general characteristics before deep diving into each log source’s details. Now, let’s take a look at the Windows event log characteristics. Below are the main specifications of the Windows event logs.

Format
/
Extension

• Default format/extension:
• .evt
• .evtx
• Can be extracted in the following formats:
• .xml
• .csv
• .txt

File Location

• The log files’ directory can differ according to the Windows OS version. MS Windows changed the event log file directory/location with Windows Vista.
• 2003 and earlier versions:
• C:\WINDOWS\system32\config
• Vista and newer:
• C:\WINDOWS\System32\winevt\Logs

View Tool

• Default:
• Event Viewer
• Alternative tools can also be used.

As explained in the table above, Windows event logs can be generated and stored in a variety of extensions and formats. The alternatives here provide great flexibility in the logging and analysis process, allowing the environment to be painlessly set up for the analysis phase and to take advantage of alternative components.

After getting familiar with the log file formats, it is timesecu to identify the event log file categories. By default, there are two main log categories in Windows. The details are shown in the table below.

Category

• Windows Logs

Image

Details

• Built-in Windows event logs.
• The main focus is system-level activities.
• Not customisable.
• Gives insights into the overall system.
• Applications and Services Logs

Image

Details

• More specific logs on individual applications and services.
• Applications and services can create their custom log formats.
• Gives insights on particular applications and services.

We’ve already mentioned that Windows systems are very generous when it comes to logging. Windows systems can generate hundreds of logs for the two categories mentioned above. Some are commonly used, and others need to be activated by configuration. Now, let’s look at common log files and find out what information they provide! The table below highlights the commonly used event logs.

The following table breaks down the anatomy of the event logs and explains the main elements. In case you can’t recall what a Windows event Log file looks like, use the given hints and image.

Click here to view the hints and images.

You can open and view the details of the Windows event log files with the “Event Viewer” tool. Details can be seen in multiple formats, as shown in the images. The first image shows the default view option, and the second image shows the detailed view option.

Note: This room focuses on the log file anatomy, so if you are unfamiliar with the given formats and details, please visit the Windows Event Logsroom.

Now, let’s cover the main parts of the log file.

Log Name

• Log Channel
• Application, System, Security, etc.

Source

• The source which generated the event.
• It could be an application or a component.

Event ID

• A number that identifies the event type and provides more insights.

Level

• The severity of the event
• Information: Events without issues (success).
• Error: Issues in the system or service.
• Warning: Potential problem.
• Critical: Significant issue.
• Verbose: Progress or success messages.

User

• The name of the user account that triggered the event.

Logged
(Event Date/Time)

• Date and time information when the event occurred.

Task Category

• The type of the event log that highlights the purpose of the log.
• Common categories:
• Process Creation
• Service Creation
• Log Clear

Keywords

• Flag/attribute that categorises events.
• Standard keywords are defined in Windows OS resources.
• None: No filtering is performed.
• Audit Failure: Security audit fail event.
• Audit Success: Security audit success event.

Computer

• The hostname of the system.

Message

• Log message details.

Remember that you will use the ULogViewer to view and filter the logs. The sample log is parsed as shown in the figure below. Field details are opened by double-clicking on the field or clicking on the “view more” section.

PID and TID data are beneficial for process tracing, correlation, and understanding the natural flow of events during log analysis. The timestamp also plays an essential role in determining which processes and threads were running at a particular time or during the workflow and, if applicable, the call times of the child processes and threads created by the user or execution flow.

Learning the anatomy of the Windows log files is essential, but you will also need to know the indicative event log IDs and details. Therefore, you will be able to track and understand the details of a potential anomaly or an intrusion attempt. Some useful Windows event log IDs are listed below.

Event Category: Event ID and DetailAccount Management

• 4720: User account creation.
• 4722: User account enabled.
• 4723: Attempt to change an account password.
• The user attempts to change their password.
• 4724: Attempt to reset the account password.
• The user attempts to reset the password of another account.
• 4725: Account disable.
• 4726: Account removal.

Account Logon

• 4624: Successful logon.
• 4625: Failed logon.
• 4634 and 4647: Logoff.
• 4779: Session disconnect.

Scheduled Tasks

• 4698: Scheduled task creation.
• 4702: Scheduled task updated.
• 4699: Scheduled task deletion.

Security

• 1100: Logging service disabled.
• 1102: Log deletion.
• 1116: Malware detection.

Note that each source produces different logs for different purposes. The Intro to logs room covers common log types, formats and standards. You will find more information in this space if you are still getting familiar with logs. You can also visit the Windows Event Logs and Sysmon rooms for more details about the event you are interested in.

Now, switch to the given VM and analyse the “Windows Questions” log file using ULogViewer.

Answer the questions below

4.1 What is the Thread ID of the user creation event?

Event ID 4720 corresponds to a user account creation, as per standard Windows Event IDs. In the provided ULogViewer screenshot, the row with event ID 4720 shows a Thread ID (TID) of 744.

Answer: 744

4.2 What is the account name that creates the new user? (Question Hint The “SubjectUserName” field highlights the account name that created the event.)

Answer: Administrator

4.3 What is the name of the created account name? (Question Hint The “TargetUserName” field highlights the account name affected by the event. Attackers can create accounts to maintain access to victim systems. They deliberately make subtle spelling mistakes to prevent or delay the detection of the accounts they create by making them look legitimate. Since the difference between a legitimate and a fake name is usually a single letter or symbol, it can be challenging to spot these account services at first glance unless you look closely. MITRE ATT&CK ID: T1136, Detection: DS0002.)

Answer: Adminstrator

4.4 What is the “SubjectLogonID” value of the “account reset attempt” event?

Answer: 0x4B666

Task 5 Zoom In: Linux Logs

Linux Logs Anatomy

Like Windows event logs, Linux logs provide in-depth footprint information on the system, security, and applications installed. Again, the logging scope is fully configurable, and the default settings are not enough for the current state of the threats. Let’s take a deep dive into the Unix-like system logs!

Format
/
Extension

• Default format/extension:
• .log
• Syslog stores logs in cleartext format.
• Systemd and Journald stores logs in binary format.

File Location

• The directory of the log files can differ according to the used services and configuration file. Still, most Unix-like operating systems share the same directory to store the log files.
• /var/log

View Tool

• Default:
• System Log Viewer
• Alternative tools and utilities can be used to view cleartext log files.
• cat, tail, more, less, grep, awk
• lnav
• Some log files can be read only with the corresponding binary.
• journalctl, who, faillog, lastlog, dmesg

Common Logs Generated by Unix-like Systems

Cleartext Formatted Logs

Cleartext logs can be viewed and analysed using multiple tools, as highlighted in the above table.

• General system logs
• /var/log/syslog
• /var/log/messages
• System boot logs
• /var/log/boot.log
• Background daemon logs
• /var/log/daemon.log
• Authentication logs
• /var/log/auth.log
• /var/log/secure
• Kernel logs
• /var/log/kern.log
• Audit logs
• /var/log/audit
• Cron daemon logs
• /var/log/cron.log
• Debug logs
• /var/log/debug
• Apt command logs
• /car/log/apt/

Logs Require Specific Binary to View

Some binary formatted logs can be viewed and analysed only with the corresponding tool.

• Failed logins only
• /var/log/faillog
• Command: faillog
• Active login records
• /var/log/wtmp
• Command: who
• Last logins
• /var/log/lastlog
• Command: lastlog
• Kernel ring buffer logs
• /var/log/dmesg
• Command: dmesg

The Syslog log file structure is defined by RFC 5424–5426. Also, it is widely used and easy to process with native granular system tools like tail and grep. The structure details are explained in the following table.

Timestamp

• Date and time information when the event occurred.

Hostname

• The machine name that created the message.

Application / Process

• The application that created the message.

Process ID

• Process ID number of the application that created the message.

Message ID

• Message ID that identifies the type of message.
• Example: Inbound TCP connection: TCPIN

Message

• Message itself.
• Encoded in UTF-8

While RFC defines the main structure, there are alternative implementations of message presentations. The common structure (IETF) is shown below.

• IETF log format:

timestamp hostname process[pid]: message

Sample Log: Syslog
Case: Command Execution

• Example log in IETF format:

2023-09-28T15:05:55.333333Z TempServer sudo[2345]: [AUTH] User alice executed command '/usr/bin/apt-get update'.

• Evaluation:

The above log entry showcases an event where the user “Alice” executed the command ‘/usr/bin/apt-get update’ using the ‘sudo’ command on the server “TempServer”. This log entry provides valuable insights into user actions, including the executed command and authentication details, which can aid in auditing and monitoring system activities.

As the previous example shows, reading and understanding a single log entry in a structured format can be relatively straightforward. However, the challenge increases when dealing with voluminous log files containing numerous lines of entries. In such cases, extracting the relevant attributes and establishing correlations becomes essential to comprehensively understand specific events within the log data.

Analysing multi-line logs requires high attention to detail and the ability to filter and extract significant details. This process involves identifying and filtering timestamps, hostnames, process IDs, and event descriptions, among other critical details. By effectively analysing and correlating these attributes, it is possible to uncover the sequence of events, diagnose problems, and gain valuable insight into system behaviour, security incidents, or operational performance.

In summary, while individual log entries may seem manageable, the complexity arises when dealing with large log datasets. The magic lies in organising the extraction and analysis of key attributes to reveal the underlying story hidden within the logs, making log management and monitoring an essential part of maintaining system health and security.

The following log examples are a series of timestamped entries from “TempServer” that show critical system events and resource management. These logs provide insight into memory issues, process terminations, and system reboots. Each log entry includes essential information such as timestamp, server name (TempServer), log source (Kernel or Systemd), process IDs (PID), and detailed event descriptions. It’s important to note that these logs are presented in a generic system log format, with no specific categorisation, providing an insight into the server’s operational challenges and the need for memory management and system recovery.

Sample Log: Syslog
Case: Restart Due to Insufficient RAM

• Example log in IETF format:

2023-09-25T08:15:20.123456Z TempServer kernel: Out of memory: Killed process 5678 (myapp); system reboot required.
2023-09-25T11:30:45.987654Z TempServer kernel: Memory cgroup out of memory: Kill process 9876 (database) score 500 or sacrifice child.
2023-09-26T03:45:10.234567Z TempServer systemd[1]: Reached memory limit at process 1234 (myapp), restarting.
2023-09-26T12:20:35.543210Z TempServer kernel: Swap space exhausted; system restarting to free up memory.
2023-09-27T07:55:50.765432Z TempServer systemd[1]: Server running low on memory, initiating reboot for recovery.

• Evaluation:

The above log entries showcase events on “TempServer” that include crucial memory-related events. It shows that processes with IDs “5678” (myapp), “9876” (database), and “1234” (myapp) caused memory problems and resulted in process terminations, memory limit violations, swap space exhaustion, and low memory warnings, ultimately requiring reboots for recovery and stability.

Now, switch to the given VM and analyse the “Linux” log file using ULogViewer. Remember that you will use the ULogViewer and select the “Linux System Log Files” profile to view and filter the logs. The sample log is parsed as shown in the figure below.

Answer the questions below

5.1 What is the number of successful sync events done by the NTPD service?

Use ULogViewer to load the “Linux” log file.

In the filter tab at the top, type “synchronized” to filter out all log entries that contain this keyword.

This action will display only the logs related to synchronization events performed by the NTPD service.

at the bottom, indicating that there are 28 occurrences of logs containing the word “synchronized.”

This means that there were 28 successful synchronization events performed by the NTPD service.

Answer: 28

5.2 Which user logged in using the SSHD service?

In the filter tab at the top, type “sshd” to filter out all log entries that contain this keyword.

In the logs, the user “THMjohn-p” appears multiple times, indicating successful SSH logins

Answer: THMjohn-p

5.3 What is the PID number of the Apache web server?

In the filter tab at the top, type “apache” to filter out all log entries that contain this keyword.

Answer: 5678

5.4 Which service is stopped due to RAM issues?

In the filter tab at the top, type “ram” to filter out all log entries that contain this keyword.

Answer: nginx

5.5 Which service is stopped due to CPU issues?

In the filter tab at the top, type “cpu” to filter out all log entries that contain this keyword.

Answer: Apache Tomcat

5.6 What is the timestamp of the second time reset event? (Question Hint

Whole timestamp data)

In the filter tab at the top, type “reset” to filter out all log entries that contain this keyword.

Answer: 03/27 15:51:56

Task 6 Zoom In: Misc Logs

Misc Logs: Application Logs

Misc logs provide in-depth footprint information on application-based events, giving more insights on application and process-based details that will help analysts in security operations, including monitoring, threat hunting, and incident response. This task will cover the details of Apache logs.

Apache Logs

Format
/
Extension

• Default format/extension:
• .log

File Location

• The directory of the log files can differ according to the used services and configuration file. Still, most Unix-like operating systems share the same directory to store the log files.
• Main directory
• /var/log/apache2/
• Access logs
• /var/log/apache2/access.log
• Error logs
• /var/log/apache2/error.log

View Tool

• Default:
• System Log Viewer
• Alternative tools and utilities can be used to view cleartext log files.
• lnav
• cat, tail, more, less, grep, awk

Apache Logs: Access.log

Access logs are invaluable records generated by web servers, containing essential attributes that form the backbone of effective log analysis. These attributes, including IP addresses, timestamps, HTTP methods, URLs, status codes, and user agent information, play a vital role in web server management and security.

These attributes enable administrators and analysts to ensure server health, diagnose problems, detect security threats, and optimise web services for a seamless user experience. Understanding the importance of these attributes is essential before embarking on log analysis, as it provides the baseline for informed decision making and effective web server management.

The anatomy of the Apache access.log’s “Combined log format” is detailed in the given table.

IP

%h

• IP address of the client (requester).

No Info

%l

• It represents the “hyphen” (-).
• This appears when the requested information is not available.

User ID

%u

• User id information.
• This is based on HTTP authentication, so if the request code is 401 (auth fails or still needs to be done), this information is unreliable.

Timestamp

%t

• Time information when the request is received.

Request

\”%r\”

• The request.
• Logged in double quotation.

Status

%>s

• Status code that is sent back to the client.

Object Size

%b

• Size of the returned object.
• It doesn’t include response headers.

Referer

\”%{Referer}i\”

• HTTP referer field that identifies the previous resource used to come to the current page.

User Agent

\”%{User-agent}i\”

• The User-Agent field identifies the application and operating system which sends the request.

Apache access logs are identified by the “Common Log Format” and “Combined Log Format”. Both formats focus on visibility but serve different purposes. The standard format is more lightweight and focuses on the basics, capturing only the essentials such as IP, URL and response. The combined format captures more details, such as referrer and user agent data, which is useful for in-depth analysis. The combined log format structure is shown in the below section.

• Combined log format

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

Sample Log: Access Log
Case: Using HTTP POST Method

• Example log in combined log format:

192.168.1.100 - THMjohn [15/Nov/2025:10:30:45 -0500] "POST /api/data HTTP/1.1" 404 1234 "https://www.LogUniverse-THM.com/page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.0.0 Safari/537.36"

• Evaluation:

The above log file showcases a client IP 192.168.1.100 made a POST request to the “/api/data” URL on the server. It also shows that the server responded with a 404 “Not Found” status code. The client was referred by the “https://www.LogUniverse-THM.com/page" using the Mozilla web browser on the Windows 10 operating system.

Note that the web browser and operating system information can be manipulated or changed by an attacker crafting custom packets or using an attack or audit tool.

The sample log is parsed as shown in the figure below.

Apache Logs: Error.log

Error logs are an essential component of web server management, providing critical insight into system health and potential issues that can impact server performance and user experience. These logs capture essential attributes such as timestamps, error messages, file paths, and originating IP addresses. These attributes are essential for administrators and analysts to diagnose and resolve errors, identify vulnerabilities, and maintain server security. Before delving into error log analysis, it is vital to understand the importance of these attributes as they provide the foundation for proactive problem resolution and continuous improvement of web server stability and reliability.

The anatomy of the Apache error.log is detailed in the given table.

Timestamp%t

• Time information when the request is received.

Level%l

• Log level of the message.

Process ID
%P

• Process ID if the current process.

Source File%F

• Source file and line number of the log call

Error Status%E

• Error status code.

Client IP%a

• Client’s IP and port info.

Actual Log Message%M

• The actual log message.

Unlike access.log, error.log is identified under a single format named “ErrorLogFormat” and focuses on providing additional information to the log message. The structure is shown in the below section.

• ErrorLogFormat:

ErrorLogFormat "[%t] [%l] [pid %P] %F: %E: [client %a] %M"

Sample Log: Error Log
Case: Resource Availability

• Example log in ErrorLogFormat:

[Thu May 12 08:28:57.652118 2011] [core:error] [pid 8777] [client ::1] File does not exist: /usr/local/apache2/htdocs/favicon.ico

• Evaluation:

The above log file showcases a loopback address ::1, which means an internal request made to the resource favicon.ico located on the same server. The message part highlights that the directory /usr/local/apache2/htdocs doesn't contain the requested file.

This error is valuable as it shows a missing resource on the server. This can help administrators to identify and fix the resource availability issue. Note that ::1 is a loopback address 127.0.0.1 in IPv6.

Now, switch to the given VM and use the tools and datasets to answer the questions.

Remember that you will use the ULogViewer to view and filter the logs. The sample log is parsed as shown in the figure below. Note that you need to use the “Raw Text In Files” format for the Apache Error.log file!

Answer the questions below

6.1 Use the Access.log file to answer the first few questions.
What is the user’s IP address who accessed “/secure.html”? (In defanged format)(Question Hint CyberChef can defang).

In the filter tab at the top, type “secure” to filter out all log entries that contain this keyword.

Open CyberChef and in the “Operations” panel, search for “Defang IP” and select the “Defang IP Address” operation, after paste the IP address into the input field and convert the IP address to its defanged

Answer: 203[.]45[.]78[.]102

6.2 Which user failed to access the settings page?

In the filter tab at the top, type “settings” to filter out all log entries that contain this keyword, for attempts to access the settings page

Answer: buyer986

6.3 Which user accessed the malicious page? (Question Hint Status codes are important to notice. Regex hint: (payload|\?|param|passwd|shell))

In the filter tab at the top, type “200” to filter out all log entries that contain this keyword, indicating a successful request

The relevant log entry shows a successful GET request for “/malware.php?payload=1”, with a 200 status code, indicating that the request was successful.

Answer: adv8779

6.4 What is the user agent that discovered the malicious page?

Answer: nikto/2.1.5 (OpenVAS)

6.5 Use the Error.log file to answer the rest of the questions.

What is the PID of the process that causes permission error?

In the filter tab at the top, type “permission” to filter out all log entries that contain this keyword

Answer: 7654

6.6 What is the request that contains an invalid method?

In the filter tab at the top, type “invalid” to filter out all log entries that contain this keyword

Answer: \x80\x03\x01\x00\x01

6.7 What pattern match triggered the access error in ModSecurity?

In the filter tab at the top, type “ModSecurit” to filter out all log entries that contain this keyword

Answer: “SELECT.+FROM”

6.8 What is the path value of the file that tries to remove data from the system?

In the filter tab at the top, type “rm” to filter out all log entries that contain this keyword

Answer: /etc/httpd/conf.d/malicious.conf

Task 7 Conclusion

Congratulations!

You just finished the “Log Operations” room. In this room, we dived deep into Windows, Linux and Apache log file configurations attributes and discovered the data carving points for in-depth log analysis operations by covering:

• Windows event log structure
• Linux syslog structure
• Apache access and error log structure

Now, you have a solid understanding of the common log formats and are ready to carve data to adopt a course of action! So, it’s time to visit the Advanced Splunk and Advanced ELK modules to fire advanced queries focused on bits of the log files!

Answer the questions below

Click and continue learning!