Web servers process HTTP requests from legitimate users, applications, and other clients. These requests generate access logs and, when relevant events occur, error logs that contain information about request methods, requested resources, response status codes, timestamps, client addresses, and server-side errors.
Individual log entries may provide useful information about a particular request, but identifying a multi-stage cyberattack requires examining how suspicious events occur over time. For example, an attacker may first perform reconnaissance, then send unusual requests to identify accessible resources, and subsequently attempt to access a restricted or vulnerable application function. When these events are investigated independently, their relationship may not be immediately apparent.
Traditional security monitoring may use predefined rules, individual suspicious indicators, or known attack signatures to identify potentially malicious requests. Such methods can be useful, but they may not adequately represent the temporal relationships between different stages of an attack. A sequence-based analysis approach can provide an additional method for investigating relationships between security events.
In this use case, an Apache HTTP Server is deployed on Ubuntu within an isolated cybersecurity laboratory. Legitimate web activity and controlled attack-related requests are generated from Kali Linux against the authorized laboratory environment. The Apache access and error logs are collected and processed to create time-ordered event sequences.
An existing public web-server security-log dataset is used as the external data source for the machine-learning experiment. The available records are reviewed, and suitable event information is extracted to establish the training and evaluation data. The dataset is not manually created for the project. The controlled attack logs generated in the laboratory are used for validation against the known attack timeline.
A Hidden Markov Model (HMM) is used to analyze sequences of observed security events and estimate patterns associated with different stages of activity. The model is evaluated using appropriately prepared event sequences, and its results are compared with the known stages of the controlled attack. The model's output is treated as an investigative indicator rather than definitive proof that an attack succeeded.
Application: Apache HTTP Server on Ubuntu
Apache HTTP Server is the web-server application used in this use case. It receives HTTP requests and records request-related information in its access logs. Relevant server-side problems and errors may also be recorded in its error logs.
The application is deployed within an isolated laboratory so that legitimate web activity and controlled suspicious requests can be generated, collected, and analyzed without targeting external systems.
A multi-stage cyberattack may produce a series of requests rather than a single clearly malicious event. An attacker might begin by examining available web resources, continue by requesting unusual paths or application functions, and then attempt to access a restricted resource.
Individual requests may appear similar to ordinary web traffic. A single HTTP response code or unusual URI does not necessarily establish that an attack is occurring. However, the order, timing, and relationship between several events may provide additional evidence for security investigation.
Rule-based monitoring can identify predefined indicators, but it may have difficulty recognizing relationships between events when the sequence differs from expected patterns. Security analysts therefore need a method to investigate the temporal structure of observed web-server events.
The proposed solution processes existing public web-server security-log data and laboratory-generated Apache logs into time-ordered event sequences. An HMM is then evaluated for its ability to identify sequence patterns associated with controlled multi-stage attack activity.
Specific Attack: Controlled Multi-Stage Web Attack Simulation Against an Apache HTTP Server on Ubuntu
The attack scenario evaluates whether a sequence-based analysis mechanism can identify suspicious relationships between multiple stages of web activity recorded by Apache.
The experiment is performed entirely within an authorized VirtualBox laboratory containing Ubuntu and Kali Linux. Normal requests and controlled suspicious requests are generated against the local web-server environment. A deliberately vulnerable local test application may be used where necessary to provide realistic application-level testing scenarios.
The attack scenario consists of controlled reconnaissance, unusual resource requests, and an authorized test of a deliberately vulnerable application function. The resulting events are recorded and compared with the known experimental timeline.
The experiment does not require attacking a public server, deploying real malware, or establishing command-and-control communication.
The primary security concept is Sequence-Based Cybersecurity Event Analysis Using a Hidden Markov Model.
The first objective is to establish a structured representation of web-server events. Relevant fields from the public security-log dataset and the laboratory-generated Apache logs are processed into event categories suitable for sequence analysis.
The second objective is to analyze the temporal relationships between observed events. A Hidden Markov Model represents a sequence using hidden states and observable events. The hidden states are not directly visible in the logs; the model estimates them from the observed sequence according to its training and configuration.
For this project, the analysis may investigate whether event sequences are consistent with ordinary web activity or with patterns associated with the stages of the controlled attack. The hidden states must be interpreted and validated against the available evidence rather than automatically assumed to represent specific attack stages.
The third objective is to evaluate the model against a controlled attack whose sequence and timing are known. The public dataset is used as the existing external data source, while the laboratory-generated requests provide additional validation evidence. The training, validation, and test data must be separated appropriately to avoid misleading results.
Security-Log Collection
Apache access and error logs are collected during legitimate and controlled suspicious activity.
Consistent Log Preprocessing
The public dataset and laboratory-generated logs are processed into compatible event representations. Available timestamps, request methods, URI characteristics, response codes, and other relevant fields are handled consistently.
Temporal Event Sequencing
Log records are ordered by timestamp and transformed into event sequences. Events are grouped according to the experiment's defined session or time-window rules.
Hidden Markov Model Analysis
The HMM analyzes the observed event sequences using the states, transition probabilities, and observation probabilities established during training or configuration.
Attack-Chain Correlation
The analyzed sequences are compared with the known timeline of the controlled attack and the corresponding captured HTTP requests.
Detection Performance Evaluation
The results are evaluated using appropriately separated test sequences and suitable metrics, such as precision, recall, F1-score, and a confusion matrix when reliable labels are available.
Evidence Preservation
Relevant log records, event sequences, model predictions, timestamps, and validation results are preserved for review.
Apache provides the target web-server environment for generating legitimate and controlled suspicious HTTP traffic.
Ubuntu hosts the Apache web server and supports the log-collection and analysis workflow.
Kali Linux provides the client environment used to generate legitimate and controlled suspicious requests against the laboratory server.
cURL sends HTTP requests to the laboratory web server and records the responses returned by the application.
OWASP ZAP supports inspection of HTTP communication between the testing client and the local web application.
Wireshark captures and inspects network traffic within the authorized laboratory network.
Python provides the implementation environment for log parsing, event categorization, sequence preparation, HMM processing, and evaluation.
Pandas loads, filters, transforms, and organizes log records into structured datasets.
NumPy supports numerical operations required by the event-processing and model-analysis workflow.
The Python library hmmlearn provides implementations of Hidden Markov Models suitable for sequence-based experiments.
Matplotlib visualizes selected event patterns, sequence scores, and evaluation results.
VirtualBox hosts the isolated Ubuntu and Kali Linux virtual machines used for the experiment.
An isolated cybersecurity laboratory is established using VirtualBox, Ubuntu, and Kali Linux, providing a controlled environment for generating legitimate and suspicious web activity.
Apache HTTP Server is deployed on Ubuntu, and its access and error logs are verified as the primary sources of web-server event evidence for the experiment.
An existing public web-server security-log dataset is reviewed and processed, providing the external data source for event extraction and HMM experimentation without manually creating the original training dataset.
A consistent log-preprocessing and event-categorization workflow is established to transform supported web-server fields into structured observations suitable for sequence analysis.
Time-ordered event sequences are prepared using documented timestamp, session, and grouping rules, preserving the relationships between relevant web-server events.
A Hidden Markov Model is configured and trained using the designated training sequences, establishing the sequence-analysis component for the controlled experiment.
Legitimate web activity and a predefined multi-stage attack scenario are generated against the authorized local environment, producing laboratory logs that can be correlated with the known test timeline.
The laboratory-generated Apache logs are processed using the established feature and sequence pipeline, allowing new event sequences to be evaluated consistently by the trained model.
The HMM outputs are compared with normal activity and the known controlled attack stages, providing evidence about whether suspicious event sequences can be identified under the experimental conditions.
The complete workflow of public security-log processing, event extraction, temporal sequence preparation, HMM training, controlled web-attack simulation, Apache log collection, sequence analysis, attack-timeline correlation, and detection evaluation is documented, including observed results, false positives, false negatives, and limitations.
Research Breakthrough Possible @S-Logix