Web applications use HTTP requests to provide authentication, data retrieval, user interaction, and other application functionalities. These requests contain behavioral characteristics such as HTTP methods, URI structures, request parameters, header information, and payload characteristics that can be analyzed for security monitoring.
If malicious requests exhibit behavioral characteristics that differ from normal application traffic, detection mechanisms based only on predefined signatures or known attack patterns may fail to identify certain variations. Behavioral anomaly detection provides an additional approach by learning the characteristics of normal HTTP requests and identifying requests that significantly deviate from the established behavioral pattern.
In this use case, OWASP WebGoat is deployed within an isolated laboratory environment. WebGoat is an intentionally vulnerable web application designed for security education and controlled application security testing. It provides a suitable environment for generating legitimate requests and testing a designated SQL injection vulnerability scenario.
A public HTTP request dataset is used as the existing data source for the machine-learning stage. Normal HTTP request records are selected and processed to establish the behavioral characteristics required for model training. The dataset is obtained from an existing public source rather than manually created for this project.
Isolation Forest is used as the machine-learning algorithm because it can identify observations that differ from the learned normal behavioral population. The model is trained using the selected normal HTTP request behavior and is not directly trained on the controlled SQL injection requests used during validation.
A controlled SQL injection test is then performed against the designated WebGoat lesson in the local laboratory. The resulting HTTP request is captured and processed using the same feature-extraction procedure used for the training data. The trained Isolation Forest model evaluates the request and generates an anomaly prediction and anomaly score.
The purpose of this experiment is to evaluate whether the behavioral characteristics of controlled SQL injection requests differ from the learned normal HTTP request population. Isolation Forest identifies anomalous behavior; the security tester must correlate the model output with the known test request to determine whether the anomaly corresponds to the controlled SQL injection activity.
Public HTTP Request Dataset → Normal Request Selection → Data Preprocessing → Behavioral Feature Extraction → Isolation Forest Training → Normal HTTP Behavior Learning → OWASP WebGoat → Controlled SQL Injection Test → HTTP Request Capture → Feature Extraction → Isolation Forest Analysis → Anomaly Identification → SQL Injection Test Validation
Application: OWASP WebGoat
OWASP WebGoat is the target web application in this use case. It provides intentionally vulnerable web functionality for learning, security testing, and controlled experimentation.
The application is deployed within an authorized, isolated laboratory environment. The designated SQL injection lesson is used to generate controlled test requests and evaluate the behavioral anomaly-detection workflow.
Web applications frequently accept user-supplied values through HTTP request parameters and other input fields. If application code handles these values unsafely, an attacker may attempt to manipulate database queries by supplying SQL syntax through an input field or request parameter.
Traditional detection mechanisms may rely on predefined signatures, known SQL injection patterns, or specific malicious strings. Such approaches may not identify every variation of an attack, particularly when a request uses an unfamiliar payload or differs from previously known patterns.
Behavioral anomaly detection provides an additional layer of analysis by evaluating the characteristics of incoming requests and identifying deviations from the learned normal request population.
The proposed solution processes an existing public HTTP request dataset, selects normal request behavior, and trains an Isolation Forest model using the extracted behavioral features. Newly captured WebGoat requests are processed through the same feature-extraction workflow and evaluated by the trained model.
Specific Attack: Controlled SQL Injection Attack Against OWASP WebGoat
The controlled attack scenario evaluates whether an HTTP request containing SQL injection test input exhibits behavioral characteristics that differ from normal HTTP requests.
The test is performed only against the designated vulnerable lesson in the local WebGoat application. The objective is to generate a controlled request that can be captured, analyzed, and compared with the normal request behavior learned by the machine-learning model.
The experiment does not assume that every SQL injection request will automatically be classified as anomalous. The detection outcome depends on the features extracted from the requests, the normal training data, and the Isolation Forest model configuration.
Behavioral Anomaly Detection and Web Application Security Testing
The primary security concept is Behavioral Anomaly Detection combined with controlled Web Application Security Testing.
The first objective is to establish a behavioral representation of normal HTTP request activity using existing public request data. Relevant request characteristics are processed into numerical features that can be evaluated by a machine-learning algorithm.
The second objective is to identify HTTP requests that differ from the learned normal behavioral population. Isolation Forest evaluates the processed features and generates anomaly predictions and scores for new observations.
The third objective is to validate the model's output against a known SQL injection test performed in WebGoat. This comparison helps determine whether the model identifies the controlled attack requests as anomalous and reveals the limitations of using behavioral features for attack detection.
The security concept defines the overall security strategy.
The following are the specific defensive mechanisms implemented in this use case.
Normal HTTP request characteristics are processed to establish the behavioral representation required by the anomaly-detection model.
Isolation Forest evaluates request features and identifies observations that differ from the learned normal behavioral population.
The same preprocessing and feature-extraction procedure is applied to the public training data and newly captured WebGoat requests.
HTTP requests generated during normal application activity and controlled SQL injection testing are captured for behavioral analysis.
The model's anomaly predictions are compared with the known SQL injection test requests and their captured application responses.
Normal and controlled attack requests are compared against the model predictions to identify correct detections, false positives, and missed attack requests.
OWASP WebGoat provides the intentionally vulnerable web application environment used to generate normal HTTP activity and controlled SQL injection test requests.
Python provides the primary environment for dataset processing, feature extraction, model training, anomaly scoring, and result analysis.
Scikit-learn provides the Isolation Forest implementation used for behavioral anomaly detection.
Pandas is used to load, filter, transform, and organize the existing public HTTP request records.
NumPy supports the numerical transformations required during feature preparation and model processing.
OWASP ZAP is used to inspect and capture HTTP communication during the controlled WebGoat security assessment.
cURL is used to generate and reproduce authorized HTTP requests against the local application where appropriate.
Kali Linux provides the controlled security-testing environment used to interact with the WebGoat application and inspect HTTP traffic.
Ubuntu provides the local environment for running WebGoat and the machine-learning workflow, depending on the laboratory deployment configuration.
VirtualBox provides the isolated laboratory environment used to host the application and security-testing systems.
An existing public HTTP request dataset provides the request records used to establish the normal behavioral population for model training.
Matplotlib is used to visualize anomaly scores and summarize the model's evaluation results.
OWASP WebGoat is deployed within an isolated laboratory environment and provides the intentionally vulnerable application required for controlled SQL injection testing.
The application network exposure is identified and validated within the authorized laboratory, establishing a controlled environment for application security experimentation.
An existing public HTTP request dataset is processed without manually creating a training dataset, providing the request records required to establish the normal behavioral population for the machine-learning experiment.
Normal request characteristics are transformed into numerical behavioral features, creating a consistent representation suitable for Isolation Forest analysis.
The Isolation Forest model is trained using the selected normal request behavior and establishes a behavioral baseline without directly training on the controlled SQL injection request used for validation.
Normal WebGoat application requests are captured and processed using the same feature-extraction workflow, providing a practical assessment of the detection pipeline's compatibility with the target application.
A controlled SQL injection test is performed against the designated WebGoat lesson, and the resulting HTTP request is captured for behavioral analysis.
The captured test request is processed and evaluated by the trained Isolation Forest model to determine whether its behavioral characteristics differ from the learned normal request population.
The anomaly prediction is correlated with the known controlled test activity, and the results are reviewed for correct detections, false positives, and missed attack requests. The experiment evaluates anomaly detection rather than assuming that every SQL injection request will be identified.
The complete workflow of public HTTP request-data processing, normal behavior learning, Isolation Forest training, WebGoat testing, controlled SQL injection request capture, behavioral feature extraction, anomaly scoring, attack correlation, and final security validation is demonstrated within the isolated laboratory environment.
Research Breakthrough Possible @S-Logix