Modern web applications generate HTTP requests, responses, application events, and security-related logs during normal operation and security testing. These records can contain useful information about SQL Injection (SQLi) attempts, suspicious request parameters, database-related errors, unusual response patterns, and other activities associated with potential application attacks.
SQL Injection is a web application attack in which an attacker attempts to manipulate database queries through application inputs. If an application does not handle user-controlled input securely, malicious input may affect database query execution and potentially expose or modify information.
During a SQL Injection attack, relevant evidence may appear across multiple records, including HTTP requests, application logs, security-tool alerts, and response information. Individually, these records may provide only partial information about the activity. Investigators must correlate the available evidence to understand the sequence of events and reconstruct the incident.
In this use case, OWASP Juice Shop is deployed within an isolated cybersecurity laboratory. Kali Linux is used as the controlled security-testing environment, while Ubuntu Linux hosts the application and log-analysis workflow. OWASP ZAP is used to inspect HTTP traffic and preserve relevant request and response evidence during authorized testing.
Existing public web-application security data is considered as an additional data source where suitable records and metadata are available. The dataset is inspected to determine whether its records can support the proposed log-processing and similarity-analysis workflow. The dataset name is not fixed in this use case; the source is selected according to data availability, suitability, and permitted usage.
Python is used to preprocess collected log records and prepare textual representations. TF-IDF converts suitable log text into numerical vectors, while Cosine Similarity measures the textual similarity between selected log entries. The resulting similarity scores are combined with available information, such as timestamps, request identifiers, source addresses, HTTP methods, request paths, and attack-test context, to identify potentially related events.
The correlated events are organized chronologically to reconstruct the observed SQL Injection test activity. SQLite stores the relevant log records, similarity results, correlation relationships, and incident evidence.
Gemma 2 2B Instruct is executed locally through Ollama to summarize the correlated evidence and generate a structured incident investigation report. The generated report is verified against the original logs and controlled testing evidence before its conclusions are accepted.
OWASP Juice Shop → Controlled SQL Injection Testing → HTTP Traffic and Log Collection → Existing Public Security-Data Review → Log Preprocessing → TF-IDF Feature Extraction → Cosine Similarity Calculation → Related Event Identification → Evidence-Based Event Correlation → SQLite Evidence Storage → Chronological Incident Reconstruction → Gemma 2 2B Instruct Analysis → Incident Summary Generation → Evidence Verification → Correlation Evaluation → Final Incident Report
Application: OWASP Juice Shop
OWASP Juice Shop is the target web application in this use case. It provides intentionally vulnerable application functionality for authorized web application security testing and security research.
The application is deployed within an isolated laboratory environment. Kali Linux is used to perform controlled security testing, while OWASP ZAP supports inspection of HTTP requests and responses associated with the application.
The application provides the testing environment in which SQL Injection-related requests and their associated evidence can be collected and analyzed.
SQL Injection testing can generate multiple HTTP requests and responses containing different inputs, response characteristics, and application events. Security tools may also produce alerts related to suspicious requests.
When these records are examined independently, investigators may find it difficult to determine which events belong to the same attack sequence. Similar requests may occur at different times, while a single request may generate multiple related records.
Manual investigation can require reviewing request parameters, timestamps, response information, and application logs separately. This can increase investigation effort and make it harder to reconstruct the sequence of a controlled security incident.
Traditional string matching may identify records containing identical text, but it may fail to associate records that contain related terms with different wording. Text-similarity techniques can help identify potentially related records, although similarity alone cannot establish that an attack occurred.
The proposed solution introduces a TF-IDF and Cosine Similarity-based log-correlation workflow to identify potentially related records. Event correlation uses additional available evidence to establish relationships between records and reconstruct the observed incident timeline.
Specific Attack: Controlled SQL Injection Attack Against OWASP Juice Shop
The controlled attack scenario evaluates whether HTTP traffic and related security evidence generated during a SQL Injection test can be collected, correlated, and reconstructed using the proposed similarity-based investigation workflow.
OWASP Juice Shop is deployed within the authorized laboratory environment. Kali Linux is used as the security-testing system, and OWASP ZAP is used to inspect the relevant HTTP communication.
A controlled SQL Injection test is performed against an appropriate test endpoint within the local application. The test is restricted to the laboratory environment and does not target external applications, production systems, or third-party data.
The resulting HTTP requests, responses, and available application or security-tool records are preserved as investigation evidence. The records are then processed using TF-IDF and Cosine Similarity to identify textual relationships between potentially related events.
The correlation results are combined with timestamps, request information, and other available evidence to reconstruct the observed test sequence.
Gemma 2 2B Instruct analyzes the resulting evidence and generates an incident summary. The summary is validated against the original records and known laboratory test conditions.
TF-IDF and Cosine Similarity-Based Security Log Correlation with LLM-Assisted Incident Reconstruction
TF-IDF and Cosine Similarity-Based Security Log Correlation with LLM-Assisted Incident Reconstruction
The first objective is to convert collected HTTP and security-related log text into numerical representations using TF-IDF. This technique assigns weights to terms based on their occurrence within a record and their distribution across the collection of records.
The second objective is to calculate Cosine Similarity between selected log representations. The resulting scores indicate how similar the textual representations are. These scores are used to identify potentially related records for further investigation.
The third objective is to combine similarity results with available contextual information, such as timestamps, request paths, HTTP methods, source information, and controlled test records. The resulting event relationships are organized chronologically to reconstruct the observed incident.
Gemma 2 2B Instruct then summarizes the correlated evidence, describes the observed sequence of events, and prepares a structured investigation report. Its conclusions are verified against the original records.
The security concept defines the overall security strategy.
The following are the specific defensive mechanisms implemented.
HTTP requests, responses, and available security-related records are collected during legitimate application activity and controlled SQL Injection testing.
Existing public web-application security data is reviewed to determine whether it contains records suitable for the proposed analysis.
Collected records are normalized into a consistent textual representation while retaining important contextual fields.
TF-IDF converts suitable log text into numerical vectors representing the relative importance of terms within the collection of records.
Cosine Similarity measures the similarity between TF-IDF vectors associated with selected log entries.
Potential relationships identified through similarity analysis are evaluated using additional available information, such as timestamps, request paths, HTTP methods, request identifiers, and attack-test context.
Correlated events are arranged chronologically using available timestamps and other reliable ordering information.
Relevant logs, similarity scores, correlation relationships, and incident findings are stored in SQLite.
Gemma 2 2B Instruct analyzes the correlated evidence and generates a structured summary of the observed incident.
The generated summary is compared with the original HTTP traffic, available application records, and controlled test information.
The correlation workflow is evaluated using known relationships from controlled test scenarios and manually verified evidence.
OWASP Juice Shop provides the web application environment used for controlled SQL Injection testing.
Kali Linux provides the controlled security-testing environment.
Ubuntu Linux hosts the application or analysis components according to the laboratory configuration.
VirtualBox provides the isolated laboratory environment.
OWASP ZAP is used to inspect HTTP requests and responses associated with the controlled testing activity.
Python provides the main implementation environment for the project.
NumPy supports numerical operations required by the text-analysis workflow.
Scikit-learn's TF-IDF Vectorizer converts suitable log text into numerical feature vectors.
Scikit-learn provides a function for calculating Cosine Similarity between numerical vectors.
Pandas is used to organize log records, contextual fields, and correlation results.
Ollama provides the local runtime for executing the selected language model.
Gemma 2 2B Instruct provides the LLM-assisted incident investigation capability.
SQLite provides local storage for security evidence and investigation results.
Python scripts and manually verified controlled-test evidence are used to evaluate the correlation workflow.
OWASP Juice Shop is successfully deployed within an isolated cybersecurity laboratory to provide a controlled environment for web application security testing.
The HTTP traffic inspection workflow is configured using OWASP ZAP, enabling relevant requests, responses, and available security evidence to be collected during normal application activity and controlled SQL Injection testing.
Existing public web-application security data is reviewed as an additional source where suitable records are available. The project does not depend on a predefined dataset name and documents the suitability of any selected external data source.
Collected HTTP and security-related records are preprocessed into a consistent textual representation while preserving the contextual information required for investigation.
TF-IDF converts the selected log text into numerical feature vectors, providing a representation suitable for textual similarity analysis.
Cosine Similarity generates scores that indicate the degree of textual similarity between selected log entries. These scores are used to identify candidate relationships for further investigation rather than independently confirming an attack.
The event-correlation workflow combines similarity results with available timestamps, request paths, HTTP methods, request identifiers, source information, and controlled test context to identify potentially related security events.
Correlated records are arranged chronologically to reconstruct the observed sequence of events associated with the controlled SQL Injection test. The resulting timeline retains references to the original records and distinguishes observed facts from inferred relationships.
SQLite stores the collected evidence, similarity scores, event relationships, incident timeline, and investigation results in a structured format.
Gemma 2 2B Instruct analyzes the correlated evidence through Ollama and generates a readable incident summary containing the observed activity, supporting evidence, possible explanations, and recommended investigation steps.
The generated incident report is verified against the original HTTP traffic and available application or security-tool records, ensuring that unsupported statements are identified before the report is accepted.
Repeated testing with normal application activity and controlled SQL Injection scenarios provides evidence for evaluating the correlation workflow, including false correlations, missed relationships, timeline quality, and report accuracy.
The complete workflow of application deployment, controlled security testing, log collection, public-data assessment, TF-IDF feature extraction, Cosine Similarity calculation, contextual event correlation, incident reconstruction, evidence storage, LLM-assisted summarization, and final validation is demonstrated.
The final project provides a practical AI for Cybersecurity implementation focused on similarity-based security log correlation and LLM-assisted incident reconstruction. Its results demonstrate how text similarity and evidence-based analysis can assist incident investigation, while documenting the limitations of textual similarity and the need for contextual validation.
Research Breakthrough Possible @S-Logix