Location Research Breakthrough Possible @S-Logix pro@slogix.in

Office Address

Social List

Service Mesh-Based Secure Communication for a Cloud-Native Distributed Research Data Management Application

Description

This project focuses on building a cloud-native distributed Research Data Management Application using multiple microservices. The application allows researchers to upload, validate, store, process, search, retrieve, and share research datasets. Services such as Dataset, Metadata, Validation, Processing, Search, Storage, and Notification communicate with each other through Istio Service Mesh. Istio secures this service-to-service communication using mTLS and provides traffic management, retries, timeouts, and monitoring.

Aim

To implement a secure, reliable, and manageable communication layer between research data management microservices using a service mesh architecture.

Objectives

01 Containerize research data management services using Docker.
02 Deploy and manage services using Kubernetes.
03 Implement Istio Service Mesh for service-to-service communication.
04 Secure communication using mTLS.
05 Configure traffic routing, retries, and timeouts.
06 Monitor service communication and performance.
07 Protect research datasets and metadata through controlled service access.

Application Workflow

01

Stage 1 – Researcher Login

Process

A researcher or administrator logs into the Research Data Management Application.

Tools
Keycloak Kong Kubernetes
Implementation

The researcher enters login credentials. Keycloak authenticates the user and provides an access token. Kong receives the request and routes it to the appropriate application service. The system verifies the user's role before providing access.

02

Stage 2 – Dataset Upload

Process

The researcher uploads a research dataset to the application.

Tools
Kong Kubernetes PostgreSQL
Implementation

The researcher selects a dataset and uploads it through the application. The Dataset Service receives the request and records the dataset information such as dataset name, researcher, file type, size, and upload date.

03

Stage 3 – Dataset Validation

Process

The uploaded dataset is checked before being made available for further processing.

Tools
Istio Kubernetes
Implementation

The Dataset Service communicates with the Validation Service through Istio. The Validation Service checks the dataset format, required metadata, file integrity, and basic data quality. The validation result is returned to the Dataset Service.

04

Stage 4 – Metadata Management

Process

Metadata is added and associated with the research dataset.

Tools
Istio PostgreSQL
Implementation

The Metadata Service communicates with the Dataset Service through Istio. Information such as dataset description, research category, creator, keywords, creation date, and data format is stored in PostgreSQL.

05

Stage 5 – Research Data Processing

Process

Researchers process datasets for analysis or further research activities.

Tools
Istio Kubernetes
Implementation

The Dataset Service sends processing requests to the Data Processing Service through Istio. The processing service performs operations such as data transformation, filtering, format conversion, and basic statistical processing.

06

Stage 6 – Dataset Search and Retrieval

Process

Researchers search for and retrieve available research datasets.

Tools
Istio PostgreSQL
Implementation

The Search Service receives the researcher's query and communicates with the Metadata and Dataset Services through Istio. The system searches metadata and returns matching datasets that the researcher is authorized to access.

07

Stage 7 – Dataset Sharing

Process

A researcher shares a dataset with another authorized researcher or research team.

Tools
Keycloak Istio PostgreSQL
Implementation

The system verifies the researcher's permissions using Keycloak. The Dataset Service communicates with the access-control-related services through Istio, and the sharing permissions are recorded in PostgreSQL.

08

Stage 8 – Notification

Process

Users receive notifications about important dataset activities.

Tools
Istio PostgreSQL
Implementation

The Dataset Service communicates with the Notification Service through Istio. Notifications can be generated for events such as successful dataset upload, validation completion, processing completion, or dataset sharing.

09

Stage 9 – Monitoring

Process

The system monitors the health and performance of research data services.

Tools
Prometheus Grafana Jaeger
Implementation

Prometheus collects service metrics, Grafana displays dashboards, and Jaeger traces requests across the research data microservices to identify delays and failures.

Cloud Infrastructure and Tools

Cloud Compute Cloud EC2

Provides virtual servers where the Kubernetes cluster and research data management microservices run.

Cloud Network Cloud VPC

Provides the private network for the Kubernetes cluster and research data services.

Operating System Ubuntu Server 24.04 LTS

Provides the operating environment for the EC2 servers.

Containerization Tool Docker

Packages User, Dataset, Metadata, Validation, Processing, Search, Storage, and Notification services into containers.

Container Orchestration Tool Kubernetes

Deploys, manages, scales, and restarts the research data service containers.

Service Mesh Istio

Secures and manages communication between research data microservices using mTLS, traffic routing, retries, and timeouts.

API Gateway Kong

Receives researcher requests and routes them to the appropriate research data service.

Authentication Tool Keycloak

Handles researcher authentication, identity management, roles, and access control.

Database PostgreSQL

Stores researcher information, dataset metadata, access permissions, and research data management information.

Monitoring Tool Prometheus

Collects application, Kubernetes, and infrastructure metrics.

Dashboard Tool Grafana

Displays service health, latency, and performance metrics.

Tracing Tool Jaeger

Tracks research data requests across multiple microservices and helps identify delays or failures.

Infrastructure as Code Tool OpenTofu

Creates and manages Cloud infrastructure such as VPC, EC2, and security groups.

Configuration Automation Tool Ansible

Configures Ubuntu servers and installs required software.

Deployment Tool Helm

Packages and deploys research data microservices and Istio configurations into Kubernetes.

Implementation Process

01
Step 1 – Design the Research Data Management Application
  • Identify the main services such as User, Dataset, Metadata, Validation, Processing, Search, Storage, and Notification.
  • Define the responsibilities of each microservice.
  • Design the APIs and communication flow between services.
  • Design PostgreSQL tables for researcher information, dataset metadata, and access permissions.
  • Define the authentication and authorization requirements.
02
Step 2 – Create Cloud Infrastructure
  • Use OpenTofu to create the Cloud VPC, subnets, EC2 instances, and security groups.
  • Install Ubuntu Server on the EC2 nodes.
  • Use Ansible to configure the servers.
  • Install and configure the Kubernetes cluster.
03
Step 3 – Containerize and Deploy Services
  • Package each research data microservice using Docker.
  • Create Kubernetes deployments and services for the containers.
  • Use Helm to deploy and manage the research data services.
  • Deploy PostgreSQL for researcher, dataset, metadata, and access information.
04
Step 4 – Configure API, Authentication and Service Mesh
  • Configure Kong as the API Gateway for incoming research application requests.
  • Configure Keycloak for researcher authentication and authorization.
  • Deploy Istio to manage communication between research data services.
  • Enable mTLS to encrypt and authenticate service-to-service communication.
  • Configure routing, retries, timeouts, and fault handling using Istio.
05
Step 5 – Monitoring and Testing
  • Deploy Prometheus to collect application, service, and Kubernetes metrics.
  • Configure Grafana dashboards to monitor service health and performance.
  • Configure Jaeger for distributed request tracing.
  • Test communication between Dataset, Validation, Processing, and Storage services.
  • Verify mTLS, traffic routing, retries, and failure handling.
  • Test the complete research dataset workflow before deployment.

Proposed Solution

The proposed solution provides a secure cloud-native distributed Research Data Management Application using Docker, Kubernetes, and the Istio Service Mesh. Research services such as User, Dataset, Metadata, Validation, Processing, Search, Storage, and Notification run as Docker containers on Kubernetes in Cloud EC2 servers. Istio secures service-to-service communication using mTLS, while Envoy sidecar proxies manage service traffic. Kong handles external API requests, and Keycloak manages researcher authentication and access control. PostgreSQL stores researcher information, dataset metadata, and access permissions, while Prometheus, Grafana, and Jaeger provide monitoring and distributed tracing. OpenTofu and Ansible automate the infrastructure and server configuration, while Helm manages application deployment.

Benefits

Secure Communication: mTLS encrypts and authenticates communication between research data services.
Traffic Management: Istio provides routing, retries, timeouts, and fault handling.
Access Control: Keycloak controls researcher access to datasets and application functions.
Fault Recovery: Kubernetes can restart failed containers.
Better Monitoring: Prometheus, Grafana, and Jaeger provide visibility into service performance.
Scalability: Kubernetes allows individual research services to scale based on workload.
Data Protection: Controlled service communication helps protect research datasets and metadata.

Challenges

Service Mesh Complexity: Istio configuration can be difficult to manage.
Proxy Overhead: Envoy proxies require additional CPU and memory.
Monitoring Data: Large amounts of metrics and traces require storage and management.
Traffic Configuration: Incorrect routing or security policies can affect service communication.
Data Access Control: Research datasets may require carefully configured permissions.
Tool Integration: Multiple cloud, Kubernetes, service mesh, and monitoring tools must be properly integrated.
← Back to Project Titles