Location Research Breakthrough Possible @S-Logix pro@slogix.in

Office Address

Social List

Serverless Data Streaming Pipeline for Real-Time User Activity Analytics in a Streaming Media Application

Description

The proposed system is a serverless streaming media analytics platform that collects and processes user activity events in real time. Activities such as video search, play, pause, watch duration, skip, and content selection are captured and processed to understand user engagement and content popularity.

Aim

To develop a serverless data streaming pipeline that processes real-time user activity data from a Streaming Media Application for analytics and reporting.

Objectives

01 To capture user activities from the streaming application.
02 To process user activity events in real time.
03 To validate and transform activity data.
04 To analyze viewer engagement and content popularity.
05 To generate useful user activity analytics.
06 To store processed analytics data for reporting.
07 To provide monitoring of the streaming pipeline.

Application Workflow

01

Stage 1 – User Registration and Login

Process

The user creates an account and logs into the streaming media application.

Tools
Keycloak
Implementation

The user enters their credentials, and Keycloak authenticates the user before allowing access to the application.

02

Stage 2 – Browse Content

Process

The user browses available movies, shows, or other streaming content.

Tools
PostgreSQL Redis
Implementation

The application retrieves content information from PostgreSQL. Frequently accessed content information can be temporarily stored in Redis for faster access.

03

Stage 3 – Search Content

Process

The user searches for a particular movie, show, or video.

Tools
PostgreSQL Redis
Implementation

The application processes the search request and retrieves matching content information for the user.

04

Stage 4 – Play Content

Process

The user selects and starts watching a video.

Tools
PostgreSQL
Implementation

The application records information such as the selected content, user, and playback activity.

05

Stage 5 – Video Interaction

Process

The user interacts with the video while watching.

Tools
PostgreSQL
Implementation

Activities such as pause, resume, seek, skip, change quality, and stop are generated during playback.

06

Stage 6 – Continue or Stop Watching

Process

The user either continues watching the content or stops playback.

Tools
PostgreSQL
Implementation

The application records the viewing activity and watch duration for the session.

Cloud Infrastructure and Tools

Cloud Data Storage Cloud S3

Stores raw user activity data and processed datasets generated by the streaming analytics pipeline.

Event Streaming Cloud Kinesis Data Streams

Receives and streams user activity events generated by the streaming media application.

Event Trigger Cloud EventBridge

Handles event-based triggers required for processing and downstream application activities.

Serverless Compute Cloud Lambda

Performs lightweight validation, transformation, and processing of incoming user activity events.

Workflow Orchestration Cloud Step Functions

Coordinates multi-stage processing when sequential processing is required.

Authentication Tool Keycloak

Provides user authentication and access control.

API Gateway Tool Kong Gateway

Handles and routes application API requests.

Application Database PostgreSQL

Stores user, content, viewing-session, and application information.

Cache Tool Redis

Stores frequently accessed content and temporary session information for faster access.

Log/Event Collection Tool Fluent Bit

Collects user activity events generated by the application and forwards them to the streaming pipeline.

Metrics Collection Tool Prometheus

Collects metrics related to application activity and pipeline processing.

Monitoring Dashboard Grafana

Displays user activity and pipeline performance metrics through dashboards.

Infrastructure as Code OpenTofu

Creates and manages the required Cloud infrastructure using configuration code.

Implementation Process

01
Step 1 – Analyze User Activity Requirements
  • Identify important user activities.
  • Define events such as login, search, play, pause, skip, and completion.
  • Define the required event fields.
  • Define analytics requirements such as watch time and content popularity.
02
Step 2 – Configure the Streaming Application
  • Configure Keycloak for user authentication.
  • Configure Kong Gateway for API access.
  • Configure PostgreSQL for application data.
  • Configure Redis for frequently accessed and temporary data.
  • Enable user activity event generation.
03
Step 3 – Configure the Streaming Pipeline
  • Configure Fluent Bit to collect activity events.
  • Send events to Cloud Kinesis Data Streams.
  • Configure Cloud Lambda to process incoming events.
  • Store raw activity data in Cloud S3.
04
Step 4 – Process User Activity Data
  • Validate incoming activity events.
  • Remove incomplete or invalid events.
  • Transform events into a standard format.
  • Calculate metrics such as watch duration and viewing frequency.
05
Step 5 – Generate Analytics
  • Analyze user engagement.
  • Identify popular content.
  • Calculate viewing patterns.
  • Store processed analytics data for reporting.
06
Step 6 – Monitor and Deploy
  • Configure Prometheus for metrics collection.
  • Create Grafana dashboards.
  • Test different user activity scenarios.
  • Verify event processing and analytics.
  • Deploy the infrastructure using OpenTofu.

Proposed Solution

The proposed solution uses a serverless data streaming architecture to process user activity from a Streaming Media Application in real time. User activities such as searching, playing, pausing, skipping, and completing videos generate events. Fluent Bit collects these events and sends them to Cloud Kinesis Data Streams. Cloud Lambda validates and transforms the events, while Cloud S3 stores raw and processed data. The processed information is analyzed to identify viewer engagement, watch duration, and content popularity. Prometheus and Grafana provide monitoring of the application and streaming pipeline.

Benefits

Real-Time Analytics: User activity can be analyzed as it occurs.
Scalability: Handles large numbers of viewer events.
Serverless Processing: No processing servers need to be managed.
Better User Insights: Helps understand viewing behavior.
Content Analysis: Identifies popular and less-engaged content.
Automated Processing: User events are processed automatically.

Challenges

High Event Volume: Large numbers of viewers can generate huge amounts of events.
Real-Time Processing: Events need to be processed with low delay.
Data Quality: Missing or duplicate events can affect analytics.
Data Storage: Continuous activity generation requires effective storage management.
Processing Failures: Failed events need proper retry and error handling.
Monitoring Complexity: Multiple streaming and serverless components require monitoring.
← Back to Project Titles