Project Background:
The clustering for streaming data revolves around addressing the unique challenges posed by continuously arriving data streams in various fields, such as IoT, sensor networks, and online platforms. Unlike static datasets, streaming data is characterized by its high volume, velocity, and potentially infinite nature, making traditional clustering techniques impractical. In this context, the project seeks to develop innovative clustering algorithms that dynamically adapt to evolving data distributions and concept drifts inherent in streaming data. These algorithms must efficiently process incoming data in real-time, minimizing computational resources and memory usage while maintaining high accuracy. Furthermore, this aims to explore techniques for handling noisy and incomplete data commonly encountered in streaming environments. Ultimately, the goal is to provide scalable and robust clustering solutions tailored to streaming data applications, enabling timely insights and decision-making in dynamic and rapidly changing environments.
Problem Statement
- Streaming data arrives continuously and in real-time, posing challenges for traditional batch-based clustering algorithms designed for static datasets.
- Underlying patterns and distributions in streaming data may change, leading to concept drift, which traditional clustering algorithms struggle to adapt.
- Clustering algorithms for streaming data must operate under constrained computational resources and memory, necessitating efficient algorithms capable of incremental updates.
-
Streaming data often contains noise and outliers, which can adversely affect the quality of clustering results if not appropriately handled in real time.
- Establishing meaningful metrics and evaluation methods for assessing the quality and performance of clustering algorithms on streaming data is challenging due to the lack of ground truth labels and evolving data distributions.
-
Streaming data may contain missing values or incomplete observations, requiring clustering algorithms to be robust to such data irregularities while maintaining clustering accuracy.
Aim and Objectives
- Develop efficient and adaptive clustering algorithms tailored for streaming data applications.
-
Design clustering algorithms capable of handling dynamic data arrival and concept drift in real-time.
-
Optimize algorithms to operate under limited memory and computational resources while maintaining scalability.
-
Develop techniques to handle noise, outliers, and missing data in streaming environments.
-
Implement online learning mechanisms to update clustering models with new data continuously.
-
Evaluate algorithm performance using meaningful metrics and validation methods specific to streaming data.
Contributions to Clustering for Streaming Data
- Developing clustering algorithms that can dynamically adapt to evolving data streams, facilitating timely insights and decision-making in dynamic environments.
- Providing efficient and scalable clustering solutions that operate under constrained computational resources, enabling the processing of large volumes of streaming data.
- Enhancing clustering algorithms to be robust against noise, outliers, and concept drift commonly encountered in streaming data, ensuring reliable clustering results over time.
- Enabling clustering models to continuously learn and update with new data without retraining from scratch, supporting adaptive clustering in real-time.
- Advancing clustering techniques to various streaming data domains, including IoT, sensor networks, and online platforms, to extract valuable insights and facilitate decision-making processes.
Deep Learning Algorithms for Clustering for Streaming Data
- Deep Embedded Clustering (DEC)
-
Deep Autoencoding Gaussian Mixture Model (DAGMM)
-
Variational Autoencoder-based Clustering (VAEC)
-
Deep Adaptive Clustering (DAC)
-
Deep k-Means (DkM)
-
Deep Affinity Network (DAN)
-
Deep Spectral Clustering (DSC)
-
Deep Reinforcement Learning for Clustering (DRLC)
-
Deep Belief Network-based Clustering (DBNC)
-
Deep Generative Models for Clustering (DGMC)
Datasets for Clustering for Streaming Data
- Online Retail II Dataset
-
Network Traffic Data
-
Sensor Data Streams
-
Twitter Streaming Data
-
KDD Cup 1999 Dataset
-
Electricity Consumption Data
-
Yahoo S5 Dataset
-
Covtype Data Streams
-
Synthetic Data Streams
-
Stock Market Data Streams
Software Tools and Technologies
Operating System: Ubuntu 18.04 LTS 64bit / Windows 10
Development Tools: Anaconda3, Spyder 5.0, Jupyter Notebook
Language Version: Python 3.9
Python Libraries:
1.Python ML Libraries:
- Scikit-Learn
- Numpy
- Pandas
- Matplotlib
- Seaborn
- Docker
- MLflow
2.Deep Learning Frameworks:
- Keras
- TensorFlow
- PyTorch
