Main Reference PaperAn Intermediate Data Partition Algorithm for Skew Mitigation in Spark Computing Environment, IEEE Transactions on Cloud Computing, 2018 [Python/Spark]
  • In spark framework, to solve the imbalance of data distribution among reduce tasks, this work proposes a two algorithms: hash based key reassigning algorithm (KRHP) and rang based key splitting algorithm (KSRP), which can generate appropriate strategy and implement the skew mitigation in shuffle phase. KSRP generates the weighted bounds to split intermediate data for the type of sort-based applications while KRHP records these reassigned keys and the new reducers these keys belong to for other applications.

Description
  • In spark framework, to solve the imbalance of data distribution among reduce tasks, this work proposes a two algorithms: hash based key reassigning algorithm (KRHP) and rang based key splitting algorithm (KSRP), which can generate appropriate strategy and implement the skew mitigation in shuffle phase. KSRP generates the weighted bounds to split intermediate data for the type of sort-based applications while KRHP records these reassigned keys and the new reducers these keys belong to for other applications.

  • To reduce the execution time of tasks.

Aim & Objectives
  • To reduce the execution time of tasks.

  • The proposed scheme is improved by further adding the concepts of reduce the wastage of resource in Spark.

Contribution
  • The proposed scheme is improved by further adding the concepts of reduce the wastage of resource in Spark.

  • M.E / M.Tech / MS / Ph.D.- Customized according to the client requirements.

Project Recommended For
  • M.E / M.Tech / MS / Ph.D.- Customized according to the client requirements.

  • No Readymade Projects-project delivery Depending on the complexity of the project and requirements.

Order To Delivery
  • No Readymade Projects-project delivery Depending on the complexity of the project and requirements.

Professional Ethics: We S-Logix would appreciate the students those who willingly contribute with atleast a line of thinking of their own while preparing the project with us. It is advised that the project given by us be considered only as a model project and be applied with confidence to contribute your own ideas through our expert guidance and enrich your knowledge.

Leave Comment

Your email address will not be published. Required fields are marked *

clear formSubmit