How to implement linear regression using Spark with R ?

Description

To implement linear regression using Spark with R

  • Set up Spark Context and Spark session
  • Load the Data set
  • Split the data into train and test set
  • Fit the linear regression model
  • Predict using the test set
  • Take the summary of the model

#Set up spark home
Sys.setenv(SPARK_HOME=”…./spark-2.4.0-bin-hadoop2.7″)
.libPaths(c(file.path(Sys.getenv(“SPARK_HOME”), “R”, “lib”), .libPaths()))
#Load the library
library(SparkR)
#Initialize the Spark Context
#To run spark in a local node give master=”local”
sc #Start the SparkSQL Context
sqlContext #Load the data set
data = read.df(“file:///…./servo.csv”,”csv”,header = “true”, inferSchema = “true”, na.strings = “NA”)
#Split the data into train and test set
splt_data=randomSplit(data,c(8,2))
trainingData=splt_data[[1]]
testData=splt_data[[2]]
xtest=select(testData,”Motor”,”Screw”,”Pgain”,”Vgain”)
ytest=select(testData,”Class”)
#Build the model
linear_model summary(linear_model)
#Predict using the test data
y_pred=predict(linear_model,xtest)
showDF(y_pred)

Leave Comment

Your email address will not be published. Required fields are marked *

clear formSubmit