How to Initialize and Stop Spark Session in R ?

Description

To Initialize Spark Session in R

  • Set up spark home
  • Load the spark library
  • Initialize the spark context
  • Now write the desired coding
  • Stop the spark session

#Set up spark home
Sys.setenv(SPARK_HOME=”/…/spark-2.4.0-bin-hadoop2.7″)
.libPaths(c(file.path(Sys.getenv(“SPARK_HOME”), “R”, “lib”), .libPaths()))
#Load the library
library(SparkR)
#Initialize the Spark Context
#To run spark in a local node give master=”local”
sc #Start the SparkSQL Context
sqlContext #To stop the spark session
sparkR.session.stop()

Leave Comment

Your email address will not be published. Required fields are marked *

clear formSubmit