How to execute scala application in Spark shell

  • The code segment given below performs the word count operation using Scala API. The data from the input file is read and map reduce operation is performed over the data to count the words. The output is stored as a file.
  • Code Segment
    scala> val file= sc.textFile(“samplewordcount.txt”)
    scala> val wordcounts = file.flatMap(line => line.split(” “)).map(word => (word, 1)).reduceByKey(_+_);
    scala> wordcounts.toDebugString
    scala> wordcounts.cache()
    scala> wordcounts.saveAsTextFile(“outputfile”)