Spark Archives - Script(s)

This script loops through hdfs files system and reads the first line and writes it to console. Most part its

Spark Scala Command line Arguments

February 9, 2018February 11, 2018 gchandra 0 Comments CommandLine, Hadoop, Scala, Spark

Its pretty straight forward to add Command Line Arguments to Spark (scala) from shell. $ ./spark-2.0.0-bin-hadoop2.6/bin/spark-shell -i ~/scalaparam.scala –conf spark.driver.args=”param1value