从Spark-Shell到SparkContext的函数调用路径过程分析(源码)

  1. 云栖社区>
  2. 博客>
  3. 正文

从Spark-Shell到SparkContext的函数调用路径过程分析(源码)

技术小哥哥 2017-11-13 14:47:00 浏览581
展开阅读全文

 

首先,我们清晰定位找到这几个。

 

1、spark-shell

 

 

2、 spark-submit

 

 

3、spark-class

 

 

 

 4、SparkSubmit.scala

 

 

 

 

 

 5、SparkILoop.scala

 

 

initializeSpark的源码

def initializeSpark() {
intp.beQuietDuring {
command("""
@transient val sc = {
val _sc = org.apache.spark.repl.Main.interp.createSparkContext()
println("Spark context available as sc.")
_sc
}
""")
command("""
@transient val sqlContext = {
val _sqlContext = org.apache.spark.repl.Main.interp.createSQLContext()
println("SQL context available as sqlContext.")
_sqlContext
}
""")
command("import org.apache.spark.SparkContext._")
command("import sqlContext.implicits._")
command("import sqlContext.sql")
command("import org.apache.spark.sql.functions._")
}

 

 createSparkContext的源码

// NOTE: Must be public for visibility
@DeveloperApi
def createSparkContext(): SparkContext = {
val execUri = System.getenv("SPARK_EXECUTOR_URI")
val jars = SparkILoop.getAddedJars
val conf = new SparkConf()
.setMaster(getMaster())
.setAppName("Spark shell")
.setJars(jars)
.set("spark.repl.class.uri", intp.classServerUri)
if (execUri != null) {
conf.set("spark.executor.uri", execUri)
}
sparkContext = new SparkContext(conf)
logInfo("Created spark context..")
sparkContext
}

 

 

 

总结


本文转自大数据躺过的坑博客园博客,原文链接:http://www.cnblogs.com/zlslch/p/5905540.html,如需转载请自行联系原作者

网友评论

登录后评论
0/500
评论
技术小哥哥
+ 关注