开发者社区> 问答> 正文

使用spark streaming连接loghub报错,是什么问题

"main" java.lang.ClassNotFoundException: Failed to find data source: loghub. Please find packages at http://spark.apache.org/third-party-projects.html

at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:652)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:159)
at com.aliyun.emr.examples.sql.streaming.RealtimeComputation.main(RealtimeComputation.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:896)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: loghub.DefaultSource

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27$$anonfun$apply$15.apply(DataSource.scala:635)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27$$anonfun$apply$15.apply(DataSource.scala:635)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27.apply(DataSource.scala:635)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27.apply(DataSource.scala:635)
at scala.util.Try.orElse(Try.scala:84)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)

展开
收起
游客lplm6xso3kx3e 2019-05-28 13:41:12 4831 0
1 条回答
写回答
取消 提交回答
  • 1、需要依赖对应的loghub的datasource com.aliyun.emr emr-logservice_2.11 ${emr.version} 2、也可以参考下面的demo https://github.com/aliyun/aliyun-emapreduce-demo/blob/master-2/pom.xml

    2020-03-21 15:04:30
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
Hybrid Cloud and Apache Spark 立即下载
Scalable Deep Learning on Spark 立即下载
Comparison of Spark SQL with Hive 立即下载

相关实验场景

更多