Sqoop commit failure

简介:

线上一个Sqoop抽取Job报如下错误:

[ERROR]: Utils - Caused by: java.sql.SQLException: Communicationslink failure during commit(). Transaction resolution unknown.
[ERROR]: Utils - at com.mysql.jdbc.Connection.commit(Connection.java:2171)
[ERROR]: Utils - atorg.apache.sqoop.mapreduce.db.DBRecordReader.close(DBRecordReader.java:167)

看报错应该是Sqoop在导出数据后,处理数据到提交超过了MySQLwait_timeout(线上是2分钟);

引起这个的原因会有很多,服务器高负载,HDFS高负载,网卡跑满等等,可以通过在jdbc url里设置interactiveClient=true(线上interactive_timeout1小时)后没有再报过错;

BTW:在写blog的时候想起其实可以尝试一下将fetch-size去掉应该也可以workaround,现在再看当时处问题前一天的日志数据其实MAP其实内存已经满了;

212514243.png

后续更新:发现在出错的前一天修改过mapred-site.xml,child.opt里的最大内存改成了1G;这样看来fetch-size去掉才是最终解决方案:)结贴



本文转自MIKE老毕 51CTO博客,原文链接:http://blog.51cto.com/boylook/1299842,如需转载请自行联系原作者


相关文章
|
关系型数据库 MySQL Java
Flink作业报错:Caused by: The connector is trying to read binlog starting at GTIDs ..., but this is no longer available on the server
Flink作业报错:Caused by: The connector is trying to read binlog starting at GTIDs ..., but this is no longer available on the server
Flink作业报错:Caused by: The connector is trying to read binlog starting at GTIDs ..., but this is no longer available on the server
|
SQL 存储 资源调度
sqoop数据导出报错Job failed as tasks failed. failedMaps:1 failedReduces:0
sqoop数据导出报错Job failed as tasks failed. failedMaps:1 failedReduces:0
|
7月前
|
SQL 监控 关系型数据库
Mysql主从同步报错解决:Error executing row event: Table zabbix.history-..
Mysql主从同步报错解决:Error executing row event: Table zabbix.history-..
【异常】svn: E200009: Commit failed (details follow)/both sides of the move must be committed together的解决办法
svn: E200009: Commit failed (details follow)/both sides of the move must be committed together的解决办法
478 0
|
12月前
|
缓存 Oracle 关系型数据库
Oracle中控制commit的三个参数 commit_write, commit_logging和 commit_wait
Oracle中控制commit的动作有三个参数 commit_write, commit_logging和 commit_wait,按重要性分别说明如下
141 0
|
存储 API 流计算
Flink - checkpoint Failure reason: Not all required tasks are currently running
Flink 程序增加 readFile 生成文件流后,最初运行期间 CheckPoint 存储没有问题,待文件流 Finished 后 CheckPoint 存储报错: checkpoint Failure reason: Not all required tasks are currently running,下面分析并解决下。
2121 0
Flink - checkpoint Failure reason: Not all required tasks are currently running
|
NoSQL MongoDB 开发工具
启动mongodb 报错 Error: Failure while executing; `git clone https://github.com/Homebrew/homebrew-serv...
启动mongodb 报错 Error: Failure while executing; `git clone https://github.com/Homebrew/homebrew-serv...
1103 0
|
分布式计算 Ubuntu Hadoop
MapReduce报错:「MKDirs failed to create file」
MapReduce报错:「MKDirs failed to create file」
275 0
MapReduce报错:「MKDirs failed to create file」
|
开发工具 git
git commit 弹出编辑器后报错: Aborting commit due to empty commit message.
使用终端提交代码 "git commit" 能正常弹出 设置的编辑器,但是直接被空消息提交上来导致无效。 git commit 使用了插件 # git-extras 简化命令 gc == git commit 解决方法: $ git config --global core.editor "subl -w -f" "subl -f" 表示设置默认启动的编辑器,-w表示等待编辑器提交之后, -f 为一个参数 让它不要 fork。
5233 0