回复
Seatunnel在Dolphinscheduler调度中报错
江南独孤客
发布于 2022-7-7 19:07
浏览
0收藏
[INFO] 2022-07-06 13:12:16.714 - [taskAppId=TASK-393-81727-908774]:[238] - process has exited, execute path:/data/dolphinscheduler/exec/process/9/393/81727/908774, processId:25215 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2022-07-06 13:12:17.201 - [taskAppId=TASK-393-81727-908774]:[138] - -> tools
size = 2813343
4574389
5445379
4238956
4253580
/data/dolphinscheduler/exec/process/9/393/81727/908774/393_81727_908774_node.sh: line 7: [: too many arguments
dbos-bigdata-clickhouse001 8000 0 2 0
dbos-bigdata-clickhouse003 8000 0 1 0
dbos-bigdata-clickhouse002 8000 0 0 0
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.nio.file.AccessDeniedException: plugins.tar.gz
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at org.apache.seatunnel.core.base.utils.CompressionUtils.tarGzip(CompressionUtils.java:62)
at org.apache.seatunnel.core.spark.SparkStarter$ClusterModeSparkStarter.buildCommands(SparkStarter.java:390)
at org.apache.seatunnel.core.spark.SparkStarter.main(SparkStarter.java:103)
方案:
查看脚本中的源码,需要代码中cd /opt/apache-seatunnel-incubating-2.1.2/
#!/bin/bash
hive_table_name='dim_dbos_mac_userid_relation_his'
hdfs_size=`hdfs dfs -du oss://dbbigdatapro/group/user/tools/meta/hive-temp-table/${hive_table_name}/ | awk '{print $1}'`
echo `whoami`
echo "size = ${hdfs_size}"
if [ $hdfs_size -eq 0 ]
then
echo "没有数据不需要导出"
else
clickhouse-client -h 172.18.2.111 --port 8000 -u default -d default -m -q "TRUNCATE TABLE dmp.dim_dbos_mac_userid_relation_his on CLUSTER cluster_clickhouse"
cd /opt/apache-seatunnel-incubating-2.1.2
/opt/apache-seatunnel-incubating-2.1.2/bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config /home/tools/bigsource/etl/DMP/new/clickhouse_ddl/${hive_table_name}.conf
fi
问题排除
标签
赞
收藏
回复
相关推荐