Seatunnel在Dolphinscheduler调度中报错

江南独孤客
发布于 2022-7-7 19:07
浏览
0收藏
[INFO] 2022-07-06 13:12:16.714  - [taskAppId=TASK-393-81727-908774]:[238] - process has exited, execute path:/data/dolphinscheduler/exec/process/9/393/81727/908774, processId:25215 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2022-07-06 13:12:17.201  - [taskAppId=TASK-393-81727-908774]:[138] -  -> tools
  size = 2813343
  4574389
  5445379
  4238956
  4253580
  /data/dolphinscheduler/exec/process/9/393/81727/908774/393_81727_908774_node.sh: line 7: [: too many arguments
  dbos-bigdata-clickhouse001  8000  0   2 0
  dbos-bigdata-clickhouse003  8000  0   1 0
  dbos-bigdata-clickhouse002  8000  0   0 0
  SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
  SLF4J: Defaulting to no-operation (NOP) logger implementation
  SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
  Exception in thread "main" java.nio.file.AccessDeniedException: plugins.tar.gz
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
    at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
    at java.nio.file.Files.newOutputStream(Files.java:216)
    at org.apache.seatunnel.core.base.utils.CompressionUtils.tarGzip(CompressionUtils.java:62)
    at org.apache.seatunnel.core.spark.SparkStarter$ClusterModeSparkStarter.buildCommands(SparkStarter.java:390)
    at org.apache.seatunnel.core.spark.SparkStarter.main(SparkStarter.java:103)
  

方案:

Seatunnel在Dolphinscheduler调度中报错 -鸿蒙开发者社区
查看脚本中的源码,需要代码中cd /opt/apache-seatunnel-incubating-2.1.2/

#!/bin/bash
hive_table_name='dim_dbos_mac_userid_relation_his'
hdfs_size=`hdfs dfs -du oss://dbbigdatapro/group/user/tools/meta/hive-temp-table/${hive_table_name}/ | awk '{print $1}'`
echo `whoami`
echo "size = ${hdfs_size}" 

if [ $hdfs_size -eq 0 ]
then
  echo "没有数据不需要导出"
else
  clickhouse-client -h 172.18.2.111 --port 8000 -u default -d default -m -q "TRUNCATE TABLE dmp.dim_dbos_mac_userid_relation_his on CLUSTER cluster_clickhouse"
  cd /opt/apache-seatunnel-incubating-2.1.2
  /opt/apache-seatunnel-incubating-2.1.2/bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config /home/tools/bigsource/etl/DMP/new/clickhouse_ddl/${hive_table_name}.conf
fi

问题排除
Seatunnel在Dolphinscheduler调度中报错 -鸿蒙开发者社区

标签
收藏
回复
举报
回复
    相关推荐