51CTO首页
AI.x社区
博客
学堂
精品班
软考社区
免费课
企业培训
鸿蒙开发者社区
WOT技术大会
IT证书
公众号矩阵
移动端
视频课免费课排行榜短视频直播课软考学堂
全部课程软考华为认证厂商认证IT技术PMP项目管理免费题库
在线学习
文章资源问答课堂专栏直播
51CTO
鸿蒙开发者社区
51CTO技术栈
51CTO官微
51CTO学堂
51CTO博客
CTO训练营
鸿蒙开发者社区订阅号
51CTO软考
51CTO学堂APP
51CTO学堂企业版APP
鸿蒙开发者社区视频号
51CTO软考题库
活动 短视频 专栏 极客Show 鸿蒙技术特刊

【我和openGauss的故事】openGauss集群故障节点替换操作

老老老JR老北
发布于 2023-9-12 11:16
浏览
0收藏

背景信息

节点故障或者节点替换(主机名和ip与原主机保持一致)的情况下，尝试使用拷贝正常节点的​​app​​​二进制文件和​​om​​​文件来恢复故障或替换节点，并通过​​gs_ctl build​​​[从备机进行​​build​​]来将节点重新加入到现有集群中。

本次验证是在测试环境下，数据库无压力，生产环境请谨慎测试。

集群信息

2023-08-04 07:43:24 [line:905] INFO <module> 94105 [   Cluster State   ]

cluster_state   : Normal
redistributing  : No
current_az      : AZ_ALL

[  Datanode State   ]

    node   node_ip         port      instance                     state
---------------------------------------------------------------------------------------
1  pghost3 192.168.56.30   26000      6001 /app/ogdata/data/dn1   P Primary Normal
2  pghost5 192.168.56.50   26000      6002 /app/ogdata/data/dn1   S Standby Normal
3  pghost6 192.168.56.60   26000      6003 /app/ogdata/data/dn1   S Standby Normal
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.

模拟故障

root@pghost6 /app# rm -rf ogdata/
root@pghost6 /app# rm -rf opengauss/
root@pghost6 /app# rm -rf ogxlog/
root@pghost6 /app# rm -rf ogarchive/

kill -9 ${GAUSSDB-PID}
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.

集群状态

omm@pghost3 ~$ gs_om -t status --detail
[   Cluster State   ]

cluster_state   : Degraded
redistributing  : No
current_az      : AZ_ALL

[  Datanode State   ]

    node   node_ip         port      instance                     state
---------------------------------------------------------------------------------------
1  pghost3 192.168.56.30   26000      6001 /app/ogdata/data/dn1   P Primary Normal
2  pghost5 192.168.56.50   26000      6002 /app/ogdata/data/dn1   S Standby Normal
3  pghost6 192.168.56.60   26000      6003 /app/ogdata/data/dn1   S Unknown Unknown
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.

恢复

安装​​python3​​

根据系统情况决定是否需要安装​​python3​​。

拷贝目录及文件

在​​/etc/hosts​​中加入节点的映射关系

omm@pghost6 ~$ more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.60 pghost6 
192.168.56.30 pghost3 
192.168.56.50 pghost5
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.

从​​pghost5​​​节点拷贝目录到故障节点​​pghost6​​对应目录下。

# pghost6 上创建对应目录
root@pghost6 /app# mkdir opengauss
root@pghost6 /app# chown omm: opengauss

# 拷贝 app 和 tool 目录
omm@pghost5 /app/opengauss$ scp -r app omm pghost6:/app/opengauss/
omm@pghost5 /app/opengauss$ scp -r tool pghost6:/app/opengauss/
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.

从​​pghost5​​​节点拷贝​​pg_hba.conf​​​、​​postgresql.conf​​​文件到故障节点​​pghost6​​对应目录下。

root@pghost6 /app# mkdir -p ogdata/data/dn1
root@pghost6 /app# chown omm: ogdata/data/dn1/
omm@pghost5 /app/ogdata/data/dn1$ scp pg_hba.conf postgresql.conf pghost6:/app/ogdata/data/dn1/
  • 1.
  • 2.
  • 3.

修改​​postgresql.conf​​对应的值

local_bind_address = '192.168.56.60'

replconninfo1 = 'localhost=192.168.56.60 localport=26001 localheartbeatport=26005 localservice=26004 remotehost=192.168.56.30 remoteport=26001 remoteheartbeatport=26005 remoteservice=26004' 
replconninfo2 = 'localhost=192.168.56.60 localport=26001 localheartbeatport=26005 localservice=26004 remotehost=192.168.56.50 remoteport=26001 remoteheartbeatport=26005 remoteservice=26004' 

synchronous_standby_names = 'ANY 1(dn_6001,dn_6002)'
log_directory = '/app/opengauss/gaussdb_log/omm/pg_log/dn_6003'
audit_directory = '/app/opengauss/gaussdb_log/omm/pg_audit/dn_6003'
application_name = 'dn_6003'
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.

在​​.bashrc​​中加入如下内容

PATH="$HOME/.local/bin:$HOME/bin:$PATH"
export PATH
export GPHOME=/app/opengauss/tool
export PATH=$GPHOME/script/gspylib/pssh/bin:$GPHOME/script:$PATH
export LD_LIBRARY_PATH=$GPHOME/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$GPHOME/lib
export GAUSSHOME=/app/opengauss/app/2.0.1
export PATH=$GAUSSHOME/bin:$PATH
export LD_LIBRARY_PATH=$GAUSSHOME/lib:$LD_LIBRARY_PATH
export S3_CLIENT_CRT_FILE=$GAUSSHOME/lib/client.crt
export GAUSS_VERSION=3.0.3
export PGHOST=/app/opengauss/tmp
export GAUSSLOG=/app/opengauss/gaussdb_log/omm
umask 077
export GAUSS_ENV=2
export GS_CLUSTER_NAME=gauss_omm
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.

​​build​​拉齐数据

# 从备机进行build
gs_ctl build -D /app/ogdata/data/dn1 -b standby_full -C "localhost=192.168.56.60 localport=26000 remotehost=192.168.56.50 remoteport=26000"

0 LOG:  [Alarm Module]can not read GAUSS_WARNING_TYPE env.

0 LOG:  [Alarm Module]Host Name: pghost6

0 LOG:  [Alarm Module]Host IP: pghost6. Copy hostname directly in case of taking 10s to use 'gethostbyname' when /etc/hosts does not contain <HOST IP>

0 LOG:  [Alarm Module]Cluster Name: gauss_omm

0 LOG:  [Alarm Module]Invalid data in AlarmItem file! Read alarm English name failed! line: 57

0 WARNING:  failed to open feature control file, please check whether it exists: FileName=gaussdb.version, Errno=2, Errmessage=No such file or directory.
0 WARNING:  failed to parse feature control file: gaussdb.version.
0 WARNING:  Failed to load the product control file, so gaussdb cannot distinguish product version.
The core dump path is an invalid directory
[2023-08-04 08:36:04.775][68234][][gs_ctl]: gs_ctl standby full build ,datadir is /app/ogdata/data/dn1,conn_str is 'localhost=192.168.56.60 localport=26000 remotehost=192.168.56.50 remoteport=26000'
[2023-08-04 08:36:04.775][68234][][gs_ctl]: fopen build pid file "/app/ogdata/data/dn1/gs_build.pid" success
[2023-08-04 08:36:04.775][68234][][gs_ctl]: fprintf build pid file "/app/ogdata/data/dn1/gs_build.pid" success
[2023-08-04 08:36:04.779][68234][][gs_ctl]: fsync build pid file "/app/ogdata/data/dn1/gs_build.pid" success
[2023-08-04 08:36:04.780][68234][][gs_ctl]: stop failed, killing gaussdb by force ...
[2023-08-04 08:36:04.780][68234][][gs_ctl]: command [ps c -eo pid,euid,cmd | grep gaussdb | grep -v grep | awk '{if($2 == curuid && $1!="-n") print "/proc/"$1"/cwd"}' curuid=`id -u`| xargs ls -l | awk '{if ($NF=="/app/ogdata/data/dn1")  print $(NF-2)}' | awk -F/ '{print $3 }' | xargs kill -9 >/dev/null 2>&1 ] path: [/app/ogdata/data/dn1]
[2023-08-04 08:36:04.812][68234][][gs_ctl]: server stopped
[2023-08-04 08:36:04.812][68234][][gs_ctl]: current workdir is (/home/omm).
[2023-08-04 08:36:04.814][68234][][gs_ctl]: set gaussdb state file when standby full build build:db state(BUILDING_STATE), server mode(STANDBY_MODE), build mode(FULL_BUILD).
[2023-08-04 08:36:04.814][68234][dn_6001_6002_6003][gs_ctl]: Get repl_auth_mode is  and repl_uuid is
[2023-08-04 08:36:04.915][68234][dn_6001_6002_6003][gs_ctl]: standby build try host(192.168.56.50) port(26000) success
[2023-08-04 08:36:04.915][68234][dn_6001_6002_6003][gs_ctl]: connected to server success, build started.
[2023-08-04 08:36:04.915][68234][dn_6001_6002_6003][gs_ctl]: clear old target dir success
[2023-08-04 08:36:04.915][68234][dn_6001_6002_6003][gs_ctl]: create build tag file success
[2023-08-04 08:36:04.916][68234][dn_6001_6002_6003][gs_ctl]: create build tag file again success
[2023-08-04 08:36:04.916][68234][dn_6001_6002_6003][gs_ctl]: get system identifier success
[2023-08-04 08:36:04.916][68234][dn_6001_6002_6003][gs_ctl]: receiving and unpacking files...
[2023-08-04 08:36:04.916][68234][dn_6001_6002_6003][gs_ctl]: create backup label success
[2023-08-04 08:36:07.391][68234][dn_6001_6002_6003][gs_ctl]: xlog start point: 0/5008718
[2023-08-04 08:36:07.391][68234][dn_6001_6002_6003][gs_ctl]: begin build tablespace list
[2023-08-04 08:36:07.391][68234][dn_6001_6002_6003][gs_ctl]: finish build tablespace list
[2023-08-04 08:36:07.391][68234][dn_6001_6002_6003][gs_ctl]: begin get xlog by xlogstream
[2023-08-04 08:36:07.391][68234][dn_6001_6002_6003][gs_ctl]: starting background WAL receiver
[2023-08-04 08:36:07.391][68234][dn_6001_6002_6003][gs_ctl]: starting walreceiver
[2023-08-04 08:36:07.391][68234][dn_6001_6002_6003][gs_ctl]: begin receive tar files
[2023-08-04 08:36:07.392][68234][dn_6001_6002_6003][gs_ctl]: receiving and unpacking files...
[2023-08-04 08:36:07.424][68234][dn_6001_6002_6003][gs_ctl]: standby build try host(192.168.56.50) port(26000) success
[2023-08-04 08:36:07.429][68234][dn_6001_6002_6003][gs_ctl]: check identify system success
[2023-08-04 08:36:07.437][68234][dn_6001_6002_6003][gs_ctl]: send START_REPLICATION 0/5000000 success
[2023-08-04 08:36:12.641][68234][dn_6001_6002_6003][gs_ctl]: finish receive tar files
[2023-08-04 08:36:12.641][68234][dn_6001_6002_6003][gs_ctl]: xlog end point: 0/5008838
[2023-08-04 08:36:12.642][68234][dn_6001_6002_6003][gs_ctl]: fetching MOT checkpoint
[2023-08-04 08:36:12.820][68234][dn_6001_6002_6003][gs_ctl]: waiting for background process to finish streaming...
[2023-08-04 08:36:18.521][68234][dn_6001_6002_6003][gs_ctl]: starting fsync all files come from source.
[2023-08-04 08:36:26.308][68234][dn_6001_6002_6003][gs_ctl]: finish fsync all files.
[2023-08-04 08:36:26.313][68234][dn_6001_6002_6003][gs_ctl]: build dummy dw file success
[2023-08-04 08:36:26.313][68234][dn_6001_6002_6003][gs_ctl]: rename build status file success
[2023-08-04 08:36:26.321][68234][dn_6001_6002_6003][gs_ctl]: standby full build build completed(/app/ogdata/data/dn1).
[2023-08-04 08:36:26.758][68234][dn_6001_6002_6003][gs_ctl]: waiting for server to start...
.0 LOG:  [Alarm Module]can not read GAUSS_WARNING_TYPE env.

0 LOG:  [Alarm Module]Host Name: pghost6

0 LOG:  [Alarm Module]Host IP: pghost6. Copy hostname directly in case of taking 10s to use 'gethostbyname' when /etc/hosts does not contain <HOST IP>

0 LOG:  [Alarm Module]Cluster Name: gauss_omm

0 LOG:  [Alarm Module]Invalid data in AlarmItem file! Read alarm English name failed! line: 57

0 WARNING:  failed to open feature control file, please check whether it exists: FileName=gaussdb.version, Errno=2, Errmessage=No such file or directory.
0 WARNING:  failed to parse feature control file: gaussdb.version.
0 WARNING:  Failed to load the product control file, so gaussdb cannot distinguish product version.
The core dump path is an invalid directory
2023-08-04 08:36:26.895 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [REDO] LOG:  Recovery parallelism, cpu count = 1, max = 4, actual = 1
2023-08-04 08:36:26.895 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [REDO] LOG:  ConfigRecoveryParallelism, true_max_recovery_parallelism:4, max_recovery_parallelism:4
2023-08-04 08:36:27.010 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] LOG:  [Alarm Module]can not read GAUSS_WARNING_TYPE env.

2023-08-04 08:36:27.010 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] LOG:  [Alarm Module]Host Name: pghost6

2023-08-04 08:36:27.010 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] LOG:  [Alarm Module]Host IP: pghost6. Copy hostname directly in case of taking 10s to use 'gethostbyname' when /etc/hosts does not contain <HOST IP>

2023-08-04 08:36:27.010 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] LOG:  [Alarm Module]Cluster Name: gauss_omm

2023-08-04 08:36:27.010 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] LOG:  [Alarm Module]Invalid data in AlarmItem file! Read alarm English name failed! line: 57

2023-08-04 08:36:27.139 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] LOG:  loaded library "security_plugin"
2023-08-04 08:36:27.144 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] WARNING:  could not create any HA TCP/IP sockets
2023-08-04 08:36:27.144 [unknown] [unknown] localhost 140411958592896 0[0:0#0] 0 [BACKEND] FATAL:  could not create lock file "/app/opengauss/tmp/.s.PGSQL.26000.lock": No such file or directory
[2023-08-04 08:36:27.159][68234][dn_6001_6002_6003][gs_ctl]: waitpid 68257 failed, exitstatus is 256, ret is 2

[2023-08-04 08:36:27.159][68234][dn_6001_6002_6003][gs_ctl]: stopped waiting
[2023-08-04 08:36:27.159][68234][dn_6001_6002_6003][gs_ctl]: could not start server
Examine the log output.
[2023-08-04 08:36:27.159][68234][dn_6001_6002_6003][gs_ctl]: fopen build pid file "/app/ogdata/data/dn1/gs_build.pid" success
[2023-08-04 08:36:27.159][68234][dn_6001_6002_6003][gs_ctl]: fprintf build pid file "/app/ogdata/data/dn1/gs_build.pid" success
[2023-08-04 08:36:27.164][68234][dn_6001_6002_6003][gs_ctl]: fsync build pid file "/app/ogdata/data/dn1/gs_build.pid" success
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
  • 63.
  • 64.
  • 65.
  • 66.
  • 67.
  • 68.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
  • 75.
  • 76.
  • 77.
  • 78.
  • 79.
  • 80.
  • 81.
  • 82.
  • 83.
  • 84.
  • 85.
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.

报错1

could not create lock file "/app/opengauss/tmp/.s.PGSQL.26000.lock": No such file or directory
 # 创建 /app/opengauss/tmp 目录，再次build。
 
 .........................................
[2023-08-04 08:45:10.215][68303][dn_6001_6002_6003][gs_ctl]:  done
[2023-08-04 08:45:10.533][68303][dn_6001_6002_6003][gs_ctl]: server started (/app/ogdata/data/dn1)
[2023-08-04 08:45:10.602][68303][dn_6001_6002_6003][gs_ctl]: fopen build pid file "/app/ogdata/data/dn1/gs_build.pid" success
[2023-08-04 08:45:10.602][68303][dn_6001_6002_6003][gs_ctl]: fprintf build pid file "/app/ogdata/data/dn1/gs_build.pid" success
[2023-08-04 08:45:11.525][68303][dn_6001_6002_6003][gs_ctl]: fsync build pid file "/app/ogdata/data/dn1/gs_build.pid" success
# 从以上日志可以看到build已经成功，查看进程和集群状态，发现集群已经恢复正常。
omm@pghost6 /app/ogdata/data/dn1$ ps x
    PID TTY      STAT   TIME COMMAND
  68150 pts/0    S      0:00 -bash
  68320 ?        Ssl    0:03 /app/opengauss/app/2.0.1/bin/gaussdb -D /app/ogdata/data/dn1 -M standby
  68377 pts/0    R+     0:00 ps x
omm@pghost6 /app/ogdata/data/dn1$ gs_om -t status --detail
[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
current_az      : AZ_ALL

[  Datanode State   ]

    node   node_ip         port      instance                     state
---------------------------------------------------------------------------------------
1  pghost3 192.168.56.30   26000      6001 /app/ogdata/data/dn1   P Primary Normal
2  pghost5 192.168.56.50   26000      6002 /app/ogdata/data/dn1   S Standby Normal
3  pghost6 192.168.56.60   26000      6003 /app/ogdata/data/dn1   S Standby Normal
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.

删除​​pg_tblspc​​无效目录

如果​​pghost6​​​节点是通过安装单节点集群以后再​​build​​​修复的话，修复成功后需要注意​​pg_tblspc​​目录下无效文件的大小，如太大，要考虑删除，避免占用较大的磁盘空间。




文章转载自公众号：openGauss

分类
数据库
标签
openGauss
已于2023-9-12 11:16:41修改
赞
收藏
回复
举报
回复
    相关推荐
    这个用户很懒，还没有个人简介
    觉得TA不错？点个关注精彩不错过
    帖子
    视频
    声望
    粉丝
    最近发布
    热门推荐
    进行中——【有奖征文活动】和大家分享你的鸿蒙开发手记​ 16回复
    聊聊 2025 年最值得关注的 IT 技术趋势，一起来投票！ 14回复
    #我的鸿蒙开发手记# 我的HarmonyOS成长之路 3回复
    #我的鸿蒙开发手记# 鸿蒙版云闪付用户头像编辑功能实践 5回复
    #我的鸿蒙开发手记#Atomgit 客户端实战，开启鸿蒙开发新旅程！ 2回复
    相关问题
    Mongodb集群节点故障恢复场景的分片中的所有Mongod节点宕机是什么？ 1回答
    为什么在 openGauss 中使用“explain timing”时显示错误？ 1回答
    docker swarm集群manager节点docker ps无反应的问题 1回答
    PolarDB 集群版最少包含多少个只读节点？ 1回答
    PolarDB 集群版最多包含多少个只读节点？ 1回答
    社区精华内容

    目录