51CTO首页
AI.x社区
博客
学堂
精品班
直播训练营
企业培训
鸿蒙开发者社区
WOT技术大会
AIGC创新中国行
公众号矩阵
移动端
短视频免费课程课程排行直播课软考学堂
全部课程厂商认证IT技术2024年软考PMP项目管理软考资讯
在线学习
文章资源问答课堂专栏直播
51CTO
鸿蒙开发者社区
51CTO技术栈
51CTO官微
51CTO学堂
51CTO博客
CTO训练营
鸿蒙开发者社区订阅号
51CTO题库小程序
51CTO学堂APP
51CTO学堂企业版APP
鸿蒙开发者社区视频号
活动 短视频 专栏 极客Show 鸿蒙技术特刊

【我和openGauss的故事】openGauss修改pg_hba导致节点无法启动及cm主备切换

老老老JR老北
发布于 2023-8-29 15:40
浏览
0收藏

一、状态正常

[omm@Euler1 ~]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Primary
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Standby
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Primary Normal
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Standby Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

二、修改pg_hba

[omm@Euler1 data]$ vi pg_hba.conf
host    all    all    172.16.221.6       sha256
host    all    all    172.16.221.118     sha256

三、关闭

[omm@Euler1 data]$ gs_om -t stop
Stopping cluster.
=========================================
Successfully stopped cluster.
=========================================
End stop cluster.
[omm@Euler1 data]$ gs_om -t start
Starting cluster.
======================================================================

^CTraceback (most recent call last):
File "/database/opengauss/tool/script/gs_om", line 837, in <module>
    main()
  File "/database/opengauss/tool/script/gs_om", line 806, in main
    impl.doStart()
  File "/database/opengauss/tool/script/impl/om/OmImpl.py", line 88, in doStart
    self.doStartCluster()
  File "/database/opengauss/tool/script/impl/om/OLAP/OmImplOLAP.py", line 183, in doStartCluster
    self.doStartClusterByCm()
  File "/database/opengauss/tool/script/impl/om/OLAP/OmImplOLAP.py", line 169, in doStartClusterByCm
    self.dataDir)
  File "/database/opengauss/tool/script/gspylib/component/CM/CM_OLAP/CM_OLAP.py", line 279, in startCluster
    result_set = CmdUtil.retryGetstatusoutput(cmd, retry_time=retry_times)
  File "/database/opengauss/tool/script/base_utils/os/cmd_util.py", line 566, in retryGetstatusoutput
    (status, output) = subprocess.getstatusoutput(cmd)
  File "/usr/lib64/python3.7/subprocess.py", line 611, in getstatusoutput
    data = check_output(cmd, shell=True, text=True, stderr=STDOUT)
  File "/usr/lib64/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.7/subprocess.py", line 490, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib64/python3.7/subprocess.py", line 951, in communicate
    stdout = self.stdout.read()
KeyboardInterrupt

重启，卡住

[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Primary
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Standby
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Degraded
redistributing  : No
balanced        : No
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Pending Starting
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Primary Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

其他节点已经启动，本节点一直处于Pending Starting

[omm@Euler1 data]$ gs_om -t stop -h Euler1
Stopping node.
=========================================
Successfully stopped node.
=========================================
End stop node.
[omm@Euler1 data]$ gs_om -t start -h Euler1
Starting node.
======================================================================
Successfully started node.
======================================================================
End start node.
Successfully started node.
[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Standby
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Primary
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Degraded
redistributing  : No
balanced        : No
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Pending Starting
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Primary Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

关闭、重启本节点，状态一直是Pending Starting，并且由于故障切换至另外节点

四、修复

仔细观察发现是pg_hba.conf中添加的ip地址格式错误，ip地址后面没有写掩码，修改完成.

[omm@Euler1 data]$ gs_om -t stop -h Euler1
Stopping node.
=========================================
Successfully stopped node.
=========================================
End stop node.
[omm@Euler1 data]$ gs_om -t start -h Euler1
Starting node.
======================================================================
Successfully started node.
======================================================================
End start node.
Successfully started node.
[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Standby
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Primary
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : No
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Standby Normal
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Primary Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

修改完成后可以正常启动，切换回来。

[omm@Euler1 data]$ gs_ctl switchover -D /database/opengauss/data
[2023-07-13 14:05:22.942][2167163][][gs_ctl]: gs_ctl switchover ,datadir is /database/opengauss/data 
[2023-07-13 14:05:22.942][2167163][][gs_ctl]: switchover term (1)
[2023-07-13 14:05:22.947][2167163][][gs_ctl]: waiting for server to switchover........
[2023-07-13 14:05:27.975][2167163][][gs_ctl]: done
[2023-07-13 14:05:27.975][2167163][][gs_ctl]: switchover completed (/database/opengauss/data)
[omm@Euler1 data]$ gs_om -t status -detail
[GAUSS-50000] : Unrecognized parameter: -d.
[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Standby
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Primary
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Primary Normal
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Standby Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal
[omm@Euler1 data]$ gs_om -t refreshconf
Generating dynamic configuration file for all nodes.
Successfully generated dynamic configuration file.

存在一个问题，Datanode切换成功，但是CMServer的Primary节点依然在2节点。

五、CM切换

目前没有发现可以手工切换cm的命令，cm_ctl依然是切换数据库主备命令，不过cm只是故障转移组件，不影响。对于强迫症患者，可以考虑将主节点全部切换到同一个节点或者在2节点复现上面的错误由cm自动去切换。
既然可以通过触发故障切换实现切换效果，那么当然也可以通过kill dn进程触发切换

[omm@Euler2 ~]$ ps ux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
omm         1627  0.0  0.0  20056  9968 ?        Ss   Jun14   3:42 /usr/lib/systemd/systemd --user
omm         1629  0.0  0.0  24124  2812 ?        S    Jun14   0:00 (sd-pam)
omm         2422  0.4  0.0  18564 13608 ?        S    Jun14 172:21 /database/opengauss/app/bin/om_monitor -L /database/opengauss/log/omm/cm/om_monitor
omm      2396593  0.0  0.0 214116  3676 pts/0    S+   11:53   0:00 -bash
omm      2611562  2.0  0.0 799340 24364 ?        Sl   14:18   4:04 /database/opengauss/app/bin/cm_agent
omm      2611575  4.7  0.3 6505600 400692 ?      Sl   14:18   9:23 /database/opengauss/app/bin/cm_server
omm      2611586  3.0  2.8 47986344 3774748 ?    Sl   14:18   6:02 /database/opengauss/app/bin/gaussdb -D /database/opengauss/data -M pending
omm      2611593  0.0  0.0 1401088 75952 ?       Sl   14:18   0:00 gaussdb fenced UDF master process
omm      2908296  0.0  0.0  15020  4772 ?        S    17:37   0:00 sshd: omm@pts/1
omm      2908297  0.0  0.0 214088  3752 pts/1    Ss   17:37   0:00 -bash
omm      2908380  0.0  0.0 215868  3228 pts/1    R+   17:37   0:00 ps ux
[omm@Euler2 ~]$ kill -9 2611575

在二节点找到cm_server进程,kill进程，kill后om会自动重新拉起

[omm@Euler2 ~]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  10.236.160.45  1    /database/opengauss/cm/cm_server Primary
2  Euler2 10.236.160.201  2    /database/opengauss/cm/cm_server Standby
3  Euler3 10.236.160.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1 10.236.160.45   6001 /database/opengauss/data P Primary Normal
2  Euler2 10.236.160.201  6002 /database/opengauss/data S Standby Normal
3  Euler3 10.236.160.221  6003 /database/opengauss/data S Standby Normal

此时发现cm主备节点已经更换。




文章转载自公众号：openGauss

分类
数据库
标签
openGauss
已于2023-8-29 15:40:25修改
赞
收藏
回复
举报
回复
    相关推荐
    老老老JR老北
    这个用户很懒，还没有个人简介
    帖子
    视频
    声望
    粉丝
    最近发布
    热门推荐
    OpenHarmony 4.1 Release的组件分析 4回复
    OpenHarmony开发-系统烧录 2回复
    可以使用中文开发鸿蒙APP啦, 中文代码实现喜马拉雅APP主页示例 4回复
    OpenHarmony投屏工具OHScrcpy Beta4发布，适配API11，支持模拟输入文本 2回复
    OpenHarmony系统之C++线程异步回调JS机制介绍 1回复
    相关问题
    为什么在 openGauss 中使用“explain timing”时显示错误？ 1回答
    通知栏及控制空心无法切换、暂停音乐。操作无反应 3回答
    分布式服务，添加FLAG_ABILITYSLICE_MULTI_DEVICE，导致无法启动本地服务 1回答
    签名打包出的hsp包主工程无法引用和打开 0回答
    PolarDB MySQL读写分离为什么当主节点负载很高时只读节点上负载却很低 1回答
    社区精华内容

    目录