51CTO首页
内容精选
博客
学堂
精培
企业培训
CTO训练营
智能汽车
开源基础软件社区
WOT全球技术创新大会
公众号矩阵
移动端
短视频免费课程课程排行直播课软考学堂
精品班厂商认证IT技术2023年软考PMP项目管理
在线学习企业服务
CTO训练营技术经理研习营LeaTech峰会
文章资源问答开源课堂专栏直播
51CTO
开源基础软件社区
51CTO技术栈
51CTO官微
51CTO学堂
51CTO博客
CTO训练营
开源基础软件社区订阅号
51CTO学堂APP
51CTO学堂企业版APP
开源基础软件社区视频号
资源 开源课堂 专栏 极客Show 鸿蒙技术特刊
汽车开发者社区

【我和openGauss的故事】openGauss修改pg_hba导致节点无法启动及cm主备切换

老老老JR老北
发布于 2023-8-29 15:40
浏览
0收藏

一、状态正常

[omm@Euler1 ~]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Primary
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Standby
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Primary Normal
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Standby Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

二、修改pg_hba

[omm@Euler1 data]$ vi pg_hba.conf
host    all    all    172.16.221.6       sha256
host    all    all    172.16.221.118     sha256

三、关闭

[omm@Euler1 data]$ gs_om -t stop
Stopping cluster.
=========================================
Successfully stopped cluster.
=========================================
End stop cluster.
[omm@Euler1 data]$ gs_om -t start
Starting cluster.
======================================================================

^CTraceback (most recent call last):
File "/database/opengauss/tool/script/gs_om", line 837, in <module>
    main()
  File "/database/opengauss/tool/script/gs_om", line 806, in main
    impl.doStart()
  File "/database/opengauss/tool/script/impl/om/OmImpl.py", line 88, in doStart
    self.doStartCluster()
  File "/database/opengauss/tool/script/impl/om/OLAP/OmImplOLAP.py", line 183, in doStartCluster
    self.doStartClusterByCm()
  File "/database/opengauss/tool/script/impl/om/OLAP/OmImplOLAP.py", line 169, in doStartClusterByCm
    self.dataDir)
  File "/database/opengauss/tool/script/gspylib/component/CM/CM_OLAP/CM_OLAP.py", line 279, in startCluster
    result_set = CmdUtil.retryGetstatusoutput(cmd, retry_time=retry_times)
  File "/database/opengauss/tool/script/base_utils/os/cmd_util.py", line 566, in retryGetstatusoutput
    (status, output) = subprocess.getstatusoutput(cmd)
  File "/usr/lib64/python3.7/subprocess.py", line 611, in getstatusoutput
    data = check_output(cmd, shell=True, text=True, stderr=STDOUT)
  File "/usr/lib64/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.7/subprocess.py", line 490, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib64/python3.7/subprocess.py", line 951, in communicate
    stdout = self.stdout.read()
KeyboardInterrupt

重启，卡住

[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Primary
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Standby
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Degraded
redistributing  : No
balanced        : No
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Pending Starting
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Primary Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

其他节点已经启动，本节点一直处于Pending Starting

[omm@Euler1 data]$ gs_om -t stop -h Euler1
Stopping node.
=========================================
Successfully stopped node.
=========================================
End stop node.
[omm@Euler1 data]$ gs_om -t start -h Euler1
Starting node.
======================================================================
Successfully started node.
======================================================================
End start node.
Successfully started node.
[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Standby
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Primary
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Degraded
redistributing  : No
balanced        : No
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Pending Starting
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Primary Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

关闭、重启本节点，状态一直是Pending Starting，并且由于故障切换至另外节点

四、修复

仔细观察发现是pg_hba.conf中添加的ip地址格式错误，ip地址后面没有写掩码，修改完成.

[omm@Euler1 data]$ gs_om -t stop -h Euler1
Stopping node.
=========================================
Successfully stopped node.
=========================================
End stop node.
[omm@Euler1 data]$ gs_om -t start -h Euler1
Starting node.
======================================================================
Successfully started node.
======================================================================
End start node.
Successfully started node.
[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Standby
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Primary
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : No
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Standby Normal
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Primary Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal

修改完成后可以正常启动，切换回来。

[omm@Euler1 data]$ gs_ctl switchover -D /database/opengauss/data
[2023-07-13 14:05:22.942][2167163][][gs_ctl]: gs_ctl switchover ,datadir is /database/opengauss/data 
[2023-07-13 14:05:22.942][2167163][][gs_ctl]: switchover term (1)
[2023-07-13 14:05:22.947][2167163][][gs_ctl]: waiting for server to switchover........
[2023-07-13 14:05:27.975][2167163][][gs_ctl]: done
[2023-07-13 14:05:27.975][2167163][][gs_ctl]: switchover completed (/database/opengauss/data)
[omm@Euler1 data]$ gs_om -t status -detail
[GAUSS-50000] : Unrecognized parameter: -d.
[omm@Euler1 data]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  172.16.220.45   1    /database/opengauss/cm/cm_server Standby
2  Euler2 172.16.220.201  2    /database/opengauss/cm/cm_server Primary
3  Euler3 172.16.220.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1  172.16.220.45   6001 /database/opengauss/data P Primary Normal
2  Euler2 172.16.220.201  6002 /database/opengauss/data S Standby Normal
3  Euler3 172.16.220.221  6003 /database/opengauss/data S Standby Normal
[omm@Euler1 data]$ gs_om -t refreshconf
Generating dynamic configuration file for all nodes.
Successfully generated dynamic configuration file.

存在一个问题，Datanode切换成功，但是CMServer的Primary节点依然在2节点。

五、CM切换

目前没有发现可以手工切换cm的命令，cm_ctl依然是切换数据库主备命令，不过cm只是故障转移组件，不影响。对于强迫症患者，可以考虑将主节点全部切换到同一个节点或者在2节点复现上面的错误由cm自动去切换。
既然可以通过触发故障切换实现切换效果，那么当然也可以通过kill dn进程触发切换

[omm@Euler2 ~]$ ps ux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
omm         1627  0.0  0.0  20056  9968 ?        Ss   Jun14   3:42 /usr/lib/systemd/systemd --user
omm         1629  0.0  0.0  24124  2812 ?        S    Jun14   0:00 (sd-pam)
omm         2422  0.4  0.0  18564 13608 ?        S    Jun14 172:21 /database/opengauss/app/bin/om_monitor -L /database/opengauss/log/omm/cm/om_monitor
omm      2396593  0.0  0.0 214116  3676 pts/0    S+   11:53   0:00 -bash
omm      2611562  2.0  0.0 799340 24364 ?        Sl   14:18   4:04 /database/opengauss/app/bin/cm_agent
omm      2611575  4.7  0.3 6505600 400692 ?      Sl   14:18   9:23 /database/opengauss/app/bin/cm_server
omm      2611586  3.0  2.8 47986344 3774748 ?    Sl   14:18   6:02 /database/opengauss/app/bin/gaussdb -D /database/opengauss/data -M pending
omm      2611593  0.0  0.0 1401088 75952 ?       Sl   14:18   0:00 gaussdb fenced UDF master process
omm      2908296  0.0  0.0  15020  4772 ?        S    17:37   0:00 sshd: omm@pts/1
omm      2908297  0.0  0.0 214088  3752 pts/1    Ss   17:37   0:00 -bash
omm      2908380  0.0  0.0 215868  3228 pts/1    R+   17:37   0:00 ps ux
[omm@Euler2 ~]$ kill -9 2611575

在二节点找到cm_server进程,kill进程，kill后om会自动重新拉起

[omm@Euler2 ~]$ gs_om -t status --detail
[  CMServer State   ]

node                node_ip         instance                             state
--------------------------------------------------------------------------------
1  Euler1  10.236.160.45  1    /database/opengauss/cm/cm_server Primary
2  Euler2 10.236.160.201  2    /database/opengauss/cm/cm_server Standby
3  Euler3 10.236.160.221  3    /database/opengauss/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node                node_ip         instance                     state            
----------------------------------------------------------------------------------
1  Euler1 10.236.160.45   6001 /database/opengauss/data P Primary Normal
2  Euler2 10.236.160.201  6002 /database/opengauss/data S Standby Normal
3  Euler3 10.236.160.221  6003 /database/opengauss/data S Standby Normal

此时发现cm主备节点已经更换。




文章转载自公众号：openGauss

分类
其他
标签
openGauss
已于2023-8-29 15:40:25修改
赞
收藏
回复
举报
回复
    相关推荐
    老老老JR老北
    这个用户很懒，还没有个人简介
    帖子
    视频
    声望
    粉丝
    最近发布
    热门推荐
    【投票结果公布】你如何看待华为Mate 60 Pro上架？ 86回复
    HarmonyOS Codelab 优秀样例——溪村小镇（ArkTS） 7回复
    OpenHarmony 文件监听开发样例介绍 10回复
    OpenHarmony：全流程讲解如何编写GPIO平台驱动以及应用程序 7回复
    OpenHarmony AI框架开发指导 9回复
    相关问题
    为什么在 openGauss 中使用“explain timing”时显示错误？ 1回答
    通知栏及控制空心无法切换、暂停音乐。操作无反应 3回答
    分布式服务，添加FLAG_ABILITYSLICE_MULTI_DEVICE，导致无法启动本地服务 1回答
    obd升级后导致obproxy服务不能启动 1回答
    PolarDB MySQL读写分离为什么当主节点负载很高时只读节点上负载却很低 1回答
    社区精华内容

    目录