作者：杨文

DBA，负责客户项目的需求与维护，会点数据库，不限于MySQL、Redis、Cassandra、GreenPlum、ClickHouse、Elastic、TDSQL等等。

本文来源：原创投稿

*爱可生开源社区出品，原创内容未经授权不得随意使用，转载请联系小编并注明来源。

一、前情提要：

我们知道Greenplum集群由Master Severs和Segment Severs组成。其中故障存在三种类别：Master故障、Segment故障、数据异常。之前我们已经聊过“Master故障”和“数据异常”的处理方式，今天将介绍Segment故障的处理方式。

二、本地模拟故障环境：

2.1、第一种情况：段故障。

[gpadmin@master ~]$ gpstate
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: 
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
...
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-Greenplum instance status summary
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Master instance                                           = Active
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Master standby                                            = standby
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Standby master state                                      = Standby host passive
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total segment instance count from metadata                = 40
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Primary Segment Status
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total primary segments                                    = 20
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total primary segment valid (at master)                   = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total primary segment failures (at master)                = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number postmaster processes found                   = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Mirror Segment Status
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total mirror segments                                     = 20
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total mirror segment valid (at master)                    = 20
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total mirror segment failures (at master)                 = 0
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number postmaster processes found                   = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments   = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number mirror segments acting as mirror segments    = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------

[gpadmin@master ~]$ gpstate -m
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -m
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--------------------------------------------------------------
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--Current GPDB mirror list and status
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--Type = Group
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--------------------------------------------------------------
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   Mirror       Datadir                            Port    Status              Data Status    
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg0    56000   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg1    56001   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg2    56002   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg3    56000   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg4    56001   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg5    56002   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg6    56000   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg7    56001   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg8    56002   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg9    56000   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg10   56001   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg11   56002   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg12   56000   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg13   56001   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg14   56002   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg15   56003   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg16   56003   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg17   56003   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg18   56003   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg19   56003   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--------------------------------------------------------------
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-4 segment(s) configured as mirror(s) are acting as primaries
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-4 segment(s) configured as mirror(s) have failed
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-4 mirror segment(s) acting as primaries are not synchronized

2.2、第二种情况：表空间故障。

[gpadmin@data05 ~]$ cd /greenplum/gpdata/mirror/gpseg10
[gpadmin@data05 gpseg10]$ ls
backup_label.old    gpmetrics               pg_clog            pg_logical    pg_stat                PG_VERSION            postmaster.pid
base                gpperfmon               pg_distributedlog  pg_multixact  pg_stat_tmp            pg_xlog               recovery.conf
fts_probe_file.bak  gpsegconfig_dump        pg_dynshmem        pg_notify     pg_subtrans            postgresql.auto.conf  recovery.done
global              gpssh.conf              pg_hba.conf        pg_replslot   pg_tblspc              postgresql.conf
gpexpand.pid        internal.auto.conf      pg_ident.conf      pg_serial     pg_twophase            postgresql.conf.bak
gpexpand.status     internal.auto.conf.bak  pg_log             pg_snapshots  pg_utilitymodedtmredo  postmaster.opts
[gpadmin@data05 gpseg10]$ rm -rf pg_tblspc/

[gpadmin@master ~]$ gpstate -e
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -e
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
20221127:23:13:30:026114 gpstate:master:gpadmin-[WARNING]:-pg_stat_replication shows no standby connections
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-Segment Mirroring Status Report
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-Downed Segments (may include segments where status could not be retrieved)
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-   Segment      Port    Config status   Status
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-   data05       56001   Up              Process error -- database process may be down

三、故障分析及解决：

3.1、针对“2.1”情况的处理：

在线生成一个配置文件：

[gpadmin@master ~]$ gprecoverseg -o ./recover1
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-Starting gprecoverseg with args: -o ./recover1
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-Configuration file output to ./recover1 successfully.[gpadmin@master ~]$ more recover1
data05|55000|/greenplum/gpdata/primary/gpseg12
data05|55001|/greenplum/gpdata/primary/gpseg13
data05|55002|/greenplum/gpdata/primary/gpseg14
data05|55003|/greenplum/gpdata/primary/gpseg19

通过生成的配置文件进行修复集群：

[gpadmin@master ~]$ gprecoverseg -i ./recover1 -a

检查状态：

[gpadmin@master ~]$ gpstate -e
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -e
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-Segment Mirroring Status Report
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-Segments with Primary and Mirror Roles Switched
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   Current Primary   Port    Mirror       Port
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data04            56000   data05       55000
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data04            56001   data05       55001
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data04            56002   data05       55002
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data01            56003   data05       55003

[gpadmin@master ~]$ psql -c "select * from gp_segment_configuration order by content asc,dbid;"dbid | content | role | preferred_role | mode | status | port  | hostname | address |             datadir              
------+---------+------+----------------+------+--------+-------+----------+---------+-----------------------------------44 |      -1 | p    | p              | s    | u      |  5432 | master   | master  | /greenplum/gpdata/master/gpseg-145 |      -1 | m    | m              | s    | u      |  5432 | standby  | standby | /greenplum/gpdata/master/gpseg-12 |       0 | p    | p              | s    | u      | 55000 | data01   | data01  | /greenplum/gpdata/primary/gpseg011 |       0 | m    | m              | s    | u      | 56000 | data02   | data02  | /greenplum/gpdata/mirror/gpseg03 |       1 | p    | p              | s    | u      | 55001 | data01   | data01  | /greenplum/gpdata/primary/gpseg112 |       1 | m    | m              | s    | u      | 56001 | data02   | data02  | /greenplum/gpdata/mirror/gpseg14 |       2 | p    | p              | s    | u      | 55002 | data01   | data01  | /greenplum/gpdata/primary/gpseg213 |       2 | m    | m              | s    | u      | 56002 | data02   | data02  | /greenplum/gpdata/mirror/gpseg25 |       3 | p    | p              | s    | u      | 55000 | data02   | data02  | /greenplum/gpdata/primary/gpseg314 |       3 | m    | m              | s    | u      | 56000 | data03   | data03  | /greenplum/gpdata/mirror/gpseg36 |       4 | p    | p              | s    | u      | 55001 | data02   | data02  | /greenplum/gpdata/primary/gpseg415 |       4 | m    | m              | s    | u      | 56001 | data03   | data03  | /greenplum/gpdata/mirror/gpseg47 |       5 | p    | p              | s    | u      | 55002 | data02   | data02  | /greenplum/gpdata/primary/gpseg516 |       5 | m    | m              | s    | u      | 56002 | data03   | data03  | /greenplum/gpdata/mirror/gpseg58 |       6 | p    | p              | s    | u      | 55000 | data03   | data03  | /greenplum/gpdata/primary/gpseg617 |       6 | m    | m              | s    | u      | 56000 | data01   | data01  | /greenplum/gpdata/mirror/gpseg69 |       7 | p    | p              | s    | u      | 55001 | data03   | data03  | /greenplum/gpdata/primary/gpseg718 |       7 | m    | m              | s    | u      | 56001 | data01   | data01  | /greenplum/gpdata/mirror/gpseg710 |       8 | p    | p              | s    | u      | 55002 | data03   | data03  | /greenplum/gpdata/primary/gpseg819 |       8 | m    | m              | s    | u      | 56002 | data01   | data01  | /greenplum/gpdata/mirror/gpseg821 |       9 | p    | p              | s    | u      | 55000 | data04   | data04  | /greenplum/gpdata/primary/gpseg930 |       9 | m    | m              | s    | u      | 56000 | data05   | data05  | /greenplum/gpdata/mirror/gpseg922 |      10 | p    | p              | s    | u      | 55001 | data04   | data04  | /greenplum/gpdata/primary/gpseg1031 |      10 | m    | m              | s    | u      | 56001 | data05   | data05  | /greenplum/gpdata/mirror/gpseg1023 |      11 | p    | p              | s    | u      | 55002 | data04   | data04  | /greenplum/gpdata/primary/gpseg1132 |      11 | m    | m              | s    | u      | 56002 | data05   | data05  | /greenplum/gpdata/mirror/gpseg1124 |      12 | m    | p              | s    | u      | 55000 | data05   | data05  | /greenplum/gpdata/primary/gpseg1227 |      12 | p    | m              | s    | u      | 56000 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1225 |      13 | m    | p              | s    | u      | 55001 | data05   | data05  | /greenplum/gpdata/primary/gpseg1328 |      13 | p    | m              | s    | u      | 56001 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1326 |      14 | m    | p              | s    | u      | 55002 | data05   | data05  | /greenplum/gpdata/primary/gpseg1429 |      14 | p    | m              | s    | u      | 56002 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1433 |      15 | p    | p              | s    | u      | 55003 | data01   | data01  | /greenplum/gpdata/primary/gpseg1539 |      15 | m    | m              | s    | u      | 56003 | data02   | data02  | /greenplum/gpdata/mirror/gpseg1534 |      16 | p    | p              | s    | u      | 55003 | data02   | data02  | /greenplum/gpdata/primary/gpseg1640 |      16 | m    | m              | s    | u      | 56003 | data03   | data03  | /greenplum/gpdata/mirror/gpseg1635 |      17 | p    | p              | s    | u      | 55003 | data03   | data03  | /greenplum/gpdata/primary/gpseg1741 |      17 | m    | m              | s    | u      | 56003 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1736 |      18 | p    | p              | s    | u      | 55003 | data04   | data04  | /greenplum/gpdata/primary/gpseg1842 |      18 | m    | m              | s    | u      | 56003 | data05   | data05  | /greenplum/gpdata/mirror/gpseg1837 |      19 | m    | p              | s    | u      | 55003 | data05   | data05  | /greenplum/gpdata/primary/gpseg1938 |      19 | p    | m              | s    | u      | 56003 | data01   | data01  | /greenplum/gpdata/mirror/gpseg19
(42 rows)

可以看到所有段都是up状态了，但存在部分段角色异常。

修复角色状态：

[gpadmin@master ~]$ gprecoverseg -r

再次检查确认状态，此处省略。

3.2、针对“2.2”情况的处理：

如果可以自动生成配置文件，就使用自动生成的。如果无法自动生成，则手工创建：

[gpadmin@master ~]$ vi recover2
data05|56001|/greenplum/gpdata/mirror/gpseg10

通过生成的配置文件进行修复集群：

[gpadmin@master ~]$ gprecoverseg -i ./recover2 -a
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Starting gprecoverseg with args: -i ./recover2 -F
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Greenplum instance recovery parameters
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Recovery from configuration -i option supplied
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Recovery 1 of 1
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Synchronization mode                 = Full
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance host                 = data05
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance address              = data05
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance directory            = /greenplum/gpdata/mirror/gpseg10
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance port                 = 56001
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance host        = data04
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance address     = data04
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance directory   = /greenplum/gpdata/primary/gpseg10
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance port        = 55001
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Target                      = in-place
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-1 segment(s) to recover
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-Ensuring 1 failed segment(s) are stopped
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-Validating remote directories
20221127:23:15:48:026332 gprecoverseg:master:gpadmin-[INFO]:-Configuring new segments data05 (dbid 31): pg_basebackup: base backup completed
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Updating configuration with new mirrors
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Updating mirrors
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Starting mirrors
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-era is c6f862530103c913_221127213422
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Process results...
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Triggering FTS probe
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-******************************************************************
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-Updating segments for streaming is completed.
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-For segments updated successfully, streaming will continue in the background.
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-Use  gpstate -s  to check the streaming progress.
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-******************************************************************

进程检查：

[gpadmin@data05 gpseg10]$ ps -ef |grep postgres
gpadmin   45364      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg13 -p 55001
gpadmin   45367      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg12 -p 55000
gpadmin   45369      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg14 -p 55002
gpadmin   45373      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg19 -p 55003
gpadmin   45378      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg9 -p 56000
gpadmin   45380      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg18 -p 56003
gpadmin   45382      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg11 -p 56002
gpadmin   47899      1  0 23:15 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg10 -p 56001
......

表空间检查：

[gpadmin@data05 gpseg10]$ ls
backup_label.old    gpmetrics               pg_clog            pg_logical    pg_stat                PG_VERSION            postmaster.pid
base                gpperfmon               pg_distributedlog  pg_multixact  pg_stat_tmp            pg_xlog               recovery.conf
fts_probe_file.bak  gpsegconfig_dump        pg_dynshmem        pg_notify     pg_subtrans            postgresql.auto.conf  recovery.done
global              gpssh.conf              pg_hba.conf        pg_replslot   pg_tblspc              postgresql.conf
gpexpand.pid        internal.auto.conf      pg_ident.conf      pg_serial     pg_twophase            postgresql.conf.bak
gpexpand.status     internal.auto.conf.bak  pg_log             pg_snapshots  pg_utilitymodedtmredo  postmaster.opts

状态检查：

[gpadmin@master ~]$ gpstate -e
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -e
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-Segment Mirroring Status Report
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-All segments are running normally

对于这种情况，一般不会存在数据节点状态异常的情况：

[gpadmin@master ~]$ psql -c "select * from gp_segment_configuration order by content asc,dbid;"dbid | content | role | preferred_role | mode | status | port  | hostname | address |             datadir              
------+---------+------+----------------+------+--------+-------+----------+---------+-----------------------------------44 |      -1 | p    | p              | s    | u      |  5432 | master   | master  | /greenplum/gpdata/master/gpseg-145 |      -1 | m    | m              | s    | u      |  5432 | standby  | standby | /greenplum/gpdata/master/gpseg-12 |       0 | p    | p              | s    | u      | 55000 | data01   | data01  | /greenplum/gpdata/primary/gpseg011 |       0 | m    | m              | s    | u      | 56000 | data02   | data02  | /greenplum/gpdata/mirror/gpseg03 |       1 | p    | p              | s    | u      | 55001 | data01   | data01  | /greenplum/gpdata/primary/gpseg112 |       1 | m    | m              | s    | u      | 56001 | data02   | data02  | /greenplum/gpdata/mirror/gpseg14 |       2 | p    | p              | s    | u      | 55002 | data01   | data01  | /greenplum/gpdata/primary/gpseg213 |       2 | m    | m              | s    | u      | 56002 | data02   | data02  | /greenplum/gpdata/mirror/gpseg25 |       3 | p    | p              | s    | u      | 55000 | data02   | data02  | /greenplum/gpdata/primary/gpseg314 |       3 | m    | m              | s    | u      | 56000 | data03   | data03  | /greenplum/gpdata/mirror/gpseg36 |       4 | p    | p              | s    | u      | 55001 | data02   | data02  | /greenplum/gpdata/primary/gpseg415 |       4 | m    | m              | s    | u      | 56001 | data03   | data03  | /greenplum/gpdata/mirror/gpseg47 |       5 | p    | p              | s    | u      | 55002 | data02   | data02  | /greenplum/gpdata/primary/gpseg516 |       5 | m    | m              | s    | u      | 56002 | data03   | data03  | /greenplum/gpdata/mirror/gpseg58 |       6 | p    | p              | s    | u      | 55000 | data03   | data03  | /greenplum/gpdata/primary/gpseg617 |       6 | m    | m              | s    | u      | 56000 | data01   | data01  | /greenplum/gpdata/mirror/gpseg69 |       7 | p    | p              | s    | u      | 55001 | data03   | data03  | /greenplum/gpdata/primary/gpseg718 |       7 | m    | m              | s    | u      | 56001 | data01   | data01  | /greenplum/gpdata/mirror/gpseg710 |       8 | p    | p              | s    | u      | 55002 | data03   | data03  | /greenplum/gpdata/primary/gpseg819 |       8 | m    | m              | s    | u      | 56002 | data01   | data01  | /greenplum/gpdata/mirror/gpseg821 |       9 | p    | p              | s    | u      | 55000 | data04   | data04  | /greenplum/gpdata/primary/gpseg930 |       9 | m    | m              | s    | u      | 56000 | data05   | data05  | /greenplum/gpdata/mirror/gpseg922 |      10 | p    | p              | s    | u      | 55001 | data04   | data04  | /greenplum/gpdata/primary/gpseg1031 |      10 | m    | m              | s    | u      | 56001 | data05   | data05  | /greenplum/gpdata/mirror/gpseg1023 |      11 | p    | p              | s    | u      | 55002 | data04   | data04  | /greenplum/gpdata/primary/gpseg1132 |      11 | m    | m              | s    | u      | 56002 | data05   | data05  | /greenplum/gpdata/mirror/gpseg1124 |      12 | p    | p              | s    | u      | 55000 | data05   | data05  | /greenplum/gpdata/primary/gpseg1227 |      12 | m    | m              | s    | u      | 56000 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1225 |      13 | p    | p              | s    | u      | 55001 | data05   | data05  | /greenplum/gpdata/primary/gpseg1328 |      13 | m    | m              | s    | u      | 56001 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1326 |      14 | p    | p              | s    | u      | 55002 | data05   | data05  | /greenplum/gpdata/primary/gpseg1429 |      14 | m    | m              | s    | u      | 56002 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1433 |      15 | p    | p              | s    | u      | 55003 | data01   | data01  | /greenplum/gpdata/primary/gpseg1539 |      15 | m    | m              | s    | u      | 56003 | data02   | data02  | /greenplum/gpdata/mirror/gpseg1534 |      16 | p    | p              | s    | u      | 55003 | data02   | data02  | /greenplum/gpdata/primary/gpseg1640 |      16 | m    | m              | s    | u      | 56003 | data03   | data03  | /greenplum/gpdata/mirror/gpseg1635 |      17 | p    | p              | s    | u      | 55003 | data03   | data03  | /greenplum/gpdata/primary/gpseg1741 |      17 | m    | m              | s    | u      | 56003 | data04   | data04  | /greenplum/gpdata/mirror/gpseg1736 |      18 | p    | p              | s    | u      | 55003 | data04   | data04  | /greenplum/gpdata/primary/gpseg1842 |      18 | m    | m              | s    | u      | 56003 | data05   | data05  | /greenplum/gpdata/mirror/gpseg1837 |      19 | p    | p              | s    | u      | 55003 | data05   | data05  | /greenplum/gpdata/primary/gpseg1938 |      19 | m    | m              | s    | u      | 56003 | data01   | data01  | /greenplum/gpdata/mirror/gpseg19

查看数据：

[gpadmin@master ~]$ psql -c "select gp_segment_id,count(*) from test_yw;"

同样可以看到所有数据节点上的数据都是正常的。