故障案例 | 主从复制环境中tokudb引擎报错排查过程

[root@redhat76-greatdb greatdb]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        33T   32T  1.1T  97% /
devtmpfs         63G     0   63G   0% /dev
tmpfs            63G     0   63G   0% /dev/shm
tmpfs            63G  4.0G   59G   7% /run
1.
2.
3.
4.
5.
6.

2021-05-08T18:31:00.210203+08:00 44458 [ERROR] Slave SQL for channel '': Worker 1 failed executing transaction 'fb18799a-afeb-11eb-a3f0-fa163e18e1d9:513684180' at master log greatdb-bin.031344, end_log_pos 8397; Could not execute write_rows event on table test.t1; Got error 28 from storage engine, Error_code: 1030; handler error No Error!; the event's master log FIRST, end_log_pos 8397, Error_code:1030
2021-05-08T18:31:00.210236+08:00 44457 [Warning] Slave SQL for channel '': ... The slave coordinator and worker threads are stoppped, possibly leaving data in inconsistent state, A restart should restore consistency automatically, althougn using non-transactional storage for data or info tables or DDL queries could lead to problems. In such cases you have to examine your data (see documentation for details). Error_code:1756 
1.
2.

MySQL [(none)]> show variables like '%tmpdir%';
+-------------------+----------------------------------------+
| Variable_name     | Value                                  |
+-------------------+----------------------------------------+
| innodb_tmpdir     |                                        |
| slave_load_tmpdir | /greatdb/tmp                           |
| tmpdir            | /greatdb/dbdata/greatdb57_data3307/tmp |
+-------------------+----------------------------------------+
3 rows in set (0.01 sec)
1.
2.
3.
4.
5.
6.
7.
8.
9.

MySQL [(none)]> show variables like '%tokudb%fs%';
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| tokudb_fs_reserve_percent | 5     |
| tokudb_fsync_log_period   | 0     |
+---------------------------+-------+
2 rows in set (0.00 sec)
1.
2.
3.
4.
5.
6.
7.
8.

variable tokudb_fs_reserve_percent
Command Line: Yes
Config File: Yes
Scope: Global
Dynamic: No
Variable Type: Numeric
Range: 0-100
Default Value: 5
This variable controls the percentage of the file system that must be available for inserts to be allowed. By default, this is set to 5. We recommend that this reserve be at least half the size of your physical memory. See Full Disks for more information.
1.
2.
3.
4.
5.
6.
7.
8.
9.