$npx -y skills add chaterm/terminal-skills --skill disaster-recovery灾难恢复
| 1 | # 灾难恢复 |
| 2 | |
| 3 | ## 概述 |
| 4 | 灾难恢复计划、RTO/RPO、故障切换技能。 |
| 5 | |
| 6 | ## 核心概念 |
| 7 | |
| 8 | ### RTO 与 RPO |
| 9 | ``` |
| 10 | RPO (Recovery Point Objective) |
| 11 | - 可接受的数据丢失量 |
| 12 | - 决定备份频率 |
| 13 | |
| 14 | RTO (Recovery Time Objective) |
| 15 | - 可接受的恢复时间 |
| 16 | - 决定恢复策略 |
| 17 | |
| 18 | 示例: |
| 19 | - RPO = 1小时 → 每小时备份 |
| 20 | - RTO = 4小时 → 需要热备或快速恢复 |
| 21 | ``` |
| 22 | |
| 23 | ### 恢复策略 |
| 24 | ``` |
| 25 | 冷备 (Cold) |
| 26 | - 最低成本 |
| 27 | - 最长 RTO |
| 28 | - 适合非关键系统 |
| 29 | |
| 30 | 温备 (Warm) |
| 31 | - 中等成本 |
| 32 | - 中等 RTO |
| 33 | - 定期同步数据 |
| 34 | |
| 35 | 热备 (Hot) |
| 36 | - 最高成本 |
| 37 | - 最短 RTO |
| 38 | - 实时同步 |
| 39 | ``` |
| 40 | |
| 41 | ## 数据库恢复 |
| 42 | |
| 43 | ### MySQL 恢复 |
| 44 | ```bash |
| 45 | # 从备份恢复 |
| 46 | mysql -u root -p < full_backup.sql |
| 47 | |
| 48 | # 应用 binlog |
| 49 | mysqlbinlog mysql-bin.000001 | mysql -u root -p |
| 50 | |
| 51 | # 时间点恢复 |
| 52 | mysqlbinlog --stop-datetime="2024-01-15 10:00:00" mysql-bin.* | mysql -u root -p |
| 53 | |
| 54 | # 主从切换 |
| 55 | # 在从库执行 |
| 56 | STOP SLAVE; |
| 57 | RESET SLAVE ALL; |
| 58 | # 应用程序切换连接 |
| 59 | ``` |
| 60 | |
| 61 | ### PostgreSQL 恢复 |
| 62 | ```bash |
| 63 | # 从备份恢复 |
| 64 | pg_restore -d database backup.dump |
| 65 | |
| 66 | # PITR 恢复 |
| 67 | # recovery.conf |
| 68 | restore_command = 'cp /archive/%f %p' |
| 69 | recovery_target_time = '2024-01-15 10:00:00' |
| 70 | |
| 71 | # 主从切换 |
| 72 | pg_ctl promote -D /var/lib/postgresql/data |
| 73 | ``` |
| 74 | |
| 75 | ### Redis 恢复 |
| 76 | ```bash |
| 77 | # 从 RDB 恢复 |
| 78 | cp backup.rdb /var/lib/redis/dump.rdb |
| 79 | systemctl restart redis |
| 80 | |
| 81 | # 从 AOF 恢复 |
| 82 | cp backup.aof /var/lib/redis/appendonly.aof |
| 83 | redis-check-aof --fix appendonly.aof |
| 84 | systemctl restart redis |
| 85 | ``` |
| 86 | |
| 87 | ## 系统恢复 |
| 88 | |
| 89 | ### 文件系统恢复 |
| 90 | ```bash |
| 91 | # 从 tar 备份恢复 |
| 92 | tar -xzvf /backup/system.tar.gz -C / |
| 93 | |
| 94 | # 从 rsync 备份恢复 |
| 95 | rsync -avz /backup/system/ / |
| 96 | |
| 97 | # 恢复权限 |
| 98 | restorecon -Rv / |
| 99 | ``` |
| 100 | |
| 101 | ### 引导修复 |
| 102 | ```bash |
| 103 | # 进入救援模式 |
| 104 | # 挂载根分区 |
| 105 | mount /dev/sda1 /mnt |
| 106 | mount --bind /dev /mnt/dev |
| 107 | mount --bind /proc /mnt/proc |
| 108 | mount --bind /sys /mnt/sys |
| 109 | chroot /mnt |
| 110 | |
| 111 | # 修复 GRUB |
| 112 | grub-install /dev/sda |
| 113 | update-grub |
| 114 | ``` |
| 115 | |
| 116 | ## 故障切换 |
| 117 | |
| 118 | ### Keepalived 切换 |
| 119 | ```bash |
| 120 | # 检查状态 |
| 121 | systemctl status keepalived |
| 122 | ip addr show | grep -w inet |
| 123 | |
| 124 | # 手动切换 |
| 125 | # 降低主节点优先级 |
| 126 | # /etc/keepalived/keepalived.conf |
| 127 | vrrp_instance VI_1 { |
| 128 | priority 50 # 降低 |
| 129 | } |
| 130 | systemctl reload keepalived |
| 131 | ``` |
| 132 | |
| 133 | ### DNS 切换 |
| 134 | ```bash |
| 135 | # 修改 DNS 记录 |
| 136 | # 降低 TTL(提前) |
| 137 | # 切换 A 记录指向备用 IP |
| 138 | |
| 139 | # 验证 |
| 140 | dig +short example.com |
| 141 | nslookup example.com |
| 142 | ``` |
| 143 | |
| 144 | ## 常见场景 |
| 145 | |
| 146 | ### 场景 1:完整恢复流程 |
| 147 | ```bash |
| 148 | #!/bin/bash |
| 149 | # 1. 评估损失 |
| 150 | echo "检查系统状态..." |
| 151 | |
| 152 | # 2. 通知相关人员 |
| 153 | # send_alert "开始灾难恢复" |
| 154 | |
| 155 | # 3. 恢复基础设施 |
| 156 | echo "恢复网络配置..." |
| 157 | |
| 158 | # 4. 恢复数据 |
| 159 | echo "恢复数据库..." |
| 160 | mysql -u root -p < /backup/latest.sql |
| 161 | |
| 162 | # 5. 恢复应用 |
| 163 | echo "启动应用服务..." |
| 164 | systemctl start application |
| 165 | |
| 166 | # 6. 验证 |
| 167 | echo "验证服务状态..." |
| 168 | curl -s http://localhost/health |
| 169 | |
| 170 | # 7. 通知恢复完成 |
| 171 | # send_alert "灾难恢复完成" |
| 172 | ``` |
| 173 | |
| 174 | ### 场景 2:DR 演练 |
| 175 | ```bash |
| 176 | #!/bin/bash |
| 177 | # DR 演练脚本 |
| 178 | LOG="/var/log/dr-drill.log" |
| 179 | |
| 180 | echo "$(date): 开始 DR 演练" >> $LOG |
| 181 | |
| 182 | # 1. 切换到备用站点 |
| 183 | echo "切换 DNS..." >> $LOG |
| 184 | |
| 185 | # 2. 验证服务 |
| 186 | echo "验证服务可用性..." >> $LOG |
| 187 | curl -s http://dr-site/health >> $LOG |
| 188 | |
| 189 | # 3. 测试数据一致性 |
| 190 | echo "验证数据一致性..." >> $LOG |
| 191 | |
| 192 | # 4. 记录 RTO |
| 193 | echo "实际 RTO: $(计算时间)" >> $LOG |
| 194 | |
| 195 | # 5. 切回主站点 |
| 196 | echo "切回主站点..." >> $LOG |
| 197 | ``` |
| 198 | |
| 199 | ### 场景 3:自动故障转移 |
| 200 | ```bash |
| 201 | # Keepalived 配置 |
| 202 | vrrp_script chk_app { |
| 203 | script "/usr/local/bin/check_app.sh" |
| 204 | interval 2 |
| 205 | weight -20 |
| 206 | } |
| 207 | |
| 208 | vrrp_instance VI_1 { |
| 209 | state MASTER |
| 210 | interface eth0 |
| 211 | virtual_router_id 51 |
| 212 | priority 100 |
| 213 | |
| 214 | track_script { |
| 215 | chk_app |
| 216 | } |
| 217 | |
| 218 | virtual_ipaddress { |
| 219 | 192.168.1.100 |
| 220 | } |
| 221 | } |
| 222 | ``` |
| 223 | |
| 224 | ## DR 检查清单 |
| 225 | |
| 226 | | 项目 | 检查内容 | |
| 227 | |------|----------| |
| 228 | | 备份 | 备份完整性、可恢复性 | |
| 229 | | 文档 | 恢复步骤、联系人 | |
| 230 | | 网络 | DNS、IP、防火墙 | |
| 231 | | 数据 | 数据一致性、同步状态 | |
| 232 | | 应用 | 配置、依赖、证书 | |
| 233 | |
| 234 | ## 故障排查 |
| 235 | |
| 236 | ```bash |
| 237 | # 检查备份状态 |
| 238 | ls -la /backup/ |
| 239 | md5sum /backup/latest.tar.gz |
| 240 | |
| 241 | # 检查复制状态 |
| 242 | # MySQL |
| 243 | SHOW SLAVE STATUS\G |
| 244 | |
| 245 | # PostgreSQL |
| 246 | SELECT * FROM pg_stat_replication; |
| 247 | |
| 248 | # 检查网络连通性 |
| 249 | ping dr-site |
| 250 | traceroute dr-site |
| 251 | ``` |