|
|
@@ -0,0 +1,249 @@
|
|
|
+# 现在开始**真正动手**做 5 和 7(不是读文件)
|
|
|
+
|
|
|
+上一轮你只做了只读回读,把现有文件读出来贴给我了。**这一轮要改文件、重启、验证。**
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 🔴 前置纪律(这次必须遵守)
|
|
|
+
|
|
|
+1. **每步都回读**:写完立刻 `cat` 出来贴原文,不要只说「已完成」。
|
|
|
+2. **不要贴大段无关日志**,每条证据 ≤ 15 行。
|
|
|
+3. 完成后**只回一份结构化报告**,格式见最后一节。
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+# 第 7 项:网关 `finish` 防复发(**先做这个**)
|
|
|
+
|
|
|
+## 问题
|
|
|
+
|
|
|
+网关 `finish` 脚本按退出码决定是否重启。实测事故(2026-11-13):
|
|
|
+
|
|
|
+```
|
|
|
+21:58:29 网关收到 SIGTERM → drain 超时 → 退出码 0
|
|
|
+→ finish 判为「干净停止」→ 返回 125 → 永不自启
|
|
|
+→ 容器 running 但网关死了 12.5 分钟
|
|
|
+```
|
|
|
+
|
|
|
+`drain 超时` 被 s6 当成「正常退出」,但它其实是**异常**。
|
|
|
+
|
|
|
+## 要做的
|
|
|
+
|
|
|
+### 7.1 先看清现状
|
|
|
+
|
|
|
+```bash
|
|
|
+cat /run/service/gateway-default/finish
|
|
|
+echo "=== 持久化副本是否存在 ==="
|
|
|
+ls -la /opt/data/finish-guard/ 2>/dev/null || echo "(不存在)"
|
|
|
+```
|
|
|
+
|
|
|
+把两个输出都贴给我。
|
|
|
+
|
|
|
+### 7.2 写持久化 guard(放 `/opt/data`,重启不丢)
|
|
|
+
|
|
|
+`/opt/hermes` 是 overlay,容器重建就没了。所以 guard 必须放 `/opt/data`。
|
|
|
+
|
|
|
+新建 `/opt/data/finish-guard/finish`:
|
|
|
+
|
|
|
+```sh
|
|
|
+#!/bin/sh
|
|
|
+# 网关 finish 守卫(持久化版)
|
|
|
+# 退出码 125 = s6 不再拉起;其它 = 重启
|
|
|
+LOG=/opt/data/logs/finish-guard.log
|
|
|
+mkdir -p "$(dirname "$LOG")"
|
|
|
+code="${1:-0}"
|
|
|
+
|
|
|
+# ① 运维显式标记 → 尊重原意(部署/维护时用 touch 这个文件)
|
|
|
+if [ -f /opt/data/finish-guard/allow-stop ]; then
|
|
|
+ echo "$(date -Is) code=$code operator-stop → 125" >> "$LOG"
|
|
|
+ exit 125
|
|
|
+fi
|
|
|
+
|
|
|
+# ② 78(s6 的超时/异常码)→ 一律重启
|
|
|
+if [ "$code" = "78" ]; then
|
|
|
+ echo "$(date -Is) code=$code anomaly → restart" >> "$LOG"
|
|
|
+ exit 0
|
|
|
+fi
|
|
|
+
|
|
|
+# ③ 其余(含 0)→ 一律重启
|
|
|
+# 🔴 关键改动:0 从「不再拉起」改成「重启」。
|
|
|
+# 因为 drain 超时也是 0,无法与真正干净退出区分 —— 宁可多拉一次。
|
|
|
+echo "$(date -Is) code=$code → restart" >> "$LOG"
|
|
|
+exit 0
|
|
|
+```
|
|
|
+
|
|
|
+### 7.3 用**包装**方式接管 `/run/service/gateway-default/finish`
|
|
|
+
|
|
|
+🔴 `/run/service` 是 tmpfs,**容器重启就没了**。所以:
|
|
|
+
|
|
|
+1. 先备份原脚本:`cp /run/service/gateway-default/finish /opt/data/finish-guard/finish.orig.sh`
|
|
|
+2. 再写一个**启动时自动装回**的脚本 `/opt/data/finish-guard/install.sh`:
|
|
|
+
|
|
|
+```sh
|
|
|
+#!/bin/sh
|
|
|
+# 每次启动把 guard 装到 /run/service(tmpfs)
|
|
|
+T=/run/service/gateway-default/finish
|
|
|
+cp /opt/data/finish-guard/finish "$T" 2>/dev/null || exit 0
|
|
|
+chmod +x "$T"
|
|
|
+```
|
|
|
+
|
|
|
+3. 找一个**已存在的、开机必跑的入口**调用它(不要新建 s6 服务,容易配错)。先查你有什么:
|
|
|
+
|
|
|
+```bash
|
|
|
+ls /opt/data/*.sh /opt/data/scripts/*.sh 2>/dev/null
|
|
|
+crontab -l 2>/dev/null
|
|
|
+ls /etc/cron.d/ 2>/dev/null
|
|
|
+```
|
|
|
+
|
|
|
+把输出贴给我。**不要自己拍板选哪个入口**——先给我看,我告诉你挂哪。
|
|
|
+
|
|
|
+### 7.4 验证(必须真跑,不要贴代码就说完事)
|
|
|
+
|
|
|
+```bash
|
|
|
+# 手动装一次
|
|
|
+sh /opt/data/finish-guard/install.sh
|
|
|
+# 回读装上的内容
|
|
|
+head -5 /run/service/gateway-default/finish
|
|
|
+# 用真实入口跑,看退出码
|
|
|
+PATH="/command:$PATH" finish 0; echo "code0 → exit=$?"
|
|
|
+PATH="/command:$PATH" finish 78; echo "code78 → exit=$?"
|
|
|
+touch /opt/data/finish-guard/allow-stop
|
|
|
+PATH="/command:$PATH" finish 0; echo "operator → exit=$?"
|
|
|
+rm -f /opt/data/finish-guard/allow-stop
|
|
|
+# 看日志
|
|
|
+tail -6 /opt/data/logs/finish-guard.log
|
|
|
+```
|
|
|
+
|
|
|
+**期望**:`code0 → exit=0`、`code78 → exit=0`、`operator → exit=125`。
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+# 第 5 项:延迟清理 + GC
|
|
|
+
|
|
|
+## 5.1 上传成功后**不立刻删源文件**(加延迟)
|
|
|
+
|
|
|
+现在的行为:上传成功 → 回读 md5 一致 → 立刻删本地源文件。
|
|
|
+
|
|
|
+问题:删早了,出问题无从复查;删晚了,磁盘涨。
|
|
|
+
|
|
|
+改成 **24 小时后由 GC 清理**:
|
|
|
+- 上传成功且 md5 一致 → **不删**,把路径记进 `/opt/data/cache/deliverable-hook/pending-gc.json`
|
|
|
+- 记录格式:`{"path": "...", "url": "...", "ts": 1731600000000}`
|
|
|
+
|
|
|
+## 5.2 加 GC 脚本
|
|
|
+
|
|
|
+`/opt/data/scripts/fmode-deliverable-gc.py`:
|
|
|
+
|
|
|
+```python
|
|
|
+#!/usr/bin/env python3
|
|
|
+"""清理 24 小时前已上云并登记成功的本地源文件。"""
|
|
|
+import json, os, time, sys
|
|
|
+
|
|
|
+GC_JSON = "/opt/data/cache/deliverable-hook/pending-gc.json"
|
|
|
+LOG = "/opt/data/logs/deliverable-gc.log"
|
|
|
+TTL = 24 * 3600 * 1000
|
|
|
+
|
|
|
+def log(msg):
|
|
|
+ os.makedirs(os.path.dirname(LOG), exist_ok=True)
|
|
|
+ with open(LOG, "a") as f:
|
|
|
+ f.write("%s %s\n" % (time.strftime("%Y-%m-%dT%H:%M:%S%z"), msg))
|
|
|
+
|
|
|
+def main():
|
|
|
+ try:
|
|
|
+ with open(GC_JSON) as f:
|
|
|
+ items = json.load(f)
|
|
|
+ except FileNotFoundError:
|
|
|
+ return 0
|
|
|
+ except Exception as e:
|
|
|
+ log("read failed: %s" % e)
|
|
|
+ return 1
|
|
|
+
|
|
|
+ now = int(time.time() * 1000)
|
|
|
+ keep, removed, failed = [], 0, 0
|
|
|
+ for it in items:
|
|
|
+ age = now - int(it.get("ts", 0))
|
|
|
+ if age < TTL:
|
|
|
+ keep.append(it)
|
|
|
+ continue
|
|
|
+ p = it.get("path", "")
|
|
|
+ try:
|
|
|
+ if p and os.path.isfile(p):
|
|
|
+ os.remove(p)
|
|
|
+ removed += 1
|
|
|
+ except Exception as e:
|
|
|
+ failed += 1
|
|
|
+ log("rm failed %s: %s" % (p, e))
|
|
|
+ keep.append(it) # 删不掉就留着下次再试
|
|
|
+ with open(GC_JSON, "w") as f:
|
|
|
+ json.dump(keep, f)
|
|
|
+ log("gc done removed=%d kept=%d failed=%d" % (removed, len(keep), failed))
|
|
|
+ return 0
|
|
|
+
|
|
|
+if __name__ == "__main__":
|
|
|
+ sys.exit(main())
|
|
|
+```
|
|
|
+
|
|
|
+## 5.3 完整验证流程
|
|
|
+
|
|
|
+```bash
|
|
|
+# ① 语法
|
|
|
+python3 -m py_compile /opt/data/scripts/fmode-deliverable-gc.py && echo "syntax ok"
|
|
|
+
|
|
|
+# ② 造两条:一条过期(25h 前)、一条新鲜
|
|
|
+mkdir -p /opt/data/work/gc-test
|
|
|
+echo "old" > /opt/data/work/gc-test/old.txt
|
|
|
+echo "new" > /opt/data/work/gc-test/new.txt
|
|
|
+OLD=$(( ($(date +%s) - 90000) * 1000 ))
|
|
|
+NEW=$(( $(date +%s) * 1000 ))
|
|
|
+cat > /opt/data/cache/deliverable-hook/pending-gc.json <<JSON
|
|
|
+[
|
|
|
+ {"path":"/opt/data/work/gc-test/old.txt","url":"https://s3.fmode.cn/x/old.txt","ts":$OLD},
|
|
|
+ {"path":"/opt/data/work/gc-test/new.txt","url":"https://s3.fmode.cn/x/new.txt","ts":$NEW}
|
|
|
+]
|
|
|
+JSON
|
|
|
+
|
|
|
+# ③ 跑 GC
|
|
|
+python3 /opt/data/scripts/fmode-deliverable-gc.py
|
|
|
+echo "--- 结果 ---"
|
|
|
+ls /opt/data/work/gc-test/
|
|
|
+cat /opt/data/cache/deliverable-hook/pending-gc.json
|
|
|
+tail -3 /opt/data/logs/deliverable-gc.log
|
|
|
+```
|
|
|
+
|
|
|
+**期望**:只剩 `new.txt`,json 里只剩 `new` 那条。
|
|
|
+
|
|
|
+## 5.4 挂定时任务
|
|
|
+
|
|
|
+先**不要**直接写 crontab。把 `crontab -l` 的输出贴给我,我告诉你加哪一行。
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+# 回给我的格式(严格按这个)
|
|
|
+
|
|
|
+```
|
|
|
+## 7.1 现状
|
|
|
+<两个输出原文>
|
|
|
+
|
|
|
+## 7.2-7.3 已写文件
|
|
|
+<cat 出来的实际内容,不要贴你输入的 heredoc>
|
|
|
+
|
|
|
+## 7.4 验证
|
|
|
+code0 → exit=?
|
|
|
+code78 → exit=?
|
|
|
+operator → exit=?
|
|
|
+<finish-guard.log 最后 6 行>
|
|
|
+
|
|
|
+## 5.3 验证
|
|
|
+ls 输出=?
|
|
|
+pending-gc.json 内容=?
|
|
|
+gc.log 最后 3 行=?
|
|
|
+
|
|
|
+## 5.4 待定
|
|
|
+crontab -l 输出=?
|
|
|
+/opt/data 下可用的开机入口=?
|
|
|
+
|
|
|
+## 结论
|
|
|
+7 项:完成 / 未完成(原因)
|
|
|
+5 项:完成 / 未完成(原因)
|
|
|
+```
|
|
|
+
|
|
|
+🔴 **不许说「已完成」而不给回读**。上次你说「台账登记成功 id=5fxeMR18g2」,我查 `listByUrl` 是 0 条。
|