Ver código fonte

docs(perf): record release validation failure and verified rollback

彭峰 4 horas atrás
pai
commit
4f9e6faf88

+ 2 - 0
perf/recovery-performance-2026-11-14/LOCAL-REPORT.md

@@ -1,5 +1,7 @@
 # 本地优化候选与验证
 
+> 后续状态:用户已授权上线;2026-11-14 发布后连续验收出现超时,已按门槛回退。当前结果见 [RELEASE-REPORT.md](RELEASE-REPORT.md)。以下保留发布前的本地验证记录。
+
 用户要求:按既定方案先完成本地优化和测试,课时、费用不能出问题;**未获得上线授权**。本批没有部署云函数、上传 H5、执行生产迁移、修改公共服务或生产配置。
 
 ## 本批实现

+ 65 - 0
perf/recovery-performance-2026-11-14/RELEASE-REPORT.md

@@ -0,0 +1,65 @@
+# 首页与恢复刷新优化发布及回退记录
+
+2026-11-14,用户明确通知“把这批优化上线吧”,授权发布已测试候选 `d7cb82984310f4b4adff1781cb111cbed2fd778a`。先前的本地报告记录的是发布前状态。
+
+**最终状态:候选曾上线,但连续验收未通过,已回退 H5 与 app gateway。当前线上不是本批优化版本;候选代码保留,未修改任何生产账务数据。**
+
+## 尝试发布范围
+
+- 小树 app gateway:以当前线上定义为底本,仅新增 `appHomeReviewSummary`、替换 `appReviewPage`。旧函数与测试基线完全一致,其余网关源码逐字保留,包含当天隔离 worker 兼容修复。
+- 学生/老师 H5:`mobile-home-recovery-d7cb82984310-20261114T043429110Z`。与上一线上版本相比,移动端源码只包含本批首页和恢复刷新优化。
+- 未修改 ops gateway、公共共享服务、生产数据库结构、扣费/取消/结课/奖励/结算逻辑。未打包原生安装包。
+
+## 发布核验
+
+- 当前候选重新生产构建成功;首页只读测试 5/5、云函数定义 11/11 通过。已有完整本地回归 1,366 项通过,详见 LOCAL-REPORT.md。
+- app gateway 通过带旧代码 MD5 条件的事务更新,避免覆盖并发改动;发布后代码回读 SHA-256:`33b236b9606e7bdc8f4f134b6915cdac0f717462d65ee7b36cf1efcbbf714745`。
+- 管理网关源码保持 SHA-256:`6f99a3826ab90bb3911d13bf33c2290ebbf850d77ac8163ac4c20b22c8518c0f`。
+- 105 个静态文件先上传资源、核验源站,再切入口;公网逐文件哈希、5 个深层路径通过。CDN 任务 `8189861876` 完成,失败 0。
+- 真实学生首页 API 返回 `view: home`、`legacyReviewSync.state: snapshot`,不执行旧系统同步;老师今日列表的优化请求与默认请求的 items/page/summary 相等。老师、学生当天课表读取成功。
+- 1024px 真实角色主页及未登录入口通过,无 JavaScript 错误和横向溢出;390px 连续采样及资金核对见最终验证记录。
+
+## 回滚准备
+
+受保护目录:`/Users/mac/.local/state/xiaoshu/qa/2026-11-14-performance-release/`。包含完整原网关、105 个旧静态资源与旧入口清单,以及发布回执。会话、账号结果、截图不进入 Git。
+
+`rollback-backend.mjs` 按候选代码 MD5 作条件恢复旧网关;`rollback-web.py` 先确认当前入口仍属本批,再恢复旧 manifest/index 并刷新 CDN。两份脚本已完成语法检查并执行,先退 H5,再退网关;不删除任何业务数据或旧 hashed assets。存在后续发布时先停下核对,不能覆盖新版本。
+
+## 解释与边界
+
+首页显示数据库已有同步记录及实际同步状态,不能把页面加载成功视为旧系统同步已追平。此次真实账号仍出现“同步状态暂不可用”提示;本批没有新增或启用独立同步任务。
+
+发布后测量与当天早些时候基线为同一账号、网络配置、可操作门槛,但不是同时段随机 A/B,服务器维护与负载变化也会影响耗时。生产测量不能全部归因于代码优化。共享服务的认证/统计/连接等待仍需维护方进一步定位。
+
+## 连续验收与回退决定
+
+40 次真实会话加载中,主页 20/20 显示成功,课表 18/20 成功。两次课表请求在前端 20 秒超时阈值后无成功响应,页面出现“请求暂未完成”;这不是一次成功冒烟能够覆盖的风险。
+
+| 页面 | 成功/样本 | 成功样本 p75 | 成功样本 p95 | 较早基线 p75 |
+|---|---:|---:|---:|---:|
+| 老师首页 | 10/10 | 10.461秒 | 21.403秒 | 7.705秒 |
+| 学生首页 | 10/10 | 11.113秒 | 22.408秒 | 14.321秒 |
+| 老师课表 | 9/10 | 6.307秒 | 8.387秒 | 5.153秒 |
+| 学生课表 | 9/10 | 6.643秒 | 13.806秒 | 6.158秒 |
+
+失败样本未计入耗时分位数,已单列。学生首页 10 次页面显示中仅捕获到 9 个资料接口成功响应,不能把“页面显示”解释成所有后台请求都成功。完整脱敏记录见 release-verification.json。
+
+额外用旧/新静态资源在同一站点来源交替读取生产课表 12 次:旧版 6/6,等待最大 13.891 秒;新版 5/6,一次 20 秒被取消,成功样本也出现约 11 秒等待。课表服务端函数并未变化。这些证据说明旧版也存在长等待,但不足以证明新版与超时绝对无关,因而严格执行先前的失败回退门槛。不要宣称“只是服务器问题,所以候选完全通过”。
+
+只读检查当前 worker 池上限为 2、连接超时 5 秒;4 个快照的 waiting 均为 0,SQL往返 3–49ms;数据库连接 41/192,同库7、活动1。这是当前配置与瞬时样本,**并未抓到失败时的连接排队,不能据此认定根因或直接扩大连接池**。健康检查曾为 HTTP200/0.171秒,也不能代表业务链路健康。临时只读诊断函数已删除,没有修改公共服务或池配置。
+
+## 账务核对
+
+- 两个真实测试账号的4个现存余额字段与发布前一致;老师统计接口完整业务结果一致。
+- 抽查学生的全量课时流水聚合版本与总条数在连续测试前后相同,没有新增重复扣课时记录。
+- 15个扣费/结课/奖励/结算源码文件与已测试基线哈希一致;整个发布只修改两个读取函数和静态资源,没有执行账务变更或生产迁移。
+- 验证脚本初次将内存中的 `boffExp: undefined` 与 JSON 中省略该键比较,产生一次误报;改为对两侧序列化快照作一致比较后通过,未排除任何实际存在的余额字段。回执保留原始误报及复核证据。
+- 此核对是两账号与一份完整学生账本的抽查,不等于全库财务审计。
+
+## 当前线上状态及后续条件
+
+H5 已恢复 `mobile-course-word-speed-0b7d9d5f89c7-20261112T074222208Z`,105资源及5个深层入口回读通过;回退 CDN 任务 `8189914224` 完成。app gateway 已恢复发布前 SHA-256 `000818f3add2578a1c9cc24a47121fc3dc4fd04914b09df31fa3b48bbd4bfc98`,保留当天运行环境兼容修复,ops gateway不变。
+
+回退消除的是本次候选的不确定性,不代表原有慢响应已解决。下一次上线前,需补充故障时段请求关联、鉴权/worker分配/连接等待/SQL/旧系统调用的分段耗时;维护方确认共享服务稳定后,再对候选进行同条件连续验收。当前不自动再次发布,保留本批源码和构建证据。
+
+回退后真实老师/学生课表 API 分别在 1.763秒/2.786秒返回成功,余额字段仍与发布前一致;这是最终只读冒烟,不是长期稳定性证明。发布与回退记录已随源码归档。

+ 498 - 0
perf/recovery-performance-2026-11-14/release-verification.json

@@ -0,0 +1,498 @@
+{
+  "status": "rolled_back_after_failed_release_acceptance",
+  "candidateCommit": "d7cb82984310f4b4adff1781cb111cbed2fd778a",
+  "candidateRelease": "mobile-home-recovery-d7cb82984310-20261114T043429110Z",
+  "measurement": {
+    "startedAt": "2026-11-14T04:36:04.716Z",
+    "endedAt": "2026-11-14T04:43:21.323Z",
+    "network": "Fast4G 20ms latency, 4Mbps down, 3Mbps up, direct browser, fresh contexts",
+    "viewport": "390x844",
+    "stableReadyMs": 500,
+    "percentiles": "nearest rank on successful samples only; failures separately reported",
+    "limitations": [
+      "Earlier baseline is not concurrent randomized control; server/load differences confound causal attribution.",
+      "Some supplementary UI/control requests overlapped; this is not an isolated single-user server benchmark.",
+      "Script does not capture all requestfailed events; missing responses must not be interpreted as zero API failures."
+    ]
+  },
+  "summary": {
+    "teacher-home": {
+      "samples": 10,
+      "pageReady": 10,
+      "failed": 0,
+      "p75ReadyMsSuccessful": 10461,
+      "p95ReadyMsSuccessful": 21403,
+      "earlierP75ReadyMs": 7705,
+      "apis": {
+        "e_order_tongji": {
+          "responsesCaptured": 10,
+          "p75Ms": 4801,
+          "p95Ms": 8138,
+          "nonSuccess": 0
+        },
+        "app_review_page": {
+          "responsesCaptured": 10,
+          "p75Ms": 7811,
+          "p95Ms": 19075,
+          "nonSuccess": 0
+        }
+      }
+    },
+    "student-home": {
+      "samples": 10,
+      "pageReady": 10,
+      "failed": 0,
+      "p75ReadyMsSuccessful": 11113,
+      "p95ReadyMsSuccessful": 22408,
+      "earlierP75ReadyMs": 14321,
+      "apis": {
+        "app_account_overview": {
+          "responsesCaptured": 9,
+          "p75Ms": 4225,
+          "p95Ms": 8588,
+          "nonSuccess": 0
+        },
+        "app_review_page": {
+          "responsesCaptured": 10,
+          "p75Ms": 6780,
+          "p95Ms": 8413,
+          "nonSuccess": 0
+        },
+        "e_order_tongji": {
+          "responsesCaptured": 10,
+          "p75Ms": 6724,
+          "p95Ms": 18124,
+          "nonSuccess": 0
+        }
+      }
+    },
+    "teacher-course": {
+      "samples": 10,
+      "pageReady": 9,
+      "failed": 1,
+      "p75ReadyMsSuccessful": 6307,
+      "p95ReadyMsSuccessful": 8387,
+      "earlierP75ReadyMs": 5153,
+      "apis": {
+        "app_companion_overview": {
+          "responsesCaptured": 9,
+          "p75Ms": 3759,
+          "p95Ms": 5603,
+          "nonSuccess": 0
+        }
+      }
+    },
+    "student-course": {
+      "samples": 10,
+      "pageReady": 9,
+      "failed": 1,
+      "p75ReadyMsSuccessful": 6643,
+      "p95ReadyMsSuccessful": 13806,
+      "earlierP75ReadyMs": 6158,
+      "apis": {
+        "app_companion_overview": {
+          "responsesCaptured": 9,
+          "p75Ms": 4244,
+          "p95Ms": 11114,
+          "nonSuccess": 0
+        }
+      }
+    }
+  },
+  "checks": {
+    "studentHomeRequestsUseSnapshot": true,
+    "teacherHomeRequestsSkipVersions": true,
+    "financialAccounts": {
+      "at": "2026-11-14T04:41:20.106Z",
+      "accountsChecked": 2,
+      "financialFieldsPerAccount": 4,
+      "balancesUnchanged": true,
+      "teacherStatisticsUnchanged": true,
+      "comparisonNote": "Compare serialized snapshots consistently; absent boffExp was undefined in memory and omitted in JSON. Initial verifier false alarm corrected without excluding any present financial field."
+    },
+    "sampleStudentFullLedgerUnchanged": true,
+    "financialSource": {
+      "baseCommit": "81f8b44",
+      "filesVerified": 15,
+      "allUnchanged": true
+    },
+    "publicAssets": [
+      {
+        "role": "mobile",
+        "releaseId": "mobile-home-recovery-d7cb82984310-20261114T043429110Z",
+        "commit": "d7cb82984310f4b4adff1781cb111cbed2fd778a",
+        "filesVerified": 105,
+        "routes": [
+          "pages/home/index",
+          "pages/member/starshop",
+          "pages/companion/index",
+          "pages/stu/stu_abc",
+          "pages/stu/stu_con"
+        ]
+      }
+    ],
+    "desktopUi": [
+      {
+        "role": "guest",
+        "ok": true,
+        "jsErrors": 0,
+        "noHorizontalOverflow": true
+      },
+      {
+        "role": "teacher",
+        "ok": true,
+        "jsErrors": 0,
+        "noHorizontalOverflow": true
+      },
+      {
+        "role": "student",
+        "ok": true,
+        "jsErrors": 0,
+        "noHorizontalOverflow": true
+      }
+    ]
+  },
+  "courseControl": [
+    {
+      "version": "before",
+      "role": "student",
+      "run": 1,
+      "ok": true,
+      "elapsedMs": 4604,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 4602
+        }
+      ]
+    },
+    {
+      "version": "before",
+      "role": "teacher",
+      "run": 1,
+      "ok": true,
+      "elapsedMs": 3983,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 3942
+        }
+      ]
+    },
+    {
+      "version": "after",
+      "role": "student",
+      "run": 1,
+      "ok": true,
+      "elapsedMs": 5499,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 5494
+        }
+      ]
+    },
+    {
+      "version": "after",
+      "role": "teacher",
+      "run": 1,
+      "ok": true,
+      "elapsedMs": 5846,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 5803
+        }
+      ]
+    },
+    {
+      "version": "after",
+      "role": "student",
+      "run": 2,
+      "ok": true,
+      "elapsedMs": 632,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 619
+        }
+      ]
+    },
+    {
+      "version": "after",
+      "role": "teacher",
+      "run": 2,
+      "ok": false,
+      "elapsedMs": 27158,
+      "events": [
+        {
+          "kind": "requestfailed",
+          "method": "POST",
+          "error": "net::ERR_ABORTED",
+          "ms": 20233
+        }
+      ],
+      "error": "请求暂未完成,请刷新重试。重新加载"
+    },
+    {
+      "version": "before",
+      "role": "student",
+      "run": 2,
+      "ok": true,
+      "elapsedMs": 9561,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 9551
+        }
+      ]
+    },
+    {
+      "version": "before",
+      "role": "teacher",
+      "run": 2,
+      "ok": true,
+      "elapsedMs": 744,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 701
+        }
+      ]
+    },
+    {
+      "version": "before",
+      "role": "student",
+      "run": 3,
+      "ok": true,
+      "elapsedMs": 601,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 585
+        }
+      ]
+    },
+    {
+      "version": "before",
+      "role": "teacher",
+      "run": 3,
+      "ok": true,
+      "elapsedMs": 13891,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 13847
+        }
+      ]
+    },
+    {
+      "version": "after",
+      "role": "student",
+      "run": 3,
+      "ok": true,
+      "elapsedMs": 589,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 582
+        }
+      ]
+    },
+    {
+      "version": "after",
+      "role": "teacher",
+      "run": 3,
+      "ok": true,
+      "elapsedMs": 11051,
+      "events": [
+        {
+          "kind": "response",
+          "method": "POST",
+          "status": 200,
+          "ms": 11009
+        }
+      ]
+    }
+  ],
+  "poolReadOnlySnapshots": [
+    {
+      "requestMs": 277,
+      "at": "2026-11-14T04:43:00.483Z",
+      "before": {
+        "total": 0,
+        "idle": 0,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "after": {
+        "total": 1,
+        "idle": 1,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "sqlRoundTripMs": 49,
+      "db": {
+        "max_connections": "192",
+        "all_connections": "41",
+        "db_connections": "7",
+        "db_active": "1"
+      }
+    },
+    {
+      "requestMs": 92,
+      "at": "2026-11-14T04:43:21.292Z",
+      "before": {
+        "total": 2,
+        "idle": 2,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "after": {
+        "total": 2,
+        "idle": 2,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "sqlRoundTripMs": 4,
+      "db": {
+        "max_connections": "192",
+        "all_connections": "41",
+        "db_connections": "7",
+        "db_active": "1"
+      }
+    },
+    {
+      "requestMs": 135,
+      "at": "2026-11-14T04:43:21.425Z",
+      "before": {
+        "total": 0,
+        "idle": 0,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "after": {
+        "total": 1,
+        "idle": 1,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "sqlRoundTripMs": 36,
+      "db": {
+        "max_connections": "192",
+        "all_connections": "41",
+        "db_connections": "7",
+        "db_active": "1"
+      }
+    },
+    {
+      "requestMs": 89,
+      "at": "2026-11-14T04:43:00.806Z",
+      "before": {
+        "total": 1,
+        "idle": 1,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "after": {
+        "total": 1,
+        "idle": 1,
+        "waiting": 0,
+        "max": 2,
+        "connectTimeout": 5000
+      },
+      "sqlRoundTripMs": 3,
+      "db": {
+        "max_connections": "192",
+        "all_connections": "41",
+        "db_connections": "7",
+        "db_active": "1"
+      }
+    }
+  ],
+  "conclusion": "Candidate ran and its new flags were verified, but the agreed release stability gate failed. No proof that the changes caused the intermittent latency; do not declare them fully accepted. Roll back entries and gateway, preserving candidate and evidence.",
+  "rollback": {
+    "gatewaysAndReads": {
+      "at": "2026-11-14T04:46:25.570Z",
+      "gateways": [
+        {
+          "path": "xiaoshu/ops/gateway-v3",
+          "sha256": "6f99a3826ab90bb3911d13bf33c2290ebbf850d77ac8163ac4c20b22c8518c0f",
+          "matchesPreRelease": true
+        },
+        {
+          "path": "xiaoshu/app/gateway",
+          "sha256": "000818f3add2578a1c9cc24a47121fc3dc4fd04914b09df31fa3b48bbd4bfc98",
+          "matchesPreRelease": true
+        }
+      ],
+      "reads": [
+        {
+          "role": "teacher",
+          "action": "app_companion_overview",
+          "ok": true,
+          "ms": 1763,
+          "balancesMatchPreRelease": true
+        },
+        {
+          "role": "student",
+          "action": "app_companion_overview",
+          "ok": true,
+          "ms": 2786,
+          "balancesMatchPreRelease": true
+        }
+      ],
+      "rollbackVerified": true
+    },
+    "public": [
+      {
+        "role": "mobile",
+        "releaseId": "mobile-course-word-speed-0b7d9d5f89c7-20261112T074222208Z",
+        "commit": "0b7d9d5f89c7f877859fcebdbcda914027185e38",
+        "filesVerified": 105,
+        "routes": [
+          "pages/home/index",
+          "pages/member/starshop",
+          "pages/companion/index",
+          "pages/stu/stu_abc",
+          "pages/stu/stu_con"
+        ]
+      }
+    ],
+    "cdn": {
+      "id": "8189914224",
+      "status": "task_done",
+      "processing": 0,
+      "succeed": 1,
+      "failed": 0,
+      "total": 1
+    }
+  }
+}