Просмотр исходного кода

fix: valid default facets and source locator reuse on archive rebuild

pengzhikang 6 дней назад
Сommit
9e6ab7b201
6 измененных файлов с 291 добавлено и 0 удалено
  1. 1 0
      .gitattributes
  2. 70 0
      README.md
  3. BIN
      ksf-unified-runtime-hotfix-20261109-r1.zip
  4. 68 0
      patches/runtime.diff
  5. 16 0
      release.json
  6. 136 0
      validation.json

+ 1 - 0
.gitattributes

@@ -0,0 +1 @@
+* -text

+ 70 - 0
README.md

@@ -0,0 +1,70 @@
+# 统一数据库运行补丁 r1
+
+修复 facets 默认分组失败及真实历史归档重建时的来源唯一键冲突。本包以已封版迁移 r3 为基线,只改两个服务模块及研究运行组件中的相同副本;主 Skill r9、教学细则、操作手册均不在包内。不要重新执行数据库迁移。
+
+## 修复行为
+
+- facets 空请求仍查默认实体;包含 social 时默认仅分 platform/category。仅查 products 时默认 platform/brand/category。显式请求 social 品牌分组仍拒绝,防止把提及当归属。
+- contract 的 retrieval.aggregation 提供 entities/groupBy 参数及可执行示例,不支持 entity/dimensions/facets/table 等猜测参数。
+- 来源按映射后的 asset_path + line_number 查重,只有 kind、指纹和原始 payload 等全部字段一致才复用既有来源 ID;冲突返回 SOURCE_LOCATOR_CONFLICT。相同来源的关联证据只有所有事实和外键都一致才复用,不屏蔽唯一约束。
+- 同一原始归档在导入器代码变化后,incomingGeneration 可能变化。仅凭归档相同不跳过重新解释;允许正常合并预检,不强求一定返回 already-imported。相同 incomingGeneration 的重复发布仍返回 already-imported。历史原件只做 apply=false,不能拿重发旧材料冒充新代际验收。
+
+## 1. 核包与部署
+
+读取本固定提交目录的 release.json,核对 ZIP 字节数和 SHA-256,解压到新目录 `/opt/ksf-unified-runtime-hotfix-20261109-r1`,逐项核对 FILES.sha256。查看 patches/runtime.diff、validation.json。保留封版 r3 及原服务备份;新目录公共代码可由服务用户读取,私有配置继续原位置和权限。
+
+先对新目录做 Python 3.9 语法和模块加载检查。沿用实际有效的 `/etc/ksf/unified-db.json`、PGSERVICEFILE、PYTHONPATH=/opt/ksf-unified-deps 和 KSF_LIBRARY_PYTHON,不改成文档中的示例路径。目标仍为 FKieRbIoGt_master / FKieRbIoGt_admin / FKieRbIoGt_main。
+
+服务预热后统一切到本目录中的对应入口:PM2 资料 HTTP、systemd 行业 HTTP、inbox、登记脚本。逐进程核对真实入口和环境,并维持旧 SQLite 停写封存。不要用旧 SQLite 恢复写入。
+
+SSH 保留已有 authorized_keys 两条受限键和 restrict 不动,只将现有短包装脚本 receiver-r3.sh 内的 receiver 代码入口改到本包 `server/voc-library/library_receiver.py`,保留原 receiver.json、依赖与 PG 环境。包装脚本名称可以保留,不再写长 forced-command。
+
+## 2. 研究运行组件
+
+使用本包 `scripts/install-unified-skills.py`,仅选择 `--component research-runtime`,对服务器已部署研究运行目录执行 plan → apply → verify。参数形式:
+
+```text
+python3.9 scripts/install-unified-skills.py plan --component research-runtime --target <现有研究运行目录> --out <新的私有plan目录>
+python3.9 scripts/install-unified-skills.py apply --component research-runtime --target <同一目录> --plan <plan目录>/plan.json --out <新的私有apply目录>
+python3.9 scripts/install-unified-skills.py verify --component research-runtime --target <同一目录> --out <新的私有verify目录>
+```
+
+实际已知服务器目录为 `/opt/ksf-library-publish-20261106-r1/skills/ksf-voc-library`,执行前确认,禁止覆盖错误目录。Agent 若有独立运行组件副本也按相同流程更新并回读核验;不要上传依赖目录或凭据。保持原 Agent `/opt/data/.env` 中 KSF_LIBRARY_CHANNEL 持久绑定,不能回落本地快照。不得安装本包未提供的主 Skill,或恢复旧 SKILL.md 活动入口。
+
+## 3. 必验:默认 Agent → 受限 SSH → PostgreSQL
+
+从原 Agent 的新非登录子进程执行默认 `python3 query_voc.py contract`,不手动指定通道;应返回 mode=ssh-channel、backend=parse-postgresql、正式研究代际。参数发现位置为 contract.retrieval.aggregation。
+
+测试脚本和输入只写 `/opt/data` 内的私有子目录,不放宽写目录限制。以下为 facets 三个独立 JSON 输入,通过 `query_voc.py facets --input <私有JSON路径>` 执行;也可按契约从标准输入传入。
+
+```json
+{}
+```
+
+应成功,分组为 platform/category。
+
+```json
+{"entities":["products"],"groupBy":["brand"],"limit":20}
+```
+
+应成功,返回 brand 记录数。再以 `{"entities":["products"]}` 验证三个默认维度。显式 `{"entities":["social"],"groupBy":["brand"]}` 应被 SOCIAL_BRAND_IS_MENTION_NOT_GROUPABLE_OWNERSHIP 正确拒绝。
+
+复验此前真实商品 get → source:`jd:100000348106`,source ID `faa74ac0044a2b51e06915524468b1820fee8bd1c7af14b7176608d93b56761e`,应保持原归档指纹及来源。原资料/行业网关、鉴权和现有代际同样回读,不拿测试环境代际替换生产。
+
+## 4. 必验:真实归档 intake 预检
+
+重用此前失败的真实 `30ff3c44…/transport.zip` 原字节和 batchId=ksf-handoff-20261106-reviewed-r1,通过原受限 receiver intake、apply=false、expectedGeneration=执行时 contract 正式代际进行预检。不得改 requestId/window/batchId 或伪造材料。
+
+应通过解包、build、mergePlan,不能再出现 source_records.asset_path/line_number 唯一冲突。此预检可以返回有效合并计划;相同导入器代际已导入时也可以返回 already-imported。核对前后研究指针与业务表未变、无新研究代际。worker 心跳导致的监测 revision 前进不算业务写入。
+
+新代际写入只有取得真实未导入材料后才按原授权任务发布,并做同代际回读;本次修复验收不以旧归档 apply 代替。
+
+## 5. 必验:正常成果保存 → artifactId → 看板归属
+
+这项已在当前迁移验收授权范围内,继续执行,不再以缺触发授权暂停。使用原 Agent、原真实项目/会话、已有真实工作坊成果,经已装 r9 的正常 workshop-save 链路保存。沿用真实 taskKey/version/归属和稳定 requestId,不虚构身份、内容、场景或批次;不得直写 Parse 表代替平台接口。
+
+按 r9 的 workshop-tasks.md 组织保存输入,保存后用返回 artifactId 经 workshop-read 回读,并在真实看板核对原小组/项目/会话归属、阶段、内容和版本。回执列 artifactId、sessionId、归属核验、看板定位链接及回读一致性,勿回显凭据。无法访问原 Agent 时明确报运行条件,不再误报未授权。
+
+## 6. 一次回传
+
+回传:核包与组件 verify、实际服务/SSH/Agent 入口、facets 正向与反向校验、真实归档 dry-run、研究指针与原件指纹、成果保存回读归属、旧库仍封存。明确分开“本次已执行通过”“故障”“仅缺真实新材料待验收”。Skill/手册联合优化留第二步;前端展示与并发性能另按原任务顺序处理。

BIN
ksf-unified-runtime-hotfix-20261109-r1.zip


+ 68 - 0
patches/runtime.diff

@@ -0,0 +1,68 @@
+--- sealed-r3/server/voc-library/retrieval.py
++++ ksf-unified-runtime-hotfix-20261109-r1/server/voc-library/retrieval.py
+@@ -170,6 +170,11 @@
+       'fields':{k:{'type':typ,'operators':['eq','in','exists']+(['min','max'] if typ in ('number','date') else []),'unit':'CNY分' if k=='priceMinor' else 'g' if k=='netWeightG' else '原文日历日期;时区未统一' if typ=='date' else None} for k,(_,typ) in FIELDS.items()},
+       'productSorts':{'reviews_asc':'已收录评论从少到多(含默认及待核)','reviews_desc':'已收录评论从多到少(含默认及待核)',
+                       'price_asc':'最低历史观察价从低到高,未知置后','price_desc':'最低历史观察价从高到低,未知置后'},
++      'aggregation':{'entityParameter':'entities','groupingParameter':'groupBy',
++                     'facetsDefaultFields':['platform','category'],
++                     'productFacetsDefaultFields':['platform','brand','category'],
++                     'socialBrandGrouping':'explicit brand grouping with social entities is rejected; mentions are not product ownership',
++                     'facetsExamples':{'allEntities':{},'productBrands':{'entities':['products'],'groupBy':['brand'],'limit':20}}},
+       'taxonomy':tax,'concepts':CONCEPTS,
+       'relationships':['products.id → reviews/prices/specs/rankings/coverage.product_id','documents.id → clauses.document_id','all.source_record_id → source_records.id → source_links/assets/collection_requests','social品牌仅为提及关系,不能推为商品归属'],
+       'questionRoutes':[
+@@ -383,7 +388,8 @@
+ 
+ def aggregate(c,request,facets=False):
+     where,values,effective=prepare(c,request)
+-    fields=request.get('groupBy',['platform','brand','category'] if facets else ['platform'])
++    defaults=(['platform','category'] if 'social' in effective['entities'] else ['platform','brand','category']) if facets else ['platform']
++    fields=request.get('groupBy',defaults)
+     require(isinstance(fields,list) and 1<=len(fields)<=4 and all(isinstance(f,str) and f in FIELDS for f in fields) and len(fields)==len(set(fields)),'INVALID_GROUP_BY')
+     require(not ('brand' in fields and 'social' in effective['entities']),'SOCIAL_BRAND_IS_MENTION_NOT_GROUPABLE_OWNERSHIP')
+     limit=request.get('limit',50);require(type(limit) is int and 1<=limit<=100,'INVALID_LIMIT')
+--- sealed-r3/server/voc-library/update_library.py
++++ ksf-unified-runtime-hotfix-20261109-r1/server/voc-library/update_library.py
+@@ -138,11 +138,22 @@
+                             entity='products' if row['entity_type']=='product' else 'reviews'
+                             candidate=c.execute('SELECT incoming_source_record_id,selected_source_record_id FROM merge_decisions WHERE entity=? AND record_id=? AND batch_sha256=? AND decision=?',
+                                 (entity,row['entity_id'],im['archive_sha256'],'kept_existing')).fetchone()
+-                            if candidate:
++                            if candidate and candidate['incoming_source_record_id']!=candidate['selected_source_record_id']:
+                                 row['detail']=canonical({'scope':'incoming_candidate_not_current_row',
+                                     'candidateSourceRecordId':candidate[0],'selectedSourceRecordId':candidate[1],'originalDetail':row['detail']})
+                         columns=list(row)
+                         old=c.execute('SELECT * FROM '+table+' WHERE '+' AND '.join(k+'=?' for k in pk),[row[k] for k in pk]).fetchone()
++                        if table=='source_records':
++                            # Asset remapping may resolve to a previously imported archive.
++                            # Rebuilt importer generations must reuse its exact source locator.
++                            located=c.execute('SELECT * FROM source_records WHERE asset_path=? AND line_number=?',
++                                              (row['asset_path'],row['line_number'])).fetchone()
++                            if located:
++                                voc_db.require(all(located[k]==value for k,value in row.items() if k!='id'),
++                                               'SOURCE_LOCATOR_CONFLICT')
++                                mapping[original['id']]=located['id']
++                                delta['duplicate'][table]=delta['duplicate'].get(table,0)+1
++                                continue
+                         if table=='batches' and old and old['archive_sha256']==row['archive_sha256']:
+                             mapping[original['id']]=old['id'];delta['duplicate'][table]=delta['duplicate'].get(table,0)+1;continue
+                         # The same timestamped price observation may be re-exported under a new source locator.
+@@ -191,6 +202,17 @@
+                         if old and table not in STABLE:
+                             voc_db.require(len(pk)==1 and pk[0] in ('id','path','asset_path'),'CONFLICTING_RELATION:'+table)
+                             col=pk[0]
++                            if col=='id':
++                                # Related evidence from older importer generations may already
++                                # have a namespaced ID. Reuse it only when every fact and FK agrees.
++                                facts=[k for k in columns if k!=col]
++                                equivalent=c.execute('SELECT '+col+' FROM '+table+' WHERE '+
++                                    ' AND '.join(k+' IS ?' for k in facts)+' ORDER BY '+col+' LIMIT 1',
++                                    [row[k] for k in facts]).fetchone()
++                                if equivalent:
++                                    mapping[original[col]]=equivalent[col]
++                                    delta['duplicate'][table]=delta['duplicate'].get(table,0)+1
++                                    continue
+                             if table=='assets':row[col]='batches/'+im['archive_sha256'][:16]+'/'+row[col]
+                             elif table=='collection_requests':raise ValueError('CONFLICTING_COLLECTION_REQUEST')
+                             else:row[col]=hashlib.sha256((im['generation']+':'+str(original[col])).encode()).hexdigest()

+ 16 - 0
release.json

@@ -0,0 +1,16 @@
+{
+  "release": "ksf-unified-runtime-hotfix-20261109-r1",
+  "archive": "ksf-unified-runtime-hotfix-20261109-r1.zip",
+  "bytes": 311703,
+  "sha256": "af755f31f3ee1f2ebe2ff14d901b2afe9ec9c060840b3f74709f4396429e5c5f",
+  "filesVerified": 75,
+  "validation": {
+    "targetedRegressionTests": 66,
+    "realPostgreSQLTests": 15,
+    "realArchiveRebuildPassed": true,
+    "realReceiverPreflightPassed": true,
+    "productionWrites": 0
+  },
+  "productionDeploymentPending": true,
+  "mainSkillAndManualDeferred": true
+}

+ 136 - 0
validation.json

@@ -0,0 +1,136 @@
+{
+  "ok": true,
+  "targetedRegressionTests": 66,
+  "realPostgreSQLTests": 15,
+  "genuineArchiveValidation": {
+    "ok": true,
+    "genuineBatchId": "ksf-handoff-20261106-reviewed-r1",
+    "genuineInputReused": true,
+    "baselineIsOlderLocalSnapshot": true,
+    "originalInputUnchanged": true,
+    "sameArchiveRebuildSourceRowsPreserved": true,
+    "counts": {
+      "batches": 2,
+      "assets": 4593,
+      "collection_requests": 2532,
+      "brands": 21,
+      "categories": 11,
+      "source_records": 13820,
+      "source_links": 25914,
+      "products": 4674,
+      "classification_assertions": 9478,
+      "product_observations": 8388,
+      "specs": 4714,
+      "prices": 4172,
+      "price_sources": 4264,
+      "reviews": 2380,
+      "social_posts": 3814,
+      "social_brand_mentions": 1108,
+      "policy_documents": 25,
+      "policy_clauses": 31,
+      "ranking_signals": 55,
+      "coverage_observations": 1478,
+      "quality_issues": 3504,
+      "review_theme_annotations": 0
+    },
+    "firstMergeDelta": {
+      "added": {
+        "batches": 1,
+        "assets": 37,
+        "source_records": 247,
+        "source_links": 247,
+        "products": 35,
+        "classification_assertions": 200,
+        "product_observations": 100,
+        "specs": 75,
+        "prices": 102,
+        "price_sources": 102,
+        "reviews": 20,
+        "ranking_signals": 1,
+        "coverage_observations": 2,
+        "quality_issues": 66
+      },
+      "changed": {
+        "products": 20
+      },
+      "duplicate": {
+        "brands": 21,
+        "categories": 11,
+        "specs": 28
+      },
+      "retained": {
+        "products": 45,
+        "reviews": 19
+      },
+      "mergeReview": {
+        "policy": "ksf.voc-merge.20261106.v1",
+        "decisions": 84,
+        "conflictRecords": 1,
+        "reasonCounts": {
+          "NEWER_COMPLETE_OBSERVATION": 20,
+          "CLASSIFICATION_EVIDENCE_WOULD_WEAKEN": 31,
+          "KNOWN_VALUES_WOULD_BE_LOST": 29,
+          "OLDER_OBSERVATION": 30,
+          "CHANGED_VALUES_WITHOUT_NEWER_OBSERVATION": 4,
+          "IDENTITY_OR_FACT_CONFLICT": 1
+        }
+      }
+    },
+    "replayDelta": {
+      "added": {},
+      "changed": {},
+      "duplicate": {
+        "batches": 1,
+        "assets": 37,
+        "brands": 21,
+        "categories": 11,
+        "source_records": 247,
+        "source_links": 247,
+        "products": 55,
+        "classification_assertions": 200,
+        "product_observations": 100,
+        "specs": 103,
+        "prices": 102,
+        "price_sources": 102,
+        "reviews": 20,
+        "ranking_signals": 1,
+        "coverage_observations": 2,
+        "quality_issues": 66
+      },
+      "retained": {
+        "products": 45,
+        "reviews": 19
+      },
+      "mergeReview": {
+        "policy": "ksf.voc-merge.20261106.v1",
+        "decisions": 139,
+        "conflictRecords": 1,
+        "reasonCounts": {
+          "EQUIVALENT_RECORD": 75,
+          "CLASSIFICATION_EVIDENCE_WOULD_WEAKEN": 31,
+          "KNOWN_VALUES_WOULD_BE_LOST": 29,
+          "OLDER_OBSERVATION": 30,
+          "CHANGED_VALUES_WITHOUT_NEWER_OBSERVATION": 4,
+          "IDENTITY_OR_FACT_CONFLICT": 1
+        }
+      }
+    },
+    "productionWrites": 0
+  },
+  "genuineReceiverValidation": {
+    "ok": true,
+    "genuineBatchId": "ksf-handoff-20261106-reviewed-r1",
+    "transportBytes": 74446,
+    "transportSha256": "2509a61503250a82f96a646a7fe6023146abeef051b411c1f2a954b7c42ed64f",
+    "apply": false,
+    "mergePreflightPassed": true,
+    "facetsReceiverReadPassed": true,
+    "persistentFilesUnchanged": true,
+    "intakeEmptyAfterDryRun": true,
+    "databaseWrites": "temporary-only",
+    "productionWrites": 0,
+    "baselineIsOlderLocalSnapshot": true
+  },
+  "productionWrites": 0,
+  "productionAcceptancePending": true
+}