feat: persist workbook and competitor catalog

README.md

@@ -27,8 +27,10 @@ Completed through 2026-07-24:
 - Owner-only owner changes, last-owner protection, and workspace-member action assignment validation.
 - Parse REST storage adapter for environments where only a Parse Server URL and application credentials are available.
 - Master-key-only `Voc*` class schemas, resumable bounded imports, direct count verification, and a Parse-backed sync worker.
+- Direct streaming import from the source Demashi workbook without a frontend-generated JSON intermediate.
+- Idempotent competitor detail backfill through the company ecommerce relay, with mapped placeholders preserved when a request fails.
 
-The current case import resolves to 2,817 operating products, 9,717 daily metrics, 40 relations, and 0 reviews. The PostgreSQL importer may create internal `relation_stub` rows to preserve foreign keys; the Parse REST model stores denormalized relation identities and therefore does not create those extra catalog products.
+The current case import resolves to 2,817 operating products, 37 mapped competitor products, 9,717 daily metrics, 40 relations, and 0 reviews. All 37 competitor product details were persisted through the company relay on 2026-07-24. The PostgreSQL importer may create internal `relation_stub` rows to preserve foreign keys; the Parse REST model stores denormalized relation identities and therefore does not create those extra catalog products.
 
 The worker calls JD product-detail and review paths only through the company `/api/voc-e-commerce` gateway. It never sends a browser request to a supplier endpoint and never returns raw gateway errors or credentials.
 
@@ -75,11 +77,13 @@ $env:PARSE_MASTER_KEY = '<process-only-master-key>'
 $env:API_AUTH_MODE = 'disabled' # isolated local development only
 $env:FMODE_API_KEY = '<process-only-company-gateway-key>'
 npm run bootstrap:parse-rest -- "E:\workspace\Saas-voc\src\assets\data\demashi-summary.json" demashi
+npm run import:workbook:parse-rest -- "E:\path\to\德玛仕产品及竟品收集0721.xlsx" demashi
+npm run sync:competitors:parse-rest -- demashi --concurrency=3
 npm run verify:parse-rest -- demashi jd
 npm run dev
 ```
 
-The bootstrap command creates or reconciles 14 isolated `Voc*` classes, seeds the workspace/member/source records, imports the normalized case in request-size-bounded batches, and is idempotent for the same source hash and verified counts. The verification command checks the schema set, class-level permissions, workspace/import readiness, exact object counts, and denial of app-id-only reads. See `docs/parse-rest-schema.md` for the class contract.
+The bootstrap command creates or reconciles 14 isolated `Voc*` classes, seeds the workspace/member/source records, imports the normalized case in request-size-bounded batches, and is idempotent for the same source hash and verified counts. The workbook command performs the same bounded import directly from Excel using a streaming reader. Competitor sync creates all mapped competitor records first, then skips already completed details on later runs. The verification command checks the schema set, class-level permissions, workspace/import readiness, exact own/competitor/detail counts, and denial of app-id-only reads. See `docs/parse-rest-schema.md` for the class contract.
 
 ### Local frontend integration without a database
 
@@ -225,7 +229,7 @@ When the dedicated database and server are available, no route or frontend contr
 
 ## Security status
 
-`parse-server@9.10.0` replaces the initially evaluated v8 line, removing all high and critical audit findings. A `ws@8.21.0` override is retained because Parse currently pins an older vulnerable patch. The remaining audit findings are moderate transitive dependencies in Parse push/Firebase and redirect support; push and LiveQuery are not configured in this template. Re-run `npm audit --omit=dev` before each deployment and do not use `npm audit fix --force`, which currently proposes an unsafe Parse downgrade.
+`parse-server@9.10.0` replaces the initially evaluated v8 line, removing all high and critical audit findings. A `ws@8.21.0` override is retained because Parse currently pins an older vulnerable patch. The remaining 14 audit findings are moderate transitive dependencies in Parse push/Firebase and redirect support plus ExcelJS's nested UUID package; push and LiveQuery are not configured in this template. Re-run `npm audit --omit=dev` before each deployment and do not use `npm audit fix --force`, which currently proposes breaking Parse/ExcelJS downgrades.
 
 ## Verification
 
@@ -239,13 +243,13 @@ npm audit --omit=dev
 Current result:
 
 - TypeScript build: passed.
-- Unit, REST client, adapter, worker recovery, local-demo, authentication, RBAC, cursor, workflow, audit, and HTTP contract tests: 33 passed.
-- Production dependency audit: 0 critical, 0 high, 13 moderate.
+- Unit, REST client, adapter, worker recovery, local-demo, authentication, RBAC, cursor, workflow, audit, and HTTP contract tests: 35 passed.
+- Production dependency audit: 0 critical, 0 high, 14 moderate.
 - Company gateway health: HTTP 200 on 2026-07-23.
 - Credentialed live JD product-detail request: passed through the company gateway for product `11266507445`; the adapter extracted the product id, title, and brand from the live double-`data` envelope.
 - Credentialed live JD review request: authenticated successfully and returned an empty first page with next-page metadata. A non-empty review sample is still required before review evidence can be accepted.
 - Live credentials are absent from the repository, fixtures, logs, and Git history. No supplier endpoint is contacted directly.
-- Parse REST development verification: 14/14 `Voc*` schemas present, master-key-only access confirmed, 2,817 products, 9,717 metrics, 40 relations, and 0 reviews.
+- Parse REST development verification: 14/14 `Voc*` schemas present, master-key-only access confirmed, 2,817 own products, 37 competitor products with 37 details ready, 9,717 metrics, 40 relations, and 0 reviews.
 - End-to-end local API mode: `/health`, SaaS context, full snapshot, frontend proxy, desktop navigation, and 390x844 responsive rendering passed against the Parse REST store.
 - PostgreSQL integration: pending a newly provisioned database. The local Docker CLI is installed but its engine was unavailable on 2026-07-23; no existing database was contacted.
 

TASKS.md

@@ -34,6 +34,8 @@ Status date: 2026-07-24
 - [x] Add a batch importer for the normalized Demashi dataset.
 - [x] Import 2,817 operating products and 9,717 daily metric rows through bounded Parse REST batches.
 - [x] Import 40 competitor relations with deterministic relation keys.
+- [x] Add a streaming importer that persists the original Demashi workbook directly through Parse REST.
+- [x] Create 37 mapped competitor products and backfill all 37 real product details through the company relay.
 - [x] Verify Parse counts and snapshot totals against `demashi-summary.json`.
 - [x] Switch `DomesticDatasetService` between static case mode and backend API mode.
 - [x] Run desktop/mobile browser smoke tests against Parse REST API mode.

package.json

@@ -16,6 +16,8 @@
     "migrate": "tsx scripts/migrate.ts",
     "bootstrap:parse-rest": "tsx scripts/bootstrap-parse-rest.ts",
     "verify:parse-rest": "tsx scripts/verify-parse-rest.ts",
+    "import:workbook:parse-rest": "tsx scripts/import-workbook-parse-rest.ts",
+    "sync:competitors:parse-rest": "tsx scripts/sync-parse-rest-competitors.ts",
     "import:dataset": "tsx scripts/import-dataset.ts",
     "test": "tsx --test test/**/*.test.ts",
     "test:coverage": "tsx --test --experimental-test-coverage test/**/*.test.ts",
@@ -24,6 +26,7 @@
   "dependencies": {
     "cors": "^2.8.6",
     "dotenv": "^17.2.3",
+    "exceljs": "^4.4.0",
     "express": "^5.2.1",
     "parse-server": "^9.10.0",
     "pg": "^8.22.0",

scripts/import-workbook-parse-rest.ts

@@ -0,0 +1,27 @@
+import 'dotenv/config';
+import { resolve } from 'node:path';
+import { loadConfig } from '../src/config/env.js';
+import { ParseRestClient } from '../src/db/parse-rest.client.js';
+import { ensureVocParseSchemas } from '../src/db/parse-rest.schema.js';
+import { normalizeDemashiWorkbook } from '../src/modules/domestic-voc/importers/demashi-workbook.js';
+import { ParseRestDatasetImportService } from '../src/modules/domestic-voc/services/parse-rest-dataset-import.service.js';
+
+const inputPath = process.argv[2];
+if (!inputPath) throw new Error('Usage: npm run import:workbook:parse-rest -- <workbook.xlsx> [workspaceId]');
+const config = loadConfig();
+if (config.storageDriver !== 'parse_rest') throw new Error('STORAGE_DRIVER must be parse_rest');
+const workspaceId = process.argv[3] || config.auth.defaultWorkspaceId;
+const client = new ParseRestClient({
+  serverUrl: config.parse.serverUrl,
+  appId: config.parse.appId,
+  masterKey: config.parse.masterKey,
+  timeoutMs: config.parse.timeoutMs,
+});
+const dataset = await normalizeDemashiWorkbook(resolve(inputPath));
+await ensureVocParseSchemas(client);
+const imported = await new ParseRestDatasetImportService(client).import(dataset, workspaceId, {
+  userId: config.auth.localUserId,
+  email: config.auth.localUserEmail,
+  displayName: config.auth.localUserName,
+});
+console.log(JSON.stringify({ source: dataset.source, summary: dataset.summary, imported }, null, 2));

scripts/sync-parse-rest-competitors.ts

@@ -0,0 +1,40 @@
+import 'dotenv/config';
+import { loadConfig } from '../src/config/env.js';
+import { ParseRestClient } from '../src/db/parse-rest.client.js';
+import { ParseRestCompetitorSyncService } from '../src/modules/domestic-voc/services/parse-rest-competitor-sync.service.js';
+import { FmodeVocEcommerceClient } from '../src/modules/domestic-voc/upstream/fmode-client.js';
+import { ParseRestVocRepository } from '../src/modules/saas-platform/parse-rest-voc.repository.js';
+
+const config = loadConfig();
+if (config.storageDriver !== 'parse_rest') throw new Error('STORAGE_DRIVER must be parse_rest');
+const workspaceId = process.argv[2] || config.auth.defaultWorkspaceId;
+const refresh = process.argv.includes('--refresh');
+const concurrencyArg = process.argv.find((value) => value.startsWith('--concurrency='));
+const concurrency = Number(concurrencyArg?.split('=')[1] ?? 3);
+const client = new ParseRestClient({
+  serverUrl: config.parse.serverUrl,
+  appId: config.parse.appId,
+  masterKey: config.parse.masterKey,
+  timeoutMs: config.parse.timeoutMs,
+});
+const repository = new ParseRestVocRepository(client);
+const gateway = new FmodeVocEcommerceClient(config.fmode);
+const service = new ParseRestCompetitorSyncService(client, repository, gateway);
+const result = await service.sync({
+  workspaceId,
+  platform: 'jd',
+  concurrency,
+  refresh,
+  onProgress(progress) {
+    console.log(`[competitor-sync] ${progress.completed}/${progress.total} ${progress.productId} ${progress.status}`);
+  },
+});
+await repository.appendAudit({
+  workspaceId,
+  actorUserId: 'system:competitor-sync',
+  action: 'competitor.sync.completed',
+  entityType: 'product',
+  entityId: null,
+  metadata: { ...result },
+});
+console.log(JSON.stringify(result, null, 2));

scripts/verify-parse-rest.ts

@@ -27,8 +27,10 @@ const masterOnly = vocSchemas.every((schema) => operations.every((operation) =>
 )));
 
 const where = { workspaceId, platform };
-const [products, metrics, relations, reviews, workspace, completedImport] = await Promise.all([
-  client.count(VOC_PARSE_CLASSES.product, where),
+const [ownProducts, competitorProducts, competitorDetailsReady, metrics, relations, reviews, workspace, completedImport] = await Promise.all([
+  client.count(VOC_PARSE_CLASSES.product, { ...where, role: 'own' }),
+  client.count(VOC_PARSE_CLASSES.product, { ...where, role: 'competitor' }),
+  client.count(VOC_PARSE_CLASSES.product, { ...where, role: 'competitor', source: 'fmode_gateway' }),
   client.count(VOC_PARSE_CLASSES.dailyMetric, where),
   client.count(VOC_PARSE_CLASSES.productRelation, where),
   client.count(VOC_PARSE_CLASSES.review, where),
@@ -53,7 +55,16 @@ console.log(JSON.stringify({
   },
   workspaceReady: Boolean(workspace),
   completedImport: Boolean(completedImport),
-  counts: { products, metrics, relations, reviews },
+  counts: {
+    ownProducts,
+    competitorProducts,
+    competitorDetailsReady,
+    competitorDetailsPending: competitorProducts - competitorDetailsReady,
+    products: ownProducts + competitorProducts,
+    metrics,
+    relations,
+    reviews,
+  },
   publicReadBlocked: !publicResponse.ok,
   publicReadStatus: publicResponse.status,
   publicReadCode: publicPayload.code ?? null,

src/modules/domestic-voc/adapters/jd-product.adapter.ts

@@ -12,7 +12,7 @@ export interface JdProductRecord {
   platform: 'jd';
   productId: string;
   productKey: string;
-  role: 'own';
+  role: 'own' | 'competitor';
   brand: string;
   title: string;
   model: string;
@@ -28,7 +28,11 @@ const PRODUCT_KEYS = [
   'brandName', 'brand', 'model', 'categoryName', 'category1', 'category2', 'category3',
 ];
 
-export function adaptJdProductResponse(response: unknown, requestedProductId: string): JdProductRecord {
+export function adaptJdProductResponse(
+  response: unknown,
+  requestedProductId: string,
+  role: JdProductRecord['role'] = 'own',
+): JdProductRecord {
   const payload = unwrapGatewayPayload(response);
   const record = findBestRecord(payload, PRODUCT_KEYS);
   const productId = firstString(record ?? {}, ['itemId', 'skuId', 'productId', 'id']) || requestedProductId;
@@ -36,7 +40,7 @@ export function adaptJdProductResponse(response: unknown, requestedProductId: st
     platform: 'jd',
     productId,
     productKey: makeProductKey('jd', productId),
-    role: 'own',
+    role,
     brand: firstString(record ?? {}, ['brandName', 'brand', 'brand_name']),
     title: firstString(record ?? {}, ['itemName', 'skuName', 'title', 'name', 'productName']),
     model: firstString(record ?? {}, ['model', 'modelName', 'skuModel', 'wareModel']),
@@ -47,4 +51,3 @@ export function adaptJdProductResponse(response: unknown, requestedProductId: st
     rawPayload: record,
   };
 }
-

src/modules/domestic-voc/importers/demashi-workbook.ts

@@ -0,0 +1,309 @@
+import { createHash } from 'node:crypto';
+import { readFile } from 'node:fs/promises';
+import { basename } from 'node:path';
+import ExcelJS from 'exceljs';
+import type { DomesticDataset, DomesticMetricSummary, DomesticProductRelation } from '../../../types/domestic-dataset.js';
+import type { ImportDataset } from '../services/dataset-import.service.js';
+
+const METRIC_SHEET = '德玛仕产品基础数据';
+const RELATION_SHEET = '竟对品牌及编码';
+
+type Row = unknown[];
+type HeaderIndex = Map<string, number>;
+
+interface MetricPoint {
+  date: string;
+  gmv: number;
+  soldUnits: number;
+  transactionOrders: number;
+  transactionCustomers: number;
+  impressions: number;
+  clicks: number;
+  views: number;
+  visitors: number;
+  cartUnits: number;
+  orderAmount: number;
+  orderUnits: number;
+  orderCount: number;
+  refundAmount: number;
+  refundUnits: number;
+  refundOrders: number;
+}
+
+type MutableProduct = ImportDataset['products'][number];
+type MutableDailyMetric = DomesticDataset['dailyTotals'][number];
+
+export async function normalizeDemashiWorkbook(inputPath: string): Promise<ImportDataset> {
+  const file = await readFile(inputPath);
+  const sheets = await readWorkbookRows(inputPath);
+  const metricRows = sheets.get(METRIC_SHEET);
+  const relationRows = sheets.get(RELATION_SHEET);
+  if (!metricRows) throw new Error(`Missing worksheet: ${METRIC_SHEET}`);
+  if (!relationRows) throw new Error(`Missing worksheet: ${RELATION_SHEET}`);
+  const metricHeaderIndex = findHeaderRow(metricRows, ['时间', 'SKU', 'SKU名称']);
+  const relationHeaderIndex = findHeaderRow(relationRows, ['类目', '德玛仕sku']);
+  const metricHeaders = indexHeaders(metricRows[metricHeaderIndex]!);
+  const relationHeaders = indexHeaders(relationRows[relationHeaderIndex]!);
+  const products = new Map<string, MutableProduct>();
+  const dailyTotals = new Map<string, MutableDailyMetric>();
+  let metricRowCount = 0;
+
+  for (const row of metricRows.slice(metricHeaderIndex + 1)) {
+    const productId = asId(read(row, metricHeaders, 'SKU'));
+    const date = asDate(read(row, metricHeaders, '时间'));
+    if (!productId || !date) continue;
+    metricRowCount += 1;
+    const productKey = `jd:${productId}`;
+    const metric = metricPoint(row, metricHeaders, date);
+    const product = products.get(productKey) ?? createProduct(row, metricHeaders, productId, productKey);
+    product.trend.push(metric);
+    addMetric(product.summary, metric);
+    products.set(productKey, product);
+
+    const day = dailyTotals.get(date) ?? { date, ...emptySummary() };
+    addMetric(day, metric);
+    dailyTotals.set(date, day);
+  }
+
+  for (const product of products.values()) {
+    product.trend.sort((left, right) => left.date.localeCompare(right.date));
+    finalizeSummary(product.summary);
+  }
+
+  const mappingGroups: DomesticDataset['mappingGroups'] = [];
+  const relations: DomesticProductRelation[] = [];
+  const competitorIds = new Set<string>();
+  const orphanMappings: DomesticDataset['quality']['orphanMappings'] = [];
+  const mappingsWithoutCompetitor: DomesticDataset['quality']['mappingsWithoutCompetitor'] = [];
+  const brandWithoutProductId: DomesticDataset['quality']['brandWithoutProductId'] = [];
+
+  for (const row of relationRows.slice(relationHeaderIndex + 1)) {
+    const ownProductId = asId(read(row, relationHeaders, '德玛仕sku'));
+    if (!ownProductId) continue;
+    const ownProductKey = `jd:${ownProductId}`;
+    const category = cleanText(read(row, relationHeaders, '类目'));
+    const model = cleanText(read(row, relationHeaders, '德玛仕型号'));
+    const competitors: DomesticProductRelation[] = [];
+    for (let slot = 1; slot <= 3; slot += 1) {
+      const brand = cleanText(read(row, relationHeaders, `竟品${slot}品牌`));
+      const competitorProductId = asId(read(row, relationHeaders, `竟品sku${slot}`));
+      if (brand && !competitorProductId) {
+        brandWithoutProductId.push({ ownProductId, slot, brand });
+        continue;
+      }
+      if (!competitorProductId) continue;
+      const competitorProductKey = `jd:${competitorProductId}`;
+      const relation: DomesticProductRelation = {
+        relationKey: `${ownProductKey}:${competitorProductKey}`,
+        ownProductKey,
+        ownProductId,
+        competitorProductKey,
+        competitorProductId,
+        competitorBrand: brand,
+        category,
+      };
+      competitors.push(relation);
+      relations.push(relation);
+      competitorIds.add(competitorProductId);
+    }
+    const ownProduct = products.get(ownProductKey);
+    if (ownProduct) {
+      ownProduct.model = model || ownProduct.model;
+      ownProduct.relationCount = competitors.length;
+    } else {
+      orphanMappings.push({ ownProductId, model, category });
+    }
+    if (!competitors.length) mappingsWithoutCompetitor.push({ ownProductId, model, category });
+    mappingGroups.push({ ownProductId, ownProductKey, model, category, competitors });
+  }
+
+  const productList = [...products.values()].sort((left, right) => right.summary.gmv - left.summary.gmv);
+  const daily = [...dailyTotals.values()]
+    .sort((left, right) => left.date.localeCompare(right.date))
+    .map((metric) => finalizeSummary(metric));
+  const dates = daily.map((metric) => metric.date);
+
+  return {
+    schemaVersion: 1,
+    generatedAt: new Date().toISOString(),
+    caseName: '德玛仕',
+    platform: 'jd',
+    source: {
+      sourceFile: basename(inputPath),
+      sourceHash: createHash('sha256').update(file).digest('hex'),
+      sheets: [METRIC_SHEET, RELATION_SHEET],
+      dateRange: { start: dates[0] ?? '', end: dates.at(-1) ?? '' },
+    },
+    summary: {
+      metricRows: metricRowCount,
+      metricProducts: productList.length,
+      mappingRows: mappingGroups.length,
+      relations: relations.length,
+      uniqueCompetitorProducts: competitorIds.size,
+      category2Count: new Set(productList.map((product) => product.category2).filter(Boolean)).size,
+      category3Count: new Set(productList.map((product) => product.category3).filter(Boolean)).size,
+      reviewCount: 0,
+    },
+    dailyTotals: daily,
+    products: productList,
+    mappingGroups,
+    relations,
+    reviews: [],
+    quality: { orphanMappings, mappingsWithoutCompetitor, brandWithoutProductId },
+  };
+}
+
+function createProduct(row: Row, headers: HeaderIndex, productId: string, productKey: string): MutableProduct {
+  return {
+    platform: 'jd',
+    productId,
+    productKey,
+    asin: productId,
+    role: 'own',
+    brand: '德玛仕',
+    title: cleanText(read(row, headers, 'SKU名称')),
+    model: '',
+    category1: cleanText(read(row, headers, '一级类目')),
+    category2: cleanText(read(row, headers, '二级类目')),
+    category3: cleanText(read(row, headers, '三级类目')),
+    source: 'excel',
+    relationCount: 0,
+    summary: emptySummary(),
+    trend: [],
+  };
+}
+
+function metricPoint(row: Row, headers: HeaderIndex, date: string): MetricPoint {
+  return {
+    date,
+    gmv: asNumber(read(row, headers, '成交金额')),
+    soldUnits: asNumber(read(row, headers, '成交商品件数')),
+    transactionOrders: asNumber(read(row, headers, '成交单量')),
+    transactionCustomers: asNumber(read(row, headers, '成交客户数')),
+    impressions: asNumber(read(row, headers, '搜索曝光次数')),
+    clicks: asNumber(read(row, headers, '搜索点击次数')),
+    views: asNumber(read(row, headers, '商品浏览量')),
+    visitors: asNumber(read(row, headers, '商品访客数')),
+    cartUnits: asNumber(read(row, headers, '加购商品件数')),
+    orderAmount: asNumber(read(row, headers, '下单金额')),
+    orderUnits: asNumber(read(row, headers, '下单商品件数')),
+    orderCount: asNumber(read(row, headers, '下单单量')),
+    refundAmount: asNumber(read(row, headers, '取消及售后退款金额')),
+    refundUnits: asNumber(read(row, headers, '取消及售后退款商品件数')),
+    refundOrders: asNumber(read(row, headers, '取消及售后退款单量')),
+  };
+}
+
+function emptySummary(): DomesticMetricSummary {
+  return {
+    gmv: 0, soldUnits: 0, transactionOrders: 0, transactionCustomers: 0,
+    impressions: 0, clicks: 0, views: 0, visitors: 0, cartUnits: 0,
+    orderAmount: 0, orderUnits: 0, orderCount: 0, refundAmount: 0,
+    refundUnits: 0, refundOrders: 0, conversionRate: 0, clickThroughRate: 0,
+    averageUnitPrice: 0, refundToGmvRate: 0,
+  };
+}
+
+function addMetric(target: DomesticMetricSummary, metric: MetricPoint): void {
+  target.gmv += metric.gmv;
+  target.soldUnits += metric.soldUnits;
+  target.transactionOrders += metric.transactionOrders;
+  target.transactionCustomers += metric.transactionCustomers;
+  target.impressions += metric.impressions;
+  target.clicks += metric.clicks;
+  target.views += metric.views;
+  target.visitors += metric.visitors;
+  target.cartUnits += metric.cartUnits;
+  target.orderAmount += metric.orderAmount;
+  target.orderUnits += metric.orderUnits;
+  target.orderCount += metric.orderCount;
+  target.refundAmount += metric.refundAmount;
+  target.refundUnits += metric.refundUnits;
+  target.refundOrders += metric.refundOrders;
+}
+
+function finalizeSummary<T extends DomesticMetricSummary>(summary: T): T {
+  summary.conversionRate = divide(summary.transactionCustomers, summary.visitors);
+  summary.clickThroughRate = divide(summary.clicks, summary.impressions);
+  summary.averageUnitPrice = divide(summary.gmv, summary.soldUnits);
+  summary.refundToGmvRate = divide(summary.refundAmount, summary.gmv);
+  return summary;
+}
+
+function findHeaderRow(rows: Row[], required: string[]): number {
+  const index = rows.findIndex((row) => required.every((header) => row.some((cell) => cleanText(cell) === header)));
+  if (index < 0) throw new Error(`Header row not found: ${required.join(', ')}`);
+  return index;
+}
+
+function indexHeaders(row: Row): HeaderIndex {
+  return new Map(row
+    .map((value, index): [string, number] => [cleanText(value), index])
+    .filter(([name]) => Boolean(name)));
+}
+
+function read(row: Row, headers: HeaderIndex, name: string): unknown {
+  const index = headers.get(name);
+  return index === undefined ? null : row[index];
+}
+
+function cleanText(value: unknown): string {
+  return value === null || value === undefined ? '' : String(value).trim();
+}
+
+function asId(value: unknown): string {
+  if (value === null || value === undefined || value === '') return '';
+  if (typeof value === 'number' && Number.isFinite(value)) return Math.trunc(value).toString();
+  return cleanText(value).replace(/\.0$/, '');
+}
+
+function asNumber(value: unknown): number {
+  if (typeof value === 'number') return Number.isFinite(value) ? value : 0;
+  const parsed = Number(cleanText(value).replace(/,/g, ''));
+  return Number.isFinite(parsed) ? parsed : 0;
+}
+
+function asDate(value: unknown): string {
+  if (value instanceof Date && !Number.isNaN(value.getTime())) return value.toISOString().slice(0, 10);
+  if (typeof value === 'number') {
+    const date = new Date(Date.UTC(1899, 11, 30) + Math.round(value * 86_400_000));
+    if (!Number.isNaN(date.getTime())) return date.toISOString().slice(0, 10);
+  }
+  const text = cleanText(value);
+  if (!text) return '';
+  const timestamp = Date.parse(text);
+  return Number.isNaN(timestamp) ? text.slice(0, 10) : new Date(timestamp).toISOString().slice(0, 10);
+}
+
+async function readWorkbookRows(inputPath: string): Promise<Map<string, Row[]>> {
+  const sheets = new Map<string, Row[]>();
+  const reader = new ExcelJS.stream.xlsx.WorkbookReader(inputPath, {
+    worksheets: 'emit',
+    sharedStrings: 'cache',
+    hyperlinks: 'ignore',
+    styles: 'ignore',
+  });
+  for await (const worksheet of reader) {
+    const sheetName = (worksheet as typeof worksheet & { name: string }).name;
+    if (sheetName !== METRIC_SHEET && sheetName !== RELATION_SHEET) continue;
+    const rows: Row[] = [];
+    for await (const row of worksheet) {
+      const values = Array.isArray(row.values) ? row.values.slice(1) : [];
+      rows.push(values.map((value) => cellValue(value as ExcelJS.CellValue)));
+    }
+    sheets.set(sheetName, rows);
+  }
+  return sheets;
+}
+
+function cellValue(value: ExcelJS.CellValue): unknown {
+  if (value === null || value === undefined || typeof value !== 'object' || value instanceof Date) return value;
+  if ('result' in value) return value.result ?? null;
+  if ('richText' in value) return value.richText.map((part) => part.text).join('');
+  if ('text' in value) return value.text;
+  return String(value);
+}
+
+function divide(numerator: number, denominator: number): number {
+  return denominator ? numerator / denominator : 0;
+}

src/modules/domestic-voc/services/parse-rest-competitor-sync.service.ts

@@ -0,0 +1,111 @@
+import { ParseRestClient } from '../../../db/parse-rest.client.js';
+import { VOC_PARSE_CLASSES } from '../../../db/parse-rest.schema.js';
+import { adaptJdProductResponse, JD_PRODUCT_DETAIL_PATH } from '../adapters/jd-product.adapter.js';
+import type { GatewayRequestClient } from './jd-sync.service.js';
+import { classifySyncFailure } from './jd-sync.service.js';
+import { ParseRestVocRepository } from '../../saas-platform/parse-rest-voc.repository.js';
+
+interface RelationObject {
+  competitorProductId: string;
+  competitorBrand: string;
+  category: string;
+}
+
+interface StoredProduct {
+  productId: string;
+  title: string;
+  source: string;
+}
+
+export interface CompetitorSyncResult {
+  total: number;
+  placeholders: number;
+  succeeded: number;
+  skipped: number;
+  failed: number;
+  failedProductIds: string[];
+}
+
+export class ParseRestCompetitorSyncService {
+  constructor(
+    private readonly client: ParseRestClient,
+    private readonly repository: ParseRestVocRepository,
+    private readonly gateway: GatewayRequestClient,
+  ) {}
+
+  async sync(input: {
+    workspaceId: string;
+    platform: 'jd';
+    concurrency?: number;
+    refresh?: boolean;
+    onProgress?: (progress: { completed: number; total: number; productId: string; status: 'succeeded' | 'skipped' | 'failed' }) => void;
+  }): Promise<CompetitorSyncResult> {
+    const relations = await this.client.findAll<RelationObject>(VOC_PARSE_CLASSES.productRelation, {
+      workspaceId: input.workspaceId,
+      platform: input.platform,
+    });
+    const mappings = new Map<string, RelationObject>();
+    for (const relation of relations) {
+      if (!mappings.has(relation.competitorProductId)) mappings.set(relation.competitorProductId, relation);
+    }
+    const targets = [...mappings.values()];
+    for (const target of targets) {
+      await this.repository.ensureCompetitorProduct({
+        workspaceId: input.workspaceId,
+        platform: input.platform,
+        productId: target.competitorProductId,
+        brand: target.competitorBrand,
+        category: target.category,
+      });
+    }
+
+    const result: CompetitorSyncResult = {
+      total: targets.length,
+      placeholders: targets.length,
+      succeeded: 0,
+      skipped: 0,
+      failed: 0,
+      failedProductIds: [],
+    };
+    let cursor = 0;
+    let completed = 0;
+    const worker = async () => {
+      for (;;) {
+        const target = targets[cursor];
+        cursor += 1;
+        if (!target) return;
+        const productId = target.competitorProductId;
+        let status: 'succeeded' | 'skipped' | 'failed';
+        try {
+          const existing = await this.client.findOne<StoredProduct>(VOC_PARSE_CLASSES.product, {
+            workspaceId: input.workspaceId,
+            platform: input.platform,
+            productId,
+          });
+          if (!input.refresh && existing?.source === 'fmode_gateway' && existing.title) {
+            result.skipped += 1;
+            status = 'skipped';
+          } else {
+            const response = await this.gateway.request<unknown>(JD_PRODUCT_DETAIL_PATH, {
+              params: { itemId: productId },
+            });
+            const product = adaptJdProductResponse(response, productId, 'competitor');
+            await this.repository.upsertProduct(input.workspaceId, product);
+            result.succeeded += 1;
+            status = 'succeeded';
+          }
+        } catch (error) {
+          classifySyncFailure(error);
+          result.failed += 1;
+          result.failedProductIds.push(productId);
+          status = 'failed';
+        }
+        completed += 1;
+        input.onProgress?.({ completed, total: targets.length, productId, status });
+      }
+    };
+    const concurrency = Math.max(1, Math.min(8, Math.floor(input.concurrency ?? 3)));
+    await Promise.all(Array.from({ length: Math.min(concurrency, targets.length) }, () => worker()));
+    return result;
+  }
+}

src/modules/domestic-voc/services/parse-rest-dataset-import.service.ts

@@ -247,7 +247,7 @@ export class ParseRestDatasetImportService {
   async counts(workspaceId: string, platform: string): Promise<Omit<ParseDatasetImportResult, 'batchId' | 'skipped'>> {
     const where = { workspaceId, platform };
     const [products, metrics, relations, reviews] = await Promise.all([
-      this.client.count(VOC_PARSE_CLASSES.product, where),
+      this.client.count(VOC_PARSE_CLASSES.product, { ...where, role: 'own' }),
       this.client.count(VOC_PARSE_CLASSES.dailyMetric, where),
       this.client.count(VOC_PARSE_CLASSES.productRelation, where),
       this.client.count(VOC_PARSE_CLASSES.review, where),

src/modules/domestic-voc/services/parse-rest-snapshot.service.ts

@@ -164,6 +164,7 @@ export class ParseRestSnapshotService {
       return date ? { ...value, reviewDate: date } : value;
     });
     const batch = batchResult.results[0];
+    const ownProducts = products.filter((product) => product.role === 'own');
     const ownIds = new Set(relations.map((relation) => relation.ownProductId));
     const competitorIds = new Set(relations.map((relation) => relation.competitorProductId));
 
@@ -183,12 +184,12 @@ export class ParseRestSnapshotService {
       },
       summary: {
         metricRows: batch?.metricRows ?? 0,
-        metricProducts: products.filter((product) => product.trend.length > 0).length,
+        metricProducts: ownProducts.filter((product) => product.trend.length > 0).length,
         mappingRows: ownIds.size,
         relations: batch?.relationRows ?? relations.length,
         uniqueCompetitorProducts: competitorIds.size,
-        category2Count: new Set(products.map((product) => product.category2).filter(Boolean)).size,
-        category3Count: new Set(products.map((product) => product.category3).filter(Boolean)).size,
+        category2Count: new Set(ownProducts.map((product) => product.category2).filter(Boolean)).size,
+        category3Count: new Set(ownProducts.map((product) => product.category3).filter(Boolean)).size,
         reviewCount: batch?.reviewRows ?? reviews.length,
       },
       dailyTotals: batch?.dailyTotals ?? [],

src/modules/saas-platform/parse-rest-voc.repository.ts

@@ -230,6 +230,16 @@ function mapReview(review: ReviewObject): DomesticReview {
   return date ? { ...value, reviewDate: date } : value;
 }
 
+function emptyProductSummary(): DomesticProduct['summary'] {
+  return {
+    gmv: 0, soldUnits: 0, transactionOrders: 0, transactionCustomers: 0,
+    impressions: 0, clicks: 0, views: 0, visitors: 0, cartUnits: 0,
+    orderAmount: 0, orderUnits: 0, orderCount: 0, refundAmount: 0,
+    refundUnits: 0, refundOrders: 0, conversionRate: 0, clickThroughRate: 0,
+    averageUnitPrice: 0, refundToGmvRate: 0,
+  };
+}
+
 export class ParseRestVocRepository implements PlatformRepository, SyncJobStore, SyncPersistence {
   private readonly snapshot: ParseRestSnapshotService;
 
@@ -683,7 +693,7 @@ export class ParseRestVocRepository implements PlatformRepository, SyncJobStore,
       platform: product.platform,
       productId: product.productId,
       productKey: product.productKey,
-      role: product.role,
+      role: existing?.role ?? product.role,
       brand: product.brand || existing?.brand || '',
       title: product.title || existing?.title || '',
       model: product.model || existing?.model || '',
@@ -692,7 +702,7 @@ export class ParseRestVocRepository implements PlatformRepository, SyncJobStore,
       category3: product.category3 || existing?.category3 || '',
       source: product.source,
       relationCount: existing?.relationCount ?? 0,
-      summary: existing?.summary ?? {},
+      summary: existing?.summary ?? emptyProductSummary(),
       trend: existing?.trend ?? [],
       rawPayload: product.rawPayload,
     };
@@ -700,6 +710,37 @@ export class ParseRestVocRepository implements PlatformRepository, SyncJobStore,
     else await this.client.create(VOC_PARSE_CLASSES.product, body);
   }
 
+  async ensureCompetitorProduct(input: {
+    workspaceId: string;
+    platform: 'jd';
+    productId: string;
+    brand: string;
+    category: string;
+  }): Promise<void> {
+    const naturalKey = key(input.workspaceId, input.platform, input.productId);
+    const existing = await this.client.findOne<ProductObject>(VOC_PARSE_CLASSES.product, { naturalKey });
+    const body = {
+      naturalKey,
+      workspaceId: input.workspaceId,
+      platform: input.platform,
+      productId: input.productId,
+      productKey: `${input.platform}:${input.productId}`,
+      role: 'competitor' as const,
+      brand: existing?.brand || input.brand,
+      title: existing?.title || '',
+      model: existing?.model || '',
+      category1: existing?.category1 || '',
+      category2: existing?.category2 || '',
+      category3: existing?.category3 || input.category,
+      source: existing?.source === 'fmode_gateway' ? existing.source : 'relation_mapping',
+      relationCount: existing?.relationCount ?? 0,
+      summary: existing?.summary ?? emptyProductSummary(),
+      trend: existing?.trend ?? [],
+    };
+    if (existing) await this.client.update(VOC_PARSE_CLASSES.product, existing.objectId, body);
+    else await this.client.create(VOC_PARSE_CLASSES.product, body);
+  }
+
   async ensureProductStub(workspaceId: string, platform: string, productId: string): Promise<void> {
     const workspace = await this.client.findOne(VOC_PARSE_CLASSES.workspace, { publicId: workspaceId, status: 'active' });
     if (!workspace) throw new Error(`Workspace not found: ${workspaceId}`);
@@ -708,7 +749,7 @@ export class ParseRestVocRepository implements PlatformRepository, SyncJobStore,
     await this.client.create(VOC_PARSE_CLASSES.product, {
       naturalKey, workspaceId, platform, productId, productKey: `${platform}:${productId}`,
       role: 'own', brand: '', title: '', model: '', category1: '', category2: '', category3: '',
-      source: 'sync_stub', relationCount: 0, summary: {}, trend: [],
+      source: 'sync_stub', relationCount: 0, summary: emptyProductSummary(), trend: [],
     });
   }
 

test/jd-adapters.test.ts

@@ -30,6 +30,11 @@ test('JD product adapter unwraps the company gateway envelope', () => {
   assert.equal(product.source, 'fmode_gateway');
 });
 
+test('JD product adapter preserves an explicit competitor role', () => {
+  const product = adaptJdProductResponse({ data: { item: { itemId: '100119862001', itemName: 'Competitor' } } }, '100119862001', 'competitor');
+  assert.equal(product.role, 'competitor');
+});
+
 test('JD review adapter maps evidence and pagination without reviewer identity', () => {
   const page = adaptJdReviewResponse({
     code: 200,
@@ -68,4 +73,3 @@ test('JD adapters classify pending collection as retryable without raw body leak
     },
   );
 });
-

test/parse-rest-competitor-sync.test.ts

@@ -0,0 +1,53 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+import type { ParseRestClient } from '../src/db/parse-rest.client.js';
+import { ParseRestCompetitorSyncService } from '../src/modules/domestic-voc/services/parse-rest-competitor-sync.service.js';
+import type { GatewayRequestClient } from '../src/modules/domestic-voc/services/jd-sync.service.js';
+import type { ParseRestVocRepository } from '../src/modules/saas-platform/parse-rest-voc.repository.js';
+
+test('competitor sync creates mapped placeholders and persists returned products as competitors', async () => {
+  const placeholders: string[] = [];
+  const products: Array<{ productId: string; role: string }> = [];
+  const client = {
+    async findAll() {
+      return [
+        { competitorProductId: 'c1', competitorBrand: 'Brand A', category: 'Category' },
+        { competitorProductId: 'c1', competitorBrand: 'Brand A', category: 'Category' },
+        { competitorProductId: 'c2', competitorBrand: 'Brand B', category: 'Category' },
+      ];
+    },
+    async findOne() { return null; },
+  } as unknown as ParseRestClient;
+  const repository = {
+    async ensureCompetitorProduct(input: { productId: string }) { placeholders.push(input.productId); },
+    async upsertProduct(_workspaceId: string, product: { productId: string; role: string }) {
+      products.push({ productId: product.productId, role: product.role });
+    },
+  } as unknown as ParseRestVocRepository;
+  const gateway: GatewayRequestClient = {
+    async request(_path, init) {
+      const productId = String(init?.params?.itemId);
+      return { code: 200, data: { code: 200, data: { item: { itemId: productId, itemName: `Product ${productId}` } } } } as never;
+    },
+  };
+
+  const result = await new ParseRestCompetitorSyncService(client, repository, gateway).sync({
+    workspaceId: 'demashi',
+    platform: 'jd',
+    concurrency: 2,
+  });
+
+  assert.deepEqual(placeholders.sort(), ['c1', 'c2']);
+  assert.deepEqual(products.sort((left, right) => left.productId.localeCompare(right.productId)), [
+    { productId: 'c1', role: 'competitor' },
+    { productId: 'c2', role: 'competitor' },
+  ]);
+  assert.deepEqual(result, {
+    total: 2,
+    placeholders: 2,
+    succeeded: 2,
+    skipped: 0,
+    failed: 0,
+    failedProductIds: [],
+  });
+});