فهرست منبع

feat: sync competitor reviews and catalog profiles

gangvy 1 ماه پیش
والد
کامیت
7963986c55

+ 5 - 5
README.md

@@ -30,7 +30,7 @@ Completed through 2026-07-24:
 - Direct streaming import from the source Demashi workbook without a frontend-generated JSON intermediate.
 - Direct streaming import from the source Demashi workbook without a frontend-generated JSON intermediate.
 - Idempotent competitor detail backfill through the company ecommerce relay, with mapped placeholders preserved when a request fails.
 - Idempotent competitor detail backfill through the company ecommerce relay, with mapped placeholders preserved when a request fails.
 
 
-The current case import resolves to 2,817 operating products, 37 mapped competitor products, 9,717 daily metrics, 40 relations, and 0 reviews. All 37 competitor product details were persisted through the company relay on 2026-07-24. The PostgreSQL importer may create internal `relation_stub` rows to preserve foreign keys; the Parse REST model stores denormalized relation identities and therefore does not create those extra catalog products.
+The current case resolves to 2,817 operating products, 37 mapped competitor products, 9,717 daily metrics, 40 relations, and 355 first-page competitor review samples. All 37 competitor product details and review requests were persisted through the company relay on 2026-07-24. The PostgreSQL importer may create internal `relation_stub` rows to preserve foreign keys; the Parse REST model stores denormalized relation identities and therefore does not create those extra catalog products.
 
 
 The worker calls JD product-detail and review paths only through the company `/api/voc-e-commerce` gateway. It never sends a browser request to a supplier endpoint and never returns raw gateway errors or credentials.
 The worker calls JD product-detail and review paths only through the company `/api/voc-e-commerce` gateway. It never sends a browser request to a supplier endpoint and never returns raw gateway errors or credentials.
 
 
@@ -78,7 +78,7 @@ $env:API_AUTH_MODE = 'disabled' # isolated local development only
 $env:FMODE_API_KEY = '<process-only-company-gateway-key>'
 $env:FMODE_API_KEY = '<process-only-company-gateway-key>'
 npm run bootstrap:parse-rest -- "E:\workspace\Saas-voc\src\assets\data\demashi-summary.json" demashi
 npm run bootstrap:parse-rest -- "E:\workspace\Saas-voc\src\assets\data\demashi-summary.json" demashi
 npm run import:workbook:parse-rest -- "E:\path\to\德玛仕产品及竟品收集0721.xlsx" demashi
 npm run import:workbook:parse-rest -- "E:\path\to\德玛仕产品及竟品收集0721.xlsx" demashi
-npm run sync:competitors:parse-rest -- demashi --concurrency=3
+npm run sync:competitors:parse-rest -- demashi --concurrency=3 --reviews --review-pages=1
 npm run verify:parse-rest -- demashi jd
 npm run verify:parse-rest -- demashi jd
 npm run dev
 npm run dev
 ```
 ```
@@ -243,13 +243,13 @@ npm audit --omit=dev
 Current result:
 Current result:
 
 
 - TypeScript build: passed.
 - TypeScript build: passed.
-- Unit, REST client, adapter, worker recovery, local-demo, authentication, RBAC, cursor, workflow, audit, and HTTP contract tests: 35 passed.
+- Unit, REST client, adapter, worker recovery, local-demo, authentication, RBAC, cursor, workflow, audit, and HTTP contract tests: 37 passed.
 - Production dependency audit: 0 critical, 0 high, 14 moderate.
 - Production dependency audit: 0 critical, 0 high, 14 moderate.
 - Company gateway health: HTTP 200 on 2026-07-23.
 - Company gateway health: HTTP 200 on 2026-07-23.
 - Credentialed live JD product-detail request: passed through the company gateway for product `11266507445`; the adapter extracted the product id, title, and brand from the live double-`data` envelope.
 - Credentialed live JD product-detail request: passed through the company gateway for product `11266507445`; the adapter extracted the product id, title, and brand from the live double-`data` envelope.
-- Credentialed live JD review request: authenticated successfully and returned an empty first page with next-page metadata. A non-empty review sample is still required before review evidence can be accepted.
+- Credentialed live JD review requests: all 37 mapped competitors returned usable first-page evidence; 355 sanitized reviews were persisted without nickname, avatar, GUID, or other reviewer identity fields.
 - Live credentials are absent from the repository, fixtures, logs, and Git history. No supplier endpoint is contacted directly.
 - Live credentials are absent from the repository, fixtures, logs, and Git history. No supplier endpoint is contacted directly.
-- Parse REST development verification: 14/14 `Voc*` schemas present, master-key-only access confirmed, 2,817 own products, 37 competitor products with 37 details ready, 9,717 metrics, 40 relations, and 0 reviews.
+- Parse REST development verification: 14/14 `Voc*` schemas present, master-key-only access confirmed, 2,817 own products, 37 competitor products with 37 details ready, 9,717 metrics, 40 relations, and 355 competitor reviews.
 - End-to-end local API mode: `/health`, SaaS context, full snapshot, frontend proxy, desktop navigation, and 390x844 responsive rendering passed against the Parse REST store.
 - End-to-end local API mode: `/health`, SaaS context, full snapshot, frontend proxy, desktop navigation, and 390x844 responsive rendering passed against the Parse REST store.
 - PostgreSQL integration: pending a newly provisioned database. The local Docker CLI is installed but its engine was unavailable on 2026-07-23; no existing database was contacted.
 - PostgreSQL integration: pending a newly provisioned database. The local Docker CLI is installed but its engine was unavailable on 2026-07-23; no existing database was contacted.
 
 

+ 4 - 3
TASKS.md

@@ -47,11 +47,12 @@ Status date: 2026-07-24
 - [x] Confirm the catalog JD product-detail path and required `itemId` parameter.
 - [x] Confirm the catalog JD product-detail path and required `itemId` parameter.
 - [x] Confirm the catalog JD review path and optional `page` parameter.
 - [x] Confirm the catalog JD review path and optional `page` parameter.
 - [x] Confirm the live double-`data` response envelope and first-page review pagination metadata.
 - [x] Confirm the live double-`data` response envelope and first-page review pagination metadata.
-- [ ] Confirm whether an empty first review page with next-page metadata represents collection-in-progress, then capture one non-empty sanitized review fixture.
-- [ ] Save sanitized response fixtures without credentials or personal data.
+- [x] Confirm the live non-empty JD review contract and capture a sanitized fixture for `commentScore`, `commentDate`, and nested `data` arrays.
+- [x] Save sanitized product and review contract fixtures without credentials or personal data.
 - [x] Implement JD product and review adapters against company-gateway contract fixtures.
 - [x] Implement JD product and review adapters against company-gateway contract fixtures.
 - [x] Process queued jobs with partial-failure events and bounded retries.
 - [x] Process queued jobs with partial-failure events and bounded retries.
 - [x] Implement idempotent product/review UPSERT and truthful job progress updates.
 - [x] Implement idempotent product/review UPSERT and truthful job progress updates.
+- [x] Fetch and persist one bounded review page for all 37 mapped competitors (355 sanitized reviews).
 - [ ] Verify source UPSERT and snapshot totals against the provisioned PostgreSQL database.
 - [ ] Verify source UPSERT and snapshot totals against the provisioned PostgreSQL database.
 
 
 ## Phase 4 - deployable closure
 ## Phase 4 - deployable closure
@@ -64,7 +65,7 @@ Status date: 2026-07-24
 - [x] Add failed-job replay/cancel APIs and automatic stale-job recovery.
 - [x] Add failed-job replay/cancel APIs and automatic stale-job recovery.
 - [ ] Add database backup, restore, and retention procedures after infrastructure is provisioned.
 - [ ] Add database backup, restore, and retention procedures after infrastructure is provisioned.
 - [ ] Resolve or formally accept remaining moderate Parse transitive advisories.
 - [ ] Resolve or formally accept remaining moderate Parse transitive advisories.
-- [ ] Complete end-to-end acceptance with Demashi product detail and real review evidence.
+- [x] Complete local end-to-end acceptance with Demashi relations, competitor product detail, and real review evidence.
 
 
 ## Explicitly deferred
 ## Explicitly deferred
 
 

+ 11 - 1
scripts/sync-parse-rest-competitors.ts

@@ -9,8 +9,12 @@ const config = loadConfig();
 if (config.storageDriver !== 'parse_rest') throw new Error('STORAGE_DRIVER must be parse_rest');
 if (config.storageDriver !== 'parse_rest') throw new Error('STORAGE_DRIVER must be parse_rest');
 const workspaceId = process.argv[2] || config.auth.defaultWorkspaceId;
 const workspaceId = process.argv[2] || config.auth.defaultWorkspaceId;
 const refresh = process.argv.includes('--refresh');
 const refresh = process.argv.includes('--refresh');
+const includeReviews = process.argv.includes('--reviews');
+const refreshReviews = process.argv.includes('--refresh-reviews');
 const concurrencyArg = process.argv.find((value) => value.startsWith('--concurrency='));
 const concurrencyArg = process.argv.find((value) => value.startsWith('--concurrency='));
 const concurrency = Number(concurrencyArg?.split('=')[1] ?? 3);
 const concurrency = Number(concurrencyArg?.split('=')[1] ?? 3);
+const reviewPagesArg = process.argv.find((value) => value.startsWith('--review-pages='));
+const reviewPages = Number(reviewPagesArg?.split('=')[1] ?? 1);
 const client = new ParseRestClient({
 const client = new ParseRestClient({
   serverUrl: config.parse.serverUrl,
   serverUrl: config.parse.serverUrl,
   appId: config.parse.appId,
   appId: config.parse.appId,
@@ -25,8 +29,14 @@ const result = await service.sync({
   platform: 'jd',
   platform: 'jd',
   concurrency,
   concurrency,
   refresh,
   refresh,
+  includeReviews,
+  refreshReviews,
+  reviewPages,
   onProgress(progress) {
   onProgress(progress) {
-    console.log(`[competitor-sync] ${progress.completed}/${progress.total} ${progress.productId} ${progress.status}`);
+    const reviews = progress.reviewStatus === 'not_requested'
+      ? ''
+      : ` reviews=${progress.reviewStatus}:${progress.reviewCount}`;
+    console.log(`[competitor-sync] ${progress.completed}/${progress.total} ${progress.productId} detail=${progress.status}${reviews}`);
   },
   },
 });
 });
 await repository.appendAudit({
 await repository.appendAudit({

+ 44 - 0
src/modules/domestic-voc/adapters/jd-product.adapter.ts

@@ -1,7 +1,9 @@
 import { makeProductKey } from '../domain/identity.js';
 import { makeProductKey } from '../domain/identity.js';
+import type { DomesticProductProfile } from '../../../types/domestic-dataset.js';
 import {
 import {
   findBestRecord,
   findBestRecord,
   firstString,
   firstString,
+  isRecord,
   type JsonRecord,
   type JsonRecord,
   unwrapGatewayPayload,
   unwrapGatewayPayload,
 } from './jd-response.js';
 } from './jd-response.js';
@@ -28,6 +30,48 @@ const PRODUCT_KEYS = [
   'brandName', 'brand', 'model', 'categoryName', 'category1', 'category2', 'category3',
   'brandName', 'brand', 'model', 'categoryName', 'category1', 'category2', 'category3',
 ];
 ];
 
 
+export function adaptJdProductProfile(
+  payload: unknown,
+  collectedAt = '',
+): DomesticProductProfile | undefined {
+  if (!isRecord(payload)) return undefined;
+  const mainImages = Array.isArray(payload.mainImages)
+    ? payload.mainImages.filter((value): value is string => typeof value === 'string' && Boolean(value.trim()))
+    : [];
+  const fallbackImage = normalizeImageUrl(firstString(payload, ['imageurl', 'imageUrl', 'mainImage']));
+  const images = [...new Set(mainImages.map(normalizeImageUrl).filter(Boolean))];
+  if (!images.length && fallbackImage) images.push(fallbackImage);
+  return {
+    imageUrl: images[0] ?? '',
+    images,
+    color: firstString(payload, ['color', 'productColor']),
+    specification: firstString(payload, ['specName', 'spec', 'size', 'salesAttr']),
+    origin: firstString(payload, ['productArea', 'origin', 'productionPlace']),
+    weightKg: firstString(payload, ['weight']),
+    dimensionsMm: {
+      length: firstString(payload, ['length']),
+      width: firstString(payload, ['width']),
+      height: firstString(payload, ['height']),
+    },
+    saleDate: firstString(payload, ['saleDate', 'onlineDate']),
+    shopId: firstString(payload, ['shopId']),
+    sellerId: firstString(payload, ['venderId', 'sellerId']),
+    skuStatus: firstString(payload, ['skuStatus', 'status']),
+    jdExpress: payload.jdexpress === true,
+    localDelivery: payload.daojia === true,
+    sellPoint: firstString(payload, ['sellPoint', 'sellingPoint']),
+    collectedAt,
+  };
+}
+
+function normalizeImageUrl(value: string): string {
+  const image = value.trim();
+  if (!image) return '';
+  if (/^https?:\/\//i.test(image)) return image;
+  if (image.startsWith('//')) return `https:${image}`;
+  return `https://img30.360buyimg.com/sku/${image.replace(/^\/+/, '')}`;
+}
+
 export function adaptJdProductResponse(
 export function adaptJdProductResponse(
   response: unknown,
   response: unknown,
   requestedProductId: string,
   requestedProductId: string,

+ 16 - 4
src/modules/domestic-voc/adapters/jd-review.adapter.ts

@@ -23,8 +23,15 @@ export interface JdReviewPage {
   hasNextPage: boolean;
   hasNextPage: boolean;
 }
 }
 
 
-const REVIEW_ARRAY_KEYS = ['comments', 'commentList', 'reviews', 'reviewList', 'items', 'list'];
+const REVIEW_ARRAY_KEYS = ['comments', 'commentList', 'reviews', 'reviewList', 'items', 'list', 'data'];
 const REVIEW_CONTENT_KEYS = ['content', 'commentData', 'comment', 'text', 'description', 'reviewContent'];
 const REVIEW_CONTENT_KEYS = ['content', 'commentData', 'comment', 'text', 'description', 'reviewContent'];
+const SAFE_REVIEW_RAW_KEYS = [
+  'commentId', 'reviewId', 'id', 'commentIdStr',
+  ...REVIEW_CONTENT_KEYS,
+  'creationTime', 'createdAt', 'commentTime', 'commentDate', 'referenceTime', 'date', 'time',
+  'score', 'rating', 'star', 'starLevel', 'productScore', 'commentScore',
+  'productId', 'productSize', 'productColor', 'praiseCnt', 'buyCount', 'isPlus',
+];
 
 
 function findReviewRows(payload: unknown): JsonRecord[] {
 function findReviewRows(payload: unknown): JsonRecord[] {
   if (Array.isArray(payload)) return payload.filter(isRecord);
   if (Array.isArray(payload)) return payload.filter(isRecord);
@@ -68,7 +75,7 @@ export function adaptJdReviewResponse(
     if (!content) return [];
     if (!content) return [];
     const reviewId = firstString(row, ['commentId', 'reviewId', 'id', 'guid', 'commentIdStr']);
     const reviewId = firstString(row, ['commentId', 'reviewId', 'id', 'guid', 'commentIdStr']);
     const reviewDate = parseDate(firstValue(row, [
     const reviewDate = parseDate(firstValue(row, [
-      'creationTime', 'createdAt', 'commentTime', 'referenceTime', 'date', 'time',
+      'creationTime', 'createdAt', 'commentTime', 'commentDate', 'referenceTime', 'date', 'time',
     ]));
     ]));
     const reviewKey = makeReviewKey({
     const reviewKey = makeReviewKey({
       platform: 'jd',
       platform: 'jd',
@@ -80,13 +87,18 @@ export function adaptJdReviewResponse(
     return [{
     return [{
       productId,
       productId,
       reviewId: reviewId || reviewKey,
       reviewId: reviewId || reviewKey,
-      rating: normalizeRating(firstNumber(row, ['score', 'rating', 'star', 'starLevel', 'productScore'])),
+      rating: normalizeRating(firstNumber(row, ['score', 'rating', 'star', 'starLevel', 'productScore', 'commentScore'])),
       content,
       content,
       ...(reviewDate ? { reviewDate } : {}),
       ...(reviewDate ? { reviewDate } : {}),
       reviewKey,
       reviewKey,
-      rawPayload: row,
+      rawPayload: sanitizeReviewPayload(row),
     }];
     }];
   });
   });
   return { reviews, hasNextPage: detectNextPage(payload, currentPage) };
   return { reviews, hasNextPage: detectNextPage(payload, currentPage) };
 }
 }
 
 
+function sanitizeReviewPayload(row: JsonRecord): JsonRecord {
+  return Object.fromEntries(SAFE_REVIEW_RAW_KEYS
+    .filter((key) => row[key] !== undefined)
+    .map((key) => [key, row[key]]));
+}

+ 61 - 2
src/modules/domestic-voc/services/parse-rest-competitor-sync.service.ts

@@ -1,6 +1,7 @@
 import { ParseRestClient } from '../../../db/parse-rest.client.js';
 import { ParseRestClient } from '../../../db/parse-rest.client.js';
 import { VOC_PARSE_CLASSES } from '../../../db/parse-rest.schema.js';
 import { VOC_PARSE_CLASSES } from '../../../db/parse-rest.schema.js';
 import { adaptJdProductResponse, JD_PRODUCT_DETAIL_PATH } from '../adapters/jd-product.adapter.js';
 import { adaptJdProductResponse, JD_PRODUCT_DETAIL_PATH } from '../adapters/jd-product.adapter.js';
+import { adaptJdReviewResponse, JD_PRODUCT_COMMENTS_PATH } from '../adapters/jd-review.adapter.js';
 import type { GatewayRequestClient } from './jd-sync.service.js';
 import type { GatewayRequestClient } from './jd-sync.service.js';
 import { classifySyncFailure } from './jd-sync.service.js';
 import { classifySyncFailure } from './jd-sync.service.js';
 import { ParseRestVocRepository } from '../../saas-platform/parse-rest-voc.repository.js';
 import { ParseRestVocRepository } from '../../saas-platform/parse-rest-voc.repository.js';
@@ -24,6 +25,12 @@ export interface CompetitorSyncResult {
   skipped: number;
   skipped: number;
   failed: number;
   failed: number;
   failedProductIds: string[];
   failedProductIds: string[];
+  reviewProductsRequested: number;
+  reviewProductsSucceeded: number;
+  reviewProductsSkipped: number;
+  reviewProductsFailed: number;
+  reviewsUpserted: number;
+  failedReviewProductIds: string[];
 }
 }
 
 
 export class ParseRestCompetitorSyncService {
 export class ParseRestCompetitorSyncService {
@@ -38,7 +45,17 @@ export class ParseRestCompetitorSyncService {
     platform: 'jd';
     platform: 'jd';
     concurrency?: number;
     concurrency?: number;
     refresh?: boolean;
     refresh?: boolean;
-    onProgress?: (progress: { completed: number; total: number; productId: string; status: 'succeeded' | 'skipped' | 'failed' }) => void;
+    includeReviews?: boolean;
+    refreshReviews?: boolean;
+    reviewPages?: number;
+    onProgress?: (progress: {
+      completed: number;
+      total: number;
+      productId: string;
+      status: 'succeeded' | 'skipped' | 'failed';
+      reviewStatus: 'not_requested' | 'succeeded' | 'skipped' | 'failed';
+      reviewCount: number;
+    }) => void;
   }): Promise<CompetitorSyncResult> {
   }): Promise<CompetitorSyncResult> {
     const relations = await this.client.findAll<RelationObject>(VOC_PARSE_CLASSES.productRelation, {
     const relations = await this.client.findAll<RelationObject>(VOC_PARSE_CLASSES.productRelation, {
       workspaceId: input.workspaceId,
       workspaceId: input.workspaceId,
@@ -66,6 +83,12 @@ export class ParseRestCompetitorSyncService {
       skipped: 0,
       skipped: 0,
       failed: 0,
       failed: 0,
       failedProductIds: [],
       failedProductIds: [],
+      reviewProductsRequested: input.includeReviews ? targets.length : 0,
+      reviewProductsSucceeded: 0,
+      reviewProductsSkipped: 0,
+      reviewProductsFailed: 0,
+      reviewsUpserted: 0,
+      failedReviewProductIds: [],
     };
     };
     let cursor = 0;
     let cursor = 0;
     let completed = 0;
     let completed = 0;
@@ -76,6 +99,8 @@ export class ParseRestCompetitorSyncService {
         if (!target) return;
         if (!target) return;
         const productId = target.competitorProductId;
         const productId = target.competitorProductId;
         let status: 'succeeded' | 'skipped' | 'failed';
         let status: 'succeeded' | 'skipped' | 'failed';
+        let reviewStatus: 'not_requested' | 'succeeded' | 'skipped' | 'failed' = 'not_requested';
+        let reviewCount = 0;
         try {
         try {
           const existing = await this.client.findOne<StoredProduct>(VOC_PARSE_CLASSES.product, {
           const existing = await this.client.findOne<StoredProduct>(VOC_PARSE_CLASSES.product, {
             workspaceId: input.workspaceId,
             workspaceId: input.workspaceId,
@@ -100,8 +125,42 @@ export class ParseRestCompetitorSyncService {
           result.failedProductIds.push(productId);
           result.failedProductIds.push(productId);
           status = 'failed';
           status = 'failed';
         }
         }
+        if (input.includeReviews) {
+          try {
+            const existingReviewCount = await this.client.count(VOC_PARSE_CLASSES.review, {
+              workspaceId: input.workspaceId,
+              platform: input.platform,
+              productId,
+            });
+            if (!input.refreshReviews && existingReviewCount > 0) {
+              result.reviewProductsSkipped += 1;
+              reviewStatus = 'skipped';
+              reviewCount = existingReviewCount;
+            } else {
+              const reviews = [];
+              const maxPages = Math.max(1, Math.min(5, Math.floor(input.reviewPages ?? 1)));
+              for (let page = 1; page <= maxPages; page += 1) {
+                const response = await this.gateway.request<unknown>(JD_PRODUCT_COMMENTS_PATH, {
+                  params: { itemId: productId, page },
+                });
+                const adapted = adaptJdReviewResponse(response, productId, page);
+                reviews.push(...adapted.reviews);
+                if (!adapted.hasNextPage) break;
+              }
+              reviewCount = await this.repository.upsertReviews(input.workspaceId, input.platform, productId, reviews);
+              result.reviewProductsSucceeded += 1;
+              result.reviewsUpserted += reviewCount;
+              reviewStatus = 'succeeded';
+            }
+          } catch (error) {
+            classifySyncFailure(error);
+            result.reviewProductsFailed += 1;
+            result.failedReviewProductIds.push(productId);
+            reviewStatus = 'failed';
+          }
+        }
         completed += 1;
         completed += 1;
-        input.onProgress?.({ completed, total: targets.length, productId, status });
+        input.onProgress?.({ completed, total: targets.length, productId, status, reviewStatus, reviewCount });
       }
       }
     };
     };
     const concurrency = Math.max(1, Math.min(8, Math.floor(input.concurrency ?? 3)));
     const concurrency = Math.max(1, Math.min(8, Math.floor(input.concurrency ?? 3)));

+ 25 - 18
src/modules/domestic-voc/services/parse-rest-snapshot.service.ts

@@ -1,5 +1,6 @@
 import { ParseRestClient } from '../../../db/parse-rest.client.js';
 import { ParseRestClient } from '../../../db/parse-rest.client.js';
 import { VOC_PARSE_CLASSES } from '../../../db/parse-rest.schema.js';
 import { VOC_PARSE_CLASSES } from '../../../db/parse-rest.schema.js';
+import { adaptJdProductProfile } from '../adapters/jd-product.adapter.js';
 import type {
 import type {
   DomesticDataset,
   DomesticDataset,
   DomesticDailyMetric,
   DomesticDailyMetric,
@@ -16,6 +17,7 @@ interface WorkspaceObject {
 }
 }
 
 
 interface ProductObject {
 interface ProductObject {
+  updatedAt?: string;
   platform: string;
   platform: string;
   productId: string;
   productId: string;
   productKey: string;
   productKey: string;
@@ -30,6 +32,7 @@ interface ProductObject {
   relationCount: number;
   relationCount: number;
   summary: DomesticMetricSummary;
   summary: DomesticMetricSummary;
   trend: DomesticProduct['trend'];
   trend: DomesticProduct['trend'];
+  rawPayload?: Record<string, unknown> | null;
 }
 }
 
 
 interface RelationObject extends DomesticProductRelation {
 interface RelationObject extends DomesticProductRelation {
@@ -102,23 +105,27 @@ export class ParseRestSnapshotService {
       }),
       }),
     ]);
     ]);
 
 
-    const products: DomesticProduct[] = productObjects.map((product) => ({
-      platform: product.platform,
-      productId: product.productId,
-      productKey: product.productKey,
-      asin: product.productId,
-      role: product.role,
-      brand: product.brand,
-      title: product.title,
-      model: product.model,
-      category1: product.category1,
-      category2: product.category2,
-      category3: product.category3,
-      source: product.source,
-      relationCount: product.relationCount ?? 0,
-      summary: product.summary,
-      trend: product.trend ?? [],
-    }));
+    const products: DomesticProduct[] = productObjects.map((product) => {
+      const detail = adaptJdProductProfile(product.rawPayload, product.updatedAt ?? '');
+      return {
+        platform: product.platform,
+        productId: product.productId,
+        productKey: product.productKey,
+        asin: product.productId,
+        role: product.role,
+        brand: product.brand,
+        title: product.title,
+        model: product.model,
+        category1: product.category1,
+        category2: product.category2,
+        category3: product.category3,
+        source: product.source,
+        relationCount: product.relationCount ?? 0,
+        summary: product.summary,
+        trend: product.trend ?? [],
+        ...(detail ? { detail } : {}),
+      };
+    });
 
 
     const relations: DomesticProductRelation[] = relationObjects.map((relation) => ({
     const relations: DomesticProductRelation[] = relationObjects.map((relation) => ({
       relationKey: relation.relationKey,
       relationKey: relation.relationKey,
@@ -190,7 +197,7 @@ export class ParseRestSnapshotService {
         uniqueCompetitorProducts: competitorIds.size,
         uniqueCompetitorProducts: competitorIds.size,
         category2Count: new Set(ownProducts.map((product) => product.category2).filter(Boolean)).size,
         category2Count: new Set(ownProducts.map((product) => product.category2).filter(Boolean)).size,
         category3Count: new Set(ownProducts.map((product) => product.category3).filter(Boolean)).size,
         category3Count: new Set(ownProducts.map((product) => product.category3).filter(Boolean)).size,
-        reviewCount: batch?.reviewRows ?? reviews.length,
+        reviewCount: reviews.length,
       },
       },
       dailyTotals: batch?.dailyTotals ?? [],
       dailyTotals: batch?.dailyTotals ?? [],
       products,
       products,

+ 5 - 1
src/modules/saas-platform/parse-rest-voc.repository.ts

@@ -3,7 +3,7 @@ import { ApiError } from '../../http/api-error.js';
 import { ParseRestClient, parseDate, parseDateIso } from '../../db/parse-rest.client.js';
 import { ParseRestClient, parseDate, parseDateIso } from '../../db/parse-rest.client.js';
 import { isVocSchemaReady, VOC_PARSE_CLASSES } from '../../db/parse-rest.schema.js';
 import { isVocSchemaReady, VOC_PARSE_CLASSES } from '../../db/parse-rest.schema.js';
 import type { DomesticDataset, DomesticProduct, DomesticReview } from '../../types/domestic-dataset.js';
 import type { DomesticDataset, DomesticProduct, DomesticReview } from '../../types/domestic-dataset.js';
-import type { JdProductRecord } from '../domestic-voc/adapters/jd-product.adapter.js';
+import { adaptJdProductProfile, type JdProductRecord } from '../domestic-voc/adapters/jd-product.adapter.js';
 import type { JdReviewRecord } from '../domestic-voc/adapters/jd-review.adapter.js';
 import type { JdReviewRecord } from '../domestic-voc/adapters/jd-review.adapter.js';
 import type {
 import type {
   EnqueueSyncJobInput,
   EnqueueSyncJobInput,
@@ -54,6 +54,8 @@ interface MemberObject {
 interface ProductObject extends Omit<DomesticProduct, 'asin'> {
 interface ProductObject extends Omit<DomesticProduct, 'asin'> {
   naturalKey: string;
   naturalKey: string;
   workspaceId: string;
   workspaceId: string;
+  updatedAt?: string;
+  rawPayload?: Record<string, unknown> | null;
 }
 }
 
 
 interface ReviewObject {
 interface ReviewObject {
@@ -837,6 +839,7 @@ export class ParseRestVocRepository implements PlatformRepository, SyncJobStore,
   }
   }
 
 
   private mapProduct(product: ProductObject): DomesticProduct {
   private mapProduct(product: ProductObject): DomesticProduct {
+    const detail = adaptJdProductProfile(product.rawPayload, product.updatedAt ?? '');
     return {
     return {
       platform: product.platform,
       platform: product.platform,
       productId: product.productId,
       productId: product.productId,
@@ -853,6 +856,7 @@ export class ParseRestVocRepository implements PlatformRepository, SyncJobStore,
       relationCount: product.relationCount ?? 0,
       relationCount: product.relationCount ?? 0,
       summary: product.summary,
       summary: product.summary,
       trend: product.trend ?? [],
       trend: product.trend ?? [],
+      ...(detail ? { detail } : {}),
     };
     };
   }
   }
 
 

+ 23 - 1
src/types/domestic-dataset.ts

@@ -24,6 +24,28 @@ export interface DomesticDailyMetric extends DomesticMetricSummary {
   date: string;
   date: string;
 }
 }
 
 
+export interface DomesticProductProfile {
+  imageUrl: string;
+  images: string[];
+  color: string;
+  specification: string;
+  origin: string;
+  weightKg: string;
+  dimensionsMm: {
+    length: string;
+    width: string;
+    height: string;
+  };
+  saleDate: string;
+  shopId: string;
+  sellerId: string;
+  skuStatus: string;
+  jdExpress: boolean;
+  localDelivery: boolean;
+  sellPoint: string;
+  collectedAt: string;
+}
+
 export interface DomesticProduct {
 export interface DomesticProduct {
   platform: string;
   platform: string;
   productId: string;
   productId: string;
@@ -40,6 +62,7 @@ export interface DomesticProduct {
   relationCount: number;
   relationCount: number;
   summary: DomesticMetricSummary;
   summary: DomesticMetricSummary;
   trend: Array<{ date: string; gmv: number; soldUnits: number; transactionOrders: number }>;
   trend: Array<{ date: string; gmv: number; soldUnits: number; transactionOrders: number }>;
+  detail?: DomesticProductProfile;
 }
 }
 
 
 export interface DomesticReview {
 export interface DomesticReview {
@@ -100,4 +123,3 @@ export interface DomesticDataset {
     brandWithoutProductId: Array<{ ownProductId: string; slot: number; brand: string }>;
     brandWithoutProductId: Array<{ ownProductId: string; slot: number; brand: string }>;
   };
   };
 }
 }
-

+ 42 - 1
test/jd-adapters.test.ts

@@ -1,6 +1,6 @@
 import assert from 'node:assert/strict';
 import assert from 'node:assert/strict';
 import test from 'node:test';
 import test from 'node:test';
-import { adaptJdProductResponse } from '../src/modules/domestic-voc/adapters/jd-product.adapter.js';
+import { adaptJdProductProfile, adaptJdProductResponse } from '../src/modules/domestic-voc/adapters/jd-product.adapter.js';
 import { GatewayPayloadError } from '../src/modules/domestic-voc/adapters/jd-response.js';
 import { GatewayPayloadError } from '../src/modules/domestic-voc/adapters/jd-response.js';
 import { adaptJdReviewResponse } from '../src/modules/domestic-voc/adapters/jd-review.adapter.js';
 import { adaptJdReviewResponse } from '../src/modules/domestic-voc/adapters/jd-review.adapter.js';
 
 
@@ -35,6 +35,27 @@ test('JD product adapter preserves an explicit competitor role', () => {
   assert.equal(product.role, 'competitor');
   assert.equal(product.role, 'competitor');
 });
 });
 
 
+test('JD product profile exposes normalized catalog fields without the raw response', () => {
+  const profile = adaptJdProductProfile({
+    mainImages: ['https://img.example/item.jpg'],
+    color: 'Black',
+    productArea: 'China',
+    weight: '12.5',
+    length: '300',
+    width: '200',
+    height: '100',
+    saleDate: '2026-07-01 10:00:00',
+    shopId: 'shop-1',
+    venderId: 'seller-1',
+    jdexpress: true,
+  }, '2026-07-24T00:00:00.000Z');
+  assert.equal(profile?.imageUrl, 'https://img.example/item.jpg');
+  assert.equal(profile?.origin, 'China');
+  assert.equal(profile?.dimensionsMm.length, '300');
+  assert.equal(profile?.jdExpress, true);
+  assert.equal(profile?.collectedAt, '2026-07-24T00:00:00.000Z');
+});
+
 test('JD review adapter maps evidence and pagination without reviewer identity', () => {
 test('JD review adapter maps evidence and pagination without reviewer identity', () => {
   const page = adaptJdReviewResponse({
   const page = adaptJdReviewResponse({
     code: 200,
     code: 200,
@@ -59,6 +80,26 @@ test('JD review adapter maps evidence and pagination without reviewer identity',
   assert.equal(page.reviews[0]?.reviewDate, '2026-07-20T02:00:00.000Z');
   assert.equal(page.reviews[0]?.reviewDate, '2026-07-20T02:00:00.000Z');
   assert.match(page.reviews[0]?.reviewKey ?? '', /^jd:11266507445:/);
   assert.match(page.reviews[0]?.reviewKey ?? '', /^jd:11266507445:/);
   assert.equal(page.hasNextPage, true);
   assert.equal(page.hasNextPage, true);
+  assert.equal('nickname' in page.reviews[0]!.rawPayload, false);
+});
+
+test('JD review adapter maps the live commentScore and commentDate fields and removes identity fields', () => {
+  const page = adaptJdReviewResponse({
+    data: { data: { data: [{
+      commentId: 'live-comment-1',
+      commentData: 'Heating is fast.',
+      commentScore: '4',
+      commentDate: '2026-07-21 08:30:00',
+      userNickName: 'private-name',
+      userImgURL: 'https://private.example/avatar.jpg',
+      guid: 'private-guid',
+    }] } },
+  }, '100095574033', 1);
+  assert.equal(page.reviews[0]?.rating, 4);
+  assert.equal(page.reviews[0]?.reviewDate, '2026-07-21T00:30:00.000Z');
+  assert.equal('userNickName' in page.reviews[0]!.rawPayload, false);
+  assert.equal('userImgURL' in page.reviews[0]!.rawPayload, false);
+  assert.equal('guid' in page.reviews[0]!.rawPayload, false);
 });
 });
 
 
 test('JD adapters classify pending collection as retryable without raw body leakage', () => {
 test('JD adapters classify pending collection as retryable without raw body leakage', () => {

+ 26 - 1
test/parse-rest-competitor-sync.test.ts

@@ -8,6 +8,7 @@ import type { ParseRestVocRepository } from '../src/modules/saas-platform/parse-
 test('competitor sync creates mapped placeholders and persists returned products as competitors', async () => {
 test('competitor sync creates mapped placeholders and persists returned products as competitors', async () => {
   const placeholders: string[] = [];
   const placeholders: string[] = [];
   const products: Array<{ productId: string; role: string }> = [];
   const products: Array<{ productId: string; role: string }> = [];
+  const reviewProducts: Array<{ productId: string; count: number }> = [];
   const client = {
   const client = {
     async findAll() {
     async findAll() {
       return [
       return [
@@ -17,16 +18,29 @@ test('competitor sync creates mapped placeholders and persists returned products
       ];
       ];
     },
     },
     async findOne() { return null; },
     async findOne() { return null; },
+    async count() { return 0; },
   } as unknown as ParseRestClient;
   } as unknown as ParseRestClient;
   const repository = {
   const repository = {
     async ensureCompetitorProduct(input: { productId: string }) { placeholders.push(input.productId); },
     async ensureCompetitorProduct(input: { productId: string }) { placeholders.push(input.productId); },
     async upsertProduct(_workspaceId: string, product: { productId: string; role: string }) {
     async upsertProduct(_workspaceId: string, product: { productId: string; role: string }) {
       products.push({ productId: product.productId, role: product.role });
       products.push({ productId: product.productId, role: product.role });
     },
     },
+    async upsertReviews(_workspaceId: string, _platform: string, productId: string, reviews: unknown[]) {
+      reviewProducts.push({ productId, count: reviews.length });
+      return reviews.length;
+    },
   } as unknown as ParseRestVocRepository;
   } as unknown as ParseRestVocRepository;
   const gateway: GatewayRequestClient = {
   const gateway: GatewayRequestClient = {
-    async request(_path, init) {
+    async request(path, init) {
       const productId = String(init?.params?.itemId);
       const productId = String(init?.params?.itemId);
+      if (path.includes('comments')) {
+        return { data: { data: { data: [{
+          commentId: `review-${productId}`,
+          commentData: `Review ${productId}`,
+          commentScore: '5',
+          commentDate: '2026-07-24 10:00:00',
+        }], pageInfo: { maxPage: 1, nowPage: 1 } } } } as never;
+      }
       return { code: 200, data: { code: 200, data: { item: { itemId: productId, itemName: `Product ${productId}` } } } } as never;
       return { code: 200, data: { code: 200, data: { item: { itemId: productId, itemName: `Product ${productId}` } } } } as never;
     },
     },
   };
   };
@@ -35,6 +49,7 @@ test('competitor sync creates mapped placeholders and persists returned products
     workspaceId: 'demashi',
     workspaceId: 'demashi',
     platform: 'jd',
     platform: 'jd',
     concurrency: 2,
     concurrency: 2,
+    includeReviews: true,
   });
   });
 
 
   assert.deepEqual(placeholders.sort(), ['c1', 'c2']);
   assert.deepEqual(placeholders.sort(), ['c1', 'c2']);
@@ -42,6 +57,10 @@ test('competitor sync creates mapped placeholders and persists returned products
     { productId: 'c1', role: 'competitor' },
     { productId: 'c1', role: 'competitor' },
     { productId: 'c2', role: 'competitor' },
     { productId: 'c2', role: 'competitor' },
   ]);
   ]);
+  assert.deepEqual(reviewProducts.sort((left, right) => left.productId.localeCompare(right.productId)), [
+    { productId: 'c1', count: 1 },
+    { productId: 'c2', count: 1 },
+  ]);
   assert.deepEqual(result, {
   assert.deepEqual(result, {
     total: 2,
     total: 2,
     placeholders: 2,
     placeholders: 2,
@@ -49,5 +68,11 @@ test('competitor sync creates mapped placeholders and persists returned products
     skipped: 0,
     skipped: 0,
     failed: 0,
     failed: 0,
     failedProductIds: [],
     failedProductIds: [],
+    reviewProductsRequested: 2,
+    reviewProductsSucceeded: 2,
+    reviewProductsSkipped: 0,
+    reviewProductsFailed: 0,
+    reviewsUpserted: 2,
+    failedReviewProductIds: [],
   });
   });
 });
 });