Content enrichment

A catalogue grows by thousands of supplier products a night, each with a thin title and missing attributes. Millstream imports the feed, skips products that have not changed, has an agent fill in category, attributes and descriptions, checks the result against the taxonomy in code, and sends the uncertain products to merchandisers.

The pipeline

flowchart LR
  T(["nightly, 02:00"]) --> I[import feed] --> D{{"dedupe: unchanged"}} --> E["enrich (agent)"]
  E --> C[check taxonomy] --> R[/merchandiser review/] --> P[update catalogue]

pipeline: catalog-enrichment
description: Enrich supplier products before they reach the catalogue

vars: { minConfidence: 0.85 }

triggers:
  nightly: { cron: "0 2 * * *", tz: Europe/Paris }

steps:
  import:
    action: lab/feed-import@1
    in: trigger.nightly
    out: products.raw
  unchanged:
    action: dedupe
    in: products.raw
    out: products.changed
    config: { key: "item.sku + ':' + item.contentHash", window: 365d }
  enrich:
    action: lab/enrich-product@1
    in: products.changed
    out: products.enriched
    concurrency: 8
    config: { locales: [en, fr] }
  check:
    action: ./actions/check-taxonomy
    in: products.enriched
    out: products.checked
  merch-review:
    action: review
    in: products.checked
    out: products.approved
    config:
      inbox: merchandising
      timeout: 5d
      onTimeout: { escalate: merchandising-leads }
      autoApprove: "item.confidence >= vars.minConfidence && size(item.issues) == 0"
      sampleRate: 0.05
      editable: [/title, /category, /attributes, /descriptions]
      reasons: [wrongCategory, wrongAttribute, inventedClaim, tone]
  publish:
    action: lab/catalog-update@1
    in: products.approved

budget: { runUsd: 60, dayUsd: 80 }

What Millstream does for it

Only changed products cost anything

• dedupe keys each product by SKU and content hash, so a product the feed resends unchanged is filtered before the agent runs.
• Each run's cost is a metric, and runUsd and dayUsd hold back agent deliveries once spent.

The agent and the rules

• enrich reads the taxonomy from the package's assets/, mounted read-only for every action of the pipeline, and answers the action's output schema.
• check-taxonomy is ordinary code: the category exists, every attribute is allowed for it, values fall in their enums.
• A change to the taxonomy file is a new revision of the package; millstream replay --step enrich --since 30d --where "item.supplierCategory == 'footwear'" reruns the affected products.

Merchandisers see what matters

• Confident products with no issues are approved by rule. 5% of them still reach a person as review samples, so the rule's agreement is measured.
• Everything else waits in the merchandising inbox, where titles, attributes and descriptions can be edited before approval.
• A rejection needs a reason code, and inventedClaim makes a hallucinated product feature countable.

How it learns

When merchandisers keep correcting the same attribute, the edits are attributed to enrich and become a lessons proposal: a rule and corrected examples, cited by the review they came from. When people approve nearly everything without an edit (98% of at least 20 decisions), the reviewPattern insight hints a config proposal that adjusts autoApprove. Every proposal is backtested against past items, and a person accepts it before it becomes a revision.

Figures to watch

Next

Pipelines · Human review · Observability