697,299 products, 397 stores, ~$31 of LLM spend
How I built a production e-commerce market dataset in about a week, solo and mostly unattended: a five-stage pipeline that scrapes European Shopify catalogs, enriches ~110K products with LLM-judged attributes via the Anthropic Batch API, then mines those answers for +72K more labels at zero cost and 97.1% measured precision. The interesting part isn't the architecture — it's the failures each stage now absorbs.
About this project. Built for my analytics startup's market-data product, so merchants can ask "what do stores like mine charge for leather bags?" and get a statistically honest answer. The generalized, reusable engine — scraper, enrichment, audit protocol, and all 18 documented production lessons — is extracted into a public, MIT-licensed repo: Super-scrap-llm-engine. Every rule in it was paid for at least once.
A benchmark nobody could act on
The question my market-data product needed to answer sounds simple: "what do stores like mine charge for a leather bag?" Raw catalog data gives you an answer — and it's useless. Pooling couture and high-street into one number produced "median leather bag: €640", a statistic that describes no store a merchant would ever compare themselves against.
Making that number honest requires attributes the raw data doesn't carry: what a product is actually made of, what colour it actually is, which price tier the store plays in. At 697K products, no solo operator can label that by hand — and I had no data team, no labelling budget, and a laptop. The constraint set was the brief: build a dataset dense and clean enough for tiered benchmarks, at approximately zero cost.
The one-line mandate
Turn one meaningless median into segment-honest benchmarks — solo, on a laptop, for the price of a lunch.
Five deliberately boring stages
Each stage is resumable, each validates its input, and each carries provenance — every enriched value is tagged llm or inferred, forever separable and revertible. The design isn't clever; it's scar tissue. Every guarantee below exists because the naive version failed first.
Scrape
Public /products.json per store — paced, resumable, and typed so throttling can never be recorded as a dead store.
Chunk
Round-robin products across stores into small work files, so one giant store can't starve the rest of the corpus.
Enrich
Anthropic Batch API at 50% off list price — 50-row requests, thinking explicitly disabled, every response validated by key set, never by line count.
Merge
Answers join back by key — never by position — with a membership guard so a stale or foreign answer can't land on the wrong product.
Propagate
Mine the LLM answers for rules (brand×type groups, name tokens), score them on a held-out split first, apply only what clears the precision bar — then audit by reading samples against source.
The economics of stage 5 are the trick: enrich a stratified ~25% of the corpus with the LLM, then let free, precision-gated rules label the lookalikes. That single $0 step added +72K labels — more than the paid run itself. And one negative result was worth as much: colour does not propagate (neighbours predict ~3% of it) — measured in 20 minutes, saving days of wishful engineering.
Every rule was paid for once
Eighteen production lessons are documented in the repo's LESSONS.md, numbered and cited throughout the code. These four cost the most and taught the most.
-
01
The first full run was a lie
The scraper "completed" with 434 of 504 stores failed — almost all fake. Shopify throttles by IP, and HTTP 429s were being written down as "not a Shopify store": a transport failure recorded as a business verdict. The tell was temporal — detection worked for the first 50 stores, then collapsed to ~0%. The fix wasn't retry logic; it was a type system where
RateLimitedcan never reach the failed state. With pacing added and a redundant probe request removed (half of all traffic!), the re-run had zero throttle events. -
02
The machine was the bottleneck, not the API
Overnight throughput collapsed from 38 stores/hr to 5/hr — looking exactly like cumulative rate-limiting. It was the Mac going to sleep. One
caffeinate -irestored 100+/hr. I lost hours misdiagnosing the wrong layer of the stack; it's now a startup warning in the CLI, because the cheapest fix is the one the tool tells you about before you run. -
03
Sonnet thinks by default — and burns your budget doing it
Omit the
thinkingparameter and Claude Sonnet runs adaptive reasoning. For bulk classification that means it will happily spend the entire token budget deliberating and emit zero answers. Great default for chat; silent killer for batch pipelines. Disabling it explicitly is now encoded in the enrichment code, not in anyone's memory. -
04
Big requests silently drop rows
At 250 products per request, the model lost 1:1 row correspondence ~40% of the time — not truncation, just missing rows with a clean
end_turn. At 50 rows per request the problem vanished. The durable lesson: validate by key set, never by line count — roughly a third of the ~$31 bill was paying twice for this one.
Three ways to label 110K products — two of them wrong
The Batch API runs at half list price, ~250 products cost about $0.07 to enrich, and precision-gated propagation costs nothing at all. Stack those three and the cost comparison stops being close.
| Approach | Cost | Time |
|---|---|---|
| Manual data team labelling | ~$5–15K | Weeks |
| Naive per-request LLM calls (thinking on, no batch) | ~$300–500 | Days + babysitting |
| This pipeline (batch + propagation) | ~$31 | ~1 week, mostly unattended |
Quality wasn't the trade-off: keyword rules for colour measured ~21% wrong (they grabbed upsells, trims, and "black friday" tags), while the audited LLM ran at ~94% precision — verified by reading 50 samples against source text, per the repo's audit protocol.
The benchmark, before and after
The same question — "what does a leather bag cost?" — asked of the same corpus, before and after enrichment:
Before — raw data
"Median leather bag: €640"
One number across couture and high-street. Describes no store a merchant would actually compare against — technically correct, commercially meaningless.
After — enriched & tiered
- Mid-tier €82 (n=202)
- Premium €197 (n=116)
- Luxury €440 (n=896)
Segment-honest numbers with sample sizes attached — a benchmark a merchant can actually price against.
Why the numbers can be trusted
Every label carries provenance (llm vs. inferred), propagation was precision-scored on a held-out split before a single row was touched (97.1%), and systematic model errors — brand priors, undocumented inference chains — were found by reading 50 samples against source and turned into explicit prompt rules. Audit samples, not aggregates: no dashboard would have caught them.
Topics covered
Read next
MPB — Warehouse Discovery & 3 Shipped AI Agents
An 11-day product discovery that named an invisible throughput constraint, framed four strategic bets, and shipped three working AI-agent prototypes.
AI · LogisticsFreight Intel — AI Margin Intelligence
Translating macro global events into SKU-level margin decisions in minutes. 3–7% margin recovery, 50+ data points automated, time-to-action cut from days to minutes.
Want this data working for you?
The full engine — scraper, enrichment, audit protocol, all 18 lessons — is open source and MIT licensed. Or skip the build entirely: I run deep store data analyses for merchants who want the benchmarks, not the pipeline.