TL;DR (aka future GT, read this first next time): I needed structured product data from 50+ D2C coffee websites. I don't know how to code. Every scraping tool I tried either cost too much or fell apart on nested product variants. The answer was right there in plain sight — Shopify and WooCommerce expose public JSON APIs that most people don't know about. The real breakthrough wasn't scraping. It was using an LLM to normalize the mess when every store returns different data shapes. I built the whole thing on n8n first, then ported it to Python with Claude and Cursor. Now it runs 92 roasters and 1,400 coffee products without me touching it. I wrote almost none of the code myself. That's the point.
I wanted a table.
Roasters. Coffees. Prices. That's it. One table with all the Indian specialty coffee that exists, in one place, structured so I could search it, filter it, and build a website around it.
Simple ask, right?
The problem: 50+ roasters, each with their own website, their own tech stack, their own way of describing a bag of beans. Some on Shopify. Some on WooCommerce. A few on Wix, GoDaddy, or custom builds that looked like they were designed in 2008 and never touched again.
I'm not a coder. I'm a marketing guy who learned just enough to be dangerous. I know what a JSON looks like but I can't write one from scratch. I know what an API is but I'd never called one deliberately.
So I did what any sane person would do. I went to Reddit.
The Tool Graveyard
The scraping subreddits have opinions. Strong ones. Everyone has the tool that will change your life.
I tried Firecrawl first. Beautiful product. Hit a URL, get structured markdown. Worked great on simple pages. Coffee roasters don't have simple pages. They have nested product variants, dynamic pricing, size options, and a general disdain for consistency. Firecrawl gave me back beautiful markdown that was missing half the data.
I tried Crawl4AI. Powerful. Open source. Deeply configurable. Felt like driving a tank to buy milk. I spent more time reading documentation than actually scraping data.
I tried parsing JSON-LD from page source. Every store embeds schema markup. Every store formats it differently. The same product from two roasters would have completely different field names, nesting structures, and occasionally a typo that broke everything.
I spent two weeks building if-else chains. If the store is Shopify, look here. If it's WooCommerce, look there. If the price is in this field, use it. If it's in that field, transform it.
It was brittle. Every new roaster broke something. I was maintaining a house of cards, and I hadn't even launched the site yet.
The Moment the Lights Turned On
I opened DevTools on a Shopify roaster's product page. Network tab. Refresh. Watched the requests fly by.
There it was. /products.json.
I clicked it. Opened in a new tab. A clean JSON array of every product on the store. Names, prices, variants, images, descriptions. Everything I needed. Public. Free. No API key. No rate limit drama.
Wait. Every Shopify store has this?
Yes. Every single one.
WooCommerce has its own pattern: /wp-json/wc/v3/products. A bit more auth involved, but the data is the same shape. Wix stores need a little more creativity, but the pattern holds.
The real skill wasn't scraping. It was knowing which URL to call. A three-line function per platform. Everything else is just plumbing.
I felt like an idiot. A relieved idiot, but an idiot.
The Part I'm Actually Proud Of
The data from different stores doesn't match. One roaster calls it "Blue Tokai Silver Oak." Another calls it "Silver Oak — 250g." A third has it as a variant inside a parent product.
I spent two weeks trying to write rules for this. Extract the name. Normalize it. Handle the edge cases.
It's impossible. There are too many edge cases.
So I gave the raw JSON to an LLM with one instruction: "Turn this into my schema."
That was the breakthrough. The LLM doesn't scrape. It translates. Takes whatever shape the store returns and maps it to the fields my database expects. The same prompt works for Shopify, WooCommerce, Wix — doesn't matter. The LLM figures out the mapping.
This is why the pipeline is maintenance-free. New roaster? The LLM adapts. New way of describing a roast level? The LLM figures it out. I haven't touched the mapping logic in months.
The No-Code to Some-Code Pipeline
First version was on n8n. Visual workflow builder. Drag, connect, configure. It worked. I could see the data flowing from store to database. It was satisfying in a way I didn't expect.
n8n is great for prototyping. Terrible for version control. One wrong click and your flow is different and you don't know why.
So I ported it to Python. I didn't write the Python. I sat in Cursor and told Claude what I needed. "Take this n8n flow and make it a Python script." "Add error handling." "Send a Slack notification when a new roaster is detected."
I didn't write the scraper. I convinced an AI to write it for me. Multiple times. Each version got cleaner.
The irony isn't lost on me. A non-coder built a production data pipeline by explaining what he wanted to six different AI tools across three platforms. The script runs every night. It classifies products, updates prices, deactivates discontinued items. It handles Shopify, WooCommerce, Wix, GoDaddy. It uploads images to ImageKit. It sends Slack messages when something interesting happens.
And I haven't touched the code in months.
Where We Are Now
92 roasters. 1,400+ products. All of them flowing through one pipeline I almost gave up on three times before it worked.
The scraper is the unattended intake. New products get discovered, classified, and inserted without me doing anything. When a roaster adds a new coffee, it shows up on ICB. Not because I manually added it. Because the pipeline caught it.
The data moat that nobody asked for, that I built by accident while trying to make a simple table — that's the foundation everything else sits on.
But here's the thing. The scraper was just the intake. Once the data started flowing, I had a new problem: how do you turn raw product data into something people can actually discover? That's when the SEO machine showed up.
[Next post: The bi-weekly SEO agent that doesn't sleep]