Data Scraper

# data-scraper

Web Data Scraper — Extract structured data from web pages using curl + parsing. Lightweight, no browser required. Supports HTML-to-text, table extraction, price monitoring, and batch scraping.

When to Use

• Extract text content from web pages (articles, blogs, docs)
• Scrape product prices, reviews, or listings
• Monitor pages for changes (price drops, new content)
• Batch-collect data from multiple URLs
• Convert HTML tables to structured formats (JSON/CSV)

Quick Start

```bash # Extract readable text from URL data-scraper fetch "https://example.com/article"

# Extract specific elements data-scraper extract "https://example.com" --selector "h2, .price"

# Monitor for changes data-scraper watch "https://example.com/product" --interval 3600 ```

Extraction Modes

Text Mode (default) Fetches page and extracts readable content, stripping HTML tags, scripts, and styles. Similar to reader mode.

```bash data-scraper fetch URL # Output: clean markdown text ```

Selector Mode Target specific CSS selectors for precise extraction.

```bash data-scraper extract URL --selector ".product-title, .price, .rating" # Output: matched elements as structured data ```

Table Mode Extract HTML tables into structured formats.

```bash data-scraper table URL --index 0 # Output: JSON array of row objects (header → value mapping) ```

Link Mode Extract all links from a page with optional filtering.

```bash data-scraper links URL --filter "*.pdf" # Output: filtered list of absolute URLs ```

Batch Scraping

```bash # Scrape multiple URLs data-scraper batch urls.txt --output results/

# With rate limiting data-scraper batch urls.txt --delay 2000 --output results/ ```

`urls.txt` format: ``` https://site1.com/page https://site2.com/page https://site3.com/page ```

Change Monitoring

```bash # Watch for changes, alert on diff data-scraper watch URL --selector ".price" --interval 3600

# Compare with previous snapshot data-scraper diff URL ```

Stores snapshots in `data-scraper/snapshots/` with timestamps. Alerts via notification-hub when changes detected.

Output Formats

| Format | Flag | Use Case | |--------|------|----------| | Text | `--format text` | Reading, summarization | | JSON | `--format json` | Data processing | | CSV | `--format csv` | Spreadsheets | | Markdown | `--format md` | Documentation |

Headers & Auth

```bash # Custom headers data-scraper fetch URL --header "Authorization: Bearer TOKEN"

# Cookie-based auth data-scraper fetch URL --cookie "session=abc123"

# User-Agent override data-scraper fetch URL --ua "Mozilla/5.0..." ```

Rate Limiting & Ethics

• Default: 1 request per second per domain
• Respects `robots.txt` when `--polite` flag is set
• Configurable delay between requests
• Stops on 429 (Too Many Requests) and backs off

Error Handling

| Error | Behavior | |-------|----------| | 404 | Log and skip | | 403/401 | Warn about auth requirement | | 429 | Exponential backoff (max 3 retries) | | Timeout | Retry once with longer timeout | | SSL error | Warn, option to proceed with `--insecure` |

Integration

• web-claude: Use as fallback when web_fetch isn't enough
• competitor-watch: Feed scraped data into competitor analysis
• seo-audit: Scrape competitor pages for SEO comparison
• performance-tracker: Collect social metrics from public profiles