Web Scraper as a Service

# Web Scraper as a Service

Turn scraping briefs into deliverable scraping projects. Generates the scraper, runs it, cleans the data, and packages everything for the client.

How to Use

``` /web-scraper-as-a-service "Scrape all products from example-store.com — need name, price, description, images. CSV output." /web-scraper-as-a-service https://example.com --fields "title,price,rating,url" --format csv /web-scraper-as-a-service brief.txt ```

Scraper Generation Pipeline

Step 1: Analyze the Target

Before writing any code:

1. Fetch the target URL to understand the page structure
2. Identify:
3. - Is the site server-rendered (static HTML) or client-rendered (JavaScript/SPA)?
4. - What anti-scraping measures are visible? (Cloudflare, CAPTCHAs, rate limits)
5. - Pagination pattern (URL params, infinite scroll, load more button)
6. - Data structure (product cards, table rows, list items)
7. - Total estimated volume (number of pages/items)
8. Choose the right tool:
9. - Static HTML → Python + `requests` + `BeautifulSoup`
10. - JavaScript-rendered → Python + `playwright`
11. - API available → Direct API calls (check network tab patterns)

Step 2: Build the Scraper

Generate a complete Python script in `scraper/` directory:

``` scraper/ scrape.py # Main scraper script requirements.txt # Dependencies config.json # Target URLs, fields, settings README.md # Setup and usage instructions for client ```

`scrape.py` must include:

```python # Required features in every scraper:

# 1. Configuration import json config = json.load(open('config.json'))

# 2. Rate limiting (ALWAYS — be respectful) import time DELAY_BETWEEN_REQUESTS = 2 # seconds, adjustable in config

# 3. Retry logic MAX_RETRIES = 3 RETRY_DELAY = 5

# 4. User-Agent rotation USER_AGENTS = [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...", # ... at least 5 user agents ]

# 5. Progress tracking print(f"Scraping page {current}/{total} — {items_collected} items collected")

# 6. Error handling # - Log errors but don't crash on individual page failures # - Save progress incrementally (don't lose data on crash) # - Write errors to error_log.txt

# 7. Output # - Save data incrementally (append to file, don't hold in memory) # - Support CSV and JSON output # - Clean and normalize data before saving

# 8. Resume capability # - Track last successfully scraped page/URL # - Can resume from where it left off if interrupted ```

Step 3: Data Cleaning

After scraping, clean the data:

1. Remove duplicates (by unique identifier or composite key)
2. Normalize text (strip extra whitespace, fix encoding issues, consistent capitalization)
3. Validate data (no empty required fields, prices are numbers, URLs are valid)
4. Standardize formats (dates to ISO 8601, currency to numbers, consistent units)
5. Generate data quality report:
6. ```
7. Data Quality Report
8. ───────────────────
9. Total records: 2,487
10. Duplicates removed: 13
11. Empty fields filled: 0
12. Fields with issues: price (3 records had non-numeric values — cleaned)
13. Completeness: 99.5%
14. ```

Step 4: Client Deliverable Package

Generate a complete deliverable:

``` delivery/ data.csv # Clean data in requested format data.json # JSON alternative data-quality-report.md # Quality metrics scraper-documentation.md # How the scraper works README.md # Quick start guide ```

• `scraper-documentation.md` includes:
• What was scraped and from where
• How many records collected
• Data fields and their descriptions
• How to re-run the scraper
• Known limitations
• Date of scraping

Step 5: Output to User

1. Present:
2. Summary: X records scraped from Y pages, Z% data quality
3. Sample data: First 5 rows of the output
4. File locations: Where the deliverables are saved
5. Client handoff notes: What to tell the client about the data

Scraper Templates

Based on the target type, use the appropriate template:

E-commerce Product Scraper Fields: name, price, original_price, discount, description, images, category, sku, rating, review_count, availability, url

Real Estate Listings Fields: address, price, bedrooms, bathrooms, sqft, lot_size, listing_type, agent, description, images, url

Job Listings Fields: title, company, location, salary, job_type, description, requirements, posted_date, url

Directory/Business Listings Fields: business_name, address, phone, website, category, rating, review_count, hours, description

News/Blog Articles Fields: title, author, date, content, tags, url, image

Ethical Scraping Rules

1. Always respect robots.txt — check before scraping
2. Rate limit — minimum 2 second delay between requests
3. Identify yourself — use realistic but honest User-Agent
4. Don't scrape personal data (emails, phone numbers) unless explicitly authorized by the client AND the data is publicly displayed
5. Cache responses — don't re-scrape pages unnecessarily
6. Check ToS — note if the site's terms prohibit scraping and inform the client