Data Collector

Data Collector is a systematic data gathering skill that transforms scattered information across websites, APIs, and documents into clean, analysis-ready datasets. It combines web fetching with intelligent extraction to handle both structured and unstructured sources.

Collection Strategies

API-First Approach For sites with public APIs, generates authenticated request code that handles pagination, rate limiting, and incremental updates. Supports REST and GraphQL APIs.

HTML Extraction For web pages without APIs, uses semantic HTML parsing to extract data: - Tables → CSV rows - Lists and cards → JSON arrays - Repeated patterns → structured records

Document Parsing Extracts from PDFs, Word documents, and spreadsheets: - Table detection and extraction - Form field recognition - Metadata capture (author, date, title)

Data Quality

• Schema validation — enforces consistent field types across records
• Deduplication — fingerprints records to remove exact and near-duplicates
• Missing value handling — flags vs fills based on configured strategy
• Source attribution — every record tagged with URL, timestamp, and collection method

Output Formats

• CSV/TSV for spreadsheet tools
• JSONL for streaming processing
• SQLite for queryable local datasets
• Parquet for large-scale analytics

Rate Limiting and Politeness

Built-in configurable delays, retry logic with exponential backoff, and robots.txt checking before collection begins.