RSS Aggregator

RSS Aggregator builds feed ingestion pipelines that collect, deduplicate, enrich, and distribute content from multiple RSS/Atom sources.

Feed Parsing

Handles RSS 2.0, Atom 1.0, and JSON Feed formats using feedparser (Python) or rss-parser (Node.js). Normalizes disparate feed schemas into a unified article model with consistent fields: title, url, author, published_at, summary, content.

Polling & Scheduling

Configurable per-feed poll intervals using cron expressions. Respects `Cache-Control` and `ETag`/`Last-Modified` headers to avoid re-fetching unchanged feeds. Conditional GET reduces bandwidth by 80-90% for cooperative sources.

Deduplication

URL-based and content-fingerprint deduplication. SimHash similarity detection catches near-duplicate articles from wire services (AP, Reuters) that appear across multiple outlets.

Full-Text Extraction

For feeds that only provide summaries, Readability.js or newspaper3k extracts the full article body from the source page. Respects robots.txt and implements polite crawl delays.

AI Enrichment

Optional LLM pipeline: 3-sentence summary, keyword extraction, category classification, sentiment scoring, and named entity recognition for people/organizations/topics.

Output

SQLite or PostgreSQL storage, REST API for querying articles, email digest renderer, and JSON Feed re-export for downstream consumption.