Data Validator

Data Validator generates data quality validation code using industry-standard frameworks to catch data issues before they propagate through pipelines.

Schema Validation

• JSON Schema — Draft 2020-12 validation with ajv (Node.js) or jsonschema (Python). Generates schema from sample data automatically.
• Pydantic — Python data models with field validators, pre/post validators, and discriminated unions for polymorphic data.
• Great Expectations — Expectation suites for batch data validation with data docs HTML reports.
• Pandera — pandas DataFrame schema validation with statistical checks (column distributions, outlier thresholds).

Rule Types

• Structural — required fields, data types, format (email, date, UUID), enum values
• Statistical — value ranges (min/max), mean/std deviation bounds, null rate thresholds, cardinality limits
• Referential — foreign key existence checks, orphan record detection, circular reference detection
• Temporal — timestamp ordering, date range validity, event sequence integrity

Anomaly Detection

Z-score and IQR methods for outlier detection. Seasonality-aware anomaly detection using STL decomposition for time-series data. CUSUM algorithm for drift detection.

Pipeline Integration

Drops into dbt tests, Airflow task validation steps, or as GitHub Actions data quality gates on PR-merged data files.

Reporting

Data quality scorecard: total records, pass/fail counts per rule, sample failing rows, and trend over time. Slack/email alerts on quality degradation.