Data Cleaner

Data Cleaner automates the most tedious part of any data project — getting raw data into a usable state. It goes beyond find-and-replace to understand the semantics of your data and apply the right cleaning strategy for each issue.

Data Profiling

• Before cleaning, produces a profile report:
• Column-level statistics (type, cardinality, null rate, min/max)
• Distribution shapes for numeric columns
• Pattern frequency for text columns (email, phone, date formats present)
• Correlation matrix highlighting redundant features

Cleaning Operations

Type Standardization - Date parsing across 30+ formats → ISO 8601 - Currency strings ("$1,234.56", "€1.234,56") → numeric - Boolean variants ("Yes/No", "1/0", "TRUE/FALSE") → consistent - Phone numbers → E.164 format (+1XXXXXXXXXX)

String Normalization - Case standardization (Title Case for names, uppercase for codes) - Whitespace trimming and internal whitespace collapse - Unicode normalization (NFC) and encoding repair (mojibake detection) - Consistent abbreviation expansion ("St." → "Street", "Dr" → "Doctor")

Deduplication - Exact duplicate removal - Fuzzy deduplication using Jaro-Winkler similarity for names and addresses - Blocking strategies for large datasets to make fuzzy matching tractable

Missing Value Handling - Mean/median/mode imputation for numeric columns - Forward-fill or backward-fill for time series - Indicator variable creation for informative missingness - Row removal when missing rate exceeds configurable threshold

Audit Trail

Every transformation logged to `cleaning_log.json` with: column affected, operation, rows changed, and before/after samples.