Systems Architect

# Systems Architecture Rules

Infrastructure Design - Design for failure at every layer — hardware fails, networks partition, regions go down - Redundancy costs money, downtime costs more — calculate acceptable risk - Prefer managed services for undifferentiated work — run less, build more - Infrastructure as code from day one — manual changes drift and break - Immutable infrastructure beats patching — replace, don't repair

Cloud Architecture - Multi-AZ minimum, multi-region for critical systems — availability zones fail together sometimes - Right-size first, auto-scale second — baseline must be correct - Reserved capacity for steady load, spot/preemptible for bursts — cost optimization requires planning - Egress costs add up — keep traffic within regions when possible - Cloud vendor lock-in is real — abstract where escape matters, accept where it doesn't

Networking - Private subnets for workloads, public only for load balancers — minimize attack surface - VPC peering and transit gateways for multi-account — plan topology before scaling - DNS for service discovery — hardcoded IPs break migrations - Zero trust: authenticate and encrypt internal traffic — perimeter security isn't enough - Network segmentation limits blast radius — flat networks let attackers roam

Integration Patterns - APIs for synchronous, queues for asynchronous — match pattern to requirements - Event-driven for loose coupling — producers don't know consumers - Service mesh for complex microservices — observability and security at network layer - Rate limiting and backpressure protect systems — don't let slow consumers crash fast producers - Dead letter queues for failed messages — don't lose data, process later

Reliability - Define SLOs before building — what does "up" mean for this system? - Error budgets allow controlled risk — 99.9% means 8 hours downtime per year is acceptable - Blast radius reduction: cell-based architecture — limit how many users one failure affects - Chaos engineering in staging first — break things intentionally before production breaks accidentally - Runbooks for every alert — 3 AM isn't debugging time

Disaster Recovery - RTO (recovery time) and RPO (data loss) are business decisions — architect for the requirement - Backups aren't recovery until tested — restore regularly - Hot/warm/cold standby each have trade-offs — cost vs speed of recovery - Cross-region replication for critical data — single region is single point of failure - DR drills reveal real problems — plan meets reality

Security - Defense in depth: multiple barriers — one layer will fail - Least privilege for services too — not just users - Secrets management centralized — no secrets in code, config files, or environment variables in images - Audit logging for compliance and forensics — you'll need it after a breach - Patch aggressively — known vulnerabilities are actively exploited

Monitoring and Observability - Metrics, logs, and traces together — each tells part of the story - Alerting on symptoms, not causes — users down matters, CPU high might not - Dashboards for each service with golden signals — latency, traffic, errors, saturation - Distributed tracing across services — follow requests end to end - Log aggregation with retention policy — balance cost and forensic needs

Capacity Planning - Measure current baseline before projecting — can't scale what you don't measure - Load test to find breaking points — theory differs from reality - Capacity leads demand — scaling takes time, be ahead - Cost modeling for growth scenarios — 10x users is rarely 10x cost - Review quarterly at minimum — patterns change

Migration and Evolution - Strangler fig pattern for legacy replacement — route traffic gradually - Blue-green or canary for infrastructure changes — test in production safely - Database migrations are hardest — plan data migration separately - Rollback plans before rollout — assume failure, prepare for it - Communicate maintenance windows — surprises damage trust