The Fallacies of Distributed Computing
Early distributed computing architects made optimistic assumptions:
- The network is reliable
- Latency is zero
- Bandwidth is infinite
- The network is secure
- Topology doesn’t change
- There is one administrator
- Transport cost is zero
- The network is homogeneous
All of these are false. The gap between assumptions and reality creates failures.
Failure Modes
Network partition: Systems can’t communicate. One consequence: split brain where disconnected partitions make conflicting decisions.
Latency: Requests timeout waiting for responses. Is the service slow or dead?
Cascading failures: One service failure causes dependent services to fail.
Resource exhaustion: One client or feature consumes all resources, affecting others.
Resilience Patterns
Timeouts: Set limits on how long to wait for responses. Prevents indefinite blocking.
Circuit breakers: Stop sending requests to failing services temporarily. Prevents hammering failing systems.
Retry logic: Retry failed requests with exponential backoff. Transient failures often recover.
Bulkheads: Isolate resources. If one feature exhausts resources, others continue working.
Redundancy: Run services in multiple locations. If one fails, others handle traffic.
The CAP Theorem
Distributed systems must choose two of three properties:
- Consistency: All nodes see the same data
- Availability: System responds to requests
- Partition tolerance: System works even when networks partition
You can’t have all three simultaneously. This fundamental constraint affects architecture.
CA systems: Traditional databases. Consistent, available, but can’t partition.
CP systems: Services prioritize consistency. If network partitions, they become unavailable.
AP systems: Services prioritize availability. During partitions, data might be inconsistent.
Most modern systems choose availability and partition tolerance, accepting eventual consistency.
Monitoring and Observability
Resilience requires knowing when things break:
- Health checks: Periodic verification that services are alive
- Metrics: CPU, memory, request latency, error rates
- Distributed traces: Track requests through system
- Alerts: Notify when problems occur
Without observability, problems aren’t detected until users complain.
Testing for Resilience
Chaos engineering: Deliberately introduce failures (stop services, add latency, corrupt data) to verify systems handle them.
Failure drills: Regular testing of disaster recovery procedures.
Load testing: Verify performance under peak load.
Incident response planning: Document procedures for common failures.
The Operational Burden
Resilient systems are complex. They require monitoring, alerting, and incident response. The operational burden is significant.
This creates tradeoff: simple systems are fragile but easy to operate. Resilient systems are more robust but require more operational expertise.
The Cultural Piece
Building resilient systems requires accepting that failures will occur. Organizations that treat failures as learning opportunities rather than disasters build better systems. “Blameless postmortems” where teams analyze failures without blame enable learning.
Future Directions
Chaos engineering as practice: Regularly test failure scenarios.
Observability as baseline: Systems should emit detailed data about behavior.
Resilience patterns in frameworks: Frameworks that embed resilience patterns (timeouts, retries, circuit breakers) reduce burden on developers.
The most resilient systems share characteristics: they accept failure, plan for it, test for it, and learn from it.