The production incidents I have been involved in over the years share a common characteristic: the failure was foreseeable. Not necessarily the specific failure, but the category. We had not thought carefully about what happened when this service was slow. We had not tested what happened when this dependency returned errors. We had not considered what happened when this queue backed up.
Resilience in distributed systems is not about predicting exactly what will fail. It is about designing so that when something fails (and something always will), the impact is limited and the system recovers.
The patterns that have reduced incident severity most consistently are these.
Timeouts on everything. Every network call, every database query, every external API call needs a timeout. Without a timeout, a slow dependency holds your threads, which fills your thread pool, which causes your service to stop responding, which cascades to dependent services. The specific timeout value matters less than having one at all.
Circuit breakers on dependencies. When a downstream service starts failing, stop sending it requests and fail fast instead. A circuit breaker that opens after ten failures in a minute prevents your service from drowning in slow, failing requests. More importantly, it gives the downstream service less load while it is struggling, which can help it recover.
Retries with exponential backoff and jitter. When a request fails, retry it. But not immediately and not in lockstep with every other client. Immediate retries hit a service that just failed with a flood of traffic. Exponential backoff with jitter spreads retry traffic over time and avoids thundering herd problems.
Graceful degradation over total failure. When a non-critical dependency fails, continue serving requests without that dependency. A recommendation engine being unavailable should not prevent users from checking out. A logging service being slow should not block user requests. Design for the happy path and for the degraded path.
Bulkheads to contain failures. Different client types or request types should use separate thread pools and connection pools. If a spike of low-priority batch jobs fills the thread pool, interactive user requests should not be affected.
The operational practice that has improved resilience as much as any technical pattern is regular chaos engineering. Not just defining failure modes in architecture reviews but actually inducing failures in production and observing what happens. The knowledge of how the system behaves under failure, built through experience rather than theory, changes how engineers design and operate it.