Designing for Failure: Resilience Patterns in Production APIs

A system that works perfectly when everything is healthy is easy to build. The engineering challenge is what happens when a database is slow, a third-party API starts returning 503s, or a network call hangs indefinitely. Without explicit failure handling, one slow dependency causes cascading failures that take down your entire service.

Timeouts: The Baseline

Every external call needs a timeout. This sounds obvious but is commonly omitted, especially in internal network calls where "it always completes in under 100ms."

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
 
try {
  const res = await fetch(url, { signal: controller.signal });
  return await res.json();
} catch (err) {
  if (err.name === 'AbortError') throw new TimeoutError('Request timed out');
  throw err;
} finally {
  clearTimeout(timeout);
}

Set timeouts based on the P99 latency of the dependency under normal load — not the median. If TMDB responds in 200ms at P50 and 1.2s at P99, a 3-second timeout is reasonable.

Retries with Exponential Backoff

Transient failures — network blips, momentary service unavailability — are common. Retrying immediately just hammers a struggling service. Exponential backoff spaces out retries, giving the dependency time to recover.

async function withRetry<T>(
  fn: () => Promise<T>,
  maxAttempts = 3,
  baseDelayMs = 200
): Promise<T> {
  let lastError: unknown;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      if (attempt === maxAttempts - 1) break;
      // Exponential backoff with jitter
      const delay = baseDelayMs * Math.pow(2, attempt) * (0.5 + Math.random() * 0.5);
      await new Promise(r => setTimeout(r, delay));
    }
  }
  throw lastError;
}

Jitter (the * (0.5 + Math.random() * 0.5) part) prevents the thundering herd: if 1000 requests all fail simultaneously and retry at the exact same interval, they create another spike. Jitter spreads them out.

Retry only on retriable errors: network failures, 429 (Too Many Requests), 503 (Service Unavailable). Do not retry on 400 (bad request), 401 (unauthorized), or 404 (not found) — these won't succeed on retry.

Circuit Breakers

Retries help with transient failures. A circuit breaker handles sustained failures — a dependency that's down for 5 minutes. Without a circuit breaker, every request retries 3 times, creating 3× the load on an already-struggling service.

A circuit breaker has three states:

• Closed: requests flow normally
• Open: requests fail immediately (no attempt to the dependency)
• Half-Open: one test request is allowed through to check if the dependency recovered

class CircuitBreaker {
  private failures = 0;
  private lastFailure = 0;
  private state: 'closed' | 'open' | 'half-open' = 'closed';
 
  constructor(
    private threshold = 5,        // failures before opening
    private resetTimeout = 30_000  // ms before testing again
  ) {}
 
  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailure < this.resetTimeout) {
        throw new Error('Circuit open');
      }
      this.state = 'half-open';
    }
 
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }
 
  private onSuccess() { this.failures = 0; this.state = 'closed'; }
  private onFailure() {
    this.failures++;
    this.lastFailure = Date.now();
    if (this.failures >= this.threshold) this.state = 'open';
  }
}

Graceful Degradation

When a non-critical dependency fails, return a degraded response rather than an error. Cognix does this in its answer synthesis pipeline: if the configured model provider fails, it falls back to the extractive local synthesizer. The answer is lower quality, but the user gets a response.

async function getRecommendations(userId: string) {
  try {
    return await recommendationEngine.compute(userId);
  } catch {
    // Fall back to trending content — not personalized but not an error
    return await getTrendingContent();
  }
}

The key question for each dependency: "If this fails, should the user see an error or a degraded experience?" For recommendations, trending content is better than a 500. For payment processing, there is no acceptable degradation — fail explicitly.

Bulkheads

A bulkhead isolates failures in one part of the system from propagating to others. In practice: separate connection pools, separate worker queues, or separate rate limits for different types of requests.

If your API allows both expensive report generation and cheap status lookups, an expensive report that consumes all database connections will starve status lookups. A separate connection pool for each class of operation prevents this.

These patterns compose. A robust external API call uses a timeout (prevent hangs), retries with backoff (handle transients), a circuit breaker (handle sustained failures), and graceful degradation (return something useful when all else fails).