System Design: Fundamentals

Instead of memorizing patterns or blindly following best practices, we'll ask the essential questions: What is this system actually doing? Why does it work this way? And how can we build it ourselves?

What is Throughput?

Throughput is the amount of work a system can handle in a given period of time. In backend systems, it is typically measured as the number of requests (API calls) processed per second (RPS).

Throughput can be increased by horizontally scaling the system and distributing data across multiple nodes.

Little's Law

What is Latency?

Latency is the the time taken for a request to travel from the client to the server and back. It primarily depends on:

Network Delay

The time taken for data to travel across the network. It is mainly influenced by three factors:

1. Physical Distance: The farther the server is, the longer it takes for data to travel. For example, a user in India hitting a server in Delhi receives response in ~10-20 ms, however, the same user hitting a server in US (California) receives it in ~200-300 ms.

2. Routing (Path Taken): Data doesn't go directly from point A → B. It passes through multiple intermediate nodes (routers, ISPs).

3. Network Congestion: When too many request are flowing through the network, packets get delayed, queues build up, and some packets may even be retransmitted.

NOTE: Routers can become overloaded and their buffers may fill up, causing some packets to be dropped before they reach the destination. In protocols like TCP, the receiver detects missing packets (for example, by noticing gaps in sequence numbers) or the sender detects a lack of acknowledgment.

Server Processing Time

The time taken by the server to handle the request, including business logic execution and data access (e.g., fetching data from in-memory stores like Redis or querying a database).

NOTE: P50, P90, and P99 are latency percentiles that describe how a system performs across all requests, rather than relying on a single average value. P50 shows normal behavior, while P90 and P99 reveal how the system behaves under stress and edge cases.

Caching is the solution for latency??

What is Scalability?

Scalability refers to a system's ability to handle increasing workloads efficiently by adding resources (e.g., servers, storage, network capacity) without compromising performance.

Data sharding helps achieve scalability.

What is Availability?

Availability refers to a system's ability to remain operational, even in the presence of failures.

Data Replication

To achieve high availability and reliability, we need to replicate data across multiple servers.

Out-of-Order Replication

What is Consistency?

In a microservices architecture, the golden rule is Database-per-Service (each service owns its own private database). This prevents tight coupling, but it makes handling distributed transactions and complex data queries incredibly difficult.

Consistency models define the degree of data consistency. Wide range if consistency models exists:

Strong Consistency

Any read operation returns the value corresponding to the result of the most updated write data item. A client never sees out-of-date data.

Usually achieved by forcing a replica not to accept new reads/writes until every replica has agreed on current write. This approach is not ideal for highly available systems because it could block new operations.

Weak Consistency

Subsequent read operations may not see the most updated value.

Eventual Consistency

This is a specific form of weak consistency. Given enough time, all updates are propagated, and all replicas are consistent.

Quorum Consensus

W + R > N ensures strong consistency - how??

Data Partitioning

There are two challenges when partitioning data:

1. Distribute data across multiple servers evenly.
2. Minimize data movement when nodes are added or removed.

Consistent Hashing is a solution to these challenges.

Consistent Hashing

It's a foundational algorithm in distributed systems that is used to distribute data across a cluster of servers.

In most traditional hash tables, a change in the number of array slots causes nearly all keys to be remapped.

In contrast, consistent hashing is a special type of hashing such that when a hash table is resized, only k/n keys need to be remapped on average, where k is the number of keys and n is the number of slots.

The core idea is to map servers and keys on to the ring using a distributed hash function.

NOTE: Consistent hashing is used in distributed caches, Dynamo-style databases, storage systems, load balancers for stateful traffic, queue partitioning, and routing layers. The exact implementation varies, but the design goal is the same: stable ownership with controlled movement.

What are Distributed Transactions?

What is Saga Pattern?

• Purpose: Manages distributed transactions and maintains data consistency across multiple microservices without locking databases.

• What it does: Instead of a traditional database transaction (which locks tables across networks), a Saga (1) breaks a business process into a series of local transactions. Each service (2) updates its own database and (3) publishes an event.

• The Catch: If step 3 fails, the Saga orchestrator triggers Compensating Transactions (backward steps) to undo the actions of steps 1 and 2, acting like a multi-service "Ctrl+Z".

• Example: Buying an item online requires: 1. Order Service (creates order) -> 2. Payment Service (charges card) -> 3. Shipping Service (fails because out of stock). The Saga then triggers a refund in the Payment Service and cancels the order in the Order Service.

The Saga pattern is primarily implemented in two distinct architectural styles:

Orchestration-based (centralized)

In an Orchestration saga, you introduce a dedicated service called the Saga Orchestrator (or Saga Manager). It explicitly tells every single microservice when to execute its transaction and waits for a response before commanding the next service.

In an orchestrated setup, the services are "dumb" workers. They don't listen to broad events, they just expose clean APIs (methods) for the Orchestrator to invoke. The orchestrator usually communicates with individual services via standard synchronous protocols (like REST or gRPC).

// --- 1. ORDER SERVICE ---
class OrderService {
    public void createPendingOrder(String orderId) {
        System.out.println("[Order Service] 📦 Order " + orderId + " created with status: PENDING");
    }
    
    public void cancelOrder(String orderId) {
        System.out.println("[Order Service] ❌ COMPENSATE: Order " + orderId + " updated to: CANCELLED");
    }
}

// --- 2. PAYMENT SERVICE ---
class PaymentService {
    public boolean processPayment(String orderId, double amount) {
        System.out.println("[Payment Service] 💳 Charging $" + amount + " for order " + orderId);
        System.out.println("[Payment Service] ✅ Payment successful.");
        return true; // Return true for success
    }

    public void refundPayment(String orderId, double amount) {
        System.out.println("[Payment Service] ↩️ COMPENSATE: Refunded $" + amount + " for order " + orderId);
    }
}

// --- 3. PROVISIONING SERVICE ---
class ProvisioningService {
    public boolean activateSubscription(String orderId) {
        System.out.println("[Provisioning Service] 🚀 Attempting to activate premium features...");
        
        // Simulating an unexpected failure (e.g., target cluster down)
        System.out.println("[Provisioning Service] 💥 ERROR: Activation failed! Server unreachable.");
        return false; 
    }
}

/* The Saga Orchestrator (The Central Brain)

The Orchestrator contains the entire workflow logic. It executes steps sequentially and 
maintains an internal state. If a boolean check returns false, it halts the pipeline and 
runs the exact rollback sequence.

*/
class SubscriptionSagaOrchestrator {
    private final OrderService orderService = new OrderService();
    private final PaymentService paymentService = new PaymentService();
    private final ProvisioningService provisioningService = new ProvisioningService();

    public void executeSaga(String orderId, double amount) {
        System.out.println("=== Starting Orchestrated Saga Workflow ===");
        
        // Step 1: Create Order
        orderService.createPendingOrder(orderId);

        // Step 2: Process Payment
        boolean paymentSuccess = paymentService.processPayment(orderId, amount);
        
        if (!paymentSuccess) {
            // If payment fails, rollback Step 1
            rollback(orderId, amount, 1);
            return;
        }

        // Step 3: Provision Service
        boolean provisioningSuccess = provisioningService.activateSubscription(orderId);
        
        if (!provisioningSuccess) {
            // If provisioning fails, rollback everything up to Step 2
            rollback(orderId, amount, 2);
            return;
        }

        System.out.println("=== 🎉 Saga Completed Successfully! ===");
    }

    // Centralized Rollback Engine
    private void rollback(String orderId, double amount, int failedAtStep) {
        System.out.println("\n--- 🚨 ORCHESTRATOR DETECTED FAILURE: Executing Rollback Sequence ---");

        // Rollback Step 2 (Refund Payment) if we made it past Step 2
        if (failedAtStep >= 2) {
            paymentService.refundPayment(orderId, amount);
        }

        // Rollback Step 1 (Cancel Order) if we made it past Step 1
        if (failedAtStep >= 1) {
            orderService.cancelOrder(orderId);
        }

        System.out.println("--- 🛡️ System Restored to Consistent State via Orchestrator ---");
    }
}

/* Running the Simulation */
public class OrchestratorSagaMain {
    public static void main(String[] args) {
        SubscriptionSagaOrchestrator orchestrator = new SubscriptionSagaOrchestrator();
        
        String uniqueOrderId = "ORD-99823";
        double subscriptionPrice = 19.99;

        // Trigger the workflow
        orchestrator.executeSaga(uniqueOrderId, subscriptionPrice);
    }
}

Output in the Console:

=== Starting Orchestrated Saga Workflow ===
[Order Service] 📦 Order ORD-99823 created with status: PENDING
[Payment Service] 💳 Charging $19.99 for order ORD-99823
[Payment Service] ✅ Payment successful.
[Provisioning Service] 🚀 Attempting to activate premium features...
[Provisioning Service] 💥 ERROR: Activation failed! Server unreachable.

--- 🚨 ORCHESTRATOR DETECTED FAILURE: Executing Rollback Sequence ---
[Payment Service] ↩️ COMPENSATE: Refunded $19.99 for order ORD-99823
[Order Service] ❌ COMPENSATE: Order ORD-99823 updated to: CANCELLED
--- 🛡️ System Restored to Consistent State via Orchestrator ---

• Advantages: Perfect for complex, enterprise-level business workflows. The entire state machine and transaction logic live in one central location, making debugging and auditing incredibly straightforward. It completely avoids cyclic dependencies.

• Disadvantages: The orchestrator can become a single point of failure if not configured for high availability. It introduces slightly tighter coupling because the orchestrator must explicitly know how to talk to every downstream service interface.

Choreography-based (decentralized)

In a Choreography saga, there is no central supervisor. Instead, each microservice acts independently, executes its local transaction, and throws an event into a message broker (like Kafka or RabbitMQ).

The next service listens for that event, does its job, and throws its own event. If a service down the line fails, it throws an event that tells the previous services to run their Compensating Transactions (the undo steps).

// Events that push the transaction forward
public record OrderCreatedEvent(String orderId, String userId, double amount) {}
public record PaymentSuccessfulEvent(String orderId, String userId, double amount) {}

// Compensation Events that rollback the transaction
public record PaymentFailedEvent(String orderId, String userId, String reason) {}
public record RollbackOrderEvent(String orderId, String reason) {}

/* The Order Service (The Initiator & Final Rollback)

The Order Service kicks off the Saga. It also listens for the ultimate rollback event 
in case something fails later, so it can change the status from PENDING to CANCELLED.

*/
public class OrderService {
    private final MessageBroker messageBroker = new MessageBroker();

    // Step 1: Start the Saga
    public void createOrder(String userId, double amount) {
        String orderId = "ORD-" + java.util.UUID.randomUUID().toString().substring(0, 5);
        System.out.println("[Order Service] Creating order " + orderId + " with status PENDING.");
        
        // Publish event to network to notify the Payment Service
        OrderCreatedEvent event = new OrderCreatedEvent(orderId, userId, amount);
        messageBroker.publish("order-created-topic", event);
    }

    // Step 6 (COMPENSATION): Undo step if things fail down the line
    public void handleOrderRollback(RollbackOrderEvent event) {
        System.out.println("[Order Service] ❌ ROLLING BACK: Updating order " 
                + event.orderId() + " status to CANCELLED. Reason: " + event.reason());
    }
}

/* The Payment Service (Success & Reversal Logic)

The Payment Service listens for the order creation, deducts money, and passes the torch. 
Crucially, it also knows how to refund the money if the next service fails.

*/
public class PaymentService {
    private final MessageBroker messageBroker = new MessageBroker();

    // Step 2: Handle successful progress
    public void handleOrderCreated(OrderCreatedEvent event) {
        System.out.println("[Payment Service] Charging $" + event.amount() + " to user " + event.userId());
        System.out.println("[Payment Service] ✅ Payment successful for " + event.orderId());
        
        // Move to the next service
        PaymentSuccessfulEvent nextEvent = new PaymentSuccessfulEvent(event.orderId(), event.userId(), event.amount());
        messageBroker.publish("payment-success-topic", nextEvent);
    }

    // Step 5 (COMPENSATION): Refund money if the downstream service crashed
    public void reversePayment(PaymentFailedEvent event) {
        System.out.println("[Payment Service] ↩️ REVERSING TRANSACTION: Refunding user for order " + event.orderId());
        
        // Trigger the final rollback event for the Order Service
        messageBroker.publish("order-rollback-topic", new RollbackOrderEvent(event.orderId(), event.reason()));
    }
}

/* The Provisioning Service (The Failure Point)

This service is responsible for activating the user's account. In our dummy scenario, an 
error happens here (e.g., the server hosting the premium feature is completely offline).

*/
public class ProvisioningService {
    private final MessageBroker messageBroker = new MessageBroker();

    // Step 3: Try to finalize the transaction
    public void handlePaymentSuccess(PaymentSuccessfulEvent event) {
        System.out.println("[Provisioning Service] Attempting to grant premium access for order: " + event.orderId());
        
        // Simulating a critical failure (e.g., integration server down)
        boolean deploymentFailed = true; 

        if (deploymentFailed) {
            System.out.println("[Provisioning Service] ❌ CRITICAL ERROR: Cannot provision features!");
            
            // Step 4: Fire the warning flare to start the chain of reversals!
            PaymentFailedEvent rollbackEvent = new PaymentFailedEvent(event.orderId(), event.userId(), "Feature Provisioning Server Offline");
            messageBroker.publish("payment-failed-topic", rollbackEvent);
        } else {
            System.out.println("[Provisioning Service] 🎉 Saga Complete! Features activated successfully.");
        }
    }
}

/* Simulating the Execution Flow

Here is how the entire distributed architecture reacts when we run the code:

*/
public class SagaSimulation {
    public static void main(String[] args) {
        // Instantiate our isolated microservices
        OrderService orderService = new OrderService();
        PaymentService paymentService = new PaymentService();
        ProvisioningService provisioningService = new ProvisioningService();

        System.out.println("--- Starting Saga Transaction Flow ---");
        orderService.createOrder("user-99", 49.99);
        
        System.out.println("\n--- Simulated Network Triggers Event Interception ---");
        // 1. Order Service fires event -> Payment Service catches it
        paymentService.handleOrderCreated(new OrderCreatedEvent("ORD-XYZ12", "user-99", 49.99));
        
        // 2. Payment Service fires success event -> Provisioning Service catches it and fails!
        provisioningService.handlePaymentSuccess(new PaymentSuccessfulEvent("ORD-XYZ12", "user-99", 49.99));
        
        // 3. Provisioning Service fires failure event -> Payment Service catches it to execute refund
        paymentService.reversePayment(new PaymentFailedEvent("ORD-XYZ12", "user-99", "Server Offline"));
        
        // 4. Payment Service finishes refund -> Order Service catches it to cancel order
        orderService.handleOrderRollback(new RollbackOrderEvent("ORD-XYZ12", "Server Offline"));
        
        System.out.println("\n--- Saga Completed Safely (System Restored to Consistent State) ---");
    }
}

Output in Console:

--- Starting Saga Transaction Flow ---
[Order Service] Creating order ORD-a7e32 with status PENDING.

--- Simulated Network Triggers Event Interception ---
[Payment Service] Charging $49.99 to user user-99
[Payment Service] ✅ Payment successful for ORD-XYZ12
[Provisioning Service] Attempting to grant premium access for order: ORD-XYZ12
[Provisioning Service] ❌ CRITICAL ERROR: Cannot provision features!
[Payment Service] ↩️ REVERSING TRANSACTION: Refunding user for order ORD-XYZ12
[Order Service] ❌ ROLLING BACK: Updating order ORD-XYZ12 status to CANCELLED. Reason: Server Offline

--- Saga Completed Safely (System Restored to Consistent State) ---

• Advantages: Very simple to set up for small workflows (2 to 3 steps). Services are loosely coupled because they only talk via an asynchronous message broker. There is no single point of failure.

• Disadvantages: It quickly becomes a nightmare to maintain when you have 5+ services. It's incredibly hard to track the overall "state" of a transaction because the logic is scattered across different codebases. You can easily create cyclic dependencies (Service A triggers B, which triggers C, which accidentally triggers A again).

What is Fault Tolerance?

Fault tolerance is a design choice to handle hardware/software failures effectively.

Achieved through replication (e.g., database replicas, redundant servers) and failover mechanisms (automatic switching to backups).

For example, imagine a cart service in an e-commerce platform like Amazon. If a user adds an item to their shopping cart, the system must not lose it, even if the cart service fails. A replicated server should take over to ensure the cart remains intact.

Data Replication is the solution.

NOTE: Redundancy has a cost, but a reliable system must invest in eliminating every single point of failure to maintain resilience.

In modern architecture, an exception doesn't always mean your app should stop or return an error code. If a third-party service is down or timing out, throwing a hard exception can bring down your entire user experience.

Systems use patterns like Circuit Breakers (via tools like Resilience4j) to catch exceptions internally and trigger a Fallback:

// Define the circuit breaker configuration
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)          // Trip if 50% of last 100 calls fail
    .waitDurationInOpenState(Duration.ofMinutes(1)) // Wait 1 minute before testing recovery
    .slidingWindowSize(100)            // Monitor the last 100 requests
    .build();

// Applying it to a method via annotations
@CircuitBreaker(name = "recommendationService", fallbackMethod = "getTrendingBackup")
public List<Product> getPersonalizedRecommendations(User user) {
    // If this remote API call starts failing, the circuit trips OPEN!
    return externalRecommendationApi.getForUser(user.getId());
}

// The Fallback: Executed instantly when the circuit is OPEN
public List<Product> getTrendingBackup(User user, Throwable t) {
    // Return a generic list from local cache instead of crashing
    return cacheRepository.getTopTrendingProducts();
}

What is a Circuit Breaker?

A circuit breaker operates as a state machine with three main states: CLOSED, OPEN, and HALF-OPEN.

1. CLOSED State (Normal Operation)

When everything is healthy, the circuit is CLOSED.

• Traffic flows normally from your application to the downstream service.

• The circuit breaker continuously monitors the responses (successes, failures, and timeouts).

• As long as the error rate stays below a certain threshold (e.g., lower than 50% failure rate), it remains CLOSED.

2. OPEN State (Failing Fast)

If the downstream service starts failing or timing out repeatedly, and the error rate crosses your configured threshold, the circuit breaker trips and flips to OPEN.

• Traffic is completely cut off. Any new requests from your application to that service are blocked instantly at the local level.

• Instead of waiting for a network timeout, the circuit breaker immediately executes a Fallback mechanism (like returning cached data or a friendly "Service temporarily unavailable" message).

• This gives the struggling downstream service breathing room to recover or restart without being blasted by traffic.

3. HALF-OPEN State (Testing the Waters)

After a configured period of time (the "sleep window," usually a few seconds or minutes), the circuit breaker automatically switches to HALF-OPEN.

• It permits a limited number of trial requests to pass through to the downstream service.

• If the trial requests succeed: The circuit breaker assumes the service has recovered, resets its error counters, and flips back to CLOSED (normal operation).

• If any trial requests fail: The circuit breaker assumes the service is still broken, resets the timer, and immediately flips back to OPEN to block traffic again.

public enum CircuitState {
    CLOSED,
    OPEN,
    HALF_OPEN
}

public class CircuitBreaker {

    private CircuitState state = CircuitState.CLOSED;

    private int failureCount = 0;

    private long lastFailureTime = 0;

    private final int failureThreshold = 3;

    private final long retryTimeout = 10000;

    public synchronized <T> T execute(
            Supplier<T> supplier) {

        if (state == CircuitState.OPEN) {

            if (System.currentTimeMillis()
                    - lastFailureTime
                    > retryTimeout) {

                state = CircuitState.HALF_OPEN;

            } else {
                throw new RuntimeException(
                        "Circuit is OPEN");
            }
        }

        try {

            T result = supplier.get();

            failureCount = 0;
            state = CircuitState.CLOSED;

            return result;

        } catch(Exception ex) {

            failureCount++;

            if (failureCount >= failureThreshold) {

                state = CircuitState.OPEN;
                lastFailureTime =
                        System.currentTimeMillis();
            }

            throw ex;
        }
    }

    public static void main(String[] args) {
        CircuitBreaker circuitBreaker =
            new CircuitBreaker();

        String response =
                circuitBreaker.execute(() ->
                        paymentService.processPayment());

        System.out.println(response);
    }

}

What is CAP Theorem?

CAP theorem states it is impossible for a distributed system to simultaneously provide more than two of these three guarantees: consistency, availability, and partition tolerance.

1. CP (consistency and partition tolerance) systems:

2. AP (availability and partition tolerance) systems:

3. CA (consistency and availability) systems: Since network failure is unavoidable, a distributed system must tolerate network parition. When a partition occurs, we must choose between consistency and availability.

What is Observability?

In a distributed cloud system (like microservices), an error in Service A might actually be caused by a timeout in Service C. Merely printing a stack trace to a local console is useless.

Modern exception handling integrates deeply with Observability Stack Tools (like OpenTelemetry, Datadog, Prometheus, or the ELK stack):

1. Correlation & Trace IDs: When an HTTP request enters the system, a unique Trace ID is generated.

2. Context Enrichment: If an exception is thrown anywhere down the line, the global handler automatically attaches the Trace ID to the error log.

3. Log Aggregation: The exception is pushed out of the application memory into a centralized log aggregator.

4. The Client Link: The system returns the Trace ID to the frontend user. If a user sees a screen saying "Something went wrong (ID: abc-9876)", a developer can copy-paste that ID into Datadog or Splunk and see the exact multi-service stack trace instantly.

What is Caching?

Caching is the process of temporarily storing copies of data in a high-speed storage (cache) to reduce the time and cost of retrieving that data from slower sources like databases, APIs, or disk.

Caching is not a Single Component, it's a Layered Strategy

Caching can exist across multiple layers of a system, with each layer designed to solve a specific performance or scalability problem.

1. Repeated network calls & poor user experience: When users repeatedly request the same static assets (like images, CSS, or JS), making a network call every time adds unnecessary latency. Client-side caching stores this data directly in the browser (HTTP cache, localStorage), allowing instant access without hitting the network. This results in faster page loads and a smoother user experience.

2. High latency due to geographic distance: If your servers are far from users, every request has to travel long distances, increasing latency. CDNs like Cloudflare solve this by caching content at edge servers close to users. Requests are served from nearby locations, drastically reducing response times and improving global performance.

3. Database overload & high read latency: Databases like PostgreSQL can become bottlenecks under heavy read traffic. External caches such as Redis or Memcached store frequently accessed or computed data, preventing repeated database hits. This reduces load, lowers latency, and allows the system to scale efficiently.

4. Network overhead to external systems: Even calling an external cache like Redis involves network latency. In-process caching stores data directly in the application's memory, eliminating network calls entirely. This makes it the fastest backend caching layer, ideal for ultra-low-latency access, though it comes with trade-offs like lack of shared state across instances.

5. Expensive disk I/O operations: Reading from disk is slow compared to memory. Databases use internal caches (like buffer pools in PostgreSQL) to keep frequently accessed data in memory, reducing disk reads and improving query performance.

Caching Strategies for Different Workloads

Different caching strategies exist because systems have different priorities. The right choice depends on the specific problem you are solving, such as reducing database load, ensuring consistency, or handling high write traffic.

Below are the common caching strategies, reframed as solutions tailored to specific use cases:

1. Cache-Aside (Lazy Loading): The application first checks the cache (e.g., Redis). On a cache miss, it fetches data from the database (e.g., PostgreSQL), stores it in the cache, and returns the response. Use this when your system is read-heavy, you want control over what gets cached, and you can tolerate slightly stale data.

2. Write-Through Caching: The application writes data to the cache, and the cache synchronously updates the database before confirming success. Use this when your reads must always return up-to-date data (strong consistency is required), such as in financial or critical user data systems.

3. Write-Behind (Write-Back) Caching: The application writes only to the cache, and the cache asynchronously persists data to the database, often in batches. Use this when extremely high write throughput is needed and eventual consistency is acceptable (e.g., analytics, logging systems).

4. Read-Through Caching: The cache acts as a smart intermediary. On a cache miss, it automatically fetches data from the database, stores it, and returns it, without the application directly interacting with the database. Use this when you want to simplify application logic and centralize caching behavior. CDNs like Cloudflare use a similar pattern.

Cache Eviction Strategies: Managing Limited Memory Effectively

Caches are memory-constrained by design, which means they cannot store everything forever.

As new data comes in, old data must be removed intelligently to make space. This is where eviction strategies come in. They decide which data to keep and which to discard, based on usage patterns and system goals like performance or freshness.

1. LRU (Least Recently Used): LRU removes the item that hasn’t been accessed for the longest time, assuming that recently used data is more likely to be used again. Use this when your workload has temporal locality (e.g., user sessions, recently viewed items), where recent data is more relevant.

2. LFU (Least Frequently Used): LFU evicts the least frequently accessed items by maintaining access counts for each key. This ensures that consistently popular data stays in the cache. Use this when certain items remain popular over time (e.g., trending videos, top playlists).

3. FIFO (First In First Out): FIFO removes the oldest inserted item, regardless of how often or recently it was accessed. Use this when simplicity is more important than optimization, though it’s rarely ideal for real-world caching due to poor hit rates.

4. TTL (Time To Live): TTL assigns an expiration time to each cache entry, automatically removing data after a set duration. Use this when data must be refreshed periodically (e.g., API responses, authentication tokens, or time-sensitive data).

Caching makes systems faster, but it also introduces new failure modes.

1. Cache Stampede (Thundering Herd): A cache stampede happens when a popular cache entry expires and many requests try to rebuild it at the same time. There is a brief window, even if only a second, where every request misses the cache and goes straight to the database. Instead of one query, you suddenly have hundreds or thousands, which can overload the database.

How to handle it:

1. Request coalescing (single flight): Allow only one request to rebuild the cache while others wait for the result. This is the most effective solution.

2. Cache warming: Refresh popular keys proactively before they expire. This only helps when using TTL-based expiration. If you invalidate cache on writes instead, warming does not prevent stampedes.

3. Cache Consistency: Cache consistency problems are some of the most commonly discussed in system design interviews. They happen when the cache and database return different values for the same data. This is common because most systems read from the cache but write to the database first. That creates a window where the cache still holds stale data.

For example, if a user updates their profile picture, the new value is written to the database but the old value might still be in the cache. Other users may see the outdated profile picture until the cache eventually refreshes. There is no perfect solution. You choose a strategy based on how fresh the data must be.

How to handle it: Cache invalidation on writes: Delete the cache entry after updating the database so it gets repopulated with fresh data. Short TTLs for stale tolerance: Let slightly stale data live temporarily if eventual consistency is acceptable. Accept eventual consistency: For feeds, metrics, and analytics, a short delay is usually fine.

When to Bring Up Caching?

Don't jump straight to caching. You need to establish why it's necessary first. Bring up caching when you identify one of these problems:

1. Read-heavy workload: "We're serving 10M daily active users, each making 20 requests per day. That's 200M reads hitting the database. Even with indexes, we're looking at 20-50ms per query. A cache drops that to under 2ms and takes most of the load off the database."

2. Expensive queries: "Computing a user's personalized feed requires joining posts, followers, and likes across multiple tables. That query takes 200ms. We can cache the computed feed for 60 seconds and serve it in 1ms from Redis."

3. High database CPU: "Our database CPU is hitting 80% during peak hours just serving reads. The same queries run over and over. Caching the hot queries will cut database load by 70-80%."

4. Latency requirements: "We need sub-10ms response times for the API. Database queries are taking 30-50ms. We have to cache."

How to Introduce Caching?

Once you've established the need for caching, walk through your caching strategy systematically:

1. Identify the bottleneck: Start by pointing to the specific problem caching will solve. Is it database load? Query latency? Expensive computations? Be specific about what's slow and why. "User profile queries are hitting the database 500 times per second during peak hours. Each query takes 30ms. That's our bottleneck."

2. Decide what to cache: Not everything should be cached. Focus on data that is read frequently, doesn't change often, and is expensive to fetch or compute. "We'll cache user profiles since they're read on every page load but only updated when users edit their settings. We'll also cache the trending posts feed since it's computed from expensive aggregations but only needs to refresh every minute." Think about cache keys. How will you look up cached data? For user profiles, the key might be user:123:profile. For trending posts, it could be trending:posts:global.

3. Choose your cache architecture: Pick a caching pattern that matches your consistency requirements. Write-through makes sense when you need strong consistency. Write-behind works for high-volume writes where you can tolerate some risk. "I'll use cache-aside. On a read, we check Redis first. If it's there, return it. If not, query the database, store the result in Redis, and return it." If you're dealing with static content like images or videos, mention CDN caching. If you have extremely hot keys that get hammered, mention in-process caching as an optimization layer.

4. Set an eviction policy: Explain how you'll manage cache size. LRU is the safe default answer. TTL is essential for preventing stale data. "We'll use LRU eviction with Redis and set a TTL of 10 minutes on user profiles. That keeps the cache from growing unbounded while ensuring profiles don't get too stale. If a user updates their profile, we'll invalidate the cache entry immediately."

5. Address the downsides: Caching introduces complexity. Show you've thought about the trade-offs.

   • Cache invalidation: How do you keep cached data fresh? Do you invalidate on writes, rely on TTL, or accept eventual consistency? "When a user updates their profile, we'll delete the cache entry so the next read fetches fresh data from the database."

   • Cache failures: What happens if Redis goes down? Will your database get crushed by the sudden traffic spike? "If Redis is unavailable, requests will fall back to the database. We'll add circuit breakers so we don't overwhelm the database with a stampede. We might also consider keeping a small in-process cache as a last-resort layer."

   • Thundering herd: What happens when a popular cache entry expires and 1000 requests try to refetch it simultaneously? "For extremely popular keys, we can use probabilistic early expiration or request coalescing so only one request fetches from the database while others wait for that result."

How to Secure a Distributed System?

What is Rate Limiting Pattern?

In a network system, a rate limiter is used to control the rate of traffic sent by a client or a service.

• A user can write no more than two posts per second.
• You can create a maximum of 10 accounts per day from the same IP address.
• You can claim rewards no more than 5 times per week from the same device.

A rate limiter prevents DOS attacks, either intentionally or unintentionally, by blocking the excess calls.

HTTP 429 suggests a user has sent too many requests.

Where should we place the rate limiter?

This determines what information we have access to and how it integrates with the rest of our system.

Approach 1:

Each application server or microservice has rate limiting built directly into the application code. When a request comes in, the server checks its local in-memory counters, updates them, and decides whether to allow or reject the request. This is really fast since everything happens in memory, no network calls, no external dependencies.

The main problem is that each server only knows about its own traffic, not the global picture. Say you want to limit users to 100 requests per minute. If you have 5 application servers behind a load balancer, and requests get distributed evenly, each server might see 20 requests per minute from a user and think "that's fine, well under 100." But globally, the user is actually making 100 requests per minute across all servers.

Even worse, if the load balancer changes how it routes traffic, or if one server gets more load than others, your limits become completely unpredictable. A user might get 100 requests through one server and 100 through another, for 200 total.

This approach only works if you have a single application server or if you're okay with approximate limits that can be off by a factor equal to your server count.

Approach 2:

The rate limiter becomes its own microservice that sits between your clients and application servers. The rate limiting service maintains global state, so it can enforce precise limits across all your application servers. If you want 100 requests per minute globally, you get exactly that regardless of how many servers you have.

The biggest downside is latency. Every single request to your system now requires an additional network round trip before it can be processed. Even if the rate limiter is fast (say 10ms), that's still 10ms added to every request. At scale, this adds up.

You've also introduced another point of failure. If your rate limiting service goes down, you need to decide: do you fail open (allow all requests through, risking overload) or fail closed (reject all requests, essentially taking your API offline)? Neither option is great.

There's operational complexity too. You now have another service to deploy, monitor, scale, and maintain. The rate limiting service itself needs to be highly available, which means you need redundancy, health checks, and probably some form of data replication.

Finally, you need to handle network issues gracefully. What if the rate limiter is slow to respond? Do you wait (adding more latency) or timeout and make a guess? What if there are network partitions between your app servers and the rate limiter?

Approach 3:

The rate limiter runs at the very edge of your system, integrated into your API gateway or load balancer. Every incoming request hits the rate limiter first, before it reaches any of your application servers. The rate limiter examines the request (checking IP address, user authentication headers, API keys), applies the appropriate limits, and either forwards the request downstream or immediately returns an HTTP 429 response.

It's the most common pattern and gives us centralized control without adding extra network calls to every request.

NOTE: Since we chose the API Gateway approach, our rate limiter only has access to information in the HTTP request itself. This includes the request URL/path, all HTTP headers (Authorization, User-Agent, X-API-Key, etc.), query parameters, and the client's IP address. While we can technically make external calls to databases or other services, it adds latency we want to avoid so we'll stick to the request itself.

How do we identify different clients so we can apply the right limits to the right users?

We first need to decide what makes a "client" unique. The key we use determines how limits get applied. We have three main options:

1. User ID: Perfect for authenticated APIs. Each logged-in user gets their own rate limit allocation. This is typically present in the Authorization header as a JWT token.

2. IP Address: Good for public APIs or when you don't have user accounts. But watch out for users behind NATs or corporate firewalls. The IP address is typically present in the X-Forwarded-For header.

3. API Key: Common for developer APIs. Each key holder gets their own limits. Most typically, this is denoted in the X-API-Key header.

NOTE: In practice, you'll probably want a combination. Maybe authenticated users get higher limits than anonymous IPs, and premium users may get even more. This is reflective of real systems that don't just enforce a global limit, but layer multiple rules.

How to reject requests when the limit is reached?

HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1640995200
Retry-After: 60
Content-Type: application/json

{
  "error": "Rate limit exceeded",
  "message": "You have exceeded the rate limit of 100 requests per minute. Try again in 60 seconds."
}

Memory Management

Types of Out of Memory errors

Caused of Memory Leaks

Static Collections Growing forever

Unclosed resources

Infinite Caching

Best Practices for Defining APIs

Use Nouns, Not Verbs

❌ Bad
GET /getEmployees
POST /createEmployee
DELETE /deleteEmployee/1

✅ Good
GET /employees
POST /employees
DELETE /employees/{id}

Use Proper HTTP Methods

Define Request and Response Schemas

Define Response Codes Clearly

Add Examples

Specify Content Types