Relevant

Log aggregation is the process of collecting logs from many different machines/services and sending them to a central system where they can be searched, analyzed, visualized, and alerted on.

Without aggregation, the engineer responding to the incident has to know where every log lives, have access to every host or tool, and line up timestamps by hand. That is slow on a good day and painful during an incident.

Logs Aggregation & Alerting System

Functional Requirements

Design a logs aggregation system that:

Non-Functional Requirements

Entire System generates 100,000 logs/second. But should be able 1M logs/second spikes in case of unexpected failures.

Design Rationale

Q. How does log aggregation work?

Applications
     ↓
Log Agents
     ↓
Kafka / Buffer
     ↓
Log Ingestion Service
     ↓
┌──────────────────────────────┐
│      Central Log System      │
│                              │
│  Parsing → Indexing → Store  │
└──────────────┬───────────────┘
               ↓
        Search / Analytics

You typically install an agent on each machine or run a collector alongside your workloads. For example:

EC2
│
├── payment-service.log
├── nginx.log
└── system.log
       │
       ↓
   Fluent Bit
       │
       ↓
   Central system

The Collector (Fluent Bit) collects logs from the servers and sends them to the central system. Suppose your application produces "Payment failed: Insufficient balance" message. That's not very useful by itself. THe collector can enrcih it with information such as:

{
  "message": "Payment failed",
  "timestamp": "2026-08-19T13:20:05Z",
  "service": "payment-service",
  "environment": "production",
  "host": "ip-10-20-1-15",
  "region": "ap-southeast-2",
  "pod": "payment-7d8f9",
  "level": "ERROR"
}

Now you can search:

service = payment-service
environment = production
level = ERROR

The simplest mental model is:

Server 1 ──┐
Server 2 ──┤
Server 3 ──┼──→ Log Collector ──→ Log Storage/Index ──→ Search/Dashboard
Server 4 ──┤
Server 5 ──┘

The central platform perform certain level of parsing to convert raw text into structured key value pairs. The system also create indexes so that the searches are fast.

The actual log data is stored separately from the index

             Log System
                 │
        ┌────────┴────────┐
        ↓                 ↓
   Log Storage         Indexes
   "actual data"       "where to find it"

Q. What is the significance of correlation ID?

A request could generate logs across 5 services. If we attach a correlation ID to the request, every service logs it:

API Gateway
correlationId=ABC123

Order Service
correlationId=ABC123

Payment Service
correlationId=ABC123

Banking Service
correlationId=ABC123

Now searching correlationId=ABC123 will give you an idea of the entire request journey.

Q. How and when to scale metrics collector?

Q. How adding queuing system help with handling time series db failure?

Q. Workflow of Splunk?

Fluent Bit
     ↓
Kafka
     ↓
Log Processing
     ↓
Elasticsearch

equivalent to:

Application
    ↓
Fluent Bit
    ↓
Splunk

Q. What is Index in Splunk?

Used to specify which index (data repository) to search and store in. When data is added, it is stored in index for efficient searching.

Q1. An API that consistently responded within 200 ms before deployment now takes 4-5 seconds after a new release. CPU, memory, and disk utilization remain within normal limits. How would you approach debugging this issue?

A. Normal CPU and memory usage usually means the application isn't busy computing, it's often waiting. Common wait sources include database queries, locks, external service calls, network latency and connection pool availability.

Q2. The response time of an application keeps increasing over time, even though the traffic remains stable. What could be happening?

A. Memory Leak is the most common cause. The application keeps allocating memory but doesn't release it, gradually increasing heap usage.

Objects which have references but are not in use are the cause of memory leak. For example, excessive static variables usage, connection pool leaks, thread pool exhaustion, resource leaks (file), garbage collection overhead, database lock contention.

Extra