System Design (Instagram): Social Networking System

Design a photo and video sharing social media platform like Instagram that enables users to connect, share visual content, and view updates from accounts they follow at scale.

Functional Requirements

1. Post Creation: Users can create posts featuring photos, videos, and a simple caption.

2. Follow Relationships: Users can follow other users.

3. Chronological Feed: Users can view a chronological feed of posts from the accounts they follow.

Non-Functional Requirements

1. Scale: Support 500 million Daily Active Users (DAU) and 100 million new posts per day.

2. Availability: Prioritize high availability for media delivery over consistency (eventual consistency up to 2 minutes is acceptable).

3. Latency: Deliver feed content with low latency (< 500ms end-to-end response time for feed requests) and ensure instant rendering of media content.

Load Estimation

Our load requirements primarily depend on the read traffic for feed requests and write traffic for post creation.

Read Traffic (Feed Requests)

Assume each user opens/refreshes their feed an average of 5 times per day.

1. Total Feed Requests per Day = 500 million DAU x 5 = 2.5 billion req/day
2. Average Read QPS = 2,500,000,000 requests / 86,400 seconds ~ 25,000 req/sec
3. Peak Read QPS (Assuming a peak-to-average ratio of 3x) = 25,000 x 3 = 75,000 req/sec

Write Traffic (Posts)

Average Write QPS = 100,000,000 requests / 86,400 seconds ~ 1000 req/sec
Peak Write QPS (Assuming a peak-to-average ratio of 3x) = 1000 x 3 = 3000 req/sec

Storage Capacity Estimation

Our storage requirements primarily depend on the posts including media such as images or videos and metadata about the post including the caption, user ID, and timestamp.

Media Storage (Photos & Videos)

Assume an 80/20 split between photos and videos. With approximately 100 million media uploads per day, this results in around 80 million photo uploads and 20 million video uploads daily.

Assuming the average compressed image size is 2 MB and the average short video clip size is 10 MB.

Total Daily Media Storage = 80 million x 2 MB + 20 million x 10 MB ~ 160 TB + 200 TB = 360 TB

Metadata Storage (Posts)

Assuming average size of post metadata to be ~1 KB per record (Post ID, User ID, caption, creation timestamp, media URL).

Daily Metadata Volume = 100 million x 1 KB ~ 100 GB

Bandwidth Estimation

The overall bandwidth requirements are primarily determined by the inbound traffic generated by media uploads (ingress bandwidth) and the outbound traffic generated when users download images and videos during feed retrieval (egress bandwidth).

Ingress Bandwidth (Media Uploads)

To estimate the total volume of data entering our system every second, we need a single representative upload size. Since not all uploads are identical, using a simple average would produce an inaccurate estimate. For example, averaging the two media sizes ((2 MB + 10 MB) / 2 = 6 MB) directly incorrectly assumes that photos and videos are uploaded in equal numbers (50% each).

In reality, the upload traffic is heavily skewed toward photos (80% photos and 20% videos). Therefore, each media type should contribute to the average in proportion to how frequently it appears in the upload traffic.

Conceptually, imagine 100 uploads arriving at the system:

• 80 uploads are photos → 80 x 2 MB = 160 MB
• 20 uploads are videos → 20 x 10 MB = 200 MB

The total data uploaded is 360 MB, so the average upload size is 360 MB ÷ 100 uploads = 3.6 MB. This is equivalent to the weighted average calculation:

Average Upload Size
= (80% x 2 MB) + (20% x 10 MB)
= (0.8 x 2) + (0.2 x 10)
= 1.6 + 2
= 3.6 MB

The ingress bandwidth is calculated as:

Ingress Bandwidth
= 1,000 posts/sec x 3.6 MB/post
= 3,600 MB/sec
= 3.6 GB/sec
= 3.6 GB/sec x 8 ≈ 28 Gbps

Egress Bandwidth (Feed Retrieval)

Unlike media uploads, a feed request typically does not return the original high-resolution photos or videos. Instead, it returns a lightweight payload containing compressed thumbnails, post metadata, captions, profile information, and URLs for the media assets.

Assuming each feed page contains approximately 10 posts, the combined payload (thumbnails + metadata) for these 10 posts can be approximately 1 MB.

If the system receives 25,000 feed requests per second, the outbound bandwidth is:

Egress Bandwidth
= 25,000 requests/sec x 1 MB/request
= 25,000 MB/sec
= 25 GB/sec
= 25 x 8 ≈ 200 Gbps

NOTE: Videos are initially represented by preview thumbnails and playback URLs, with the actual video streamed only when the user starts watching.

Design Rationale

Designing a platform like Instagram requires carefully evaluating multiple architectural approaches to address challenges such as handling billions of media objects, serving personalized feeds with low latency, supporting global traffic, and maintaining high availability.

The following design rationale explains the key design decisions, the alternatives considered, and the trade-offs behind each choice.

How can Instagram upload large media files efficiently?

(continue from here...)

Instead of sending media through the backend, the server generates a pre-signed object storage URL. The client uploads directly to storage, after which an event triggers asynchronous processing such as compression, thumbnail generation, and video transcoding. This reduces backend bandwidth usage and allows application servers to remain stateless.

How should Instagram generate the home feed for billions of users?

A hybrid fan-out strategy balances read and write efficiency. Regular users use fan-out on write, where posts are pushed to followers' feed timelines for fast reads. Celebrity accounts use fan-out on read, where posts are merged dynamically during feed requests, avoiding massive write amplification.

How can Instagram provide low-latency image and video delivery worldwide?

Media is stored in object storage and distributed through a Content Delivery Network (CDN). Frequently accessed content is cached at edge locations close to users, minimizing latency and reducing load on origin servers.

Upload a Photo

Follow Users

We can model this relationship with just a Followers table in our database that stores the followerId and followedId. Each time we receive a new POST /follows request, we'll insert a single new row into our table.

View Feed

1. Get followees: Query the Follow table to get a list of user_ids that the current user follows.

2. Get Posts: For each of those followed users, query the Post table to get their recent posts.

3. Merge and Sort: Combine all the retrieved posts and sort them chronologically (by timestamp or postId).

4. Return: Return the sorted posts to the client.

NOTE: These queries would be incredibly slow if we needed to look through every single Followers row for every user and then search through every Post row to find the ones we want. To avoid these full table scans, we can add a few indexes to our database.

1. The client makes a GET request to the API Gateway with the cursor and limit.

2. The API Gateway routes the request to the Post Service.

3. The Post Service queries the Follow table to get the followed users of the current user.

4. The Post Service queries the Post table for each followed user to get their recent posts.

5. The Post Service combines and sorts the posts and returns them to the client, limited by the cursor and limit.

Post Service

"fan-out on read" approach could work for a small app, but it's not going to scale to 500M DAU.

Our first concern would be that for users following 1,000+ accounts, we need 1,000+ queries to the Posts table to get posts from each of their followed accounts.

• Read Amplification: Every time a user refreshes their feed, we generate a large number of database reads. With 500M daily active users refreshing their feeds multiple times per day, this quickly becomes unsustainable. This is going to get expensive fast.

• Repeated Work: If two users follow many of the same accounts (which is common), we're repeatedly querying for the same posts. At Instagram's scale, popular posts might be retrieved millions of times.

Let's put this in perspective with some numbers:

• Each feed refresh might need to process 10,000 posts (1,000 followed accounts × 10 posts/day)
• With 500M DAU, if each user refreshes their feed 5 times daily, that's 2.5 billion feed generations per day
• During peak usage (e.g., evenings, major events), we might see 150,000+ feed requests per second

A much better approach is to precompute the feeds. Instead of generating the feed when the user requests it (fan-out on read), we generate it when a user posts (fan-out on write).

Follow Service

We've added a dedicated Follow Service to handle follow/unfollow operations separately from the Post Service. Since following users happens less frequently than posting and viewing content, this separation lets us optimize and scale each service based on its specific needs.

Sequence Flow

handling post creation requests. It will receive the media and caption, store the post metadata in the database, the actual bytes on the media in a blob store, and return a postId to the client.

1. The client makes a POST request to the API Gateway with the media and caption.

2. The API Gateway routes the request to the Post Service.

3. The Post Service receives the media and caption, stores the post metadata in the database, and the actual bytes on the media in a blob store.

4. The Post Service returns a postId to the client.

High-Level Design

API Design

Create Posts

POST /posts -> postId
{
  "media": {photo or video bytes},
  "caption": "My cool photo!",
}

Follow Request

POST /follows
{
  "followedId": "123"
}

View Feed

GET /feed?cursor={cursor}&limit={page_size} -> Post[]

We'll use a cursor for pagination, and a limit to control the page size.

Data Model Design

Post Table

For the Post table, we'll make the partition key the userId since most of our queries will be to get the posts of a given user. We can make the sort key a composite of createdAt and postId to ensure chronological ordering while maintaining uniqueness.

Follow table

For the Follower table, we'll make the partition key the followerId and the sort key the followedId. This allows us to efficiently query for all users that a given user follows.

Similar Case Studies

1. Design Twitter

2. Design Facebook