Design a conversational AI system similar to ChatGPT that allows users to have natural language conversations and receive intelligent responses.
NOTE: We treat the LLM as a black box we call, not something we train or run the internals of.
Functional Requirements
Create Chat
Send Message
- Users should be able to view past chats and resume a conversation, with the chat's prior context carried into the prompt.
NOTE: We'll also scope this to text in, text out only, with no images, audio, or video, and no editing or branching of existing messages.
Non-Functional Requirements
-
Latency to the first token matters more than total completion time.
-
Serve ~200M daily active users.
5.8 billion visits per month
1 billion weekly active users sending 2.5+ billion prompts per day
Your database can't store 500TB per year. API costs will hit $145 million per month. And you'll need 11.4 million concurrent connections that no single server can handle.
Sequence Flow
- A user types a prompt, the system runs it through a large language model with the prior turns of the chat as context, and tokens stream back as they are generated.
The user might create a new chat session or might have existing chat session.
Load Estimation
- 10K new messages per second at peak.
GPUs are scarce and load is bursty, so requests have to be queued, batched, and shed under pressure rather than dispatched one per worker.
Data Model
users chats
Design Rationale
ChatGPT's workload is 90% reads, 10% writes. Hence, we can use Single Primary PostgreSQL (not sharded).
A session storage being used for storing session information (e.g., conversation history, user preferences) across multiple requests. For each user request chatGPT retrieve the relevant session and inject it into the model's input to maintain context.
High Level Architecture
Conversation Service
Massive write throughput. Chat history is append only??
Context Builder Service
Memory Service
Memory service is semantic. ??