Scaling From One Server to Millions of Users

Most real systems didn’t start distributed. They started on one server and grew. Retracing that growth is the best way to understand why each component exists, and it is exactly the story an interviewer wants you to tell.

Stage 1: Everything on one machine

flowchart LR
  U[Users] -->|DNS → IP| S["One server<br/>web app + database"]

This is fine for thousands of users. Problems: the app and the database fight for CPU and memory, and if the machine dies the whole product is down.

Stage 2: Separate the database

Move the database to its own machine. Now each can be sized for its job (the DB wants memory and fast disks; the app wants CPU), and they scale independently.

Vertical vs horizontal scaling

• Vertical scaling (scale up): buy a bigger machine. Simple and needs no code changes, but there is a ceiling, cost grows faster than capacity, and it remains a single point of failure.
• Horizontal scaling (scale out): add more machines. No hard ceiling and it tolerates failures, but you need a load balancer and your app must be designed for it.

Real systems use both: scale up until it gets expensive, then scale out.

Stage 3: A load balancer and many app servers

flowchart LR
  U[Users] --> LB[Load balancer]
  LB --> A1[App 1]
  LB --> A2[App 2]
  LB --> A3[App 3]
  A1 & A2 & A3 --> DB[(Database)]

The load balancer spreads requests across app servers and stops sending traffic to any that fail a health check. For this to work, app servers must be stateless: they must not keep anything about a user (sessions, uploaded files, in-memory carts) that the next request needs. Put session data in a shared store such as Redis or the database, and files in object storage. Then any server can serve any request, and you can add servers in seconds.

Stage 4: Read replicas

The database is now the bottleneck. Most apps read far more than they write, so add replicas: copies of the database that apply the primary’s changes and serve reads.

flowchart LR
  A[App servers] -->|writes| P[(Primary)]
  A -->|reads| R1[(Replica 1)]
  A -->|reads| R2[(Replica 2)]
  P -. replication .-> R1
  P -. replication .-> R2

Trade-off: replication lag. A user who just updated their profile may read the old version from a replica a few milliseconds later. Fixes: read your own writes from the primary, or route a user to the primary for a few seconds after they write.

Replicas also give you failover: if the primary dies, promote a replica.

Stage 5: A cache

Even replicas hit disk. Put frequently read data in memory with a cache like Redis or Memcached in front of the database. A cache hit takes well under a millisecond, while a database query takes several. With a 90% hit rate, the database sees one tenth of the reads. (Caching gets its own lesson.)

Stage 6: CDN for static content

Images, video, CSS and JavaScript don’t need your servers at all. A CDN copies them to edge locations near users, such as Mumbai, Chennai and Singapore, so they load fast and never touch your origin.

Stage 7: Asynchronous work with queues

Sending emails, resizing images and generating reports shouldn’t make the user wait. The app puts a job on a message queue and returns immediately; workers process jobs in the background and can be scaled separately.

Stage 8: Sharding

When writes or total data outgrow one primary, split the data across several databases, for example users A–M on shard 1 and N–Z on shard 2, or better, by a hash of the user ID. This is powerful but expensive in complexity: cross-shard queries and transactions become hard, and rebalancing takes work. Do it last.

Stage 9: Multiple data centres

For lower latency worldwide and to survive a whole region going down, run the stack in several regions with geo-DNS routing users to the nearest one. The hard part is keeping data in sync across regions.

The whole picture

flowchart TB
  U[Users] --> DNS[DNS]
  U --> CDN[CDN: static files]
  DNS --> LB[Load balancer]
  LB --> A1[App] & A2[App] & A3[App]
  A1 & A2 & A3 --> C[(Cache)]
  A1 & A2 & A3 --> P[(DB primary)]
  A1 & A2 & A3 --> RR[(Read replicas)]
  A1 & A2 & A3 --> Q[[Queue]] --> W[Workers]
  W --> OS[(Object storage)]