• Minimal Data loss
  • Eventually consistent
  • Scalable
  • Low latency
  • Minimum operation overhead

Collecting Logs and putting in storage

App Server

Message Transportation — Kafka

Data Transportation — Out of Kafka Store in Object Store


  • Operability — Broker go down, Rolling restart so restart all related service

Solution — Replication at Kafka


Decouple System — Logging Agent failure to Application Failure

Minimize impact of message queue failure

Dynamic Configuration Detection — Adding new pipelines, removign existing pipelines

Adjustable Logging latency



Facebook Algorithm to count distinct elements — Approximation Algorithm

Simplifies cardinality estimation, or count-distinct problem. Ex count number of views on a webpage. Takes input data and hashes it

A HyperLogLog is a probabilistic data structure. It counts the number of distinct elements in a list.

In order to count values with exact precision you need memory similar to size of unqiue values as the only way to detect if the value is already seen is by comparison with the previous value.

Since memory is a limited resource, doing this becomes problematic when working with large sets of values.

A HyperLogLog solves this problem by allowing to trade memory consumption for precision with a standard error of 2% using only 1.5 kilobytes of memory


Web Server

Always On Open Connection

Tradition Start thread for per incoming connection, blocking read write, exhaust memory, allocating 1000 stacks for 1000 stack, pin down cpu by context switching

Non Blocking Async IO

Read and write all callbacks for all open connections on single thread, scales much better

Application complicated, developer to keep track of all threads ,cant use thread stack as thread shared between connections , do that by event or state machine , Use Netty

Push Register

Store in Redis, Serialize Mapping

Data Store should have following char:

  • Low Read Latency
  • Record Expiry
  • Sharding
  • Replication

Message Processing (Routing, Queue , Deliver)

Kafka — Message Broker , Decouple…

Curated list of information from HighScalablity Facebook Twitter


  • Not Scalable
  • Easy system overload if we poll every five second

Push Based Approach

  • Storing Association between user and content is easy but keeping up with 16 million connections per second is not.
  • Every page load requires multiple writes and each write requires a read to make sure the…


News Feed is a feature of the social network Facebook. The web feed is the primary system through which users are exposed to content posted on the network. News Feed highlights information that includes profile changes, upcoming events, and birthdays, among other updates. (Source Wiki)

Functional Requirements

  • Users should be able to see live posts
  • Users should be able to scroll down
  • User should be able to like and comment
  • Need Feed should display text, video , links and images

Non Functional Requirement

  • Low Latency
  • Eventual Consistency
  • Real time news feed in 2sec and…


YouTube is the most popular video streaming website worldwide. With over 2 billion monthly active users the scale at which youtube operate is massive.

Functional Requirements

  1. Upload Metadata and Thumbnail
  2. Like , Subscribe and Comment on Videos
  3. Smooth Video Streaming

Non Functional Requirement

  1. Eventual Consistency
  2. Average Latency 200ms
  3. Highly reliable and no uploaded should be lost

Capacity Estimation

  1. Users watch 5 videos per day
  2. Out of those 5 million users 200,000 users upload 1 video per day
  3. Average size of video 200MB
  4. Total disk space 200,000 x 200MB = 40TB

High Level Design

Simplest view of youtube architecture (further read)
  • User — Phone, Desktop, Tablets

Architect Within

Senior Manager, Director, CTO

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store