Collecting Logs and putting in storage
Message Transportation — Kafka
Data Transportation — Out of Kafka Store in Object Store
Solution — Replication at Kafka
Decouple System — Logging Agent failure to Application Failure
Minimize impact of message queue failure
Dynamic Configuration Detection — Adding new pipelines, removign existing pipelines
Adjustable Logging latency
Facebook Algorithm to count distinct elements — Approximation Algorithm
Simplifies cardinality estimation, or count-distinct problem. Ex count number of views on a webpage. Takes input data and hashes it
A HyperLogLog is a probabilistic data structure. It counts the number of distinct elements in a list.
In order to count values with exact precision you need memory similar to size of unqiue values as the only way to detect if the value is already seen is by comparison with the previous value.
Since memory is a limited resource, doing this becomes problematic when working with large sets of values.
A HyperLogLog solves this problem by allowing to trade memory consumption for precision with a standard error of 2% using only 1.5 kilobytes of memory
Always On Open Connection
Tradition Start thread for per incoming connection, blocking read write, exhaust memory, allocating 1000 stacks for 1000 stack, pin down cpu by context switching
Non Blocking Async IO
Read and write all callbacks for all open connections on single thread, scales much better
Application complicated, developer to keep track of all threads ,cant use thread stack as thread shared between connections , do that by event or state machine , Use Netty
Store in Redis, Serialize Mapping
Data Store should have following char:
Kafka — Message Broker , Decouple…
News Feed is a feature of the social network Facebook. The web feed is the primary system through which users are exposed to content posted on the network. News Feed highlights information that includes profile changes, upcoming events, and birthdays, among other updates. (Source Wiki)