“News feed is the constantly updating list of stories in the middle of your home page. News Feed includes status updates, photos, videos, links, app activity, and likes from people, pages, and groups that you follow”. News feed is the main highlight of social media applications. In this post we’ll learn how to design News Feed System
News feed we will design today will support 3000 friends per user, daily 10 million post requests and image and videos both will be supported. Whenever a person uploads any image/video it will be visible on news feed.
News Feed functions can be divided in two parts:
Generating Feed (Publishing posts): When a user publishes a post, its corresponding data will written to a cache and database. Post will then be published to friend’s news feed.
Building Newsfeed: News feed in our system will be built by aggregating friends’ posts.
For above mentioned functions we will create various APIs:
- API to publish feed: POST API to publish the feed. Its body will contain content (text of the post) and authentication token.
- API to retrieve feed: GET API to retrieve news feed and will contain authentication token in its parameter
Feed Generation/Publishing
User: User1 makes a post with content “ABC” through POST API: /v1/user1/feed?content=ABC&auth_token={auth_token}
Load balancer: distribute traffic to web servers.
Web servers: web servers redirect traffic to different internal services.
Post service: persist post in the database and cache.
Fanout service: push new content to friends’ news feed. Newsfeed data is stored in the cache for fast retrieval.
Notification service: inform friends that new content is available and send out push notifications.
Feed Building
User: User1 sends a request to retrieve her news feed. The GET request looks like this: /v1/user1/feed.
Load balancer: load balancer redirects traffic to web servers.
Web servers: web servers route requests to newsfeed service.
Newsfeed service: news feed service fetches news feed from the cache.
Newsfeed cache: store news feed IDs needed to render the news feed.
Feed Generation Design
Feed Generate/Publish Design
Web servers
It helps in communicating with clients and enforces authentication & rate-limiting. So only users signed in with valid authentication token are allowed to make posts. With rate limiter it can control number of posts a user can make within a certain period, essential to prevent spam and abusive content.
Fanout service
Sending a message to all of your friends is known as fanout. Fanout on write (also known as a push model) and fanout on read (sometimes known as a pull model) are two different types of fanout models. Each model has benefits and drawbacks.
Fanout when writing: This method pre-calculates the news feed while it is being written. The moment a new post is published, it is sent to friends’ cache.
Pros:
- The news feed can be promptly distributed to friends and is created in real-time.
- Because the news feed is pre-computed during writing time, fetching it is quick.
Cons:
- When a user has a large number of friends, retrieving the friend list and creating news feeds for each of them takes time and is sluggish. It is known as the hotkey issue.
- Pre-computing news feeds wastes processing resources for inactive users or those who log in seldom.
Fanout on the read: The news feed is created while people are reading. This model is based on demand. When a user opens her home page, recent posts are withdrawn.
Pros:
- Fanout on read is preferable for people who are rarely or never online because it saves computing resources.
- There is no hotkey issue because data is not pushed to firends.
Cons:
- The news stream cannot be pre-computed, thus it must be fetched.
Hybrid Model can be used for better results:
- For users who has lots of followers, stop fanout write for their new posts. Instead, the followers fanout read the celebrities’ updates.
- When users send new posts, limit the fanout write to only their online followers.
How fanout service is working?
- Retrieve friend IDs from the connection database. Managing friend relationships and friend recommendations is a good fit for graph databases.
- Access the friend’s data stored in the user cache. Afterward, the algorithm excludes friends based on user preferences. For instance, even though you are still friends, muting someone would prevent her posts from appearing in your news feed. A user may choose to share material just with particular friends or keep it hidden from other users, which is another reason why posts might not appear.
- Add the new post ID and friends list to the message queue.
- The message queue is fetched by fanout workers, who also cache news feed data. The news feed cache can be compared to a mapping table. If we keep all user and post objects in the cache, the memory use could increase significantly. Only IDs are saved as a result. We established a programmable restriction to keep the memory size minimal. It is unlikely that a person will scroll through thousands of posts in their news feed. The cache miss rate is minimal since the majority of users are only interested in the most recent content.
- Store <post_id, user_id > in news feed cache
Feed Retrieval Design
CDN : It is used for fast retrieval of stored media content such as images, videos, etc.
How feed retrieval works?
- A user requests to get their news feed back.
- Requests are redistributed to web servers by the load balancer.
- To retrieve news feeds, web servers make a call to the news feed service.
- The news feed cache provides a collection of post IDs to the news feed service.
- The news feed of a user is more than just a collection of feed IDs. It includes the user name, the profile picture, the post’s text and image, etc. In order to create the completely hydrated news feed, the news feed service pulls the whole user and post objects from caches (user cache and post cache).
- The client receives the completely hydrated news feed in JSON format for rendering.