Designing a news feed service using Document DB as storage

The goal: design a news feed service that can quickly generate and fetch both the news feed and the profile feed for every user in a social network.

Model design

• A user table, to store user info.
• A followers table, storing a following id and a follower id, both foreign keys to the user id.
• A post table, storing posts and a user_id foreign key.

Fetching the news feed

To generate the feed for a user when they log in, you fetch everyone they follow and then fetch those users' posts. In SQL that means joining the followers table with the user table on user_id to get the followings, then joining the posts table for all of them to get the posts.

Problems with the initial design

Fetching all posts at login is very time-consuming, especially if the user follows a lot of people. Generating the feed this way also involves multiple queries (three in our case).

Solution: pre-compute the feed

Companies like Twitter solve this by pre-computing the feed for each user using a write fan-out model: when a user posts, that post gets appended to the feeds of all their followers. The pre-computed feed can then be stored in a cache, so whenever the user logs in the feed is already generated and read access is much faster.

Problems with the cache

Even though it scales, a cache has limited memory, and cache memory is much more expensive than disk. To solve for a single point of failure you would need multiple replicas, which adds to the cost.

How Document DB fits in

We can overcome those problems by storing the pre-computed feed in a NoSQL database. Why NoSQL over SQL? We want a flexible schema and we care mainly about scalability here, so we can compromise on consistency and go with an eventually-consistent design. Why a document DB specifically? Because document databases have a flexible schema, they can store documents with different attributes and values.

Using a document database, you can store each user's news feed and profile feed efficiently. When a user adds or removes a post, their document is simply replaced with an updated version. Document databases handle this level of individuality and fluidity easily.

MongoDB and CouchDB are both good choices, the main difference being that MongoDB stores data in BSON (a binary variant of JSON) while CouchDB stores data in JSON. I built a news feed generator using this write fan-out logic with CouchDB and Spring Boot; feel free to reach out if you think there's a better design.