Firestore Many-to-Many: Part 6 - The Final Backend Solution

Firestore Many-to-Many: Part 6 - The Final Backend Solution

14 min read

# Follower Feed Finale I have been writing so many posts about a Follower Feed, I got burned out on the subject. However, here is the finale, which shows you how to do it. I am finally turning [my theory into practice](https://dev.to/jdgamble555/how-to-build-a-scalable-follower-feed-in-firestore-25oj). Okay, not completely, I don't want to build everything, but I do point out the problems in all my options. ## Schema / Data Model Whether you're using a GraphDB, noSQL, or SQL, the data model is still generally the same. You're going to have `posts`, `users`, and `follows`. ```typescript Posts id title createdAt userId ... Users id username ... Followers follower following ``` The Followers Table in SQL will have a unique key based on the composite index of the two fields. In noSQL, since we do not need to query based on one user's followers, but based on who one user is following, we have two possible models: 1. **Subcollection** - `users/{userId}/followers` 2. **Root Collection** - `followers` And, we will create some index collections using FB Functions later. Now, we need to think about what we are trying to accomplish as an end goal. ## Queries These are generally the same, although I have seen more complicated versions using `exists` etc for better indexing. The GraphQL obvious will be a little different depending on the framework **SQL 1** ```sql SELECT * FROM Posts p WHERE userId IN (SELECT following FROM Followers WHERE follower = $UID) ORDER BY createdAt DESC ``` **SQL 2** ```sql SELECT * FROM Posts p JOIN Followers f ON f.following = p.userId WHERE f.follower = $UID ORDER BY createdAt DESC ``` **GraphQL** ```typescript query { queryPost( where: { userId: { follower: { id: $UID } } }, order: { desc: createdAt } ) { id title createdAt ... } } ``` So, what does this mean? Well, we know we really have two many-to-manys, and one connection. ```typescript Posts <- Followers -> Users ``` But, we can't go backwards trying to query the users first, since we need to sort by `createdAt`, which is on the post document / table / node. So, we know we have to, and can only, query the posts... which is ultimately what we are trying to get, just a filtered, sorted version. Technically we can do something similar to this: ```typescript const followersRef = db.collection('followers') .where('follower', '==', $UID); const following = (await followersRef.data()).following. db.collection('posts').where('userId', 'IN', following); ``` But we are limited to following 10 people, and we are really doing two queries on the frontend, instead of a backend join. So, let's add some automatic indexes: ## Indexes First, we are going to add a new collection called `feed` to hold our main indexes, then: 1.) Create `followers_index` ```typescript - Followers OnWrite - arrayIndex function ``` 2.) Update `followers_index` ```typescript - Users onWrite - updateJoinData function ``` See [Scalable Arrays](https://dev.to/jdgamble555/firestore-many-to-many-part-5-scalable-arrays-1hoh) 3.) Create `feed`, which is a special type of post index ```typescript a. Post onWrite - CREATE b. get userId from doc as $UID c. grab all documents from users/$UID/followers_index d. foreach document DOC, create feed/DOC e. DOC contains postId, userId, createdAt, followers array f. Post onWrite - DELETE - make sure to handle deleting all these feed posts when a post is deleted by searching for all feed/docs where postId = id ``` This is pretty basic code. I may go back and add this later, but nothing special here. Your first question may be about C. Keep this in mind: If each `followers_index` holds 10,000 users, then we only grab 100 documents for 1 million followers. If you're lucky enough to have 100 million followers, then yes, you would need to create 10,000 documents. That is actually doable in a Firestore function with my [Bulk Update Function](https://github.com/jdgamble555/adv-firestore-functions/blob/master/HELPER.md) in theory. The most amount of followers ever recorded on any platform is under 200 million. My Bulk Update Function is far from perfect. Since the Firestore Batch Function cops out at [500 actions](https://firebase.google.com/docs/firestore/manage-data/transactions#batched-writes), you need to use my bulk-update function, which basically loops through that. Yes, bulk-update uses `set`, so it can create documents. The other problem with the Bulk Update is that Firestore Functions have a [9 minute limit](https://firebase.google.com/docs/functions/manage-functions#set_timeout_and_memory_allocation) you have to manually configure. You can also speed up the functions in general if you write them [in Go](https://medium.com/google-cloud/firebase-developing-serverless-functions-in-go-963cb011265d). You can use the [Bulk Delete](https://github.com/jdgamble555/adv-firestore-functions/blob/master/HELPER.md) for deleting the feed documents. If the Post document or User Document gets updated in this case, you don't need to do anything. You're just storing their IDs. COMPLEXITY `N` = # followers_index documents (between 1 and 20,000) 4.) Update the connection when a user follows / unfollows This is equally as bad as step 3. If a user follows or unfollows another user, which is actually a rare action, all the feed documents need to be updated. ```typescript a. users/{userId}/followers_index - onWrite b. also add / remove user from all feed docs: .where('userId', '==', userId) .where('followers', 'array-contains', followerID) ``` COMPLEXITY `N` = # Posts / Tweets Youtube max videos (1.4 million), Twitter max tweets (37 million) So, here in lies the real problem with this model... following and unfollowing... However, I think most situations people are creating blogs where most users don't make it to 1000 blog posts. So, how do we solve this problem? I talked about a theoretical [self-triggering function](https://dev.to/jdgamble555/how-to-build-a-scalable-follower-feed-in-firestore-25oj) in my other post. I think this is still possible. I may write it one day, but you really need to look at your data model. ## The Query ```typescript db.collection('feed') .where('followers', 'array-contains', userId) .orderBy('createdAt', 'desc'); ``` You then need to use **rxjs** functions to join the user document and the post document on the frontend with either: 1. A [custom pipe function](https://fireship.io/lessons/firestore-joins-similar-to-sql/) like from Fireship.io 2. You could save the reference docs instead of the docIds and use my [expandRef functions](https://dev.to/jdgamble555/firestore-using-reference-types-for-joins-4194) ## Final Thoughts So what do you do if you MUST use Firestore for a follower feed? It depends on what you want to and can limit. **LIMITS** - [Fireship Model](https://fireship.io/courses/firestore-data-modeling/models-social-feed/) - Limit to only each user's last few posts (5-20ish) - [Version 2](https://dev.to/jdgamble555/firestore-many-to-many-part-4-working-follower-feeds-3k9o) - Limit the number of tweets all together - so not scalable at all - [Version 3](https://dev.to/jdgamble555/firestore-many-to-many-part-4-working-follower-feeds-3k9o) - Limit the number of people a user can subscribe to but not subscribers (my favorite) - [This Version](https://dev.to/jdgamble555/firestore-many-to-many-part-6-the-backend-solution-42io-temp-slug-899082/edit) - Limit the complexity of following / unfollowing, but overall scalable - [Use Extra Database](https://dev.to/jdgamble555/firestore-redisgraph-cloud-the-best-of-both-worlds-405d) - No limits, just not technically using Firestore directly --- Redis Graph here. This is the safest and best option if you are building a serious app. I believe Firestore is made to scale, but not made for complex apps. This is a complex app. So, with a little help from one of the options above, you can still stick with Firestore. **What would I honestly do?** I would use Version 3 for a simple app that is scalable and limit the number of people you can follow. I would not use this version personally, but again, it depends on your needs. Redis Graph is a good option too. **What am I honestly doing?** I am not using Firestore for any serious app. Yes, for play apps, not serious for the reasons [here](https://dev.to/jdgamble555/dear-firebase-team-p17). I love Firestore as a hobby, but IMHO that is all it is. I am about to switch all my production level apps to another database platform... that subject is TO BE CONTINUED IN A FUTURE POST... Let me know any ideas I may have missed, as I tried to include ever detail of possibilities in this post. J
followerfeed
manytomany
datamodeling