Firestore Many-to-Many: Part 4 - follower feeds

Firestore Many-to-Many: Part 4 - follower feeds

13 min read

So, I made a theoretical version of a scalable follower feed [here](https://dev.to/jdgamble555/how-to-build-a-scalable-follower-feed-in-firestore-25oj). However, I need to put the theory into practice. Sometimes while putting the pedal to the medal, you realize your theory is wrong. So, I am going to start from simple and work my way to more complex. This post is simply about a working version before I end up writing some complex Firebase Functions. Let's get something actually working shall we! ## Version 1 The only working version I have seen is [Fireship.io](https://fireship.io/courses/firestore-data-modeling/models-social-feed/). I highly recommend you check it out, as all his courses are great. Here he basically has a user's last 3 posts aggregated on the follower doc in a followers collection. He also has a followers array and uses `array-contains` to find the relevant userId. He then sorts the results with all the latest 3 posts from his followers. Check out his course for his code, great stuff there. The problem with this version is that you can only see aggregated posts (3 in his example) from a user. They are not necessarily sorted, and a user is limited by the number of followers. What if my top 10 posts are from 1 user? ## Version 2 So, without using any backend indexing, let's try a different approach. This approach will limit the number of people you can subscribe to, but a user can have as many subscribers as possible. This approach also displays sorted posts, always up-to-date, and requires no backend. However, it will do a lot of reads and sorting on the front end. This can work for some situations, but I am posting it to be thorough. Note: There are limits to other social media's number of people you can follow: - Twitter - 5,000 - Instagram - 7,500 - Facebook - 5,000 - Snapchat - 5,000 - TikTok - 2,000 I googled these answers, so I apologize if I am not perfect on the numbers. You get the gist. There are also daily follow limits. **So here is the basic model** **userDoc** ```typescript users/{userId} --> { displayName: 'Jon', email: 'tester@me.com', following: [ 32321a2, f2232k3, fkelses, ... ] } ``` Keep the users you are following in your user doc. All of them are readily available in an array by reading one doc. Let's give a 500 limit for the sake of argument. You could possibly store more than 10k, but that is not why we have the limit as you shall see. **postDoc** ```typescript posts/{postId} --> { title: 'some posts', content: 'some stuff here', creatorId: 'user id of creator' } ``` This could obviously be snap, or tweet, or whatever. Now let's use the tactics from [Part 3](https://dev.to/jdgamble555/firestore-many-to-many-part-3-maps-23nh) of this series. #### Query First get the followers from the user doc... ```typescript const followers = (await this.afs.doc(`users/${userId}`) .ref.get() as any).data().followers; ``` If this were a where AND clause, no problem. However, since this is an OR clause, which Firestore does not support, we are stuck with frontend indexing... ```typescript const result = combineLatest(followers.map( (student: string) => this.afs.collection('posts', (ref: any) => ref.where('creatorId', '==', follower) ).valueChanges({ idField: 'id' }) )).pipe( // combine results map((a: any[]) => a.reduce( (acc: any[], cur: any) => acc.concat(cur) // filter duplicates ).filter( (b: any, n: number, a: any[]) => a.findIndex( (v: any) => v.id === b.id) === n // sort by createdAt ).sort((a: any, b: any) => { const f = 'createdAt'; if (a[f] < b[f]) { return -1; } if (b[f] < a[f]) { return 1; } return 0; })) ); ``` But we have **severe** limitations here as well. We are basically grabbing all posts, and filtering and sorting them on the front end. If this were an AND query, it would be no problem. But Firestore does not support any kind of OR queries except **array-contains**, which is limited to 10. Back to the drawing board... ## Version 3 So I had another brainstorm [here](https://stackoverflow.com/a/66783307/271450) where we index a feed on the frontend by date, then just read the feed. I have been thinking about this problem for a long, long, time. 😩 Let's see if this version can work (theoretical up to this point). **Model** ``` users/{userId} => { displayName: 'Jon', following: [ 3929293ssks, jeons202swpeo, ... ], ... } posts/{postId} => { createdBy: 2l2l2l32 ... } users/{userId}/feed ``` There is another step here similar to Version 2. The difference is we only have to aggregate a user's feed so we don't copy the same posts over and over. **Get Docs to Aggregate** ```typescript // how many days old const x = 100; const date = new Date(Date.now() - x * 24 * 60 * 60 * 1000); const feed = (await combineLatest(authors.map( (author: string) => this.afs.collection('posts', (ref: any) => ref .where('authorId', '==', author) .where('createdAt', '<', date) ).valueChanges({ idField: 'id' }) )).pipe( take(1), // combine results map((a: any[]) => a.reduce( (acc: any[], cur: any) => acc.concat(cur) // filter duplicates ).filter( (b: any, n: number, a: any[]) => a.findIndex( (v: any) => v.id === b.id) === n )), ).toPromise() as any[]) // all we need is id and createdAt, user might update their post .map((r: any) => { return { id: r.id, createdAt: r.createdAt } }); ``` **Save last update date** ```typescript // get userId const userId = (await this.afa.user.pipe(take(1)) .toPromise())?.uid; // save now date this.afs.doc(`users/${userId}`) .set({ lastFeedUpdate: firebase.firestore.FieldValue.serverTimestamp() }); ``` You should probably have a button called "Update Feed" that does all this. To make the above code more concise... **New Date or Use `lastFeedUpdate`** ```typescript // get user const userId = (await this.afa.user.pipe(take(1)) .toPromise())?.uid; // get last update time const lastFeedUpdate = (await this.afs.doc(`users/${userId}`) .valueChanges().pipe(take(1)).toPromise() as any).lastFeedUpdate; // use new date if never created const date = lastFeedUpdate ? lastFeedUpdate : new Date(Date.now() - x * 24 * 60 * 60 * 1000); ``` **Save Data and Create Feed** ```typescript const batch = this.afs.firestore.batch(); feed.map((data: any) => { // save each doc in feed const newDoc = this.afs.doc(`users/${userId}/feed/${data.id}`).ref; batch.set(newDoc, data); }); await batch.commit(); ``` You could save the whole doc here and just read the feed as any collection, but then the posts may not be up-to-date. **Read the Feed** ```typescript const read = this.afs.collection(`users/${userId}/feed`, ref => ref.orderBy('createdAt') ).valueChanges({ idField: 'id' }).pipe( // map each id to a post doc switchMap((r: any[]) => combineLatest(r.map( (d: any) => this.afs.doc(`posts/${d.id}`).valueChanges() ))), ); ``` You might want to keep it an observable so it is always up-to-date, or at the very least re-run the get promise if there is an update. If you get a result of 'undefined', you may want to display a blank doc, or 'user has removed their post' UX. So I proved 2 types of follower feeds are possible. > But what about a scalable follower feed!!!!???!!! So, I am going to work on my [theoretical feed](https://dev.to/jdgamble555/how-to-build-a-scalable-follower-feed-in-firestore-25oj) using Firebase Functions. I am going to try and simplify everything I can, and I have to start with just one feature at a time. TO BE CONTINUED... J
rxjs
followerfeed
index
datamodeling
manytomany