Awsome video dude! Just one thing missed CDC from TOPICS table to Flink using message queues partitioned on userID.
@fangao9977
Жыл бұрын
Questions here: I'm assuming that the publishMsg API will finally trigger the service to push SSE msg to the Frontend Users, so the consumer servers as the ones who pushes the messages, but how does it know that the client is connecting to itself, what if the notification consumer picks up a message and find that the client is not connected to it ? Is it being controlled by the UserId Sharding queue ? so that the queue always know which consumer to send to ?? (and what if a consumer is down, does the queue know it and take care of the reconnected consumer ?)
@jordanhasnolife5163
Жыл бұрын
Yeah you basically said it. We're partitioning based on userID. If the consumer goes down, consistent hashing will ensure that those messages and the client frontend will connect to a different consumer, but the same one.
@AnkitGaurav-qt1hs
6 ай бұрын
What changes would you do in your design if a requirement comes in where notifications have different priority, like P0 notifs need to be sent before any P1 and P2 notifictaion?
@jordanhasnolife5163
6 ай бұрын
Probably just ensure that they're going to the same kafka queue via some sort of partitioning key. Kafka queues are consumed one at a time, so that should ensure the ordering for me. If you mean that you expect the P2 to be delivered after a P0 but it is published first, well I think you'd have to clarify your requirements a bit more. Clearly you'd need to buffer the events for a bit and use some sort of priority queue to occasionally poll one and send it to the world.
@dibll
Жыл бұрын
Why do we need to use Zookeeper to store metadata like partitions in a topic , IP Addresses of those partitions etc. Why can't we use a DB for that purpose? What are the advantages of using Zookeeper over a normal key value DB
@jordanhasnolife5163
Жыл бұрын
Zookeeper is strongly consistent which is very important to have when dealing with actual configuration of your architecture so you don't make stale reads Recommend watching my ZooKeeper video
@TheImplemented
Жыл бұрын
Did you intentionally skipped Subscriptions table (user_id, topic_id) ?
@jordanhasnolife5163
Жыл бұрын
I included a parameter userlist in the topics table that does that but yeah this is probably better
@vanderer_prashant
7 ай бұрын
I see only Flink in between topics message queue and user message queue in the diagram. Who puts message from topic MQ put into user MQ?
@jordanhasnolife5163
7 ай бұрын
Yeah very possible I forgot it here, you'd probably want some other stateful consumer that is aware of which queue a message from a given topic should be routed to.
@prateekaggarwal3305
11 ай бұрын
Can you desribe a bit more about how Configuration service is helpul here, like what information it holds and how that information is used.
@jordanhasnolife5163
10 ай бұрын
Mostly important for your sharing config. Each device is connected to one notification server, so we need to always be aware of that mapping.
@Gerald-iz7mv
8 ай бұрын
what about celebrity problem? ... would you do pull vs push in this case? you also mention user queues - how do you batch the messages for a user? ... imagine the user doesnt want to receive 20 messages per day - instead 3 or 4 message might be a better user experience...
@jordanhasnolife5163
8 ай бұрын
Yeah I think using dedicated celebrity queues which the user connects to makes a lot of sense there. If you did need to batch messages, you could use stream processing to do some sort of time windowed joining (see the stream processing concepts video)
@Gerald-iz7mv
8 ай бұрын
@@jordanhasnolife5163 how does the celebrity queue work - one queue per celebrity? how does the message get to the 100 million followers than?
@ariali2067
6 ай бұрын
Curious why we use SSE/websocket here instead of long polling? I would assume notification is more like uni-directional communication instead of bi-directional here.
@ariali2067
6 ай бұрын
Also not sure whether I missed anything, is HBase use single leader replication? It mentioned in this video but iirc this is the first time I heard of this.
@jordanhasnolife5163
6 ай бұрын
Long polling is uni directional.
@jordanhasnolife5163
6 ай бұрын
Yeah H-Base replicates in the same way that HDFS does which means that there's typically a single place where writes first go to for a given file.
@ariali2067
6 ай бұрын
Yeah but isn't notification more like uni directional pattern which push notification from server to client? So why we don't use long polling here?@@jordanhasnolife5163
@SlashTagged
Жыл бұрын
On using bloom filters for idempotence: Wouldn't we start getting a lot of false positives at some point (due to hash collisions)? That would lead to erroneously rejecting non-duplicated notifications, which seems counterintuitive to idempotency. We're essentially being so "aggressive" that we're actually rejecting "good" (read: non-duplicated) notifications.
@jordanhasnolife5163
Жыл бұрын
While this is true, you can always just occasionally reset the bloom filter, say every hour. Then you no longer have to deal with this issue.
@SlashTagged
Жыл бұрын
@@jordanhasnolife5163 I had thought about that but was worried about the case when you clear the BF but then process an actual duplicated notification. Probably an edge case but worth mentioning nonetheless. Appreciate the reply and thanks for putting in the time to make all these videos!
@jordanhasnolife5163
Жыл бұрын
@@SlashTagged I think in the worst case scenario you'd just check if you've seen it before - remember the bloom filter can only tell you with certainty if you HAVEN'T seen something, so we have to check again if there's a chance we have seen it anyways
@sanampreetsingh9638
2 жыл бұрын
Great design ! Of restricting user to just one web socket connection with server by queuing messages ! Couple of questions ! 1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock , so our general topic can be product in stock but user is only interested for product X in stock notification ! ? Thanks
@jordanhasnolife5163
2 жыл бұрын
I think that the consumer node would basically just have to go ahead and read it and determine whether it needs to send the message to any users.
@TOKYODRIFT00000
11 ай бұрын
I am trying to implement an API for one of the notification services. Let me explain the architecture. There is a gateway service that has methods for creating configurations. A person can interact with the service and create a configuration for sending a message via email, for example. Then the person calls the Notify method, which accepts the following DTO: 1. Consumer information 2. Transport configuration 3. Message: [type (simple, broadcast), content] The Notify method looks at the transport and understands that the message should be sent using the email service, so it sends the DTO to the email service. Ok, so I described how gateway and email transport service work. However, there is a transport called "client" that has not been implemented yet, and it needs to be designed. The main idea is that the client can be a web application, some desktop application, etc. It is clear that delivery will happen through a message broker, but how will it all look? I don't quite understand
@jordanhasnolife5163
11 ай бұрын
I'm confused: you're just asking how the message gets to the client after someone tries to send them a notification?
@TOKYODRIFT00000
11 ай бұрын
@@jordanhasnolife5163 maybe I’m bad in explanation and my english also bad but yes! Lets imagine we have rabbitmq broker with different topics. We want to send notification to a user so what topic to use? Maybe you have an example api?
@jordanhasnolife5163
11 ай бұрын
@@TOKYODRIFT00000 You can make a topic per user
@TOKYODRIFT00000
11 ай бұрын
@@jordanhasnolife5163imagine user is offline, we keep all notifications in the database, but what if there are a lot of them? if a person becomes online and calls fetch_notification, they will immediately receive a lot of notifications and this is bad. how to solve this problem?
@VivekMahadevanv
2 жыл бұрын
Followup question.. Why do you think a userId sharded 2nd level of MQs' are necessary? In most practical applications such as say chat for example, the notifications still go through multiple mediums based on what the user subscribes to (say push notification, email but not SMS). This information could be made part of the payload as tags that the broker can aggregate on for easier consumption. For any medium, it would then make sense to have a storage mechanism to hold messages that were not delivered. I don't think the current design saves any resources on the server(threads?) on the server either since you probably need to maintain a connection with each device the user has in order to send the message. Thoughts?
@jordanhasnolife5163
2 жыл бұрын
Yep, the point was actually to optimize for the client device. If a client has to maintain an active connection for every single topic that they are subscribed to, they need to maintain potentially hundreds of active connections, which to me at least seems excessive. I'd agree that while perhaps a second set of message queues aren't necessary, it seems pretty clear to me at least that the client should only be pulling from one device as opposed to actively maintaining many TCP connections. For a phone at least, that could really use a lot of data/drain battery life.
@VivekMahadevanv
2 жыл бұрын
@@jordanhasnolife5163 interesting. How many of these connections must one hold ? Unless it's real time (chat or something of that nature) isnt it mostly a fire and forget (email/sms) . Either ways worth having a conversation with the interviewer I guess ..
@jordanhasnolife5163
2 жыл бұрын
@@VivekMahadevanv If it's fire and forget I agree - however I've seen notification service talks that assume you're actually directly pushing to the service, in which case all the messages for a user have to go to the same place
@simin2371
2 жыл бұрын
@@jordanhasnolife5163 isn't the second queue also necessary to ensure the order of delivered messages for a given user? the first queue is topic-based, this is necessary as it enables per-topic heavy post-processing right after the first queue, which is quite likely. Then the second queue makes sure messages for each user are in the same queue and shard, hence ensuring the order of message consumption. Finally the notification sender server is 1-to-1 mapped to a user partition so to have only one websocket per mobile client. Am I right?
@jordanhasnolife5163
2 жыл бұрын
@@simin2371 That was my logic, thanks for helping to communicate it!
@povdata
10 ай бұрын
Thank you for video. I still have one question. How to to avoid sending duplication messages? You say that unique id is attached to each notification message. How to use it then? It which step?
@jordanhasnolife5163
10 ай бұрын
A consumer will receive messages on a message queue, it will then cache the unique IDs locally. If it's seen that I'd before, don't handle the message again.
@povdata
10 ай бұрын
@@jordanhasnolife5163 So, where is cached? Because it might be some consumers (backend) and each one caches locally, so one consume does not know about cach of another, right?
@povdata
10 ай бұрын
and how is possible to get duplicate message if message is fetched from queue can not be added back?
@jordanhasnolife5163
10 ай бұрын
@@povdata if the message gets delivered to a node but the node is unable to send an ack back to the broker due to network issues the broker may deliver it a second time
@povdata
10 ай бұрын
@@jordanhasnolife5163 Thank you, but I did not get about how to send message using SSE to concrete client user? Could you explain please?
@jayshah5695
2 жыл бұрын
we should probably add a notification acking mechanism from client back to service?
@jordanhasnolife5163
2 жыл бұрын
I think this is definitely useful from the perspective of not dropping notifications, though if it's okay to drop the occasional notification then it isn't a huge deal if we don't acknowledge received notifications and stick them in a database somewhere.
@jayshah5695
2 жыл бұрын
@@jordanhasnolife5163 we should discuss delivery rate and delivery SLA as part of requirements I guess.
@jordanhasnolife5163
2 жыл бұрын
@@jayshah5695 Very fair point
@sanampreetsingh9638
2 жыл бұрын
Great design ! Of restricting user to just one web socket connection with server by queuing messages ! Couple of questions ! 1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock , so our general topic can be product in stock but user is only interested for product X in stock notification ! ? 2- we are keeping Queue per user design which might hog resources as user base increases? Thanks
@jordanhasnolife5163
2 жыл бұрын
1) There will likely have to be some logic by a consumer to figure out which messages are relevant for which user 2) There's not 1 queue per user, there's just 1 user per queue. You can have many users getting messages through the same queue
@Glashutte1900
Жыл бұрын
The problem is that we have a chat app and when the app is in kill state in android or ios the push notification does not work so we go for FCM and we dont want to use firebase as google data issiues so do you think it is possible to build your own push notification in kill state like fcm. i will be looking forward for your answer
@jordanhasnolife5163
Жыл бұрын
Not sure - I imagine there's a way but from what I know about mobike dev most people outsource their push notifications to FCM, one service (I think it's called), or SNS
@Glashutte1900
Жыл бұрын
@@jordanhasnolife5163 Thanks for your kind reply
@bet_more_pls
2 жыл бұрын
How would you deal with spiky traffic - e.g. load increases by 10-100x because of some special event (black friday sales or something)?
@jordanhasnolife5163
2 жыл бұрын
Well at the end of the day your only option for something like that is to add more resources, right? Assuming you're doing this in AWS I'd imagine you'd have some autoscaler to give you more when traffic increases.
@sanampreet3045
2 жыл бұрын
Hi Mate , can u please elaborate more on local state management using flink here ? Like consumers are sharded on hash of user id and topic has list of users to notify , how state management is happening here !
@jordanhasnolife5163
2 жыл бұрын
Basically the consumers know for each userId, which topics they are subscribed to, so they can easily send messages to the appropriate queues. We just get better data locality out of doing this than having to consult a database for this information. It effectively is just a cache.
@imutkarshy
2 жыл бұрын
Following up on the user id shards. Do you mean using user ids as partitions for 2nd log based queue? If yes, then will it not be unoptimised to have 1 billion partitions (as per capacity planning) even tho there is no limit on the numb of partitions?
@jordanhasnolife5163
2 жыл бұрын
I think that you're maybe slightly misinterpreting - being sharded by userId doesn't mean that there will be only one userId per partition, but rather that we just partition on the hash range of the userId field - there could potentially be many userIds per partition, it's just important that all the messages for a given userId are sent to the same node so that we can keep a persistent websocket connection on it
@imutkarshy
2 жыл бұрын
Thanks, @@jordanhasnolife5163 for the quick reply. It's making sense now. These videos are super helpful. Keep them coming. Love from India
@PerfectSylex
10 ай бұрын
hello everyone. this is application servar and web server. which one do you like more? web servae is my crush
@xmnemonic
Жыл бұрын
Good video. Can hear siren in background at 1:55.
@jordanhasnolife5163
Жыл бұрын
That's NYC for ya :(
@art4eigen93
Жыл бұрын
wow! that's so fast I literally had to watch in 0.75x :p
@jordanhasnolife5163
Жыл бұрын
The ladies also always tell me how fast I am :(
@art4eigen93
Жыл бұрын
😂
@user-se9zv8hq9r
2 жыл бұрын
how about temporal for that 2 phase commit
@jordanhasnolife5163
2 жыл бұрын
Can you elaborate on this?
@RinayraMotwal
2 жыл бұрын
Can you please also make video on distributed messaging queue
@Elena-kx5lw
9 ай бұрын
Ппц сложно
@nocode659
2 жыл бұрын
Start a paid course.. ITs worth it
@jordanhasnolife5163
2 жыл бұрын
Haha no need truthfully - I'd rather put my videos out there for people to see, as I think in the long run it's better to get the exposure. Plus, I'm not really qualified to be charging money for this stuff as I'm mostly just ripping it off of other people for the time being. I appreciate the view!
@sanampreetsingh9638
2 жыл бұрын
Great design ! Of restricting user to just one web socket connection with server by queuing messages ! Couple of questions ! 1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock , so our general topic can be product in stock but user is only interested for product X in stock notification ! ? 2- we are keeping Queue per user design which might hog resources as user base increases? Thanks
Пікірлер: 77