Hi Arpit, Got some questions. If you can answer it will be awesome. 1. Consistent hashing will be used for writing the data as well to DB right? 2. What happens if we remove one node from a ring of 4 & that removed node contains some data. So how to determine which Node we should transfer our data from that removed node? 3. How the cache will be updated in real time? Does it got updated when user create/update a rating, or it will fetch data from DB? Cache is of limited size so what would be the best approach for storing data: I think it should be LFU. What your views? 4. How to route the request in case a node goes down? Are we adding that item (another node from other AZ1) to consistent hashing ring or how?
@AsliEngineering
Жыл бұрын
1. Yes. It spits out data ownership. 2. Consistent hashing spits out that info 3. Depends on the usecase not one correct answer here. LFU with exponential decay can also work fine. 4. That's the recovery handler written in the routing layer (API servers) in this case. Or a separate component.
@abhis3kh
Жыл бұрын
@@AsliEngineering Thank you :)
@susantaghosh504
Ай бұрын
I believe all these complexities will be abstracted by distributed database providers like DynamoDB, ScyllaDB, or CockroachDB.
@architshukla8076
Жыл бұрын
Very Informative video Arpit...Thanks :)
@gauravraj2604
Жыл бұрын
liked your explanation Arpit. Thanks a ton
@竹下-d4d
9 ай бұрын
In Japan, last year 2023, some hotels owners did large lawsuit agajnst Booking, because of payment delay. Booking side execuse was System trouble.
@ankitKumar-yd6iv
4 күн бұрын
Why can’t we use NoSQL here?
@AbdurraffaySyed
Ай бұрын
Hi Arpit, Thankyou for such an amazing explanation. I wanted to ask you a question. Is it possible that our hash space gets skewed up at a certain point like the files are inserted on the left side of the hash space and the storage nodes are inserted on the right or vice versa? If yes, so wouldn't it result in the celebrity problem? I might be missing something here. Would love to hear your thoughts. Thankyou
@sarthuaksharma9609
Ай бұрын
Although hash functions are used in such a way such that we close to uniform distribution but in case if such a condition happens you can shift some data to another node but yeah this will require some changes in the review service hashing logic.
@Aditya-us5gj
Жыл бұрын
Hi Arpit, thanks for this channel. I was utilising it to fullest before I discontinued because of lack of dedication and consistency. And now when I visit videos section, I'm f**ked up how to cover all these gems. Not sure to bing watch them on weekends or to keep them aside for a while and start covering daily videos consistently.
@adianimesh
Жыл бұрын
30 minute a day ! thats how I do it .. will catch up eventually
@koustavdas2519
Жыл бұрын
Hi Arpit. If we use NOSQL here what DB would you suggest a Cassandra or MongoDB? I know Cassandra is write efficient. But what about reads? And personally which DB would you choose? @AsliEngineering
@utsavprabhakar5072
Жыл бұрын
Do we shard databases ourselves? Like in dynamo db, sharding is handled on its own. There is a provision I guess to shard but thats not usually used by anyone. Can you give some real world examples of where consistent hashing is done nowadays? (given db requirements are handled by cloud service providers and autoscaling, provisioning and sharding is done by the database automatically?
@AsliEngineering
Жыл бұрын
depends on the database you are using. in managed db we need not do anything but if you are self hosting then yes.
@utsavprabhakar5072
Жыл бұрын
@@AsliEngineering got it, thanks!
@d4devotion
Жыл бұрын
So it turned out to saying that even reviews on your System-Design-Master-Class are playing a vital role. (You will come to know what I meant to say in your next cohort :) )
@AsliEngineering
Жыл бұрын
heheheh. suspense :) looking forward to it :)
@piyush3168
Жыл бұрын
very good explanation ! does writing service will scale ? as we have only one master node , or Cassandra ( multiple writers) will be good choice for that ? , as we are not looking for acid compliant behaviour
@girishanker3796
3 ай бұрын
A Cassandra cluster would be a great choice. +1 but since he mentioned trade offs I think we can trade off consistency for availability (so maybe 2 write masters which will be eventually consistent)
@ng.manisha
6 ай бұрын
What will be the structure of the redis key and value? Also, for DB sharding, we don't generally recommend sharding for rdbms systems right? because joining data across multiple shards are costly? How does this get managed? Shouldn't we use cassandra here?
@sarthuaksharma9609
Ай бұрын
Cross sharing is frowned upon but if you are using shards on the basis of accommodation id you will never have to do cross shards. I think he used sql in the example because that's what booking also does. Also Cassandra is a good fit generally when there is more write to read ratio here the case is different. Although I am not sure about prematerialized view performance wrt to any other Nosql DB. I think that is the deciding factor here
@microtech2448
Жыл бұрын
How would data integrity be handled when shards are added or removed?
@shubhamtyagi5219
Жыл бұрын
When we are creating new ring, how do we know which requests need to be relocated because we would be moving some keys from the older shard to new shard. How to find which key?
@AsliEngineering
Жыл бұрын
Changes happen atomically all at once.
@jayantprakash6425
Жыл бұрын
when the new ring is being prepared, are reads/writes served by old ring? Is there any downtime associated?
@AsliEngineering
Жыл бұрын
nope. no downtime needed.
@atuljoshi6182
Жыл бұрын
Excellent explanation .
@Amritanjali
Жыл бұрын
thanks🙏
@mahendratonape27
11 ай бұрын
what if routing decision is made by db router instead of consistent hashing algo written in routing service, i think db written routing alog is better than our own consistient hashing algo
@maheshkumartangella5516
6 ай бұрын
I had the exact same question, why do we want to recreate what database could already do, routing its requests to corresponding shards
@robinpaulification
6 ай бұрын
08:50 it is two regions and not AZ
@tesla1772
Жыл бұрын
Hi aprit, i have one question that all this data migration that we have to do how is it done?. Do we write our own scripts for it to first migrate and then inform services about new node . What is the right way of handling this
@AsliEngineering
Жыл бұрын
Yes. We write the scripts.
@tarunpahuja3443
Жыл бұрын
What if the master shard goes down during write re and quest. I guess here we could avoid strong consistency for available
@sarthuaksharma9609
Ай бұрын
As he already said there are read replicas. You can have leader election Master goes down slave takes over and yes consistency can be comprised for such use cases as availability and latency has been prioritised
@bharatarya7929
Жыл бұрын
Hi! Nice video can u share your thoughts on how Rapido is keep a constant OTP of 4 digits for all users.
@AsliEngineering
Жыл бұрын
Have never used Rapido. So no idea about it.
@bharatarya7929
Жыл бұрын
@@AsliEngineering so while booking any cab via any app like Uber, Ola etc. They all share a OTP for the ride which keeps on changing. But rapido keeps a constant OTP for users which is 4 digits so how they are managing large scale
@AsliEngineering
Жыл бұрын
@@bharatarya7929 then OTP are not unique just a random number assigned to a user.
@niksgupt
Жыл бұрын
Great use case and amazing presentation. Wondering, any specific reason for using relational db? Can't we solve it using NoSQL db?
@AsliEngineering
Жыл бұрын
You can so long as it fits the usecase. No hard bounds.
@boombasach
Жыл бұрын
Thanks
@AsliEngineering
Жыл бұрын
Thank you so much for the kind gesture :)
@hackwithharsha
Жыл бұрын
00:19:00 Thank You… If we use asynchrous replication.. How do we handle replication lag ? For example, Imagine If I have written a review for hotel which goes to master and then immediate read query went to replica which has’t recieved replica data yet… In the userinterface, User might think something went wrong with his submitted review and they will resubmit it too ? How do we handle this situation ?
@AsliEngineering
Жыл бұрын
You will ensure Read Your Write consistency by redirecting the critical reads to the master.
@gauravraj2604
Жыл бұрын
@@AsliEngineering Read Your Write consistency, is this any specific case we should wonder about while using master-slave replication?
@SwikarP
10 ай бұрын
Best best best awsome
@abhis3kh
Жыл бұрын
Can we get twitter design video based on the diagram shared by Elon Musk ?
@AsliEngineering
Жыл бұрын
I thought about it, but that diagram is really very very very high level. THey are literally just a bunch of microservices. If I create a video on it, no one will learn anything concrete from it. It will just be a list of microservices and guessing what it does. Say, I create a 20 min video on it, that will definitely get me views, but will add nothing to the viewer. Thank you so much for suggesting this, but I really do not want to waste the time of anyone watching my video. I like to keep the video information dense and useful. Hope you understand.
Пікірлер: 51