It would be great if you build up slowly for the tech/tool to pick up for the design rather than directly putting the cassandra or redis or kafka. There can be possibility that the Interviewer is highly good at those tools and will start digging deep into that as soon as we name a stack which can surely bring us in trouble sometimes.
@debasish2332
2 жыл бұрын
Which make sense
@AlbertoRodriguez-oe6jo
2 жыл бұрын
As an alternative, you can read about the tool being discussed in video after watching it. It will keep the videos shorter and packed with more relevant content.
@isachinq
2 жыл бұрын
Isn't a token service a single point of failure? Even if we use multiple token service, how will we synchronise all of them? Please answer 🙏
@ANILKHANDEI
2 жыл бұрын
@@isachinq i think the token service is being load balanced to avoid a single point of failure. Having miltiple tokenservices will not impact the design as it is only used to get the next range from same mysql cluster.
@isachinq
2 жыл бұрын
@@ANILKHANDEI how will you synchronise all the MySQL in MySQL cluster? By definition, horizontal scaling will bring down the consistency
@mahorasanli
2 жыл бұрын
I watched 4 different Tiny URL System Design Video. This one is by far the best
@aakashgq
2 жыл бұрын
I read many different ways this can be designed. But this design looks the best of all as it is highly scalable, very efficient and something that can easily work on a global scale. Thanks Sandeep for so much of hard work at your end to come up with such wonderful designs. I am watching your entire playlist.
@aman1893_
Жыл бұрын
Hey do you want to study system design together? I have a lot of exp but I feel most of the online resources are incorrect or incomplete. We can make a solid understanding of the common designs in next 10-15 days by brainstorming everything together.
@rahular4596
Жыл бұрын
@@aman1893_ I would be happy to learn from you😀
@vozone10
4 жыл бұрын
A diagram with Kafka for Analytics would be a cherry on the cake. Overall great job!
@ersinerdem7285
3 жыл бұрын
There is another approach which pre-calculates the short urls, and use when requested. This way, there will not be range loses when servers go down. Your way is also very good, thank you!
@anonymousgod2006
Жыл бұрын
Ya I think if Token Service layer is removed and Url shortener service simply do what Token service is doing, that will be your case, and it seems fine to me
@chepaiytrath
2 жыл бұрын
After encoding the Base10 token to Base62/Base64 for readability, doesn't taking out the first few characters(ex. 7 chars) and using them to store the short URLs increase the collision probability. Basically, from all the token ranges provided to us by token service, we get unique tokens. Cool. Then we convert those Base10 tokens to Base64. Cool. Isn't it possible to get the first 7 characters of two completely different base64 encoded tokens to be the same and thus resulting in a collision. How do you handle this. Similar situation would also arise even if you take the "Encoding URLs" approach
@ashishbhardwaj1546
2 жыл бұрын
I would rather choose last 6-7 chars. What do you think ? Less probability of having collision
@chirut4327
2 жыл бұрын
@@ashishbhardwaj1546 it still might have collision and we may not prove that in the scale of billions of numbers.
@annafalvello8138
2 жыл бұрын
"in technical terms it's called a collision, but for us it's a problem" made me laugh. thanks for the great content
@JangBahadur3028
2 жыл бұрын
Your content is great ... However, your confidence seems low. You should work on it Sir!👌 Kindly consider it as a constructive review....
@Arjun-tg1go
2 жыл бұрын
Just a thought- instead of using the token service we can generate unique tokens per service within the service itself. Steps could be as follows 1. When service node starts, it will registers itself with DB and gets itself an ID and it’s sequence will start with 1 2. Now, a particular node can generate token based on its ID+Today’s date+sequence 3.When a particular node goes down, new node will spin up and performs step 1 4. This will avoid complexity of calling token service totally
@alpitanand20
Жыл бұрын
You are talking about Zookeeper here.
@Arjun-tg1go
Жыл бұрын
@@alpitanand20 use anything (db/zk) that is a singleton service.
@KingKingSofa
Жыл бұрын
MD5 Hash of the user IP Address + Time Stamp encoded to base 62 would also be valid. Both these approaches save us from the complexity of managing another set of services and their connections to a DB. We can more easily scale horizontally.
@manojkp80
3 жыл бұрын
Great video, as always very helpful. If you could 1) add custom key support - user can specify their own tiny url 2) talk more about what could be other ways like, md5(main url) -> base2_encode, and their drawbacks etc 3) add diagram on a analytics part, that would really be helpful. Do you also feel a cache can be added in front of Cassandra to serve hot urls? Thanks, keep helping.
@p111calcutta1
11 ай бұрын
@codeKarle how do you handle the case when token service goes down and we wait for data to persist in sql db before token service return the ranges to short url service ? what if sql db goes down ? if we keep replica of db then do we want data to be synced before we return range to user ?
@deepanshunagpal7587
Жыл бұрын
One issue is that sequentially generating shortcode could be security threat as it would be predictable either we should append a random number at starting or end before converting base 62 conversion
@abhinee
2 жыл бұрын
how is redis is single point of failure, most cloud providers support HA for redis clusters?
@foodfan4512
4 жыл бұрын
Amazing explanation! Can you please provide pdf or image format of the architecture like you provided for other videos? It really helps to see everything all together in 1 place to digest everything. Thank you.
@codeKarle
4 жыл бұрын
Sure, we'll get it done in a few days :)
@samirhere4341
4 жыл бұрын
Was looking for the same. Thank you
@codeKarle
4 жыл бұрын
There you go: www.codekarle.com/system-design/TinyUrl-system-design.html You'll find the architecture & summary here :)
@CaribSurfKing1
2 жыл бұрын
you forgot 80 20 rule for caching short to long request flow
@alianwar2857
2 жыл бұрын
Very informative and the way you show the simple way to design first and then discard it again telling the problems and how we could tackle that, is really great.
@hokayantra_
4 жыл бұрын
Custom shorlink support is also most of the user wants plus its not good to have incremental shorlink ... better to have something randomized nature of shortlink.. share your thoughts
@codeKarle
4 жыл бұрын
Custom Shortlink is a good feature, that can be implemented easily I believe. You are right about the incremental shortlinks, that's a trade-off you need to choose. Random is good, but you'll end up in high collisions and higher latencies. If the NFR's are okay with some high latency/less throughput, then a randomized solution would be a much better choice since it's hard to predict the next short URL.
@anandratn8339
4 жыл бұрын
@@codeKarle another possibility is to add salt in the incremented number being used by shortener service that will maintain the uniqueness as well as hard to guess
@hanspeterpfister2253
4 жыл бұрын
Great work. Can you also upload a video for Dropbox/Google Drive like service? In case of Dropbox, most videos drop the ball at, there will be a Notification service and it will communicate with clients asynchronously using Queues and every client will have its own Queue. They don't talk about, if there are billions of clients, do we expect to have a billion queues? Is that scalable? Do focus on this as well if you make the video.
@codeKarle
4 жыл бұрын
Sure, we'll try to make that in a few weeks
@sankarsattari5921
2 жыл бұрын
I think there is still a possibility of duplicates because we are using a substring of length 6 or 7 of the base62 encoding of the numbers, which can collide. for example base62(0000001) is 107Zzj5ex0 and base62(0000002) is 107Zzj5ex0 as you can see the prefixes are the same.
@sagardafle
2 жыл бұрын
Hey, how is base62(0000001) = 107Zzj5ex0?
@MegaBelltone
Жыл бұрын
@@sagardafle Doesn't matter. The thing is that there is always possibility of collision if we do substring. I can't find any explanation of it anywhere, as how to resolve.
@LifeWithSeb99
5 ай бұрын
How can you request a number from Redis Cluster? Isn't that just a in-memory database? Would you need to program some kind of a logic into redis cluster?
@dongiveajack
3 жыл бұрын
One of the best video i found out on youtube for url shortening.
@prashanthkumar914
Ай бұрын
If token value is 100M, Base62 gives 5 char length output only. I think in order to generate 7 character length base62, the token value should start from 62^6. How to keep those ranges of tokens which are in between 62^6 and 62^7?
@VibeWithSingh
3 жыл бұрын
Great video. Thanks for putting all the effort and explaining different choices and corresponding trade offs. 👍
@chaitanyawaikar382
4 жыл бұрын
Excellent explanation !!! Even though there are a bunch of system design videos out there, your videos stand apart from them by discussing various situations and pitfalls of using a certain tool/ database. Just one quick suggestion from my side regarding upcoming videos - can you please create any video that explains capacity estimation of a database. For example how much space will a users table with let's say 6 attributes having almost 100 million records take is postgres or mongodb or any other database. This is also commonly asked in interviews now a days and given your breadth of experience, I think you would be able to create awesome content in this space also. Once again Thanks for videos :)
@codeKarle
3 жыл бұрын
Thanks Chaitanya! We'll put this in some smaller video that comes out in the future. Just that it's a time taking thing to go over the calculations so we skipped it in all the videos till now :)
@ambikabc
3 жыл бұрын
I have a doubt here.. You said that if we have multiple redis then it will be tricky but then you added more redis so I kind of got lost there or may be I dint get what you said... Can you please elobarate?
@j4s0n39
3 ай бұрын
7:45 Checking the DB to avoid a collision isn't a valid solution. That introduces a race condition. If service0 queries and finds nothing, and while it's processing, service1 queries the same token and finds nothing, it will try to create a duplicate.
@rolandmackintosh5163
2 жыл бұрын
Would anyone be able to criticize this solution to the uniqueness problem and show why it sucks? My solution is to fragment a mongoDB database with every possible unique short URL from 1 to, say, 10 trillion, across N machines (say between 3 and 20). When the shortener service wants a new URL, it asks the mongoDB instances for a new option and picks randomly from ones that aren't marked as taken. This means that all but 1 of the mongoDB clusters can go down and we'll still have access to a unique pool of shortened URLs. It should be efficient for reading - just get any non-taken URL. It should be efficient for writing: Just mark a given URL as taken or not taken. What's wrong with my solution?
@gauravradioactive
4 ай бұрын
Thank you for the great content ! It looks like the Token Service is a single point of failure as well. And if we create multiple instances of the Token Service, how do they ensure that each instance provides a unique URL range, and no 2 instances provide overlapping ranges? If the Token Services are supposed to communicate each other before deciding the range for an incoming request, this would again add to overhead and slow down the process. Can someone please share their thoughts on this ?
@ayushtewari2826
7 ай бұрын
What if the Token Service Goes down? We cannot have multiple deployments for this service. We want that there should be some body that is a single point of truth, that provides us the next range. But what if it goes down? Any solutions? Or if my question is wrong. Please explain.
@neerajramachandran4647
4 ай бұрын
Does this approach assume that we do not care about idempotency? In this model, if the long to short URL service receives multiple requests of the same long URL, the token service will assign that request to different short URLs.
@A_Aaa816
5 ай бұрын
I don't undestand why not redis ? following you concern about Single Point of Failure --> Redis is designed for horizontal scalability and can be deployed in a clustered configuration for high availability and fault tolerance. Can you explain better ?
@nagendrs
2 ай бұрын
but at 16th minute - token service becomes single point of failure.. even if you say you fetch from DB, that also becomes single point of failure. isn't it ?
@krishankantsharma196
Ай бұрын
Why do we not use MongoDB here. Simply attach _id with tiny url and return? _id always unique with unique index.
@yashendragoyal
3 жыл бұрын
A small doubt here, let's say we pick n as 6 for short URL chars and we use base 64 instead of 62. If you are starting the range from 1000 to 9999, then the base64 encoding will contain 6 chars, but as soon as you move to 10000 the chars will be 7. Doesn't it diverge from the initial design to keep the short URL chars as 6 only, also we are only using 9000 URLs in this range? If we follow this route, we might have to go to a very high range to convert to base64 and create a short URL (which will not be short anymore)
@santoshgupta1745
2 жыл бұрын
6^64 can contains 6.3340287e+49 unique numbers
@Uppathebest
6 ай бұрын
great explanation Karle. Pity you stopped making videos! If you resume, you may want to invest in a better microphone for the benefit of your viewers :)
@reynasebasti3635
Жыл бұрын
It's unclear... Functional requirements then in between non-functional then redis then token service then scaling. It needs 4, 5 videos to explain logic/ algo to convert long to short with clear roles of redis and token service, then go for scaling of redis as non functional req. U r jumping from complex logic to scaling... First spend time in logic to convert completely in detailed with all components involved..then go for non func. I think, only logic to convert will take 1 hr video, if role of redis n token service would b explained.
@arghyasen9682
5 ай бұрын
One question - Whenever the user asks for a short URL do we check in the DB if there is an existing short url for the same? If so, will that not again slow down the application?
@kritikajain6677
4 ай бұрын
Let's say we have scenarios where one url U1, is called from two different users for the first time, and both the request R1 and R2 come at the same time, but being sent to different nodes, since all the server nodes have different range of tokens being given to them, same url will be using two tokens, it can also result in decreasing the amount of tokens we have?
@LuveenWadhwani
3 ай бұрын
How do you scale the DB for token service across multiple DCs and regions without repeating token ranges? Do the token ranges need to be seeded manually per DC/region to prevent the same range from being reused in more than one DC/region? Does this solution constrain us to using a DB technology like Spanner so there is one consistent data set that spans geographies and is replicated in near real time?
@adarshmv261
2 ай бұрын
Watched couple of your videos, idk what's the prob with your mic, voice is not all clear! pls use a better quality mic
@chessmaster856
7 ай бұрын
Who will actually generate short url? Do that first. Anybody can deploy it.
@PulkitGarg-z4b
9 ай бұрын
cant we simply hash the 61 character set with new guid. guid will always be unique. i think then we dont need redis or any distributed cache service to generate us a unique number. just a thought..
@rakshashanbhag2723
2 ай бұрын
This is amazing! But quick question, isn't the token service also a single point of failure?
@mehrdadshademan1873
Жыл бұрын
what are the tables rows contains information you keep in the cassandra?
@pawansaggu5787
4 ай бұрын
Can anyone how will the Shortner Service or the Token service handle ranges that cannot be represented by any integer types in a language? Do we need to represent numbers in strings here?
@hemantagrawal25
3 жыл бұрын
can we use instance id and utc time to generate token instead of taking separate token generation service and maintain token range as per different instance.
@ajaytiwary5798
Жыл бұрын
Token service can also be single point of failure.If we will use multiple instances then again will same issue of duplication.
@akagragupta9968
6 ай бұрын
If I generate multiple instances of token service it can also send same range to multiple services....?
@rajgupta3043
4 жыл бұрын
I am having trouble understanding, that let's say there are two datacenters. Each datacenter has its own token service and DB. How do you make sure that token service in two different data centers don't end up assinging same range to the SHORTTOLONGURL service? I am assuming DB contains the range and token service simply gets the range from DB, and it's an atomic transaction. But how do you manage ranges in DB across data centers? Would you have another service for doing so?
@codeKarle
4 жыл бұрын
The main idea of two DCs for token service is for redundancy. Let's say if you have Token Service in DC1 and DC2, the Master of DB can be in DC1 and it can have a Slave in DC2. if Master goes down, or DC1 is not available, then slave in DC2 can become the master, but for all other transactions it can make a cross DC call to the Database in DC1 thus making sure that the range is always unique. Latency is not a concern here because it's once in a few hours kind of an API call to assign tokens.
@DeepakKumar-yl3ok
Жыл бұрын
How will base62 of a 4 digit number, give a seven character output?
@navdeep3469
4 ай бұрын
how will you build optimized short->long url search for such scale ?
@akakop
Жыл бұрын
how token service is preventing a single point of failure and in distributed environment how is it guaranteeing unique ids?
@mohitbhagat9922
5 ай бұрын
token service is also single point of failure. can we scale that also if yes then how we can manage tokens ?
@srinivasanvk57
3 жыл бұрын
Great job on your side. A big thank you from my end. Can you please answer the query on handling duplicate requests? Same URL requested 3 times generates 3 tinyURL? How to handle it in this design?
@srinivasanvk57
3 жыл бұрын
Ok I just verified in bitly and found that they generate different short URL everytime the same long URL is passed. So this is not a concern apparently. Thanks anyways
@jatingoswami5469
Жыл бұрын
What if tokenService gets down ( Single Point of Failure )
@AbhishekKumar-py3kg
7 ай бұрын
This does not cover how a single host will handle concurrent requests. We will fall into the same problem within a single host. Also how different token services will guarantee unique range being allotted to different hosts? Too many open issues.
@rjarora
3 ай бұрын
1. Concurrent requests will be a breeze to handle, as we are returning the short URLs from the RAM inside the server. 2. Different token services will guarantee a unique range because all of them are talking to a single database (with M/S arch). They don't have their own separate DBs.
@adityakommu344
Жыл бұрын
why only cassandra should be used? why not MongoDB or a Datawarehouse
@akash8004
5 ай бұрын
Isnt token service became single point of failure?
@coderix1404
Жыл бұрын
What happens if I try to generate short url for the same long url twice? Will it check in the database for every new query if short url is already generated for the request ?
@reallifegambits
9 ай бұрын
No. every time a new short url will be generated.
@harinimurali8724
Жыл бұрын
What if the token service is down due to some reason? Is that not a single point of failure? When it comes up, it would again send indexes from beginning
@rjarora
3 ай бұрын
There are many instances of token service. How is it a SPOF?
@sushantgawade9289
2 жыл бұрын
Hey is your content available on any other platform ?
@isachinq
2 жыл бұрын
Isn't a token service a single point of failure? Even if we use multiple token service, how will we synchronise all of them? Please answer 🙏
@manhalrahman5785
9 күн бұрын
In an RDBMS 3 column table. Index, Range, is_assigned
@prototypebharrat
3 ай бұрын
Brilliant. How can I write code based on that
@weightwatcher221
4 жыл бұрын
1 of the best system design video. Might not be reasonable question but possible. How do we scale it to some arbitrary large(billion) of txns per day?
@codeKarle
4 жыл бұрын
Thanks!! I believe that with this exact same architecture, you can scale it to not just billions, but potentially even trillions of transactions per day. All you need to change is - Modify the range of token service to now return a larger range thereby reducing the hits on token service, increase the length of the short URL, serve this service out of even more Data centers that are spread across the globe based on your traffic patterns, and throw in more hardware.
@weightwatcher221
4 жыл бұрын
@@codeKarle thank you for the quick response
@varunsadhu9060
4 жыл бұрын
what is the token service give the same range, how token service work in a distributed environment and not generate the same range ?
@codeKarle
4 жыл бұрын
Token service is a single service sitting on a transactional DB, so on multiple calls it would not repeat the same sequence. Just that the service is hosted in multiple data centers to make it redundant/fault tolerant/disaster recovery, but the Database would have one master that would power all the queries to fetch the next sequence.
@randomguy4822
2 ай бұрын
Can you please make HLD of Book my Show ?
@ujjwaldave8646
Жыл бұрын
You could have given some more thought on the short url to long url flow. Fetching data from Cassandra for each and every request could be very time consuming an latency needs may not be met. May be we can use caching in that flow to reduce the fetch latency.
@kartikvaidyanathan1237
2 жыл бұрын
Token service is a single point of failure no?
@user-fw1hj3vv2k
Жыл бұрын
I am thinking what's the maximum of base10 unique number that could be generated and could be hold by variable in languague like java. Though we have capacity to hold 62^^7 unique URL , it could be limited by the maximum base10 unique number that could be stored in the applicaiton.
@caballerosalas
Жыл бұрын
I couldn't understand a word with that accent
@ashishbhardwaj1546
2 жыл бұрын
During Colliosion of BASE64 Scenario ~~ Create hash from LONG url and then create hash function from BASE64(last 7 chars ) + hash code of long URL RESULT = (keep it to 7 chars) Same I will save in cache(reddis/etdc) KEY=7 chars || VALUE = hashcode of long URL If any collission happened I wil check cache Key then retry to generate new!! Any thoughts??
@andriylytvynskyy8359
Жыл бұрын
How we can make sure that we don't "shorten" urls which have already been "shortened" - probably with some cache... otherwise we would need to query Cassandra and find out but is it not a cheap operation?
@Spiritualleace
3 жыл бұрын
Great stuff! Why would the asynchronous request for analytics fail?
@karnveerayush
3 жыл бұрын
there can be multiple reasons such as: Network failure, etc.
@codeKarle
3 жыл бұрын
It's a good idea to assume everything can fail. Could be any reason, network related, service outage, firewalls, etc. Just that in Async requests, you usually don't bother if it fails, so there it becomes important to keep track of what happens when they fail.
@Sagarvilas
2 жыл бұрын
I have heard in many videos that checking in DB if the URL exists is not efficient, I do not understand that, you are designing a system with 1:200 write to read ration, how much of an overhead it is to check the database if the URL exists?
@jdr3808
Жыл бұрын
Thanks, I have seen many solutions talking about Event based decoupled systems. However I never encountered a robust way of making sure there is no Consistency and Integrity being effected due to any failures during those Async processing of the Event. What are various techniques for ensure Decoupled systems ensures no loss
@amitagrawal4660
Жыл бұрын
Why two different cassandra DB is used.. one for Long to short url request and other short to long url ? How will the second DB get those information via replication?
@SaikrishnaA-b4c
7 ай бұрын
Are you going to maintain the mappings from counter range to its availability in an extra datastore, or a table so that your token service could handle them?
@seelamrameshreddy702
Жыл бұрын
It is really nice video. I have couple questions. 1. Is duplicates URLs would be stored in DB every time when we hit a request ? 2. Why would you consider caching during GET ?
@SumanDas-fx5vu
2 ай бұрын
Tushar Roy of system design
@Bb-cz3fq
Жыл бұрын
HI I have a question regarding the Collision it this case is avoided if we have two same shortURL ,because it uses tokens right?
@domicio1577
Жыл бұрын
Awesome video! Lots of insights. As a piece of feedback, I would change microphones. Thank you.
@navyathamarreddy2807
Жыл бұрын
Can someone please explain how collisions are handled? I did not understand the counter approach. The tiny URLs are generated using hash on Long URL and a Base62 as per the initial discussion
@vaibhavchhabra800
Жыл бұрын
How are you going to deal with 3.5Trillion req being hit to your Cassandra DB (Just 1 instance) . Scenario where all the 7 letter combinations are full and everyone tries to get long url from their short url , the DB(Single instance )will be overwhelmed , no ?
@bhargav3099
2 жыл бұрын
1. why are we using cassandra db as we already know that we get lot of queries.Why not prefer mongo over cassandra ?
@rohanskoshti
2 жыл бұрын
What if token service goes down ? I was unable to see your view on this or why it won't go down ? Thanks for letting me know of it.
@abhinee
2 жыл бұрын
do we really to talk about technicality of unique url like character set etc, we are not doing LLD of url generation service?
@mycomputer6826
2 жыл бұрын
When one tiny URL is redirecting to two different large url, it means integrity issue, but what if both url belongs to same domain although the API are different, can we find a solution for that?
@kunal4350
4 жыл бұрын
Please make new video. I am waiting for a month. Simple request from your fan :)
@codeKarle
3 жыл бұрын
Thanks buddy!! We'll get started again soon, not sure how soon :) Some of us got a bit too busy with work, and because of the whole COVID thing we moved to different cities and it got a bit difficult to record.
@kipkip6712
3 жыл бұрын
@@codeKarle Glad to hear you are not abandoning this! Got a feeling since there werent any new videos!. Will be worth the wait whenever that is!
@maheshnavani5294
2 жыл бұрын
Great Video, would appreciate if you can do 1. Elevator System 2. Discount System at SuperMart
@nikitasinghchauhan6239
4 жыл бұрын
You explained so well. nice video .
@codeKarle
4 жыл бұрын
Thanks a lot 😊
@kumarmanish9046
2 жыл бұрын
But how will the number in any range be converted/mapped to the unique url? That wasn't shown.
@earthpligrim5757
2 жыл бұрын
this is the first video im waching on your channel and i just loved the explanation. im sure its gonna help me with my interivew. the very first thing i did post watching the video is subscribing. thanks alot for the detailed information.
@yodude2493
8 ай бұрын
what language do you speak?
@godolsss2139
4 жыл бұрын
Keep posting..love your vids...very simple and understandable content...
@codeKarle
4 жыл бұрын
Glad you like them :)
@rohitrangera8766
2 жыл бұрын
Awesome video, can you share system design on Voting Machine (EVM)?
@sreechalasani9268
2 жыл бұрын
1) why can’t the long url be hashed to create a tiny url 2) can the instance handle the volume of writes ?
@phoneix24886
2 жыл бұрын
What if two token services generate the same range? They might be single threaded but they are running in parallel.
@antrapurohit8010
2 жыл бұрын
for base62 hashing, is it ensured that for different hash key, we will get different value?
@jasper5016
Жыл бұрын
Bhai thanks so much. I got your course on Udemy as well. There is not better channel than this on System Design.
Пікірлер: 246