Kafka -> Loki -> S3 If you're looking for an existing solution :)
@interviewpen
8 ай бұрын
Yep, S3 does a lot of the things discussed here behind the scenes. Thanks for watching!
@developerjas
8 ай бұрын
Great Video man! Would how would you go about designing the data ingestion part?
@interviewpen
8 ай бұрын
Great point! There’s a lot that goes into ingesting logs while optimizing network performance and maintaining context. Check out our full video on monitoring systems on interviewpen.com :)
@sahanahunashikatti3935
8 ай бұрын
😊😊 ok 0@@interviewpen
@supragya8055
4 ай бұрын
i dont understand , if under same bucket lets say for (2021-2022) we have multiple nodes , how are reads any faster ? for the same bucket logs will be distrubuted across servers and still need to be queried across servers which is slow . Bucketing didnt help in improving read performace , is my understanding .
@interviewpen
4 ай бұрын
Yes, sharding improves write performance at the expense of query latency (unless we shard by something more clever!). However, we can still handle a high throughput of reads. This latency vs throughput problem is a common tradeoff with large-scale systems! Hope that helps :)
@prakharsrivastava6644
Ай бұрын
I love the cute computer in the background
@interviewpen
25 күн бұрын
Thank you :)
@GoofGoof-cs6ny
4 ай бұрын
So in 2018 every service was writing logs to node 3, didn't we went back to bad write complexity by doing bucketing?
@interviewpen
4 ай бұрын
Yep, bucketing makes query performance better, so we introduce sharding as well to distribute writes within a bucket.
@lunaxiao9997
6 ай бұрын
great video,very clear
@interviewpen
6 ай бұрын
Thanks!
@sahanagn4485
5 ай бұрын
Great video!!! Please slow down the speed of video as someone new to topic its bit fast to grasp the concept.
@interviewpen
5 ай бұрын
Ok, noted!
@michatobera6049
8 ай бұрын
Great video
@interviewpen
8 ай бұрын
Thanks!
@didimuschandra6680
8 ай бұрын
Greatt video!! thanks! but, can you create video to develop Effective and efficient Ticketing System?
@interviewpen
7 ай бұрын
Sure, we'll add it to the backlog. Thanks for watching!
@weidada
8 ай бұрын
Suppose every two years, it ingest 2PB and migrate 1PB, how could three sets be enough to cycle after 12 years?
@interviewpen
8 ай бұрын
Great question! At any given time, we have three "hot" nodes--two are migrating data to cold storage and one is ingesting new data. We only showed one cold storage node in the example, but we would need at least 2 to make this work long-term. Hope that helps!
@ankushraj3599
3 ай бұрын
Why not use Kafka for high through put?
@interviewpen
3 ай бұрын
Kafka is an event streaming platform, so it wouldn't solve any of the log storage problems we're addressing here. But if you have any thoughts on how to incorporate it, feel free to share!
@RaushanKumar-co3wj
3 ай бұрын
@@interviewpen Use kafka stream + cassandra . process the event through consumers and save inside a Hbase db for analytics .
Пікірлер: 24