Amazing explanation. The original blog post was so confusing but this is so well explain that I understood everything 👏
@rachitgoel4159
6 ай бұрын
Awesome Explanation. Learning a lot from this System Design video series. One question since we are duplicating so much amount of data at the Core Table data layer and then at the Derived Table layer & then some of the tables are getting duplicated in Scylla DB doesn't it increase the cost of the data management & it's maintenance.
@AsliEngineering
6 ай бұрын
storage as cheap as compared to compute. so storing redundant data is better than doing on-the fly computations.
@rachitgoel4159
6 ай бұрын
@@AsliEngineering Got it Thanks for the reply.
@abhis3kh
Жыл бұрын
Awesome explanation. Really simplified. THanks for the effort
@hackwithharsha
Жыл бұрын
Finally our data reaches ScyallDB which is a NOSQL database :)
@a.nk.r7209
Жыл бұрын
Appreciate the effort 👏🏽👏🏽👏🏽
@hackwithharsha
Жыл бұрын
00:13:10 Thank You Arpit !! What is the difference between merging with existing data or appending with existing data…. Merge seems to be like UPSERT Queries.. Right ?
@AsliEngineering
Жыл бұрын
merge means updating existing data. append is blunt append. upsert is one way to merge.
@hackwithharsha
Жыл бұрын
@@AsliEngineering Thank You !!
@piyushjha7222
Жыл бұрын
Hi Arpit, i have a small doubt. you have connected the K8s pod directly to the transactional table then where is the core table in the architecture?
@tesla1772
Жыл бұрын
great video. just one question ,since we are duplicating how will we manage update in data. lets say user changed his name but our derived data will still contain past name
@AsliEngineering
Жыл бұрын
The jobs are continuous and not one time.
@tesla1772
Жыл бұрын
@@AsliEngineering yeah i get that. Latest job will get updated data but the jobs which ran earlier fetched old data and had populated data right which is outdated now
@AsliEngineering
Жыл бұрын
@@tesla1772 no. The job will fetch the updated rows and merge into existing derived table. The iteration can be on updated_at column.
@girishanker3796
3 ай бұрын
@@AsliEngineeringthat's great.
@rohithj578
Жыл бұрын
Arpit, Why do we need the core tables layer? What happens if we directly create derived tables based on transactions tables?
@AsliEngineering
Жыл бұрын
Common aggregations and structured data layer. NoSQL to structured data happens here.
@kutalaabhiram2398
Жыл бұрын
Can we use kafka to update the big query table(core tables) instead of firing a select query on prod database?
@arijeetbhakat
Жыл бұрын
Thank you
@Vaishravana07
Жыл бұрын
Hey Arpit, I(Fresher) want to desperately work at discord because its the most awesome thing that has happend to me could you please give me some insights/advice or ask you colleagues on how one can land at Discord, Thanks
@jayeshdalal7
Жыл бұрын
Arpit can we use views here instead of tables of core & derived if it is updated via scheduler in a day and used partition view for high latency . just thought
@AsliEngineering
Жыл бұрын
Views are limited to single database. You need access to data spread across multiple databases hence views wouldn't work.
@OwaisAthar1
Жыл бұрын
Seems #AsliEngineering
@myurbantiffin
Жыл бұрын
Do a video once on how chat apps are build
@hackwithharsha
Жыл бұрын
00:12:10 Thank You Arpit !! Once i store data to derived tables, is it safe to delete data from core tables ? because my downstream queries hit derived tables not core tables..
@AsliEngineering
Жыл бұрын
no. we never delete data. other jobs would need it. jobs would be populating core tables from transactional as well.
Пікірлер: 30