Great video, as an aspiring data engineer in college (where none of this is taught), these videos are worth their weight in gold.
@SeattleDataGuy
3 жыл бұрын
Yeah! I never heard about data engineering in college
@kennylaikl299
3 жыл бұрын
Request - perhaps a deep dive videos on each ETL component like Data QA, Error handling and logging, dashboards for tracking, data lineage and cataloging, metadata database?
@SeattleDataGuy
3 жыл бұрын
I will have to go back on that! My plan is to continue going through each of the various high level points to go from raw data to machine learning. However, I will circle back at some point.
@tethadam4929
2 жыл бұрын
Can’t state enough how helpful these videos are. I know most people love the “5 things _____” videos however, these provide immense real world utility.
@SeattleDataGuy
2 жыл бұрын
I am so glad this video was helpful..but I do occasionally make the "5 x" videos as well
@higiniofuentes2551
3 жыл бұрын
Interesting video, thank you!
@SeattleDataGuy
3 жыл бұрын
My pleasure!
@AndoresuPeresu
Жыл бұрын
Marvelous Videos!!
@SeattleDataGuy
Жыл бұрын
Thank you!
@nhelpalana1729
2 жыл бұрын
Takeway, understanding of Slowly Changing Dimensions. Thank you.
@andrew3068
2 жыл бұрын
You’re videos are amazing
@SeattleDataGuy
2 жыл бұрын
Thank you!
@marcosoliveira8731
3 жыл бұрын
Hi. Do you think that dimensional modeling still relevant ? My question comes from seeing denormalized data being used with tools like Apache Spark( ELT), Elasticsearch... where the modeling actually does not apply.
@SeattleDataGuy
3 жыл бұрын
I think there is. still value in dimensional modeling. It might change a bit over the next few years but it all depends on the systems companies chose as well as their data governance. Often I find a lot of teams will still have a base layer in a more traditional data warehouse set up and then they will develop a denormalized layer for analysts. That way they can still have a single source of truth while the analysts don't have to run complex queries. There is also a lot to be said about how modern data warehouses are often read and insert focused. Meaning running update statements might not be possible. This poses a different kind of challenges. Overall, I think at the very least it's good to understand dimensional modeling and how it can drive future data warehouses and analytics.
@marcosoliveira8731
3 жыл бұрын
Thank you for expressing your opinion. I believe that DM & DW still has a important role in data infrastructure these days and for a good number of years ahead. It´s good to know other peoples opinions.
@SeattleDataGuy
3 жыл бұрын
@@marcosoliveira8731 I agree, there are multiple reasons I think you are right. Enough to fill a post. One of the biggest reasons being that switching entire concepts is hard.
@alexfilo7929
Жыл бұрын
Amazing video!
@SeattleDataGuy
Жыл бұрын
Thank you!
@alecryan8733
2 жыл бұрын
Great video, I'd be interested in seeing how to automate this ETL process. A lot of resources I see use managed services but that cost for someone working on resume projects seems too high
@SeattleDataGuy
2 жыл бұрын
I might have to put that together
@alecryan8733
2 жыл бұрын
@@SeattleDataGuy Maybe an airflow example on EC2 with RDS backend using dbt and docker? Asking for a friend... lol
@sanjayplays5010
Жыл бұрын
Hey Ben, where do the QA checks go? Do you add them to your ETL script, or is it a separate script that runs after/independently of the actual ETL script?
Пікірлер: 23