The guys from Mage.ai asked me to check out their way of building ETL, ELT and Streaming pipelines.
So, here's what I found out. Spoiler: It's really fun!
▶︎ Checkout Mage.ai: www.mage.ai
▶︎ Learn Data Engineering Academy: learndataengineering.com/p/ac...
Jump marks:
0:00 Introduction
1:00 Setting up Mage
1:42 Scaling
02:23 The user interface
03:01 ETL pipeline Data Loader
03:36 Data Transformer
04:00 Data Exporter
04:51 Data exploration with charts
05:50 Arranging pipelines
07:28 Running tests
08:10 Scheduling pipelines
09:13 dbt and Apache Spark
10:16 Summary
#dataengineering #datascience #bigdata
Негізгі бет The best alternative to Airlfow!? Mage.ai
Пікірлер: 40