In AWS Glue, you can use workflows to create and visualize complex extract, transform, and load (ETL) activities involving multiple crawlers, jobs, and triggers.
In this video , I have explained about this concept in-depth.
You can explore more about Glue Workflow from this below link:
docs.aws.amazon.com/glue/late...
Code:
------------
github.com/SatadruMukherjee/D...
Architecture Diagram:
-------------------------------------
github.com/SatadruMukherjee/D...
Prerequisite:
------------------------
An automated data pipeline using Lambda, S3 and Glue - Big Data with Cloud Computing
• An automated data pipe...
How to Use AWS Glue with Snowflake | PySpark-Snowflake Connectivity
• How to Use AWS Glue wi...
Set up the necessary AWS services to query the data inside an Amazon S3 (Datalake) using AWS Athena
• Set up the necessary A...
Transform Data Using AWS Glue and Amazon Athena
• Transform Data Using A...
Schema Evolution in AWS Glue using Glue Crawler | AWS Athena
• Schema Evolution in AW...
Simplify Amazon DynamoDB data extraction and analysis by using AWS Glue and Amazon Athena
• Simplify Amazon Dynamo...
AWS Glue as Hive catalog
• Using the AWS Glue Dat...
A very frequent technical requirement in big data domain--
You have to write spark dataframe but with kms encryption, if you are using Glue , then this is one approach you can try to improve the security of your pipeline by enabling server side encryption
• Security Configuration...
Incremental Glue crawling using Amazon S3 Event Notifications
• Incremental Glue crawl...
Tracking Processed Data Using AWS Glue Job Bookmarks | Incremental ETL In-depth intuition
• Tracking Processed Dat...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZitem CHANNEL
Негізгі бет Ғылым және технология AWS Glue Workflow in-depth intuition with Lab
Пікірлер: 9