Amazon Kinesis Data Firehose is the easiest way to capture and stream data into a data lake built on Amazon S3.
By default, records are written in Json Format. But optimized columnar formats are highly recommended for best performance and cost-savings when querying data in S3.
Kinesis Data Firehose can now save data to Amazon S3 in Apache Parquet or Apache ORC format.
This video demonstrates the process of converting Json data to parquet using Kinesis Firehose and AWS Glue Data Catalog--
Prerequisite:
--------------------------
Building Serverless Data Stream pipeline using Kinesis data streams and Firehose for Snowflake
• Building Serverless Da...
Change Data Capture for DynamoDB Streams using Amazon Kinesis in-depth intuition
• Change Data Capture fo...
AWS Lambda function as a Kinesis consumer
• AWS Lambda function as...
Architecture:
------------------------
github.com/SatadruMukherjee/D...
Reference Document:
-------------------------------------------
Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena, and Amazon Redshift
aws.amazon.com/blogs/big-data...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZitem CHANNEL
Негізгі бет Ғылым және технология Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena
Пікірлер: 3