Im working as AWS data enginner I found Your video very helpful pls keep uploading session 🙂
@AWSTutorialsOnline
Жыл бұрын
I will try my best
@puneetbhatia
Жыл бұрын
keep posting sir, your videos are really helpful. God bless
@AWSTutorialsOnline
Жыл бұрын
So nice of you
@60sfan1
Жыл бұрын
So nice video, I like the simplicity and short to the point. very clear. I suggests you link this with other videos for using Glue and S3 for ETL. Thank you
@AWSTutorialsOnline
Жыл бұрын
Thanks for the tip!
@nagrotte
4 ай бұрын
Good job - helpful content
@user-jy7cs7ry3u
Жыл бұрын
I wanted to express my gratitude for your AWS tutorials, which have greatly helped me in understanding the latest concepts. I am truly thankful for that. However, I'm encountering an issue with creating a job that was working fine when I attempted it for the first time. Now, I'm receiving an error message stating something like "auto job not supported" I'm unsure if this feature is still available on AWS. Can you please confirm its availability
@marino092109
Жыл бұрын
Hello sir, your videos are very helpful and I learned a lot. If you could cover also making continuous data ingestion from mysql workbench to aws s3 for study purposes.
@nishantkumar7751
9 ай бұрын
hello sir , why we use s3 as intermediate , can we direct copy data from warehouses to redshift
@dudilevi1
Жыл бұрын
Thanks for the informativ video, any ideas on how does it happen behind the scenes? Is there an sqs that is been created and an eventbridge rule for the copy command?
@AWSTutorialsOnline
Жыл бұрын
I don't know exactly as there is no documentation about it. But I think it should be raising S3 event which gets in queue and then processed to copy to Redshift database. I am just assuming - If I have to do - I will design like this :)
@sakshamjain1005
11 ай бұрын
what if i have ongoing updates in S3 bucket, I want old data + any new data that is coming in s3 bucket in RDS(not redshift) is that possible
@rahulpawar6908
Ай бұрын
Copy job works in large files ??
@peterpan9361
Жыл бұрын
do I have to execute the copy command manually everytime a new file is loaded in s3 bucket? how will the copy command trigger when a new file is uploaded to s3 bucket? What is the connection to trigger the copy job for new files coming in?
@AWSTutorialsOnline
Жыл бұрын
You just need to create a copy job out of copy command. Job will make sure it runs the copy command every time a new file comes to the s3 bucket.
@peterpan9361
Жыл бұрын
@@AWSTutorialsOnline how do I create copy job? just a copy command in Redshift query editor, does this means copy job? Or I need a proper glue job and inside that a copy command script?
@AWSTutorialsOnline
Жыл бұрын
You can create copy job using query editor. link for the job - docs.aws.amazon.com/redshift/latest/dg/r_COPY-JOB.html
@PraveenKumar-ic5zo
Жыл бұрын
I followed the same instructions. But files not copied. Please explain your IAM role/policies.
@AWSTutorialsOnline
Жыл бұрын
Please check this link. Hope it helps. docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html#copy-usage_notes-iam-permissions
@PraveenKumar-ic5zo
Жыл бұрын
@@AWSTutorialsOnline Thank you so much. it worked.
@mjr8771
Жыл бұрын
but how do you avoid insert duplicate data?
@AWSTutorialsOnline
Жыл бұрын
You cannot with this out of box method. You might want to look into merge scenario which can be used with Glue Job or SQL in Redshift. I already created a video about using merge operation in Glue Job, please have a look.
@imarunn
Жыл бұрын
Hlo sir, I need to stream s3 data to kinesis stream before that I need to confirm the event data in s3 is filtered and send only certain event type data. Please suggest me here or if possible make a video.
@AWSTutorialsOnline
Жыл бұрын
You can use Lambda transformation with Kinesis for that purpose. I will try to make a video on that.
@imarunn
Жыл бұрын
@@AWSTutorialsOnline Thanks a lot sir. If possible please put a video of how to read s3 data for given from date to to date in timestamp partitioned s3 bucket. (Pre-filter) Ex: - bucket/year/month/day/ Given date: Ex: 2022-july-1 to 2023-feb-30 I couldn't find a resource or solution for the problem statement. It would be helpful for me & others too.
@AWSTutorialsOnline
Жыл бұрын
Sure. I will try.
@praveensharma7178
Жыл бұрын
When i used redshift with glue, it will increase the cost of S3, it said the number of requests on s3 is very high.
@AWSTutorialsOnline
Жыл бұрын
The number of requests would grow because Glue is just a catalog and real data access is happening from the s3 bucket. What are you trying to achieve - copy data or redshift spectrum type of implementation?
@PraveenKumar-ic5zo
Жыл бұрын
please make sure to close your cluster and other resources used to avoid huge billing.
@AWSTutorialsOnline
Жыл бұрын
Thanks. I delete all my resources right after the recording :)
@sumitinnova
Жыл бұрын
How read only current date csv file using glue on daily basis
@AWSTutorialsOnline
Жыл бұрын
Can you please elaborate your question?
@sumitinnova
Жыл бұрын
@@AWSTutorialsOnline in s3 bucket there is a csv file coming on daily basis i need to read current date file only.
@AWSTutorialsOnline
Жыл бұрын
@@sumitinnova It works with new file only. so new files coming daily will be automatically copied.
@sumitinnova
Жыл бұрын
@@AWSTutorialsOnline ok...csv files are coming every day in same bucket but diff folder...same code will work or any change need to do in code.
@sumitinnova
Жыл бұрын
I tried to create auto job but redshift is giving error auto copy job operation not supported
@ArvindRawat-bn4pz
Жыл бұрын
can we do the same thing in Redshift Serverless bcz i tried in serverless query editor tool i found below error SQL Error [XX000]: ERROR: S3ServiceException:The operation is not valid for the object's storage class,Status 403,Error InvalidObjectState,Rid NA5Y24JKKNKCJBQK,ExtRid uR9CHHOzdGok5OYWHL80vRhXShE34hsimiTxYs5/nQ/HCI5wkP1nfi9Gzr4wbU73CSr+5Z7VNC8=,CanRetry 1 Detail: ----------------------------------------------- error: S3ServiceException:The operation is not valid for the object's storage class,Status 403,Error InvalidObjectState,Rid
@AWSTutorialsOnline
Жыл бұрын
I think it support Serverless. Please use this link to provision your cluster. docs.aws.amazon.com/redshift/latest/dg/loading-data-copy-job.html
@ArvindRawat-bn4pz
Жыл бұрын
@@AWSTutorialsOnline I fixed it already just after the comment posted. Issue was with my file in S3. Thanks anyways to look into it.
@avuthusivavardhanareddy5178
Жыл бұрын
i got below error ERROR: Auto copy job operation not supported [ErrorId: 1-63ff2326-5eeb20972d70b44761282298]
@akshaykadam1564
6 ай бұрын
@@avuthusivavardhanareddy5178 same here, were you able to resolve it?
Пікірлер: 45