Thanks for providing this video. It was very helpful. Keep up the good content
@listentolearn2363
Жыл бұрын
Glad it was helpful! Thanks for your support! :)
@NextGenDevs_
Жыл бұрын
This video is incredibly informative; it's just what I needed, and it conveniently appeared in my feed. I do have one suggestion, though. I'm not sure if I'm the only one experiencing this, but the audio seems to be quite low. I found myself needing to set the volume to 100% to fully grasp the content.
@listentolearn2363
Жыл бұрын
glad you found it useful :) sure, thanks for letting me know about the audio. I will increase the volume level for the future videos.
@vishal0815
Жыл бұрын
Thanks for posting. Very well explained.
@listentolearn2363
Жыл бұрын
you are welcome! Thanks Vishal :)
@nhujashakya5075
4 ай бұрын
Thanks for the video, would you be able to show the video on how to do this using cloud formation template and how to hit the opensearch domain endpoint and ingest and query data
@vivekgowda1576
3 ай бұрын
Do we need to create index before uploading the file? I wanted to just insert the data dynamically is this possible? For example. I have a sample json using that I wanted to create an index without creating it manually.
@isaaclee3714
Жыл бұрын
Hi Host, hourly spark job dumps huge amount of data to S3 like 100GB. How can I configure Lambda to handle this case? Requirement is save data to S3 and also to OS clusters (but filter to 10%). Trying to find best option. Your help is highly appreciated. I hope I know how to do this.
@listentolearn2363
11 ай бұрын
Hello, You can try setting up an event driven arch to handle this - sample here: kzitem.info/news/bejne/sGignp9to2uef4I
@something3348
7 ай бұрын
can we access s3 buckets files in open search dashboard, search the any files from the s3 bucket via search option and download it from there? Please let me know
@listentolearn2363
7 ай бұрын
Hello, you can use ingestion pipeline to push the data either way (s3 to opensearch or vice versa). - docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-s3.html
@ashinshenoy84
3 ай бұрын
why is the volume so low ?? rest information is good.
@ramdevlinguberi8157
Жыл бұрын
Please do the needful, the requirement is vpc flow log or cloudtrail log in s3 should be ingested into opensearch. Your help is highly appreciable. We got stuck at integration part only.
@listentolearn2363
Жыл бұрын
The sample data that I have used in the demo is in json format and the VPC flow logs are not of json type. So you would need to change the python code to accommodate that.
@ramdevlinguberi8157
Жыл бұрын
How to split the content into segregated fields into Opensearch index, as data from s3 is goin plain and each line appearing as one field. How do we split a line into a number of fields.
@listentolearn2363
Жыл бұрын
you can do it by writing custom code in python and then write to the index as a single document.
@mamounjanfor4409
10 ай бұрын
Thanks for the explanation, i am getting an error when i try the send the a data from s3 to opensearch. i am using a lambdafunction and the error is 403
@listentolearn2363
10 ай бұрын
403 is generally a forbidden error. Could you please check if your IAM roles and policies have enougt privileges set?
@billgilbor7041
Жыл бұрын
Hi host, this video is quite helpful, can you make a video of the same project streaming s3 data into OpenSearch using Amazon Cloud formation. Also can you share the json code used in creating the index on OpenSearch dashboard.
@billgilbor7041
Жыл бұрын
If you can share the json or paste the code on your GitHub repo. Thanks
@listentolearn2363
Жыл бұрын
here is the json for creating an index - github.com/listentolearn/aws-opensearch-s3-event/blob/main/books-index.json
@listentolearn2363
Жыл бұрын
Please refer these videos to know cloudformation: CloudFormation: kzitem.info/news/bejne/kW-cwHx4mJ2YdnY CloudFormation Stacks: kzitem.info/news/bejne/woScqIOsr5SJmHo CloudFormation StackSets: kzitem.info/news/bejne/p2ewnniFk4Zzo20
@gangireddy7973
Жыл бұрын
How to load pre existing dynamodb data to Opensearch can you please do on this concept
@listentolearn2363
Жыл бұрын
Hi Gangireddy, There are few options to load existing dynamodb data into opensearch depending on the number of items in your table and long time maintenance Below are some of my suggestions, 1. You could setup dynamodb stream as explained in the video and have an additional attribute in the table say "marked" or "touched" and run update items against all items in the table. That will help in pushing existing data as well as all the new data coming in. However, if you are dealing with data in production env, there might be risk factor associated with modifying the production data direclty 2. If the number of items is less, then you can just do a scan and push it to elastic search. You can then setup a stream for on going updates. 3. If the number if items is really huge, then you can use data pipeline to export the data and then use a separate script or kibana to load the extracted data into elastic search. Hope this helps. Thanks for your interest.
@jonnetg
11 ай бұрын
I want to oposite, export opensearch index to s3 bucket, is it possible ?
@listentolearn2363
10 ай бұрын
You can create snapshots of opensearch index and store it in s3
@yaminit3013
10 ай бұрын
Hi ,Iam unable to see the result in opensearch will you pls help.
@listentolearn2363
9 ай бұрын
Are you seeing any errors in the lambda logs?
@ramdevlinguberi8157
Жыл бұрын
Hi Host, I tried the way you told. But when executing lambda function, getting an error stating that "Unable to import module 'lambda_function': bad magic number in 'charset_normalizer':b'U\ \ \ '", "error type": "Runtime.ImportModuleError". Please look into it, your reply is most appreciated 👍
@listentolearn2363
Жыл бұрын
Hi Ramdev, This error generally occurs if the python version of the lambda doesn't match the python packages version in the zip file that you are uploading to the lambda. Could you please confirm the python version that you are using for the lambda? The python packages in the zip file are of version 3.8. Please make sure you are using the same version. If not, please update the packages in the zip file according to your python version.
@ramdevlinguberi8157
Жыл бұрын
We tried now with python 3.8, JSONDECODEERROR on line 357, 340. Can we get any other resolution. Now this time json error is coming
@listentolearn2363
Жыл бұрын
JsonDecodeError implies that there is an issue with the json format. Make sure the json data that you are using is formatted correctly and follows json syntax. You can validate your json file here - jsonlint.com/. If you are using different data than the one provided in the sample, make sure you update the python lambda code to match your data attributes.
@ramdevlinguberi8157
Жыл бұрын
Hi I tried with own vpc flow log data uploaded into S3, the last step you run in dev tools is ignored. If you don't mind, can you create a video on the same but for a change the data format is vpc flow log which put in S3 and load as index into opensearch. Can you help me to send this log by s3 and lambda into opensearch. I am struggling from past 3 days and landed on your video. But still facing issues.
@sriharireddy4993
Жыл бұрын
Can you provide elaborate video. Because we are confusing last two minutes of your video.
Пікірлер: 36