I've never seen anyone explain Azure Databricks as well as you.
@vaibhavrana4953
3 жыл бұрын
you have explained Spark & Azure Databricks very well. Thank you
@harishjulapalli448
4 жыл бұрын
Great Intro to Databricks and Spark. Thank You.
@pashersil
2 жыл бұрын
Wow you mentioned the SSIS problems and ETL problems I totally relate to .. you have earned cred with me.
@rmravilla
2 жыл бұрын
Thanks for the presentation. It is very useful if one wants to learn Spark & Azure Databricks
@josecarlossilva3670
2 жыл бұрын
Great content!! It really helped me a lot. Congrats!
@chan7354
3 жыл бұрын
Explanation is very good and for us helped to understand the topic
@TheSQLPro
4 жыл бұрын
Excellent video!
@svapneel1486
3 жыл бұрын
Great Video. Made it very easy to explain
@Atmosera-
3 жыл бұрын
Great to hear!
@Pravinamadoori
2 жыл бұрын
Can someone suggest a good book useful to automate or testing ETL on AWS S3 using databricks?
@Shradha_tech
2 жыл бұрын
Thank you so much for this video 😀
@goselvam
Жыл бұрын
Thanks for the great video. Just wanted to let you know that the slide at 38:59 has the incorrect expansion for DAG, which is shown as Directed Acrylic Graph instead of Directed Acyclic Graph.
@Pravinamadoori
2 жыл бұрын
Instructor has given clear demo.. does he have any courses on Udemy?
@mangeshxjoshi
4 жыл бұрын
good video and explanation, one question , if on-premise Informatica ETL tool need to migrate to cloud platform . is there any equivalent cloud tool which can replace Informatica ? or can we use Informatica cloud integration on Cloud platform ( as PaaS ) service. How these traditional ETL tool like informatica will replace with cloud infrastructure . i believe , Databricks is for processing power . but i believe we cannot do ETL transformation in databricks . please suggest
@PhilipHoyos
4 жыл бұрын
You can use databricks for ETL.
@anmoltrehan6060
2 жыл бұрын
Azure Data Factory and Databricks will do the trick
@shiladityachakraborty9826
2 жыл бұрын
We can use IICS and Databricks. Infa BDM will cater to relevant ETL rules and Databricks will be there in order to visualise data. Now having said that it's always better to avail native functionalities of any tool, so if there is no tech upscaling issue it's better to handle etl through pyspark in databricks itself. This is cost effective as well.
@amjds1341
3 жыл бұрын
Great
@gopinathrajee
2 жыл бұрын
@ 29:50, when you say Azure, you mean the Azure PaaS? And by ExpressRoute, you mean Microsoft Peering? How does a VNet get created on the PaaS though? If it is a VNet does it not fall under Corp Network?
@Atmosera-
2 жыл бұрын
Azure PaaS can connect to VNet using private endpoints. ExpressRoute enables on premises connectivity back into the VNet.
@denwo1982
2 жыл бұрын
Hi, I'm coming from a SQl Datbase background and at the moment I am not seeing a benefit of using Azure Databricks? there is nothing stopping me using ADF to pick up a file from ADL gen2, put it into a staging table and then createding a stored proc to do the transformation and then inserting that into a destination table. Or am I missing something here?
@Atmosera-
2 жыл бұрын
Databricks is a way of doing ETL on Azure. ADF can do a lot, but it's much more limited in its scope. Doing complex transformations in Databricks tends to be easier. But if you're using ADF, there's nothing wrong with that.
@denwo1982
2 жыл бұрын
@@Atmosera- thanks for your reply, do you have any material in regards to deltas? So for example checking a csv file on ADL gen2 for new rows entered within the last hour based on the modified date? Or would it be a case of loading the file into a SQL staging table and compare this to the destination table, find the new rows and then putting that data into another csv file on ADL gen2 folder. Which tool would be more efficiency and cost effective?
@Atmosera-
2 жыл бұрын
@@denwo1982 I would not use deltas. That is a costly comparison to do. The best thing to do is partition your data in ADL into separate files that are timestamped and load new records that way rather than use a single file.
Пікірлер: 29