Thanks for the amazing tutorials but still waiting for part III though 😁.
@wiss1998
9 ай бұрын
Thank you for your videos,waiting for more complex dataflow on azure data factory and how to automate dataFlow on azure.thank you
@CodeWithYu
9 ай бұрын
Yes, soon
@offisongemmanuel6932
10 ай бұрын
Nice video. Well done! Can the whole process be automated? Do we need to manually select the latest csv for every adf run?
@CodeWithYu
10 ай бұрын
Thanks for watching. To answer your question: potentially, it's a yes! But, it wont be on the UI. Would you be interested in a video showing the automation of pipelines on ADF? Please let me know and I can put a video up for you. Don't forget to like, subscribe and turn on notification to stay updated. Cheers, Yusuf.
@wiss1998
9 ай бұрын
why do we need to copie data from DL1 into DL2 ?
@CodeWithYu
9 ай бұрын
Usually you want to keep raw data separate for processed data, it's best practice! It also allow you to regenerate your data end to end.
@shandeepraula
11 ай бұрын
please create project with aws
@CodeWithYu
11 ай бұрын
Hi Shandeepraula7776, Thank you for watching and for your feedback! I'm glad you're interested in seeing similar projects on different platforms. AWS is indeed a powerful and popular cloud platform with a wide range of data engineering tools. I'll definitely consider creating a similar data engineering project using AWS services in the future. If you have specific AWS services or topics in mind that you'd like to see covered, please let me know. Your input helps shape the content and ensures it's relevant to the community. Thanks again for your support and suggestion. Stay tuned! 😊 Cheers, Yusuf.
@RafaVeraDataEng
6 ай бұрын
I am experiencing a big stuck on the moment to deploy airflow and sending data to Azure. as I run pip install adlfs, it comes another library pywin32 by default which make the docker container to crash... I'm struggling with this, trying to remove pywin32 from the requirements.txt, and when I do so, the container terminal shows a bunch of libraries versions incompatibilities... I've tried just copying Yusu's requirement.txt but still not working for me.. And every time I make pip freeze, the library versioning change by it self! Ohhh :( I keep on this trying to solve this.. Any suggestions, please are welcome!
@RafaVeraDataEng
6 ай бұрын
solved: ERROR: Could not find a version that satisfies the requirement pywin32==306 (from versions: none) remove pywin32==306 from requirements.txt set apache-airflow==2.8.0 in requirements.txt The conflict is caused by: 2024-02-19 13:29:25 The user requested Flask-AppBuilder==4.3.6 2024-02-19 13:29:25 apache-airflow 2.8.0 depends on flask-appbuilder==4.3.10 The conflict is caused by: 2024-02-19 13:31:02 The user requested fsspec==2023.9.2 2024-02-19 13:31:02 adlfs 2023.9.0 depends on fsspec>=2023.9.0 2024-02-19 13:31:02 apache-airflow 2.8.0 depends on fsspec>=2023.10.0 The conflict is caused by: 2024-02-19 13:32:06 The user requested cryptography==41.0.4 2024-02-19 13:32:06 apache-airflow 2.8.0 depends on cryptography>=0.9.3 2024-02-19 13:32:06 azure-identity 1.14.0 depends on cryptography>=2.5 2024-02-19 13:32:06 azure-storage-blob 12.18.2 depends on cryptography>=2.1.4 2024-02-19 13:32:06 msal 1.24.1 depends on cryptography=0.6 2024-02-19 13:32:06 pyopenssl 23.3.0 depends on cryptography=41.0.5 pip install "cryptography==41.0.5"
@CodeWithYu
6 ай бұрын
What Python version are you using? Have you tried downgrading your Python version?
@RafaVeraDataEng
6 ай бұрын
@@CodeWithYu python 3.11.4 on my env, python 3.9 on the docker image. I didn't downgraded python, I seeked for the compatible libraries instead xD (the hard way) Just followed the error logs retrieved by Docker terminal for airflow Server and Scheduler. The changes above worked for me. Anyway, it seems that the adlfs library add pywin32 by default and it crashes into Docker containers (Linux)
Пікірлер: 13