absolutely brilliant. best scraping video so far. I like how you take your time to did down into the webpage and show us small tricks that otherwise i would have missed. Great work!
@codeRECODE
Жыл бұрын
Glad it was helpful!
@shubhamsaxena3220
2 жыл бұрын
One of the best video of web scraping i have watched....
@codeRECODE
2 жыл бұрын
Thank you!
@bkcy18
3 жыл бұрын
This channel is so underrated. It's truly a gold mine. Keep up the good work sir!
@codeRECODE
3 жыл бұрын
Thank you Binayak
@StefanoPollice-oz1bf
Жыл бұрын
Very good explanation of how JSON format for data is used on the web and how to manipulate it with python. I was looking for this so far. Amazing man. Bye all web scrapers
@0x007A
2 жыл бұрын
Thank you for demonstrating it is possible to avoid the overhead and complexity of Selenium when scraping websites. I am using wget to retrieve web pages and awk to extract data from those pages for a current project. I had almost resigned myself to using Python and Selenium until you reminded us how to use the built-in tools of a many web browsers to see the network activity and what is being retrieved by dynamic websites. I have used the Python + request approach without Selenium for other projects.
@codeRECODE
2 жыл бұрын
Glad that I was able to help you. Thanks for sharing!
@idk6465
2 жыл бұрын
You saved me so much time and trouble!
@pushkarsingh9081
3 жыл бұрын
Your video are really informational. We would like to see you more often
@codeRECODE
3 жыл бұрын
Thank you. Publishing a new video in few hours :-)
@haideralihassan5053
3 жыл бұрын
Very helpful video. Thanks 👍
@codeRECODE
3 жыл бұрын
Glad it was helpful!
@python360
3 жыл бұрын
Nice video! Also....Worst case: you can use Selenium to login and then GET the json in the way you demonstrated once you are logged in. ☺️
@codeRECODE
3 жыл бұрын
Thanks :-) Yes, Selenium has to be the worst-case scenario, always :-)
@nadyamoscow2461
3 жыл бұрын
Thanks a lot. It`s just a fantastic lesson!
@codeRECODE
3 жыл бұрын
You are welcome!
@KhalilYasser
3 жыл бұрын
Awesome. Thank you very much.
@codeRECODE
3 жыл бұрын
Glad you liked it!
@ch1.q476
2 жыл бұрын
Very useful, Thanks a lot
@codeRECODE
2 жыл бұрын
Glad it was helpful!
@AnilNandam
3 жыл бұрын
This is great, Thank you for the details. What if the data is loading dynamically via a websocket or something? How do we capture the data continuously?
@codeRECODE
3 жыл бұрын
Interesting idea for the next video!
@tnssajivasudevan1601
3 жыл бұрын
Great Sir 🙏
@codeRECODE
3 жыл бұрын
Thank you :-)
@shayhalfon5460
3 жыл бұрын
Thanks! How would you read the context of .PHP? I am getting error and no results.
@codeRECODE
3 жыл бұрын
Can share specific examples for me to look at? You can share it here but note that KZitem removes comments with links. Use this page instead - coderecode.com/submit-request/
@willistan3562
Жыл бұрын
Great tutorial, these videos are valuable! I have a question that relates to my day-to-day work, and I'm not sure if you've covered this in past videos. Essentially, I'm trying to automate the process of logging in to a payment gateway site (DLocal) and downloading financial statements. 1. With Selenium, only a blank page is returned when going to the login page -->> which makes me think there's some sort of anti-bot measure in place. Other sites like AMEX gave me no problems. 2. Since the entire page is rendered from JS, requests, requests-html, BS4 didn't yield much better results. The render() method from requests-html didn't work since only a blank page (empty HTML body) is returned ->> circles back to the antibot idea. Without the recaptcha token generated from the JS script, I can't make a POST request to authenticate. I'm fairly new to this whole scraping thing, so I'm wondering if there's any sort of workaround from your experience? Or if it's even feasible/recommended? Thanks in advance!
@codeRECODE
3 ай бұрын
Hey - sorry missed this comment. Hope you found the solution
@Rusputin-eo6tv
6 ай бұрын
does work with microservice website ? I can't find that json file
@codeRECODE
4 ай бұрын
Nothing to do with underlying tech. What matters is if that website is using an api or not.
@mnainwal2013
3 жыл бұрын
Thank u sir.
@codeRECODE
3 жыл бұрын
All the best :-)
@pythonically
2 жыл бұрын
how to convert json list to a dictionary?
@codeRECODE
2 жыл бұрын
in python, there is no such thing as json list. Json module helps to convert strings into python objects such as list, dictionary, depending on what the content is. As you have a list, look at he individual items to see what these contain.
@suryagawarikar4041
3 жыл бұрын
SIR YOUR TELEGRM OR SOME GROUP LINK WHERE WE COULD JOIN
@haideralihassan5053
3 жыл бұрын
Sir, can you share a CSV file of this code?
@codeRECODE
3 жыл бұрын
I didn't keep the CSV Haider. Why not try the code yourself?
@rostranj2504
3 жыл бұрын
interesting. nice 'the right tool for the right job' showcase. .. no need to bring in the big guns like scrapy
@codeRECODE
3 жыл бұрын
Thank you :-)
@pythonically
2 жыл бұрын
how to convert json list into json string? >>>type(json.loads(res.json())) 339 else: 340 if not isinstance(s, (bytes, bytearray)): --> 341 raise TypeError(f'the JSON object must be str, bytes or bytearray, ' 342 f'not {s.__class__.__name__}') 343 s = s.decode(detect_encoding(s), 'surrogatepass') 345 if "encoding" in kw: TypeError: the JSON object must be str, bytes or bytearray, not list i'm facing this error
@codeRECODE
2 жыл бұрын
res.json() returns a list (or a dictionary) To convert this to a string, you can use json.dumps() Or use response.text instead of response.json
@pythonically
2 жыл бұрын
@@codeRECODE thanks for the quick reply 👍let me try this
@pythonically
2 жыл бұрын
@@codeRECODE Hi , i'm still facing issue why in your case it is showing "dict" . but in my mine it show "list" >>>data = json.loads(res.json()) TypeError: the JSON object must be str, bytes or bytearray, not list can you tell me proper steps to do it ? l'm beginner so can understand your answer properly
@olebohm2092
3 жыл бұрын
Hey, I get this error Traceback (most recent call last): File "/Users/olebohm/Documents/checksandstripes/code/Price Compare/fbgm.py", line 18, in data = json.loads(res.json()) File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not dict and I don't know how to fix it :( I would be pretty grateful about help :)
@codeRECODE
3 жыл бұрын
Notice: the JSON object must be str, bytes or bytearray, *not dict* You already have a dict. the purpose of using Json.loads() is to convert string to dict. And you already have a dict. Note that you don't always have to call json.loads two times. most of the time, response.json() just works without any need of json.loads()
Пікірлер: 48