Unfortunately, does not work anymore after changes. Returns 404 constantly.
@Kemmydai
Жыл бұрын
Presently doing my MSc project and this video just pushed me forward so much! Thank youuuuu
@robmulla
Жыл бұрын
Really glad to hear that.
@mulberryJamz
Жыл бұрын
Hi Rob, I'm new to data science and I was wondering if you can make a video that shares the basics to webscraping from webpages. More specifically how to go about navigating and reading the parsed HTML texts of data to locate the strings we want like say content specific to a table or reviews on a webpages. Thanks for posting such easy to digest videos and sharing valuable tips to do things in a more efficient way!
@robmulla
Жыл бұрын
Edit: It looks like this package is no longer working for some people (Thanks Elon) If you have issues here are some tips: From the SNScrape github page: ``` If you discover an issue with snscrape, please report it at github.com/JustAnotherArchivist/snscrape/issues. If you use the CLI, please run snscrape with -vv and include the log output in the issue. If you use snscrape as a module, please enable debug-level logging using import logging; logging.basicConfig(level = logging.DEBUG) (before using snscrape at all) and include the log output in the issue. ```
@chaymabarkaoui6506
Жыл бұрын
even I install the last version, it still won't work
@AbdulRehman-nu2pb
Жыл бұрын
Yes it's not working 😭😭
@andygarcia6113
Жыл бұрын
You make it look so easy. Amazing vid!
@robmulla
Жыл бұрын
Thank you! 😊 Hopefully you can put it to some good use.
@JohnFallot
Жыл бұрын
oh man, this is incredible and exactly what I was looking for 🤩 Let us know when you decide to launch a Patreon!
@robmulla
Жыл бұрын
Thanks man! No Patreon for now, just share with others who might also apprecaite it.
@joaogollucci7877
Жыл бұрын
Amazing, will help me a lot with a project. Keep up whit the awesome content!!
@robmulla
Жыл бұрын
Great to hear! Thanks for commenting!
@philipomelo
Жыл бұрын
Awesome. Simple and straight 👌
@robmulla
Жыл бұрын
Thanks for watching!
@7382932
Жыл бұрын
Hi, thanks for the video super useful. I am quite new to all of this. What software are you using? And do you make those titles/write in each box separately? I am currently using PyCharm.
@rosiefish
Жыл бұрын
He's using Jupyter Notebook
@stellamukuhi5646
10 ай бұрын
This is awesome, nice and straight to the point!
@ritamchatterjee3307
9 ай бұрын
and its not working
@ahmadjibrilhemdi7639
Жыл бұрын
thanks! your progress bar is awesome!
@robmulla
Жыл бұрын
I appreciate it! I actually have a whole video on tqdm (the progress bar package) you should check out.
@nurulhanisahmuhammadhaasin885
Жыл бұрын
hi, thank you for the tutorial. do you know how to set the data for specified location or range?
@beethovennine
Жыл бұрын
Another nice vid! Good job!
@robmulla
Жыл бұрын
Glad you enjoyed! Share with a friend!
@beethovennine
Жыл бұрын
@@robmulla Always do!
@zoe-mv6ji
Жыл бұрын
I wish you were my lecturer/ tutor, I’m doing a conversion masters and the way you explain code is stopping me from having a breakdown 🤣🤣
@robmulla
Жыл бұрын
Ha! Thanks, I apprecaite that. I try to make things easy to follow, glad to hear you're learning from it!
@zoe-mv6ji
Жыл бұрын
@@robmulla you’re doing an amazing job at making it easy to follow. Thank you!
@vtulin3097
8 ай бұрын
Hello,Would you mind if ask if you can demonstrate how to use a crawler to capture twitter space cations. If the audio file is very long (more than ten or twenty hours), how to quickly capture it?
@omondifellow3120
Жыл бұрын
I'm geting 403 Forbidden error while trying to scrape tweets using the library. Does it means there is a restriction?
@gustavolabrador6192
5 ай бұрын
Hello, I wrote the code for the video, but I'm getting an error. It still runs as of this date due to the issue with the APIs that need to be paid for on Twitter.
@Laufeyson_
Жыл бұрын
Thanks for the awesome video, are you planning on doing NLP project?
@robmulla
Жыл бұрын
Thanks for watching! Have you seen my video on sentiment analysis? That has some NLP kzitem.info/news/bejne/sqaws4uHr6qJiqA
@CaribouDataScience
Жыл бұрын
Good stuff! Why did use a list first (data) rather than a data.frame?
@robmulla
Жыл бұрын
Thanks! Appending to a list is much faster than appending to a dataframe so typically I prefer to just creat the dataframe after all the data is collected.
@huyfe1347
Жыл бұрын
Hi Rob, thank you a lot for this video. I have two questions: 1. How to proceed if we want to scrap data on twitter with more queries like : #father; #mother; #baby; #foot etc.. Are we going to write one code for each query? 2. Did you make video on scrapping Facebook, Instagram, Mastodon, Reddit, Telegram like you did for twitter?
@devngecu
Жыл бұрын
use a for loop
@AminuteCrafts
Жыл бұрын
Hi I am getting an error at line 6 for loop that Error retrieving , errors blocked 403 and request failed to giving up
@sonnix31
Жыл бұрын
Sir, do you have a chat group where people ask questions? I followed your flight EDA, do you think predictions could be done with random forest ? Do you have something about this? (newbie here) . Thanks. BTW i like your videos.
@robmulla
Жыл бұрын
Yes I do! Join the discord: discord.gg/HZszek7DQc
@hemalatha_karuppusamy
10 ай бұрын
Thanks for the knowledge sharing .... When i am trying the same i am getting the 404 error .. Can you help me with the error rectification..
@ernestcholopray3545
Жыл бұрын
Excellent Rob. Would you happen to have tutorial sessions? if so kindly link me to it
@robmulla
Жыл бұрын
Yes! Subscribe and hit the bell icon and you’ll be notified when I go live. Also follow me on twitch.
@ggggg-hr6en
Жыл бұрын
Thank you so much. I have one question. When I use snscrape, I can only use "tweet.content, count, date, id, index, outlinks, outlinkess, tcountlinks, tcountlinksss, url". But I want to explore like counts, image url and so on. Would you tell me how to scrape them?
@robmulla
Жыл бұрын
I believe likes are in there as well as the attached images. But you might want to check test it out. If snscrape doesn’t work you’ll need to get a Twitter api key.
@j.4880
10 ай бұрын
This is gold. 6 minutes and so much information given without any unnecessary fluff... I'm amazed. Thank you so much.
@ozgurbuldum6829
10 ай бұрын
Could you make it work? I guess the most recent reply is yours, I get " blocked (404) " error with Python 3.10.
@mahlatsembooi3375
9 ай бұрын
@@ozgurbuldum6829 yeah, blocked!
@mariodiaz2700
9 ай бұрын
@@ozgurbuldum6829 it does not work anymore right?
@elu1
Жыл бұрын
Very effective teaching! could you have a video for exploring data scrapped from twitter?
@robmulla
Жыл бұрын
Thanks! I'd suggest checking out one of my live streams!
@milesonme
Жыл бұрын
Mate, how are u so good? What's your strategy? You code every single day? Or maybe you have many yrs of experience? What sort of courses or books do you study? How often do you read papers? I'm asking cz I learn so much jst thru recreating some of your projects and usually u have a very unique approach that I am used to, with very accurate results. I sure wanna be like you some day. Also could u keep the projects coming mate? I know there are tons of them on kaggle but it's quite frustrating to get stuck for us newbies a video explanation really helps.
@robmulla
Жыл бұрын
Thanks for the kind words. I really don’t think of myself as being better than any other person with the same amount of experience. I think the main thing I try to do is learn constantly and be patient. Things take time to learn. Check out my video about how to get started in ML for a more detailed breakdown of the skills to learn. Good luck.
@uzairmughal9383
5 ай бұрын
Instead of capturing tweets based on some hashtag, can we pull tweets from specific accounts? Like I have 5 accounts for whom I want the tweets data for a given time frame, is it doable from this method?
@DarthKidd
Жыл бұрын
Hi Rob. Thanks for the video. Did twitter update their security? Using your exact code from the first 3 minutes of the video, I keep getting an error retrieving api requests. Any tips on how to get past that?
@eve9587
Жыл бұрын
Sameee. Did you work around it?
@DarthKidd
Жыл бұрын
@@eve9587 I did not unfortunately. What about you?
@eve9587
Жыл бұрын
Dude use top=True It works now
@clintongomez
Жыл бұрын
@@eve9587 Hi, I am having the same problem with snscrape recently. I am quite beginner on this. Can you explain where to apply top=true? Thanks in advance.
@robmulla
Жыл бұрын
Looks to be an issue everyone is experiencing. You can follow the github issues here and they may update. Hopefully just a temporary issue... github.com/JustAnotherArchivist/snscrape/issues
@yunes7305
Жыл бұрын
Hi Rob! How much time it takes per 100k tweet? Just to compare with my scraper built in another language which takes approximately 10 to 15min to scape each 100k tweets.
@robmulla
Жыл бұрын
I think if you watch the end of the video the progress bar shows 35 tweets per second. So like 1 hour for 100k tweets?
@skez4774
4 ай бұрын
Is your method still working today ?
@Azouni01
Жыл бұрын
Thank you for this wonderful tutorial. Q: How can I retrieve infos like tweet content , username ... ,for a specific tweet using its id or url?
@AIvarogarciaa
Жыл бұрын
Hi guys, I have a problem with snscrape. The code works well but without any reason the process early stop at 188 tweets :/
@nikolakalev4914
Жыл бұрын
It seems like the snscrape module doesn't work anymore, due to returns an error. Do you have any information?
@FRANKWHITE1996
Жыл бұрын
Just subscribed ❤
@robmulla
Жыл бұрын
Thanks for subbing. Hope you like the videos.
@CarlosReyes-ku6ub
Жыл бұрын
I'd really love to see you Forecasting Time Series using Transformers based Neural Nets.
@robmulla
Жыл бұрын
I plan on making a video like this at some point. Only downside is that in my experience these models don't perform well unless the data is huge.
@CarlosReyes-ku6ub
Жыл бұрын
@@robmulla You're right, but given the versatility of Transformers Architecture and the actual importance on the State of the Art it's worthy to know it, I personally haven't found good materials and since your videos are awesome I'd very much like to see it.
@rishisharma6302
9 ай бұрын
Can i use this same code in vscode
@khouloudsafi1445
10 ай бұрын
Thank you very much for the video,,, SNScrape used to work for me before but now it shows blocked 404, after some research I found that twitter now is blocking scraping without login..... any idea how to solve the problem ? Many thanx for the efforts
@magdieladames7233
9 ай бұрын
Interesting, I was just going to start using this. 😢
@tarunanand4308
Жыл бұрын
Hey I am getting a 403 blocked error, I suppose the snscrape is going through API
@CaribouDataScience
Жыл бұрын
Everything works (in VS Code notebook) except the progress bar isn't displaying? It runs without error. Here is the code: n_tweets = 5000 for i, tweet in tqdm(enumerate(scraper.get_items()),total=n_tweets): data = [ tweet.date, tweet.id, tweet.content, tweet.user.username, tweet.likeCount, tweet.retweetCount, ] tweets.append(data) if i >n_tweets: break
@robmulla
Жыл бұрын
try importing the base tqdm instead of tqdm notebook: replace: from tqdm.notebook import tqdm with: from tqdm import tqdm
@CaribouDataScience
Жыл бұрын
@@robmulla Thanks that did the trick!!!
@jp-sc5du
Жыл бұрын
How do I get a specific tweet and retrieve the quotes and replies?
@ketannanoti20
Жыл бұрын
I tried but showing error after "for" statement, Error like : blocked (403), Error retrieving and message "Bad Authentication data" code 215 what I do now? Actually I have some twitter link and from that link I want like , view , RT count .. can you help me.. ?? How I code
@Technomic-o
Жыл бұрын
Amazing man
@robmulla
Жыл бұрын
Thanks 🔥
@mmfr3
10 ай бұрын
Does it still work with new API packages?
@valerionetophdcandidate
10 ай бұрын
with this twitter x update is not working anymore? I tried to run the "for tweet in..." and it did not work "Error retrieving ...."
@robmulla
10 ай бұрын
Yea, I think Elon broke it. :(
@ahmedshalaby185
Жыл бұрын
Hello, Thanks for the nice tutorial, I just wonder if there is a way to scrape only tweets instead of reply and tweets
@robmulla
Жыл бұрын
I’m not sure but it should be possible in the query method.
@ruirossi6241
Жыл бұрын
very nice. Thanks
@robmulla
Жыл бұрын
Most welcome
@iliecosminbiltan2639
Жыл бұрын
HI, Rob. I don't understand why i have this error: Errors: blocked (403), blocked (403), blocked (403), blocked (403) that's when i call scraper.get_items()
@sohaib5086
8 ай бұрын
same errror
@waleedmandourlecturer
Жыл бұрын
Thank you for sharing this amazing tool and wonderful tutorial. Q: How can I retrieve all replies to a selection of tweets?
@robmulla
Жыл бұрын
Thanks. I don’t know if that’s possible with snscrape. You might need the official Twitter api.
@abdelbaseermohamed1396
Жыл бұрын
The code is no longer working; it reports a problem of tweets blocking. Could you please check and help solve this issue?
@ayomideonikoyi
Жыл бұрын
Yh noticed thusalso, something seems to be blocked.
@amiinofficial6272
5 ай бұрын
how can i scrape specific tweets? pls answer me it is very important .
@user-jf3rf7jy9d
Жыл бұрын
if I have the tweet's url or ID, can I get its content?
@haileyjoseph7522
Жыл бұрын
May I ask if I want to scrape from one specific user? How can I do that?
@robmulla
Жыл бұрын
Great question. You can add any query search terms like you would in the twitter search bar. Here is a cheat sheet: media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf So to search a specific user you should be able to do something like "from:Rob_Mulla". Hope that helps.
@haileyjoseph7522
Жыл бұрын
@@robmulla Thanks for answering me! I'll definitely try that method 😁
@kanaparthyjaswanth9484
Жыл бұрын
The data seems to be truncated. could anyone please tell me how to deal with that ?
@a.d.a.k2143
Жыл бұрын
how do i scrape the english tweets alone?
@stanleychukwuemeka8841
13 күн бұрын
Is scraping millions of tweets still possible because i would like to build an AI model using the text data. Please if it's possible how can I go about it?
@tommyrieder4562
Жыл бұрын
When I run the code the progress bar turns red and becomes "1001/?". Did I make a mistake in the code or can I fix this?
@scarfface
Жыл бұрын
I think it's because that loop will actually gather 1002 results with n_tweets set at 1000. I think Rob was just trying to get ~1000, not 1000 exactly. Reason: enumerate starts from 0, and the loop is only broken once that number goes ABOVE n_tweets (e.g. 1001), so you end up with an extra result at both ends. Quick Fix: break the loop when i > (n_tweets - 2)
@robmulla
Жыл бұрын
Good catch! This is a bug. It can be fixed by subtracting 2 from the number of tweets in the line before the break.
@farhadnikhashemi8681
28 күн бұрын
Your shared videos are fantastic, Rob. Well done. By the way, I applied the above, but I was unable to scrape the data, assuming that Twitter does not allow escape without API. If anyone has done this successfully, please advise.
@huyfe1347
Жыл бұрын
how can we install snscrape?
@tantomanontroppo8582
Жыл бұрын
hi rob, i think this api doesn't work anymore. i think twitter blocked it
@marioaguilar8735
Жыл бұрын
How can I solve the 404 request error that I get when looping through the scraper object? Any idea?
@robmulla
Жыл бұрын
I think they’ve blocked it unfortunately
@CaribouDataScience
Жыл бұрын
What is the mark after the "for i ,"
@scarfface
Жыл бұрын
The comma after the "i" is to separate out the two things enumerate returns (index, value) in to the index "i" and value "tweet". Does that make sense?
@robmulla
Жыл бұрын
What Tim said! Thanks Tim!
@CaribouDataScience
Жыл бұрын
What would be helpful for me would be a video on how to connect Python to a SQL Server database.!!
@robmulla
Жыл бұрын
It's really easy with pandas. pd.read_sql() - you just need to create the connection object which depends on your database type.
@andyyang8876
Жыл бұрын
is there a way to pull say, latest 50 tweets from a specific user?
@robmulla
Жыл бұрын
Yes! I mentioned in a different response but you can use twitter's query to filter to specific users or dates. Check this cheat sheet: media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf So do something like "from:Rob_Mulla"... And setting your loop to only go through 50 would get you the latest tweets.
@impac143
Жыл бұрын
Is it possible to scrape profile information and followers list? thank you
@robmulla
Жыл бұрын
That’s a great question. There are ways to do this but I don’t believe snscrape can do it. You might need to apply for the Twitter api and use something like tweepy
@vedantpanchal2291
2 ай бұрын
big headstart in a project
@josephdickson5082
2 ай бұрын
does it still work? i doubt that. Was getting a 404 and 429 status codes last time i tried
@magicbrushes0
8 ай бұрын
it's not working for me I'm getting and error in retrieving the data 🤕🤕
@lumineersgaming4322
Жыл бұрын
why the tweets extracted showing only half text? why not full text plz help
@robmulla
Жыл бұрын
Oh no, I haven't experienced that. Have you checked the github issues to see if others have the same problem? github.com/JustAnotherArchivist/snscrape/issues
@techyoungster2713
8 ай бұрын
It is not working now days. Do you have any other alternatives?
@ishangoinyambo6523
4 ай бұрын
is this still valid with x though?
@behdad0979
Жыл бұрын
Does Twitter block you if create a bot with and get tweets every day?
@robmulla
Жыл бұрын
I’m not sure. Things are changing every day at Twitter. This package has been relatively unstable lately so you might want to check out their GitHub (linked in the description)
@TheFJudge-yc1ts
Жыл бұрын
an error 403 from twitter api occurs at get items
@rishavbhardwaj8044
Жыл бұрын
Hi, just wanted to ask, we need twitter API access to do this scraping, right? Cause when I run the same code, it sends a request to twitter API website and then an error is displayed.
@khadijadar3156
Жыл бұрын
did you find any solution
@rishavbhardwaj8044
Жыл бұрын
@@khadijadar3156 yes, I can scrape tweets now. It was a very simple code.
@TemporaryForstudy
Жыл бұрын
@@rishavbhardwaj8044 can you tell me how you fixed that?
@jaiswalrohan
Жыл бұрын
tell us how!
@rishavbhardwaj8044
Жыл бұрын
@@TemporaryForstudy I was using this code, but as of 2 days ago, it stopped working: import snscrape.modules.twitter as sntwitter import pandas as pd limit = 1000 tweets_list = [] for tweet in sntwitter.TwitterSearchScraper("NestleIndia").get_items(): if len(tweets_list) == limit: break else: tweets_list.append([tweet.date, tweet.content, tweet.user.username, tweet.user.displayname, tweet.user.description, tweet.replyCount, tweet.retweetCount, tweet.likeCount, tweet.lang]) df = pd.DataFrame(tweets_list, columns=['Date', 'Content', 'Username', 'Description', 'Language']) # Save the DataFrame to a CSV file df.to_csv('tweets_nestle.csv', index=False) print("Data saved to tweets.csv")
@louloumattinson994
11 ай бұрын
Hi Rob, I cannot seem to make it work, I am not sure if its something I am doing but it produces a 'Errors: blocked (404), blocked (404), blocked (404), blocked (404)' message and states 'failed, giving up', with a lot of links to twitter that show I don't have access. Do you have any suggestions? Thanks!
@MrChefMedz
10 ай бұрын
same issue
@yugpatel588
5 ай бұрын
snscrape no longer works for twitter. Elon has changed the policies making it hard to scrape without using the official API.
@aidanheffernan652
Жыл бұрын
This with tweepy makes for a very interesting reposting bot
@robmulla
Жыл бұрын
Interesting idea.
@user-dy9yo4yq9s
9 ай бұрын
My code is not working
@tonymok7752
Жыл бұрын
Why your jupyter notebook interface looks different than mine?
@robmulla
Жыл бұрын
I got into detail about my jupyter setup towards the end of this video: kzitem.info/news/bejne/lqaclpOYqYCTnqw - Hope that helps. In short: jupyterlab with solarized dark theme.
@tonymok7752
Жыл бұрын
@@robmulla Thanks. Imma do it with my setup
@milesonme
Жыл бұрын
Can u limit the tweets to only tweets in a certain language say English?
@robmulla
Жыл бұрын
Yes, use something like "lang:en" in the query. Check out this cheatsheet on twitter queries. media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf
@mohamed-muneer
Жыл бұрын
I have seen that you use jupyter lab in dark mode, how to activate dark mode?
@robmulla
Жыл бұрын
Solarized dark theme! Check out my tutorial video where I explain how: kzitem.info/news/bejne/lqaclpOYqYCTnqw
@EmailXpertise
Жыл бұрын
what software do you use for write the code ?
@OfficialB23
Жыл бұрын
it's look like jupyterlab
@robmulla
Жыл бұрын
Yes, jupyterlab - I have a whole video about it!
@murtazahussain8485
Жыл бұрын
The request was blocked by the twitter.
@hangyuli3328
Жыл бұрын
Sir, is it legal to scrape data on twitter without using twitter api now? I heard twitter shut down their free api.
@robmulla
Жыл бұрын
I don’t know. I’m not a lawyer but you should be ok.
@hangyuli3328
Жыл бұрын
@@robmulla thank you
@Ketan3101
Жыл бұрын
How can I get follower count of different users using this??
@robmulla
Жыл бұрын
I don’t think so. The Twitter api should give this easy enough though.
@jaiswalrohan
Жыл бұрын
ok people this aint working now dont waste your time here
@Pscytlk
Жыл бұрын
how do I get only people account and also their follower and following count?
@robmulla
Жыл бұрын
Not sure if you can do this with snscrape, but with tweepy you could.
@Pscytlk
Жыл бұрын
@@robmulla thank you sir!
@AbhishekKumar-wf9ey
Жыл бұрын
please help with the 403 error if you can.
@robmulla
Жыл бұрын
It’s possible they may have blocked this
@pedropablocalderon5012
Жыл бұрын
@@robmulla Fix it
@aliathar891
Жыл бұрын
can we access tweets older than 7 days?
@robmulla
Жыл бұрын
At least you used to be able to
@siddharthmahendran8186
Жыл бұрын
Will I be able to scrape data using google collab?
@robmulla
Жыл бұрын
You should be able to!
@evansyaran233
Жыл бұрын
sure
@muchammadfahd-a1985
Жыл бұрын
i have an error "Unable to find guest token"
@robmulla
Жыл бұрын
I'm not sure why that would be, works for me! Let me know if you get it working.
@aidanheffernan652
Жыл бұрын
came across this error you need to initialize the scraper object after before every run of .get_items()
@ananth4595
3 ай бұрын
the coding is easy but currently, it's not working. I tried many times 🤕
@muchammadfahd-a1985
Жыл бұрын
i've try now but sntwitter cannot get any items
@robmulla
Жыл бұрын
Oh no. Did Elon shut it down?!
@IshaqKhan010
Жыл бұрын
not working anymore
@danielhall6759
Жыл бұрын
Does anyone know if this method still works today?
@jaiswalrohan
Жыл бұрын
nope
@bishnudas8638
Ай бұрын
Hi Rob - I tried but it is not working anymore.
@patrickonodje1428
Жыл бұрын
Awesome
@robmulla
Жыл бұрын
Thanks Patrick!
@startupschool4895
Жыл бұрын
What shall we do for reddit, Make one analysing WallStreetBets.
@robmulla
Жыл бұрын
Great idea. Maybe I’ll make another video.
@WetBet
Жыл бұрын
What is the thing you’re using to run the script? I know nothing about coding so I’m confused
@amasoharnold4762
Жыл бұрын
Juypter notebook just google it and you’ll find it … or search anaconda It’s not the snake I promise…
@manarma7536
6 ай бұрын
it is not working anymore !
@muchammadfahd-a1985
Жыл бұрын
hi rob? are you have a new trick for scrapping? i need this for college. thank you for attention rob GBU.
@ipakolaf
Жыл бұрын
brother can you make a video on followers scarping
@robmulla
Жыл бұрын
Ohh. Great idea. I need to figure it out first. I think the Twitter api would handle that
Пікірлер: 247