NOTE: To apply this method to a classification, replace SSR with Gini Impurity (or Information Gain or Entropy or whatever metric you are using). Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
@speakers159
3 жыл бұрын
I read some reference books like Introduction to Machine learning and Hands on Machine learning, but I didn't find any details about Decision Trees, or lets say other methods too like you covered!! Could you suggest some more references to get more deeper understanding?
@speakers159
3 жыл бұрын
I really wanted to try out the offline material too, but I'm still a student and can't afford it
@speakers159
3 жыл бұрын
And, Thanks a lot, this is really the best place for me to learn about machine learning as my first source, the explanations are both deep yet still accessible to a beginner
@statquest
2 жыл бұрын
@Pradeep Tripathi We use predicted and actual classes to calculate gini. For details, see: kzitem.info/news/bejne/wIJpn6aEb5mrao4
@NIHIT555
4 жыл бұрын
Josh: you have truly cracked how to use technology(slides/basic animation) to change the way we are teaching for decades. I wish all universities take a note from you and revise the way they are teaching
@statquest
4 жыл бұрын
Thank you very much! :)
@yogeshbharadwaj6200
3 жыл бұрын
yes, 100% true, lot of knowledge sharing just with simple visualisation, as you mentioned, way of teaching matters a lot.....Tks to our MASTER Josh Starmer once again for this awesome video/content !!!
@sagargoswami998
3 жыл бұрын
@@statquest I can confirm this. I am pursuing a Masters Degree in Data Analytics Engineering. And I have this course that is giving me a headache: Statistical Modelling. Your videos have helped me a lot, this is the the perfect stuff I was looking for. BAM!!!! BTW, I saw the video where you explained how your pop helped you and the StatQuest community in general, I totally get it why your videos are perfect. Double BAM!!!!! Seriously man, thanks a ton.....
@Moiez101
Жыл бұрын
seriously yes! I'm taking an online course from MIT....brilliant faculty but just oh so removed from the every day regular experience of learning as a student. Their slides, and teaching methods leave much to be desired. I don't understand anything they are teaching in stats or ML but Starmer is saving my life atm.
@gayathrigirishnair7405
6 ай бұрын
This is the best explanation of regression trees that I could find online. Professors are always too mathematical and programmers are too practical. You're explanation is juusssst right. Thanks a bunch for this!
@statquest
6 ай бұрын
Thanks!
@andyjiang8988
4 жыл бұрын
The channel with the lowest Gini score of likes vs. dislikes
@statquest
4 жыл бұрын
You get a DOUBLE BAM for that comment. Funny! :)
@pratyanshvaibhav
Жыл бұрын
sir i am learning ML from your videos and everyday i am forced to comment expressing the beauty with which the concept is explained..and the best part is you still clear our doubts even after 3 years..for those who don't know sir has also written a book which is too good
@statquest
Жыл бұрын
Thank you very much! :)
@pratyanshvaibhav
Жыл бұрын
Sir can i connect with you on linkedin
@statquest
Жыл бұрын
@@pratyanshvaibhav Linkedin limits the number of connections anyone can have, and I've hit that limit.
@pratyanshvaibhav
Жыл бұрын
Okay sir
@jiaxuzhang3527
3 ай бұрын
Absolutely brillian videos!!! I watched everything from the 1st one to this one in the list and understood so many things that I never understood in schools. I love your videos so much!
@statquest
3 ай бұрын
Awesome, thank you!
@gunupurugirija7201
8 ай бұрын
lol, that intro one who watches friends and Stat Quest would get it!! love your content its the best machine learning tutorials available
@statquest
8 ай бұрын
Thank you very much! :)
@auzaluis
Жыл бұрын
I don't know why I spend a lot of time googling if I always end up watching statquest haahahha
@statquest
Жыл бұрын
BAM! :)
@eramitjangra4660
3 жыл бұрын
I searched lot of thing for my project on ML to start from scratch. Then i landed here You nailed it. 🔥🔥🙏 Now i am on edge of completing my project thank a lot
@statquest
3 жыл бұрын
Awesome! Glad my videos are helpful. :)
@alecvan7143
4 жыл бұрын
Re watching after practising I can even further appreciate the quality of your explanations, thanks Josh :)
@statquest
4 жыл бұрын
Thank you very much! Good luck with your practicing. :)
@munawersheikh00
4 жыл бұрын
I love the way you say "BAMM"!!! Gives great relief during the video :) I want to say your style of teaching is great. The way you are explaining is making very easy for us to understand. In my opinion I can say "A difficult subject with easy to understand using your video lectures!" Thank you very much.
@statquest
4 жыл бұрын
Thank you! 😃
@mujeebrahman5282
4 жыл бұрын
The good thing about his videos is you just have to watch any video once and the concept will not leave you for a long time.
@statquest
4 жыл бұрын
Awesome! :)
@aop2182
4 жыл бұрын
This is awesome. I remember I was a bit confused when I was reading tree based methods in An Introduction to Statistical Learning. This really helps me understand it much easier when I can visualize it other than read some formulas. Thank you!
@statquest
4 жыл бұрын
Hooray! I'm glad the this video is helpful. :)
@vipul5340
4 жыл бұрын
I am reading the same book now and without this video series it's impossible to understand.
@ldk5007
Жыл бұрын
What a beautiful content! I'm not an English speaker, but His video is more helpful than the Korean lecture provided by the college I attending.
@statquest
Жыл бұрын
bam! :)
@triplefruition
4 жыл бұрын
Oh my Buddha! I'm falling in love with funny of your voice when you're explaining. Before I met your channel, my head is spinning round and round. I don't know what to do with my learning, but you came in and took me by big surprise. You made the abstract concept to be simple! Thank you!!!!! :)))))))
@statquest
4 жыл бұрын
Wow, thank you!!!
@dhananjaykansal8097
4 жыл бұрын
Ahhh Phobeee from Friends aka Smelly Cat. Haha good one Josh.
@statquest
4 жыл бұрын
Thanks! :)
@dengzhonghan5125
3 жыл бұрын
Thanks, Josh, every time I watch your video, I feel like the concept is very easy to understand! lol
@statquest
3 жыл бұрын
Happy to help!
@Serenity_Whisper_Music
Жыл бұрын
Thanks so much for your video. I've watched 8-times and have still one question. It is about 13:17~14:08 Could you possibly explain more details about the sentence 13:17 “Use the alpha values we found before to build trees(full and sub) that minimize the tree score”? My questions about this sentence are 1. How can we use alpha in building trees process? - I thought the way we build trees(full and sub) is the same as how we did in your video ‘regression tree’ *I understand tree score, alpha, and how alpha plays role in changing the tree scores of different size trees(full, sub trees) my question about the role of alpha(from whole data) in creating trees.. 2. If we can build trees with the same way as we did in ‘regression tree video’, why do we need this process of 13:17~14:08?
@statquest
Жыл бұрын
The idea is that when alpha is 0, we build a full tree just like in the original regression tree video. Then we increase alpha to the first value and that causes us to prune that tree a little bit. Then we increase alpha some more and then we prune the tree some more, until we've used all of the values from alpha we identified earlier.
@preranadas4037
3 жыл бұрын
Best video on pruning and tree selection till date!!!!!
@statquest
3 жыл бұрын
Wow, thanks!
@subashp7925
10 ай бұрын
Excellent explanation, you are the master in teaching, thank you so much much for your valuable effort
@statquest
10 ай бұрын
Many thanks!
@definitelynosebreather
3 жыл бұрын
Damn. I was finding only scientific articles and I was having trouble understanding the CCP, now you made it very clear! Thanks.
@statquest
3 жыл бұрын
Hooray!
@user-jj3we9jv9i
10 ай бұрын
Liked and Commented to help you with the KZitem algorithm.
@statquest
10 ай бұрын
Triple bam! :)
@rrrprogram8667
4 жыл бұрын
There are some notifications... Right when it shows on ur phone... ur feelings says.. I going to learn something today... MEGAAA BAMMMMM
@statquest
4 жыл бұрын
Hooray!!!! :)
@dprmiestwd5075
4 жыл бұрын
Thank you very much
@rashigupta1813
3 жыл бұрын
BEST CHANNEL! no i m not just saying, i m shouting!
@statquest
3 жыл бұрын
BAM! :)
@tymothylim6550
3 жыл бұрын
Thank you very much for this video! I really enjoyed the full step-by-step process of building the various trees using different alpha values and the use of cross validation to select the best alpha!
@statquest
3 жыл бұрын
Double bam! :)
@enlighteninginformation7647
4 жыл бұрын
This video really helped me to clearly understand the concept. Thank you for this good work
@statquest
4 жыл бұрын
Awesome! :)
@heerbrahmbhatt6917
4 ай бұрын
I'm 50% here for stats and 50% here for the sound effects!
@statquest
4 ай бұрын
double bam! :)
@LQNam
3 жыл бұрын
Thanks, Josh Starmer. The way of using train + test data to find a list of alpha, then use K-fold CV on train data to find out the optimal alpha leads to the data leakage.
@statquest
3 жыл бұрын
Noted
@bobbyfischer1672
3 жыл бұрын
At 12:05 you said that full tree has lowest tree score when alpha = 0. By default, pruned trees have higher SSR. Won't increasing alpha increase their tree scores even further, instead of lowering them?
@statquest
3 жыл бұрын
Increasing alpha increases the scores of all of the trees (it never lowers the scores). However, remember, we multiply alpha by the number of leaves in a tree. So a large tree will get a much larger penalty (alpha * number of leaves) than a smaller tree.
@bobbyfischer1672
3 жыл бұрын
@@statquest Actually I just figured out earlier that we have to increase the alpha on the previous tree structure to see the comparison. I thought we only need to increase the alpha only on pruned trees (to make it smaller). Thanks for the reply.
@graceqin5024
2 жыл бұрын
@@statquest Hi Josh, I am a beginner in learning data analytics. First I have to say our professor uses your videos in class and they really helped in understanding the course materials! As for the problem mentioned by Bobby, I also had problem with it until I saw your interaction here. I was stopped by "increase alpha until pruning leaves will give us a lower tree score."My problem was the same as Bobby. I thought increasing alpha only increase the tree scores and where should I start pruning leaves? The alpha and T get fuzzy in my mind. However, when i think "increasing alpha only increases the tree scores, i was comparing with just the full tree. Therefore the increasing of tree scores seems to have no stop. While actually, the stop is the SSR of the next tree where the leaves got pruned. At that point, the alpha is the knot which will be assigned to the next tree. (I hope I am right. Please correct if not.) Having this in mind, when I look at your words "increase alpha until pruning leaves will give us a lower tree score.", I understand it completely and I think it's very accurate. I would recommend to have some explanation here or make it more obvious in arranging of the graphical illustrations in the video. very lengthy comments. I was trying to organize my thoughts. Thank you again for wonderful videos!
@statquest
2 жыл бұрын
@@graceqin5024 I'm glad you were able to understand it! BAM! :)
@arda8206
3 жыл бұрын
One good idea for cross-validation maybe is we can split data first to train and test and again split train to train and validation sets. Therefore, we can guarantee that our test set is totally new to environment. This will result in more realistic scores.
@statquest
3 жыл бұрын
yep
@yrrep27
Жыл бұрын
Was looking for this comment.
@marieblanchemanche9217
Жыл бұрын
love the reference to Phoebe ! also thank you, all your videos are very helpful
@statquest
Жыл бұрын
Glad you like them!
@Bennilenny
4 жыл бұрын
Your intros make me smile :)
@statquest
4 жыл бұрын
Thank you! :)
@apurvagupta6217
3 жыл бұрын
The way you explain..and the amount of effort you put in the videos is great. I have learnt a lot from you sir. I always feel so positive and motivated while learning from you. Thank a lot🎈
@statquest
3 жыл бұрын
Thank you! :)
@thanhtungnguyen7500
4 жыл бұрын
thank you Josh for your very easy understanding explanation & lovely rhythm
@statquest
4 жыл бұрын
Thanks! :)
@visheshkushwaha9446
4 жыл бұрын
Explanation is TRIPLE BAM!!!
@statquest
4 жыл бұрын
Hooray! :)
@user-xn7qt3zl4m
4 жыл бұрын
The Best! The Best! The Best! video I've ever seen about Tree Pruning. Thanks a lot. Now I got the concepts. BAMM!
@statquest
4 жыл бұрын
Hooray!!! Thank you very much! :)
@thepresistence5935
2 жыл бұрын
Teachers all over the world, must learn from josh bro!
@statquest
2 жыл бұрын
Thanks!
@ahmedrejeb8575
2 жыл бұрын
this guy is a living legend ❤
@statquest
2 жыл бұрын
Thank you!
@dingusagar
6 ай бұрын
Thanks a lot for this. I came here after getting confused reading this concept from a book. I am inspired by your teaching style. Your style of teaching by examples is the best way to transfer knowledge without losing the audience at any point. May I ask how much time do you spend to create a tutorial like this? Also what kind of tools do you use to make these videos.
@statquest
6 ай бұрын
Each video takes a long time - maybe 6 weeks or more. And I talk about how I do everything in this video: kzitem.info/news/bejne/xKiCvn59Zndym6A
@soniasu2744
Жыл бұрын
you are literally doing god's work
@statquest
Жыл бұрын
Thank you!
@flaviofreire9323
3 жыл бұрын
Love the song in the beginning!!
@statquest
3 жыл бұрын
Bam! :)
@carabidus
4 жыл бұрын
Josh, the videos on this channel are nothing short of superb. I have only one suggestion: how about a dark theme for these presentations? That white background is like a supernova, especially on my 55" TV.
@chirags9774
4 жыл бұрын
Best intro of all the statquest videos which I have seen 😍
@statquest
4 жыл бұрын
Hooray! :)
@hellochii1675
4 жыл бұрын
As always the clearest explanation! Thank you so much 😊 and BAM BAM BAM
@statquest
4 жыл бұрын
Thank you! :)
@pushkarparanjpe187
4 жыл бұрын
So good! Consistently high quality across across videos and time! Keep going. Many thanks!
@statquest
4 жыл бұрын
Thank you very much! :)
@xinranwen4849
4 жыл бұрын
This video is awesome! Why didn't I know StatQuest earlier, it really helps, THX!!! btw: I'm confused about the built of trees in 13:19, how could we know that it is the bottom 2 leaves should be cut for a new training set when α=10000(pre-calculated), is that just a coincidence? Or the cut depends on which leaves give the lowest Tree Score?
@xinranwen4849
4 жыл бұрын
My understanding of the pruning process is this: ①use the whole data set to build a full-sized tree, and then increase alpha from 0 to get different sub-trees that are pruned and have lower Tree Score corresponding to different alpha ②build a new training set, testing set from whole data, then build a full-sized tree, using the alpha we have before to build sub-trees, then calculating SSR on testing data ③repeat ② til we done 10-fold cross validation ④for each iteration, choose the alpha that has the lowest SSR ⑤calculate the average among all alpha in ④ to get its final value Am I get it right?
@statquest
4 жыл бұрын
You're correct about everything except 5. We don't calculate the average of the alphas, we calculate the average of the sum of the squared residuals for each level of alpha, and select the level of alpha that corresponds to the lowest average sum of the squared residuals.
@xinranwen4849
4 жыл бұрын
@@statquest Thx so much for your reply and correction! I think I might get the process and concept a little bit, but still need more time to fully understand it, and your videos are just the most helpful, vividly, clearly! Thx again for all the efforts and sharing! I'm moving to SVM right now hahaha
@xinranwen4849
4 жыл бұрын
@@statquest and BAAAM!
@_curiosity...8731
3 жыл бұрын
@@statquest I am also having same doubt, can you please answer the original question? "how could we know that it is the bottom 2 leaves should be cut for a new training set when α=10000(pre-calculated), is that just a coincidence? Or the cut depends on which leaves give the lowest Tree Score?"
@andersk
2 жыл бұрын
At 12:20, let's say the right-hand side of the tree had node instead of just a leaf, and that node led to two leaves. In this situation, what would be pruned first to created the pruned comparison tree: prune the two leaves at the bottom of the left side only first because it's deeper? or the two leaves at the bottom of the right only first? or prune all ends with two leaves at the same time?
@statquest
2 жыл бұрын
We always remove the leaves that result in the smallest increase in SSR.
@yigithangediz2769
3 ай бұрын
excellent explanation. thank you
@statquest
3 ай бұрын
Thank you!
@karannchew2534
3 жыл бұрын
(Making notes for my own future reference) Tree Score, SSR + αT, is used to create a set of Prune Trees. Then, apply data (cross-validationally), to all Prune Trees. The Prune Tree (and its α value) that give lowest SSR (with testing data set) is the winner.
@statquest
3 жыл бұрын
Noted
@gundamdhinesh5379
3 жыл бұрын
Really love your content. Must watch content for any learner
@statquest
3 жыл бұрын
Thanks!
@stedev2256
2 жыл бұрын
Hello, first of all thanks for the great material you produced and shared, certainly among the clearest and effective I've come across. My questions are about the cross-validation trees to determine the right alpha values. As a premise, if I understood correctly, we first determine candidate alpha values by : a) create a "full" tree from the full training+testing datasets b) produce the corresponding family of "pruned" versions (and I guess asses their SSRs in preparation for the next step) based on the morphology of the "full" tree (meaning, all possible pruned trees are considered - is that correct?) c) identify the candidate alpha values as those by which the "full" tree's score becomes higher than one of the pruned versions. Assuming the above is correct, when we move on to cross-validate in order to ultimately determine the right alpha, I understand that we resample a training set (and a corresponding test set) for a number of times. Each time, we build a new tree from the training set, and its associated set of pruned versions (let me call these tress a "cross-validation family of trees" (CVFTs)), and assess their SSRs based on the test set for the current round in order to contribute to ultimately calculate the actual alpha to use. First question: how come every CVFTs in your slides has a number of members that equals the number of candidate values for alpha? couldn't a resampled training set might give rise to trees with more or even fewer leaves - and corresponding pruned versions - than the tree that was used to identify the candidate alpha values? And in that case, the candidate alpha values might be in larger or smaller number than the possible number of trees in the CVFTs at hand. I imagine that a possible answer is that the number of members in a CVFTs can actually be different than the number of candidate alphas, and that the pruned tress in a CVFTs are actually identified through their Tree Scores when each of the alpha candidate values is applied -- and if so I guess the issue is that perhaps this mechanism does not stand out 100% from the presentation... Second question: if we assess the trees in each CVFTs only by their SSRs, wouldn't always the tree with more leaves (therefore alpha=0) win? Thanks much
@statquest
2 жыл бұрын
What you wrote for b) "all possible pruned trees are considered" is not correct. When we remove a leaf, we don't just create all possible subtrees with one leaf removed. Instead, we pick the one subtree that, when we remove one leaf, results in the smallest increase in the sum of squared residuals.
@stedev2256
2 жыл бұрын
@@statquest Josh, OK, that makes sense -- so this is repeated on each new subtree to produce the set of trees where the candidate alpha values are then formulated as at minute 13:03, correct? If so, are my subsequent questions still standing?
@statquest
2 жыл бұрын
@@stedev2256 Each time we do cross validation we get a new "full sized tree", which may have a different size than the original. We then use the pre-determined alpha values to prune that new tree and use the test dataset to find out which tree (and alpha value) is best for that iteration. As for your second question, this is where the "testing" data comes in handy. A full sized tree with the most leaves (and alpha=0) will probably overfit the training data, and thus, do a pretty bad job predicting the testing data. So in practice, the full size tree (with alpha = 0) performs great with the training data (low SSR) but poorly on testing data (high SSR).
@stedev2256
2 жыл бұрын
@@statquest Josh, thanks, I was actually rephrasing / correcting my last post, and clarified a number of things to myself while doing that... I didn't think you could see my second post while I was editing it... sorry. But it was not all in vain, as what you wrote last confirms what I was getting to while revising my question and in light of your previous answer, and things seem clear now. Thanks much
@statquest
2 жыл бұрын
@@stedev2256 bam! :)
@user-bz8nm6eb6g
4 ай бұрын
Great explanation!
@statquest
4 ай бұрын
Thanks!
@beshosamir8978
2 жыл бұрын
Hi Josh , I hope u answer my question, I was searching for 3 days till now and i got nothing I have 2 problem which is : 1_ How to determine alpha where there is more one leaf in the bottom of tree (i.e : u said increase alpha till pruning this leaf get lower score) , so if i have more than one leaf in the last level of tree, which one should i cut or should i look for all subtrees every time increasing the alpha it seems like it will get high complexity? 2_ in implementation when i will give the model the ideal alpha to implement the decision tree, how the model will know when building it in every step he take is that will lead to the subtree related to this alpha finally , u r such amazing i really enjoyed every lesson i took from this channel
@statquest
2 жыл бұрын
1) You remove the leaf that results in the smallest increase in SSR. 2) You build the full tree, and prune just like we did before.
@beshosamir8978
2 жыл бұрын
@@statquest Thank u josh ,I really appreciate ur time for answering my questions
@mathematicalninja2756
4 жыл бұрын
This channel is gold mine i am telling ya :D can you cover box cox theorem (power transformations)
@sadranezam3367
4 ай бұрын
came for the pruning technique , stayed for the "dididodidodido"
@statquest
4 ай бұрын
bam!
@wabsy845
3 жыл бұрын
It is such a great explanation. Super helpful! Clear and fun! Really appreciate your time in making the video. Thank you!
@statquest
3 жыл бұрын
Glad it was helpful!
@amnont8724
Жыл бұрын
Hey Josh, how can we choose our alpha wisely? So that the tree with the minimum tree score will really work well for testing data too. Is there a specific rule of thumb?
@statquest
Жыл бұрын
I give a practical tutorial on building trees with real data here: kzitem.info/news/bejne/0m9mu3h7oIuimH4
@juliankoch5704
2 жыл бұрын
i'm curious as to how we find sensible alphas. there seems to be an explanatory gap here since in the section "comparing pruned trees with alpha" it says (in the NOTE at 8:23) that we find it during cross validation, but in the cross validation section at 13:18 we are supposed to "use the a values we found before". sensible alpha values would probably vary widely depending on the SSRs of the trees (and ultimately the variable ranges) and even more so once we do classification since gini and entropy give small values for which alphas of 10k would not work usefully. surely simply guessing various alpha levels and finding which of the guessed ones work best in cross validation is not the best method or am i misunderstanding something here?
@statquest
2 жыл бұрын
We use all of the data to find candidate values for alpha (this is demonstrated at 11:18). Once we have candidate values, we test each one with cross validation to find the optimal value for alpha.
@amarakbar2374
4 жыл бұрын
Wonderful explanation. Thank you very much
@statquest
4 жыл бұрын
You are welcome!
@rrrprogram4704
4 жыл бұрын
we love you DOSS.. hope me too will surely one day be a patreon member
@jcatlantis
4 жыл бұрын
How new trees are build by imposing previous alpha values? Maybe it is not possible to find new continuously smaller trees of reduced Tree Scores for fixed alphas. Before alpha was a parameter and during cross validation is a constraint :(
@knightedpanther
Жыл бұрын
Thank you so much for this amazing video. Very Amazing!
@statquest
Жыл бұрын
Thank you!
@skumarr53
4 жыл бұрын
Just a random thought, what if we prune the tree directly based on ssr computed on validation set instead of adding penalty. Anyway the tree that works well on validation set is selected. Why are adding penalty?. Does it helps controlling fluctuations in the number of leafs selected across cross validation folds during Hypertuning .
@BalistikJumbo
4 жыл бұрын
As always awesome! Thank you Josh!!! Horraaaayyyy
@statquest
4 жыл бұрын
Thanks! :)
@bibiworm
3 жыл бұрын
At 12:33, it reads on the video that "we increase alpha again until pruning leaves will give us a lower tree score". My question is lower than what? My understanding is lower than tree score of the full sized tree at that specific alpha value, in this example, at alpha=10000, because we have already established the tree score of the full sized tree at alpha = 0 has the smallest tree score. Similarly at 12:43, the third tree at alpha = 15,000 is chosen because it has lower tree score than the second tree at alpha = 15000. Please let me know if this is correct. Thanks.
@statquest
3 жыл бұрын
Yes
@dongli7157
4 жыл бұрын
Hi Josh, those explanatory vidoes are incredible. Thanks for the great work!
@statquest
4 жыл бұрын
Thank you!!!
@zhengwu7240
4 жыл бұрын
Hey Josh, Great video ! Quick questions: for classification tree, do we simply replace SSR with gini impurity and follow similar steps?
@statquest
4 жыл бұрын
Yes.
@ishikajohari1508
2 жыл бұрын
You got me at Smelly Stat!
@statquest
2 жыл бұрын
bam!
@charlottel9534
4 жыл бұрын
Thank you so much for posting this!
@statquest
4 жыл бұрын
Hooray! :)
@lolikpof
Жыл бұрын
Cost Complexity Pruning is otherwise known as post-pruning, while limiting the tree depth, enforcing a minimum amount of samples per leaf/split, a minimum impurity decrease is known as pre-pruning, correct? My question is, can you apply pre-pruning first, and then apply post-pruning to the pre-pruned tree? If yes, then i assume that the alpha parameters will be found from the pre-pruned tree, right? Not the initial full tree? And then cross validation will also be performed with the pre-pruned tree, not the initial full tree, to determine the final optimal alpha score? And on a separate note, should only those alpha obtained from the tree trained on all the data be used when cross validating, and why? Is there no chance that some other, random alpha, might result in better performance? Considering that cross validation is done on several different test/train splits, and there will be those that do better with one alpha, and those that do better with other alphas, doesn't it make sense to try all possible alphas (from 0 to infinity) in the cross validation, not only those that give the best tree scores for the full tree? Isn't there a chance that some other alpha will give, on average, a lower sum of squared residuals than those obtained from the full tree?
@statquest
Жыл бұрын
I believe you are correct about how pre-pruned trees are used. And this is just how the algorithm is spelled out - possibly to keep the running time down.
@InsightsWithAkshay
4 жыл бұрын
In the video, you selected 4 values of alpha i.e. 0, 10k, 15k and 22k. How did you come up with exact alpha values? In the real-world, the dimensions of data will be huge so how to decide value alpha?
@statquest
4 жыл бұрын
In the real world, you keep track of the sum of the squared residuals at each point in the tree. You can then figure out what values for alpha work directly from that information.
@graceqin5024
2 жыл бұрын
Hi Josh, It is my understanding that we 1) build a full tree will full data; calculate SSR for the full tree and all tiers of pruned trees as 7:19 shows; 2) define different alpha gates and the correspondent pruned trees as shown 12:53; 3) Use 10-fold cross validation to decide alpha and the tree leaves (T) comes with alpha. I would like to confirm if I understand it correctly. Thank you and also thank you for wonder videos!
@statquest
2 жыл бұрын
I think that is correct.
@graceqin5024
2 жыл бұрын
@@statquest Thank you very much!
@karannchew2534
3 жыл бұрын
Hi Josh, I don't get this part: 12:10 "increase alpha until pruning leave give a lower tree score" What does "lower tree score mean"? Lower than the the score from the previous round? The first (full) tree use alpha of zero. The second tree, pruned, will have a higher SSE, hence higher Tree Score. A non-zero alpha will only increase the Tree Score further. So, how is it getting a "lower tree score"? Thanks, Kar Ann --- Notes for my own future reference: *WHY PRUNING* To avoid over fitting, a less full tree is needed. Pruning procedure uses tree score that balance between classification error and number of leaves. *OVERALL STEPS* 1. Create a set of Prune Trees. Use Tree Score to decide how much to prune. The trees should have comparable "quality" in terms of error and number of node. Tree Score = SSR + αT 2. Then, apply different subset of data (cross-validationally) to all Prune Trees. The Pruned Tree that give lowest SSR (with testing data set) is the winner. *DETAILED STEPS* Tree Score, TS = SSR + alpha* T (T=no. of leaves) TS is used to decide how much to prune, and to create different trees. *1. Create a set of Prune Trees, which have comparable quality, in terms of error and number of leaves* Create the 1st trial tree. The fullest tree. Tree1. α1=0. Get TS1. Create the 2nd trial tree. A new tree. A pruned version of Tree1. Tree2. α2=a new value bigger than 0. Compare the Tree Score between the old (full) and new trees (of smaller size i.e. pruned). Even though the newer (pruned) tree has a bigger SSR, it has a lower T. At α2 = 0, the newer tree has a higher TS than the old tree. But the new tree TS increases slower than the old TS, because the old TS has a higher T. Eventually, with a certain α2 value, the newer tree has a higher TS. Get TS2, using the new α2. Create the 3rd tree. A pruned version of Tree2. Get TS3, using the new α3. Repeat and create more prune trees. Each with a α values. Now, there are many set of tree. First tree is a fill tree. Last tree is only one leaf. All tree comparable quality, based on fine balance between classification errors and number of leaves. *2. Apply different set of data, cross-validationally, to all trees. The Prune Tree that give lowest average SSR is the winner* .Create different values of alpha from the above eg. α1 - α5 .Use different set of training and test data i.e. 10 folds cross validation .From the 10 folds training and testing data, get SSR, with α1 - α4 .Get the SSR of each α, averaged over 10 sets of training and testing data .Pick the tree, and its associated α, that gives the lowest SSR The fullest tree, though gives the lowest SSR with the training data, it doesn't give the lowest SSR with test data due to over fitting. Cross validation also helps to minimise impact of over fitting.
@statquest
3 жыл бұрын
The tree score is the SSR + alpha*number of leaves. We increase alpha until it makes sense to reduce the number of leaves in order to get a lower tree score. Although reducing the number of leaves will increase SSR, if alpha is larger enough, it can overcome that increase. For example, if a full tree has 10 leaves and SSR = 10, and the same tree with 9 leaves has SSR = 20, then when alpha = 11, the tree score for the tree with 9 leaves will be smaller than the tree score for the tree with 10 leaves. In other words: 10 + (11 * 10) > 20 + (11 * 9) -> 120 > 199
@karannchew2534
3 жыл бұрын
@@statquestI got it now. Thanks! But one more question.... Why "We increase alpha until it makes sense to reduce the number of leaves in order to get a lower tree score"? Doesn't it make better sense to say "We reduce the number of leaves and increase alpha until a lower tree score"?
@karannchew2534
3 жыл бұрын
@@statquest Thanks Josh
@karannchew2534
3 жыл бұрын
@@statquest Sorry, Josh, Re-watching it... and more questions sprung up. Q1) At 13:17: "Using the training data... use the alpha values we found before to build a full tree and a sequence of subtrees that minimise the Tree Score" Why do we need to build trees again? Weren't they already created in the previous steps, when the alphas of subtrees were evaluated? Q2. The objective is to apply different folds of data to find out which subtrees give the lowest average SSR. Why is alpha needed here again? Alphas were used in Tree Score, to create the subtrees. But SSR doesn't need alpha, right?
@statquest
3 жыл бұрын
@@karannchew2534 1) The first set of trees were built with all of the available data (not including a validation dataset if you have one), the subsequent sets of trees are built with different subsets of all of the available data and this allows us to pick the best value for alpha using k-fold cross validation. 2) Correct. Ultimately we pick the tree with the lowest average SSR after cross validation.
@JoRoCaRa
Жыл бұрын
you are awesome! clear! to the point!
@statquest
Жыл бұрын
Thank you! :)
@theblindcritic5876
4 жыл бұрын
This is brilliant work! Thanks a ton!
@statquest
4 жыл бұрын
Thanks! :)
@user-fi2vi9lo2c
11 ай бұрын
Dear Josh, thanks a lot for this video! It's awesome! You told us how to prune regression trees and your explanation was very clear. I've got a question, how can I prune classification trees? What is the biggest difference between regression trees and classification trees when pruning? I guess that the Tree Score is calculated in a different way when we prune classification trees. Can we simply add tree complexity penalty (alpha*number of leaves) to Gini Impurity to get a Tree Score in case of classification problem?
@statquest
11 ай бұрын
You pretty much add alpha to the total gini impurity for the tree. See: scikit-learn.org/stable/auto_examples/tree/plot_cost_complexity_pruning.html
@hopelesssuprem1867
9 ай бұрын
Hi Josh! There is a mistake in your explanation. In the comments, you said that in order to find alpha, you need to do a complete search of them, adding 1 to the alpha value until it becomes large enough to make sense to prune, but this is a fundamentally incorrect statement. Alphas from cost_complexity_pruning_path are found using weakest link pruning and this is done as follows: for all subtrees in the tree, alpha is calculated as the difference between the weighted impurity of the root node of the subtree and its total weighted impurity in the leaves. The node/subtree with the smallest alpha is deleted and falls into the cost_complexity_pruning_path array, and then this procedure is repeated from the bottom to the root node in the tree. After the selection of the best alpha from cost_complexit _pruning_path is carried out on a test set or using cross-validation.
@statquest
9 ай бұрын
That's one way to implement the algorithm. Increasing alpha by 1 and checking, is another. So, to be clear, you could achieve the same goal either way, and I should have been clearer with my language that increasing alpha by 1 was just one option.
@Rectalium
4 жыл бұрын
Your channel is amazing man. Great job
@statquest
4 жыл бұрын
Thank you! :)
@sushilchauhan2586
4 жыл бұрын
black kitty white kitty STATQUEST is best!
@statquest
4 жыл бұрын
Nice! :)
@shashanksundi5669
3 жыл бұрын
Thank You !! Just perfect :)
@statquest
3 жыл бұрын
Thanks!
@leozhang4450
3 жыл бұрын
Hi Josh, Thanks for this GREAT video!!! Just wanted to ask what's the principal to choose several alpha as the starting values? ie. in the video, you choose these 3 alpha values as the candidates to determine the final alpha: 10,000, 15,000, 22,000, how did you come up with these three alpha?
@statquest
3 жыл бұрын
I started with alpha =0. I then increased alpha until I got a lower SSR + alpha score by pruning a branch. That was my second value for alpha. I then increased until I got a lower SSR + alpha by pruning another branch...etc.
@leozhang4450
3 жыл бұрын
@@statquest Thanks! Make perfect sense to me. Hooray! : )
@fredrikedin8880
Жыл бұрын
@@statquest Thank you Josh! Really nice videos! I am looking at most of them! I rank them by the total BAM count of each video :) One thing I don't understand is: in the example above the full tree and the tree with one pruned leaf would have an equal tree score at alpha = 5494.8 - 543.8 = 4951.0, whereas the trees with one and two pruned nodes would have an equal score at 19243.7 - 5494.8 = 13748.9. So instead of choosing a candidate alpha of 10000 as you did in the very first step, you could have chosen any value 4951.0 < alpha < 13748.9. Would perhaps the best value to choose for a candidate alpha then be the mean of 4951.0 and 13748.9 = 18699.9 rather than 10000?
@statquest
Жыл бұрын
@@fredrikedin8880 What time point, minutes and seconds, are you asking about?
@fredrikedin8880
Жыл бұрын
@@statquest Sorry for my unclear question. Between 12:00 and 12:54 you describe that you increase alpha until pruning one leaf renders a lower score. So the border between the full tree and the tree with one pruned leaf is at 10000 here, and the border between the trees with one and two pruned leaves is at 15000. Then couldn't the candidate alpha for the tree with one pruned leaf be set to 12500, i.e. the middle of the interval where this subtree is optimal, rather than at the border to the full tree?
@hellochii1675
4 жыл бұрын
Happy early Thanksgiving 🦃 💥 💥 💥
@statquest
4 жыл бұрын
Thank you! I'm sooooo excited about the holiday. :)
@SuperNexus14
4 жыл бұрын
Thank you for the video, I just have one question. We use pruning to figure out the optimal number of leaves for the tree we build using the original training data. The purpose of the original validation/ testing data was to validate the performance of this tree, as we did not use that data in the creation of said tree. Now in the pruning process, the original testing/ validation data is used in conjunction with the original training data to determine alpha. This means, that the original validation data had a large impact on the tree we created from the orinal training data and is therefore no longer independant from it. How is this procedure justified and would it be necessary to get a third independant data set for a true validation of the trees predictive performance?
@statquest
4 жыл бұрын
Often people reserve a 3rd set of data only to be used after pruning for true validation.
@psyysp171
3 жыл бұрын
The term testing data is used a bit misleading here. You usually have 3 sets of data: Training, Validation and Testing. The Validation data is used to optimize the model (in this case the pruning). The test data is seperate and should - as you said - be completely independent from the training data. I've noticed that quite some people use 'validation' and 'test' data synonymously, when they are in fact two different things.
@ltoco4415
2 жыл бұрын
Thank you for the amazing content Josh! I had a doubt, do we change the alpha and the number of nodes simultaneously cause I thought that we change just alpha and then check which tree performs better (keeping one thing constant and changing the other gives us a better way to compare). Also at 11:32, you used the entire data (train+test) to build the model and find different values of alpha and then used those values on train split to build the model, compare test scores and finally find the best value of alpha. If we use the train+test data to find alpha, isn't that causing data leakage? Also, why do we compute the alpha values beforehand? why don't we do it the usual way(the way we do in ridge regression) where we find the optimal value of regularization parameter using GridSearchCV because anyhow alpha is analogous to regularization parameter (I think).
@statquest
2 жыл бұрын
1) We find values for alpha to result in better performance after pruning. 2) We can keep some data set aside for validation. 3) Perhaps for performance reasons.
@shawnkim6287
Жыл бұрын
@@statquest sorry. I don't get your answer for #1. Can you please elaborate more? Why do you find alpha using test+training data? If you just use the training data to find alpha, you will get an alpha to result in a good performance, but it's better if we use both training and testing set. CAn you please confirm?
@statquest
Жыл бұрын
@@shawnkim6287 Presumably using more data results in better performance.
@shawnkim6287
Жыл бұрын
@@statquest Thank you. Can you please confirm what I understand from this video? You pick a number of alphas from a set which combines both training and test set. First, you have the full (untouched) tree where alpha is 0. Then, you increase alpha until pruning leaves will give us a lower tree score. You increase alpha until pruning additional leaves will give us a lower tree score. Long story short, a set of alphas are derived from the (training and testing) sets. You train the models using a set of alpha's that we calculated before. We then test the trained models by picking the optimal alpha value from the alphas by selecting the lowest SSE in the first fold which is the testing set. You pick the next optimal alpha value by selecting the lowest SSE in the second fold as the next testing set. If you are using 5-fold CV, then you have to pick 5 optimal alpha's on 5 different testing set. You just select the best alpha that appear the most. can you please verify if what I understand is correct? Thank you, Josh!
@statquest
Жыл бұрын
@@shawnkim6287 That seems reasonable to me. If you want to double check, you can find the algorithm in The Introduction to Statistical Learning in R (which is a free PDF if you look for it).
@yashyadav5605
4 ай бұрын
Smellyyy cat Smellyyy caaatttt What are they feeding youuu😂😂😂😂😂😂
@statquest
4 ай бұрын
bam!
@yashyadav5605
4 ай бұрын
@@statquest your reply,reply double bam
@moiseiloo3809
Жыл бұрын
Great explanation. I really enjoyed the video, but I'm a bit confused. Why use all of the data to build the initial tree in step 1? If we do that there is no test data left for testing the tree.
@statquest
Жыл бұрын
This is assuming you've already separated out your testing and validation data.
@itisakash
4 жыл бұрын
Hello, This is a clearly explained video and thanks for this entire series. Can you please let us know how to implement the pruning using sklearn in python?
@statquest
4 жыл бұрын
That's a good idea! I'll put it on the to-do list.
@yulinliu850
4 жыл бұрын
Thanks Josh!
@statquest
4 жыл бұрын
Thanks! :)
@corrinechou5271
2 жыл бұрын
Well explained, thank you so much
@statquest
2 жыл бұрын
Glad you liked it!
@longma7042
4 жыл бұрын
greate , thank you for your work . very clear
@statquest
4 жыл бұрын
Thanks! :)
@simaogoncalves1957
21 күн бұрын
Phoebe lives on
@statquest
21 күн бұрын
YES! :)
@mcapro
4 жыл бұрын
You are the best.
@statquest
4 жыл бұрын
Thank you, and thank you for supporting me! :)
@cuonghoba681
2 жыл бұрын
Very clear. Thank you very much
@statquest
2 жыл бұрын
Thank you!
@fiammettastrazzerapernicia3287
4 ай бұрын
you had me at the first 5 seconds
@statquest
4 ай бұрын
bam!
@tamaskiss3237
Жыл бұрын
Thank you for these videos Josh, I really love learning from them. Just one question, when we do the cross validation, should not the alphas be different compared to those in the full sized training data and also on the different cross validation set? If yes, how should we decide which alpha should get the most vote as they are basically different on every training data?
@statquest
Жыл бұрын
To be honest, I don't know how it is implemented in practice, but I would guess that the alphas are in comparable ranges.
@mohamedhanifansari9224
4 жыл бұрын
Thank Josh, for your lucid explanation. We are longing for the XGBoost Videos. Any updates on that ?
@statquest
4 жыл бұрын
This was actually the very first XGBoost video. XGBoost uses unique trees and to understand why, you have to know everything about normal regression trees. This information was originally in a XGBoost StatQuest, but it was too much, so I made it a stand alone video. That being said, the next video I put out, in the next two weeks or so, will be on XGBoost, then the next and the next, etc. XGBoost is a huge algorithm with a lot of parts. I've got 3 videos worth of material so far and I've only scratched the surface. I'm expecting to have at least 4 XGBoost videos, maybe more.
@mohamedhanifansari9224
4 жыл бұрын
@@statquest Thanks so much for the reply. I've seen all your tree based videos. For the last two days, I've been reading about XGBoost all over the internet and it was quiet difficult to grasp the whole picture of XGB. I genuinely thought it would be lot more easier and intuitive if it had been explained by you. Appreciate all your work. Couldn't be more excited for the future XGBoost videos. BAM!
@statquest
4 жыл бұрын
@@mohamedhanifansari9224 Just a few more weeks to wait! (and I'm just as excited as you are about this XGBoost thing - it's become an obsession!)
@mohamedhanifansari9224
4 жыл бұрын
@@statquest Thanks so much! :')
@swethanandyala
Жыл бұрын
hi Josh. There is small typo error in the video @ 9.45 . The tree score for the tree with 2 leaves is 19243.7+(10000 * 2) but it was written as 19243.7+(10000 * 3)
@statquest
Жыл бұрын
Ooops!
@kerimbasbug
2 жыл бұрын
Perfect!
@statquest
2 жыл бұрын
Thanks!
@divyagupta432
4 жыл бұрын
Please publish a session on reduced error pruning also.
Пікірлер: 533