for everyone trying to understand this concept even more thoroughly, towardsdatascience's article "The intuition behind Shannon’s Entropy" is amazing. it gives added insight on why information is the reciprocal of probability
@avaodonnell128
6 ай бұрын
Thank you!!
@benjahnz
4 ай бұрын
And it's paywalled...
@bluejays440
9 ай бұрын
Please make more videos this is literally the only time I've ever seen entropy be explained in a way that makes sense
@caleblo8498
Жыл бұрын
it is confusing from some part of the concept, but the rethinking progress is helpful, as I can surely say: the expected possible outcomes (the surprises) * the probability = entropy (lower the entropy, the less surprising it will be)
@Sars78
18 күн бұрын
Well done, Adian. I just found out-though I'm not surprised at all, in the Shannon sense 🤓 -that you're doing a PhD at Cambridge. Congratulations! Best wishes for everything 🙂
@zizo-ve8ib
7 ай бұрын
Bro really explained it in less than 10 mins when my professors don't bother even if it could be done in 5 secs, true master piece thus video keep it up man 🔥🔥🔥
@nicholaselliott2484
8 ай бұрын
Dude, took information theory from a rigorously academic and formal professor. I'm a little slow and under the pressure of getting assignments done, couldn't always see the forest for the trees. Just the sentence "how much information, on average, would we need to encode an outcome from a distribution" just summed up the whole motivation and intuition. Thanks!
@charleswilliams8368
5 ай бұрын
Three bits to tell the guy on the other side of the wall what happened, and it suddenly made sense. Thanks.
@caleblo8498
Жыл бұрын
each slice of probablity requires log2p_i number of bits to represent, and the total number of outcomes (they call it entropy) requires the sum of all slices of probability. Each slice of probability is basically one of the expected outcome, say, geting the combination ABCDEF in a six letter scramble. (correct me if I am wrong)
@xyzct
2 ай бұрын
Excellent. Short and sweet.
@kowpen
2 жыл бұрын
At 4:54, may I know the reason to consider 10 bits and triples? why not any other combination? Thanks.
@adianliusie590
2 жыл бұрын
I was just showing arbitrary examples but I could have chosen many different examples too. The triples (when there were 8 outcomes) was to show this could be easily extended to any power of 2, and the 10 outcomes was to show that this generalises too to non-powers of 2.
@mathy642
3 ай бұрын
Thank you for the best explanation
@anusaxena971
2 жыл бұрын
You CERTAINLY DESERVE MORE VIEWS 👏 👍👍👍👍
@murilopalomosebilla2999
2 жыл бұрын
Nice explanation. Keep up the good work, man!
@prateekyadav7679
Жыл бұрын
What I understand is that Entropy is directly related to the number of outcomes, right? So, I don't get why we need such a parameter/term when we could simply do by stating the number of outcomes of a probability distribution? What new thing does entropy bring to the table?
@derickd6150
9 ай бұрын
Consider the case that a biased coin is flipped. There are two outcomes, just like an unbiased coin, but let's say this biased coin has a (0.1)^10000 chance of being heads. Do you have exactly the same information about the outcome before hand as you do with an unbiased coin?
@maxlehtinen4189
9 ай бұрын
@@derickd6150 yes, it makes sense that a non-uniform distribution should have an effect on the uncertainty of a distribution, but can you explain how the bias affects the outcome via the entropy formula?
@derickd6150
9 ай бұрын
@@maxlehtinen4189 I'm not sure what you mean by bias here? Edit: Oh right you're referring to my answer, not something in the video. Yes well the entropy formula says something along the lines of: "How many bits do we need to represent the outcome of the coin"? That is a very natural measure of how much information you have about the outcome. If the coin is unbiased, you need two bits. If it is so severly biased like I describe above, and you plug the numbers into the entropy formula, it will essentially tell you "Well... we really only need one bit to describe the outcome right? We essentially certain it will be tails" Something intuitively along these lines. Edit 2: to see this, plot y(p) = -p log(p) - (1-p) log(1-p) for p in [0,1]. That is the expression for the entropy of the coin, whatever its bias. You will see that when p is very close to 1 or to 0 (which it is in my example), y(p) is almost 0. This is to say, you need almost no information to represent the outcome. It is just known. You need not transfer any information to someone, on the moon say, for that person to guess that the biased coin I described gives tails. However, when p is 0.5, the entropy is maximised, and so you would need to transfer the most information to someone on the moon to tell them the outcome of the coin, because they cannot use their prior knowledge at all to make any kind of educated guess
@nyx8017
8 ай бұрын
god this is an incredible video thank you so much
@mansoor9894
Жыл бұрын
Fantastic job in explaining this,
@avatar00001
2 ай бұрын
thank you codexchan
@user-wi1rj4iw9y
2 жыл бұрын
Thank you for your video.Keep it up! 感谢你的视频. 再接再厉!
@chetanwarke4658
Жыл бұрын
Simple and precise!
@AdeshBenipal
Ай бұрын
Nice video
@RodrigodaMotta
2 жыл бұрын
Blew my mind!
@MaximB
Жыл бұрын
Great job. Thank you
@lennerdsimon9117
2 жыл бұрын
Great video, well explained!
@tanjamikovic2739
Жыл бұрын
this is great! i hope you will film more!
@debasishraychawdhuri
3 ай бұрын
It does not explain the most important part - how the formula for non-uniform distribution came about
@derickd6150
9 ай бұрын
Great video!
@sirelegant2002
7 ай бұрын
Thank you!
@aj7_gauss
8 ай бұрын
can someone explain the triplets part
@alixpetit2285
2 жыл бұрын
Nice video, what do you think about set shaping theory (information theory)?
@morphos2
Жыл бұрын
I didn't quite understand the 4:36 rational.
@huibosa2780
2 жыл бұрын
excelent video, thank you!
@prateek4546
2 жыл бұрын
wonderful explaination !!
@azerack955
Жыл бұрын
I dont quite understand the very last step. what does summing over all the probability outcomes give us?
@vibhanshugupta1729
Жыл бұрын
That is the way we calculate expectation values. For a random variable X which takes values {xi}, E(X) = sum P(xi) * xi
@AkashGupta-th2nm
11 ай бұрын
Intuitively, u sum over it to get some understanding of the average uncertainty
@user-cf2yo5qf3h
5 ай бұрын
Thankkkk youuuuu.
@robertwagner5506
2 жыл бұрын
great video thank you
@zgz97
2 жыл бұрын
beautiful explanation :)
@Justin-zw1hx
Жыл бұрын
awesome!
@karlzhu99
Жыл бұрын
Uncertainty is a confusing way to describe this. For the lottery example, wouldn't you be very certain of the outcome?
@TUMENG-TSUNGF
Жыл бұрын
It’s about the numbers not whether you win the lottery or not.
@whoisray1680
Жыл бұрын
Why 1/p?????????
@lsacy8347
Жыл бұрын
im not too sure but I think its just a bitwise expression of M possible outcomes. considering there are M probabilities with equal probabilities (p), so p = 1/M -> 1/p = (1/(1/M)) = M
@bodwiser100
7 ай бұрын
I appreciate your effort, but the video is quite confusing. For example, in the example about 8 football teams, you explain why 3 bits are required by flat out stating as a starting premise that 3 bits are required! It's a circular argument.
@corydkiser
2 жыл бұрын
awesome
@AniketKumar-dl1ou
10 ай бұрын
Your should have written H[U(x)] = logM / M to better relate the entropy explanation.
@diy_mo
9 ай бұрын
I expected something else, but it's also ok.
@axonis2306
2 жыл бұрын
Most of your understanding is good, but 4:50 is an unnecessary leap of logic. At level this introductory is probably best to assume outcomes to be at 2^n.
@energy-tunes
Жыл бұрын
This seems so intuitive why did it take so long to get "discovered"
@jjwhittle8873
Жыл бұрын
"so long" relative to what?
@2011djdanny
2 жыл бұрын
Example is even difficult than the concept itself 🤦🏼♂️😃 Nice try by the way
Пікірлер: 60