1 - Basic neural network theory | 8:30 2 - "Residual" neural network theory | 12:40 3 - Ordinary Differential Equations (ODEs) | 17:00 4 - ODE Networks | 22:20 5 - Euler's Method to Optimize an ODENet | 27:45 6 - Adjoint Method for ODENet Optimization | 29:15 7 - ODENet's Applied to time series data | 30:50 8 - Future Applications of ODENets | 33:41
@buenaventuralosgrandes9266
5 жыл бұрын
Thanks broo
@flyingzipper
5 жыл бұрын
Np !
@valken666
5 жыл бұрын
Only PyTorch implementation as of now? rtqichen's torchdiffeq Github.
@FaizalSyed
5 жыл бұрын
I have faith in humanity because of people like you 👏🙏
@praveenb9048
5 жыл бұрын
Is there a popular term that KZitem people use for a list of video bookmarks like this one?
@thoyo
5 жыл бұрын
I love how you're always excited about what you're talking about. It's infectious.
@pranavsreedhar1402
5 жыл бұрын
thank you siraj for putting the effort to enclose a much larger, broader audience. Everyone benefits from this.
@OnlyGoodJawn
5 жыл бұрын
Siraj dropped the most fire freestyle of 2019 in this video.
@SirajRaval
5 жыл бұрын
@@marketsmoto3180 wait 10 hours for my next video
@OnlyGoodJawn
5 жыл бұрын
Siraj Raval I cant eat or sleep until I get these new bars Siraj!
@arnau7915
5 жыл бұрын
I'm only half way through the video and I can already tell this is my favorite one of 2019, and possibly my favorite research paper ever! Thanks, Siraj!
@jithendrayenugula7137
4 жыл бұрын
He doesnt understand this completely
@motog9464
5 жыл бұрын
I am feeling more happy and proud now for learning Mathematics as my favourite subject. Another interesting reason to explore the AI more and more ..... Thanks, Sirj :)
@notjustwarwick4432
5 жыл бұрын
I agree, I'm studying maths at university and it is awesome to see differential equations pop up in AI.
@65343739
5 жыл бұрын
Ramesh is that you?
@malolan98
5 жыл бұрын
Hey, siraj! Please make a video on Spiking Neural Networks!
@mlguy8376
5 жыл бұрын
This could be interesting for me as someone that spent many years during his PhD looking at nonlinear ODEs. Now as a ML guy this would be great to relate back to my original work. There is a caveat that I was not clear on, there is a difference between stability conditions for ODEs which was not clear in the paper how they treat this.
@Bbb78651
Жыл бұрын
This made me fall in love with AI and ML again. Thank you so much. I was going through a slump, but when watching this I couldnt stop smiling throughout the entire video
@earthbjornnahkaimurrao9542
5 жыл бұрын
This looks more and more to me like consciousness is simply a sophisticated set of mathematical operations. This Neural Network architecture is able to optimize its own structure, like how many layers it has, in order to best solve a given problem. The set of equations looks a lot like the same equations used in optimized control theory where an observed state is compared to a desired state to give error state which is then applied by a multiplier and fed back into the system so as to move the system one order of magnitude closer to the desired state.
@offchan
5 жыл бұрын
I like that you said "I know that sounds complicated but don't go anywhere."
@irisgu8890
5 жыл бұрын
Thank you! I watched many videos on ODE with ResNet and yours is the best!!!
@vman049
5 жыл бұрын
I regularly watch Siraj’s videos and this is one of the best I’ve seen... got my adrenaline pumping when I saw that list of topics to be covered at 8:30!
@trycryptos1243
5 жыл бұрын
Siraj... please tell me that you have travelled back in time to help us catchup with the future. I am just flabbergasted by the volume & intensity you handle.! I have no words to comment just a dropped jaw in pure awe!!!😘
@theaichannel242
5 жыл бұрын
Really interesting research, AI is moving so fast right now. There is so many doors going to be opened. Modelling more complicated functions but still keeping the memory tied in. Amazing stuff, your videos are first class!
@sashas5390
3 жыл бұрын
The input-times weight-add a bias-activate song is brilliant and should be used in elementary schools
@carlsilverberg8700
7 ай бұрын
You've gotten way better than the last time I checked you out. That was 4 years ago, lol, so I guess thats just normal. But great man! Loved it! Absolutely amazing content.
@carosare6700
5 жыл бұрын
Ohh come on! I needed this for my differential equations proyect last semester:/ such an interesting topic!
@amanasci2481
5 жыл бұрын
Only channel on KZitem that motivates me to study Maths..
@saratbhargavachinni5544
5 жыл бұрын
About a week back, I started working as Teaching Assistant to Under grad Differential Equations course, I wondered when I was reading the text, I had learnt all these theory myself I was in fresh men year but very rarely used these differential equations after the course and I wondered if I can use these in Machine learning (my area of interest). I am really excited after watching your video.
@vuppumadhuri9546
4 жыл бұрын
The code which was shown in this video at the end of the video, doesn't show the ODE definition block. I mean, where the ODE was specified, except for the solver. Without defining ODE, how's it possible to solve dx/dt or d2x/dt2?
@AndriyDrozdyuk
3 жыл бұрын
"ODE block" is not really a block. Shameless plug, here is my explanation of this paper: kzitem.info/news/bejne/1oaalnZmkJp5aKw
@John-bb5ty
5 жыл бұрын
Last night when I was going to sleep I had a great idea for a self-evolving non-parametric neural-network. I was wondering for the longest time how I can get the integral of a function of the learning rate with multiple variables. Today I saw this, thank you.
@yasinilulea
5 жыл бұрын
This is awesome, you're killing it mate!
@Rednas34
5 жыл бұрын
Your videos are a continues stream of super high quality learnings about new computing mechanisms! Thank you!
@darogajee3286
5 жыл бұрын
When.... Cocaine meets anxiety.... .. 😄
@CSryoh
5 жыл бұрын
you try explaining this shit.
@matthewburson2908
5 жыл бұрын
@@CSryoh 1. Download ODE ML library. 2. Use library in code. 3. ??? 4. PROFIT! 🤔
@tamerkaratekin9074
5 жыл бұрын
I haven't finished watching yet, but this type of videos is what makes Siraj shine in the world of AI teaching. Latest AI paper explained in a very exciting and motivational way. He is very right when he says that you cannot find this type of lecture anywhere else.
@ozzn.t.8050
5 жыл бұрын
Please keep posting such videos for new interesting papers. It feels like, something under our noses with math, and we just need to notice it to completely solve AI in an unexpectedly simpler way. Delicious thing to watch. WTG.
@asharkhan6714
5 жыл бұрын
I like this style of video where you talk freely, just like your livestreams.
@GReddy567
5 жыл бұрын
Awesome, Thanks Siraj! The physics community is going to love this! Looking forward to you making more videos on this when this research expands!
@elciohumphreys2596
5 жыл бұрын
Thank you for the video. One thing that I believe it's a kind of frustration it's when you try to solve a differential equation and you don't have any function initial value because actually it results in a serie of functions, not just one. Watching that video I just realized you already have those function initial values: simply they are those data you use to train the network!
@loaywael
5 жыл бұрын
Awesome Video, Hopping to cover more about new research papers in that simple way, I really enjoyed even I'm not mathematician.
@36squared
5 жыл бұрын
Excellent video. It may be self evident, but It's important to conceptualize these improvements from both a mathematical and programming understanding. You tackled a tough concept beautifully!!! Good job, mate
@macmos1
5 жыл бұрын
This is an incredible research paper.
@naxyytt
5 жыл бұрын
Even though I'm good at math, I would have never imagined myself using differential equations again after high school... and here I'm
@setlonnert
5 жыл бұрын
Interesting that more and more abstract concepts are added to the deep learning mix. Once found to be a more of a bottom up idea. Besides GANs which I found to be adding higher concepts of the mimax to lower ones as the neural networks, there are also developments in structuring networks from a point of view in abstract algebra, or now by this ODE. It's good to get an overview of the developing flow ....
@kfique
5 жыл бұрын
Great video Siraj! Thanks and keep up the great work!!
@mapleandsteel
5 жыл бұрын
Nice job, bruv. Keep making the diaspora proud!
@avatar098
5 жыл бұрын
13:37 when Siraj is about to drop some hardcore ML knowledge
@morainaxel8499
3 жыл бұрын
HAHAHAHA, shit is getting serious
@kick-tech4691
5 жыл бұрын
Hey mota bhay.....I think in this video you really tried to make things simpler , oh ...yeah . Thanks for considering my suggestion . Keep rocking bro , keep educating the people.
@jackkensik7002
5 жыл бұрын
I have been exploring differential equations and am so happy I found this video, it puts the calculus in a context that is really interesting and applicable!!
@weishenmejames
5 жыл бұрын
Your vids are always of super high quality, often the topic is completely new to me yet you explain it in simple and easy to understand terms with clear examples. Well done!
@taranveersinghanttal
5 жыл бұрын
the movement of ur hands always inspire me ;p
@yooda1458
5 жыл бұрын
Cool! Maybe we could predict the earthquake by it.
@MarkPederson
5 жыл бұрын
+Siraj Raval I tried (and failed) to implement ODE nets on a gnn just before the end of the year. It was difficult not only because of the data source structure-ML in graph DBs is still in it's infancy-but also due to the relative dearth of info on this technique. Your explanations were helpful and (maybe even more important) your enthusiasm inspired me to go back and tackle it again; I'd forgotten why ODEnets are so appealing in the first place. Thank you!
@SuvradipDasPhotographyOfficial
5 жыл бұрын
Awesome siraj. You made my day.
@FractalMannequin
5 жыл бұрын
To whoever pointing out he's speaking and going too fast: this is video is not a course in deep learning, and you shouldn't expect to be able to actively apply notions starting from here; it's a (very good, imho) panoramic view of the subject just to give you a taste. If you're willing to get somewhere, you first need to study: some linear algebra, some probability theory, some multivariable calculus, deep learning dedicated libraries in whatever programming language you wanna use and, last but not least, study from some books about deep learning. I've really appreciated this video, I come from "pure mathematics" (even if I don't really like this term), and I had just an intuitive idea of how deep learning is implemented, but now my understanding is a lot less fuzzy. Thank you very much.
@art-sauce
3 жыл бұрын
fuzzy logic?
@MostafaElhoushi
5 жыл бұрын
Thank you for the great effort you put in
@1wisestein
5 жыл бұрын
Thanks Siraj, you're doing a great job!
@Madferreiro
5 жыл бұрын
Cant thank you enough! Thank you very much man, your channel is the best!
@hyunsunggo855
5 жыл бұрын
I just wanna keep stare at the evolving convolutional layer output with this one. Must be fun! :)
@KnThSelf2ThSelfBTrue
5 жыл бұрын
Programmer: This function has too many conditionals to write. Can't be done. Data-scientist: Have you tried using Stochastic Gradient Descent to write them? *DNNs are born* Programmer: This function needs too many layers to generate. Can't be done. Data-scientist: Have you tried Stochastic Gradient Descent to pick the right number of layers? *ResNets are born* Programmer: Each feature of this function needs a dynamic, potentially non-integer number of non-linearities added in order to be generated. Can't be done. Data-scientist: Have you tried Differential Calculus to just generate the function? *ODEs are born* Programmer: This function is nowhere-differential. Can't be done. Data-scientist: Uh... *Pulls out box-counting* Programmer: This function can't be described by its fractal dimension. Can't be done. Data-scientist: Oh god... *Pulls out Neural Multifractal Analysis* Programmer: This function can't be described by its singularity spectra. Can't be done. Data-scientist: *Pulls out Neural Multifractal Analysis, but harder this time* Programmer: This function can't be described by its singularity spectra. Can't be done. Data-scientist: [Maximum Call-Stack Exceeded]
@WildeMike49
5 жыл бұрын
God-tier lulz lad, bravo
@shadfurman
5 жыл бұрын
I have no fucking clue what this means... ...but it's fucking hilarious and I like it.
@---wu1so
4 жыл бұрын
one day ill come back to this and understand...
@jackingmeoff5594
5 жыл бұрын
You are such a clever brain. Great work man thanks.
@bender2752
4 жыл бұрын
You shouldn't expect 'any' people to watch or understand this video cuz there will not be a guy who know nothing about AI or deep learning things click on this video and try to find out what is "neural ode". You should just focus on those who knows a bit about this area, which will make this video a lot more better to those who really want to understand neural ode.
@nano7586
5 жыл бұрын
30:29 You know shit is about to get serious when Siraj takes on a ninja posture
@david0aloha
5 жыл бұрын
This is amazing. You are amazing. Thank you.
@aion2177
5 жыл бұрын
Freaking fucking awesome!! Streched my brain quite a lot😂 Thanks.
@Stan_144
5 жыл бұрын
That solution seems like more math, but less analogy to neurons in the brain, right ? So it goes in the opposite direction to brain inspired solutions, eg like HTM.
@Cleon7177
5 жыл бұрын
Awesome breakdown of very involved topics, Siraj. Keep it up!
@ILikeWeatherGuy
5 жыл бұрын
so this paper essentially makes vertical wormholes for marbles to skip specific air current layers, then digs valleys so the marble has more time to fall into the appropiate grouping.
@robicjedi
5 жыл бұрын
I have read the paper: arxiv.org/pdf/1806.07366.pdf It seems that in a ResNet, the parameters of the layers are not the same in each layer, while in the ODE fitting problem the parameters are the same. This clearly reduces the degree of freedom in choosing the parameters. ODE parameter fitting is not new, there are even some limited references in the paper. It seems that now one can use standard machine learning libraries, too.
@iameman9856
5 жыл бұрын
I am also confused at this, since every layer would have to have the same weight?
@CrimsonTheOriginal
5 жыл бұрын
Thank you Siraj, ive been reading over this paper for the last two weeks seeing how I can use it for my Forex predictions
@mingc3698
5 жыл бұрын
Very interesting! Looking forward to seeing this applied in action with time series data. I'm still don't understand how this design would help irregular time series data prediction.
@akashthoriya
5 жыл бұрын
I'm Artificial intelligence enthusiastic, please bring some more videos like this. it'll be helping a lot!
@nicholasperkins4655
5 жыл бұрын
"infinitesimally small and infinity big" once again Leibniz's monad is still schooling the world.
@fusillertube
Жыл бұрын
KZitem listed this video to me this morning just after I spent time looking for information on Liquid Neural Networks. Raj understood the importance of ODE 4 years ago ! Well done
@zzziltoid
5 жыл бұрын
Thank you for the attempt, my suggestion is that you should use the time in the video more efficiently. This is a pretty advanced paper, and noone who doesn't know the basics of neural networks or what a differential is will attempt/succeed to understand it.
@frede1k
5 жыл бұрын
Feeding the next layer plus the input reminds me of Mandelbrots fractals f(z) = z^2 + c. Here the input and output are complex numbers though
@junkseed
5 жыл бұрын
Thanks for this good intro into this topic!
@robertweekes5783
5 жыл бұрын
Breakthroughs like this are why AGI is closer than we think !
@macmos1
5 жыл бұрын
Really glad I studied math and CS in college.
@waeljaber9284
5 жыл бұрын
Thank you for making these videos!
@ahilanpalarajah3159
5 жыл бұрын
Dear Siraj you promised an explanation of the paper; but what we got is a (very enthusiastic) coverage of material, that frankly if we're watching this video, already knew. I feel the actual nuance and complexity of the paper at the end was rushed and missed. Thanks for the effort but should I be concerned with my ability to understand this, or is it as easy as you say? May be it is, brb ;)
@SirajRaval
5 жыл бұрын
You’re right. I could’ve done better. It’s a difficult paper lol
@tissuebox1229
5 жыл бұрын
hi siraj, could you make a video about implementing an ODE block? after rewatching the video twice it still is a mistery as to what concretly is happening in them, thanks!
@SirajRaval
5 жыл бұрын
absolutely, its not your fault. even the researchers are still fully defining this
@vonderasche2963
5 жыл бұрын
Really badass presentation.
@Avivalious
5 жыл бұрын
Very intersting~ The way to illustrate maths(derivative, integral, partial derivative) is intuitive, I will spend time on Euler Function which I still not very clear. Thank you for uploading such a great introduction which is both profound and intuitive.
@saitaro
5 жыл бұрын
I'm so glad you don't stop rapping from time to time, man
@armansa
Жыл бұрын
There are so many bits and pieces of misleading information in this video. The presenter clearly got many parts of this wrong-he didn't quite fully understand some important ideas in the neural ODE paper, and also had a flawed view of how numerical solution of ODEs work. At some point, he even mentions that the adjoint method was an alternative to the Euler's method!
@CarlosVazquezMonzon
Жыл бұрын
There something I don't quite understand. If Neural ODEs have a "continuum" of layers meaning there are no discrete layers, why do you initialize the class ODEBlock as an ODEFunc, which has two layers?
@phil.4688
5 жыл бұрын
At ~11:00 "That, in essence, is how deep learning research goes. Let's be real, everybody." You just won LeInternet for today ;-)
@giancarlosanchez4171
5 жыл бұрын
You're a real triple og for doing this
@nerdtek
5 жыл бұрын
Could you post a video on using the adjoint method to solve odes. I would just really appreciate a concise presentation. All of the material I have found on it, is hard to digest.
@hamid7011
5 жыл бұрын
thank you so much Siraj, I think you just opened my eyes on my next paper title.
@ianprado1488
5 жыл бұрын
Wow, something more interesting than capsule networks
@igormorgado
5 жыл бұрын
integration isn't the oposite of derivative, anti-derivatives exists for that. Of course integration and derivations are deeply connected (and that is a very unexpected thing). There are integrable functions that are not differentiable and vice-versa.
@bhuvaneshs.k638
5 жыл бұрын
Do a video on Document semantic segmentation or Learning to Extract Semantic Structure from Documents
@LiLi-or2gm
5 жыл бұрын
I have a huge interest in this subject!
@bhuvaneshs.k638
5 жыл бұрын
Have you tried architecture?
@Fu11meta1-v8
5 жыл бұрын
One of your best videos.
@DrAhdol
5 жыл бұрын
So to summarize, the ODEblock essentially takes all those (ODEfunc) layers and represents them as one large layer? Also what is the need for the initial resblocks at the start of the model? It's definitely an interesting approach to NN and I'm curious about it's applications in time-series (or anything that has a sequential relationship) data.
@АндрейДынин-л8т
5 жыл бұрын
hi, dark matter or no time for energy exchage short version Energy exchange limit or limit for two point to interact. it is a bit hard to write down this thought for me. if two points have relative speed more then speed of light, they not able to interact. but they can interact through the third point. (exactly like dark matter) long version For a long time trying to communicate with physics to clarify my theory. with all and all main point here. -dark matter in our galaxy, (most likely particles emitted by central black hole) is particles that moving faster than light. (most likely you do not "belive" in this) if i assume it is correct, then big amount of hydrogen on edge of galaxy, is where this "dark matter particles" decay after losing speed. (decay like new particles from hadron collider) -parts of dark matter alredy found, but we do not about it. (perseption(particles from hadron collider)) -particles found with hadron collider behave like a dark matter after loosing speed. -most likely there is a energy exchange speed limit in betwen two points (not sound speed), most likely it is a speed of light. (that about why we do not see dark matter, but see it interction with other(slower for it/faster for us) particles) -particles from hadron collider will be stable if placed in faster then light speed. whant to tell more, I hope this is enough to contact me. the key is a energy exchange speed limit (i want my Nobel for showing you dark matter) Best regards, Dynin A.I.
@marverickbin
5 жыл бұрын
You can calculate at irregular time, but different times for different paths of the network? (as figure at 28:37 suggests)? Wow, this allows even more possibilities than just non regular sampling.
@EctoMorpheus
5 жыл бұрын
Very interesting stuff! However, what I don't quite understand is how the ODEs fit in with gradient descent. If the layers of the network can be represented as an ODE at some time t, and algorithms like Euler's method can be used to solve such equations, why is gradient descent necessary? Or if I understood incorrectly and Euler's method is used for computing the gradients rather than the weights, what is the benefit of this compared to using current methods? Does it allow for non-differentiable activation functions?
@veronmath3264
5 жыл бұрын
Math is awesome i like that bru and my first time ever to hear about reinforcement learning.
@thexhitij
9 ай бұрын
that was really great, thanks a lot!
@manoharmanuuu
5 жыл бұрын
Since the math is being appreciated increasingly throughout your videos, I wanna recommend a beautiful book that describes how math evolved through centuries of understanding mostly inspired from the nature: 'God Created the Integers by Stephen Hawking'. I enjoyed the video btw :)
@urugulu1656
5 жыл бұрын
wow we ultimately found a way to describe human Fingerprints in yet another different way (besides many many vectors or even a Bitmap)...
@86Nicholson
5 жыл бұрын
Why is it optimized with a ODE, and not a PDE? I thought PDE's solve solutions for multiple variables...Please help me understand. I enjoy your videos, and you are great at teaching complex subjects. Thank you.
@samcoding
8 ай бұрын
When we're predicting timestep t+h is it that we just forecast this in one step, or do we subdivide the gap (between t and h) into lots of sub-timesteps where the output is evaluated and passed into the algorithm again (almost like autoregression)?
@jenniferkwentoh
5 жыл бұрын
Thanks for explaining this. Genius
@Stan_144
5 жыл бұрын
Next Einstein ?
@iszotic
5 жыл бұрын
Thanks for explaining all these concepts
@chiminglee8325
5 жыл бұрын
I would like to learn more about the code started from 30:50 though. But I love this video! Thanks for sharing.
Пікірлер: 391