Great series covering different kinds of positional encodings! Love it!
@WhatsAI
3 жыл бұрын
Thank you for pursuing this series! Love the NLP-related videos as I am not in the field of AI. Always well explained, seriously, props! Please do more of them 🙌
@AICoffeeBreak
3 жыл бұрын
Hey, thanks! But now ai am confused: What's AI is not in the field of AI? 😅
@WhatsAI
3 жыл бұрын
@@AICoffeeBreak I haven't worked with NLP tasks!
@AICoffeeBreak
3 жыл бұрын
@@WhatsAI aaa, you mean "in the field of NLP". I think you typed AI instead of NLP by mistake. Unless you think that NLP == AI. 😅
@hkazami
Жыл бұрын
Great explanations of important technical key points in an intuitive way!
@AICoffeeBreak
Жыл бұрын
Thank you so much! 😁
@amphivalo
2 жыл бұрын
Such a good explanation! Thank you so much
@justinwhite2725
3 жыл бұрын
Oh how awesome. I've been thinking about positional encodings for images where I jabe broken the image into grids. I've been wondering exactly whether I should track both the x and y positionals or just treat it like an array and only have one dimension for all segments. My hypothesis was that the neural net would figure it out either way.
@AICoffeeBreak
3 жыл бұрын
And did they? Ah, you do not have any experimental results yet. Plot twist: order does not matter anyway (I am half-joking and referring to those papers in NLP showing that language models care unexpectedly little about word order). Reference: "Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little" by Sinha et al. 2021 arxiv.org/pdf/2104.06644.pdf
@SuilujChannel
2 жыл бұрын
Thank you very much for these videos! I like your style too :D
@AICoffeeBreak
2 жыл бұрын
Thanks! 😁
@mkamp
10 ай бұрын
Great video as always. ❤ No RoPE video yet, right?
@AICoffeeBreak
10 ай бұрын
No, sorry. So many topics to cover, so little time.
@seyeeet8063
2 жыл бұрын
Thanks for the video, I have a general suggestion that I think will improve the quality of your videos. it would be really helpful if you can provide simple numerical example in addition to the explanation of the video to make the viewer better understand the concept. considering your talent in visualization it would not be really hard for you and will add a huge help for understanding
@jqd3589
2 жыл бұрын
Really a good job,I learn lot from this series of video. while could you please list some of the paper about relative position using in graph .very gratefull
@subhadipnandi294
2 ай бұрын
This is incredibly useful. Thank you so much
@AICoffeeBreak
Ай бұрын
You're very welcome!
@user_2439
2 жыл бұрын
So awesome explanation!! It really helped to understand this concept. thank you:)
@hannesstark5024
3 жыл бұрын
👌
@DANstudiosable
3 жыл бұрын
I asked for this first long back but my comment not mentioned in the video😢 Anyway, great explanation as always🎉🥳
@AICoffeeBreak
3 жыл бұрын
Sorry, I could not remember where you made that comment to screenshot it. I only went to the comments of the first video in the positional encoding series where I asked if people want to see relational position representations. But you are the reason I was motivated to do the whole encoding series, so thanks!
@ambujmittal6824
3 жыл бұрын
Notification squad, where are you?
@AICoffeeBreak
3 жыл бұрын
Quickest comment in the history of this channel. 😂 I pushed the "publish" button just a few seconds ago!
@AliAhmad-vm2pk
8 ай бұрын
Great!
@AICoffeeBreak
8 ай бұрын
@diogoaraujo495
Жыл бұрын
Hello! Awesome explanation! I just got a small doubt (hope that someone can explain it). So, self-attention is itself permutation-invariant unless you use positional encoding. It makes sense that absolute positional encoding makes the self-attention mechanism permutation-variant. However, I couldn't figure out if the same happens with relative positional enccoding. Beacause, if the in relative positional encoding we only care about the distance between the tokens, shouldn't this make the self-attention mechanism permutation-invariant ? So my question is: Does the use of relative positional encoding make the self-attention mechanism permutation-invariant (unlike if we use absolute positional encoding) ?
@AICoffeeBreak
Жыл бұрын
Thanks for this question, I'm happy I finally find some time to respond to this. The short answer is: relative positional embeddings do not make / keep the transformer permutation invariant. In other words, both absolute and relative positional embeddings make the transformer permutation variant. Take for example a sentence of two tokens A and B. Both relative and absolute encodings assign a different value to the two positions. So exchanging A and B will assign them different vectors.
@diogoaraujo495
Жыл бұрын
@@AICoffeeBreak Okay, thanks!!
@ludvigericson6930
Жыл бұрын
They are invariant to isomorphisms of the graph. In a path digraph such as for sequences, there are no isomorphisms. However for cyclic path digraphs of K vertices there are K symmetries. For an undirected path graph, we would have two isomorphisms: forwards and backwards.
@ludvigericson6930
Жыл бұрын
For a 2D lattice graph, I think mirroring is symmetrical but I’m not sure. This is assuming that you have an undirected graph.
@ludvigericson6930
Жыл бұрын
Undirected implies, in terms of the notation in the video, that a_ij = w_{j-i} = w_{i-j} = a_ji.
@hassenhadj7665
2 жыл бұрын
pleeease can you explain the Dual Aspect Collaborative Transformer
@ScottzPlaylists
11 ай бұрын
I think I see an error in some slides. The slide you show at kzitem.info/news/bejne/pa2XqIWYqppmlHo and several times earlier than that, seems to be wrong. The token X3 column, seems to have a number pattern for positional embeddings, that doesn't match the patterns in the other columns. It seems it should be a31...a35 instead of a31,a12,a13,a14,a15. Am I missing something?
@gouthaamiramesh714
Жыл бұрын
Can you please make video on sensor data with transformer and video on hybrid cnn transformers
@neilteng4161
Жыл бұрын
I think there is typo in the matrix at 5:11
@wilfredomartel7781
17 күн бұрын
🎉❤
@AICoffeeBreak
15 күн бұрын
@wibulord926
2 жыл бұрын
heello
@nurullahates4585
2 жыл бұрын
Good, but you speak too fast
@SuilujChannel
2 жыл бұрын
I think it's the perfect speed. You could slow down the video in the KZitem player settings yourself.
Пікірлер: 41