Absolutely well written, clearly explained and very valuable content as always Umar. Keep perfecting your craft. 100k subs by Dec 2024 , you are opening lots of doors in AI education.
@RayGuo-bo6nr
8 ай бұрын
Thanks! Great Job! 谢谢 !
@umarjamilai
8 ай бұрын
谢谢你的支持
@RudraPratapDhara
8 ай бұрын
Thanks for listening for the request made last time for moe, thanks. You explain and elucidate the stuff in a very understandable way
@saeidghafouri8501
Ай бұрын
Thank you Umar! Please keep up the great work!
@pauldevillers797
2 ай бұрын
Amazing explanations, best channel around to dive deep into LLM !!!! Only note is that Mixtral7x8B paper clearly states that they did not observe any pattern on topic selection for a given expert, but they did exhibit some patterns on syntactic.
@andikunar7183
8 ай бұрын
Amazing content, you are a great explainer/teacher, thanks a lot!!!
@manishsharma2211
8 ай бұрын
One heck of a video umar, thank you. PS : @ 16:44 the kernel will move in the next 3*3 grid only when stride is 1 [ just FYI who might have doubt in this ]
@goelnikhils
8 ай бұрын
What a explanation of Sliding Window Attention, KV Cache , Rolling Buffer Cache , Mistral . Amazing Work. Amazing Content. I have been following Umar and whatever content he creates that is top notch.
@andikunar7183
8 ай бұрын
Danke!
@umarjamilai
8 ай бұрын
Thank you for your support!
@rajgothi2633
6 ай бұрын
Really good explanation... Please keep uploading such content. It inspire many researcher.
@unclecode
8 ай бұрын
👏 I support and subscribe to anyone who demystifies AI and helps democratize it. Keep up the fantastic job, Umar! Thanks!
@umarjamilai
8 ай бұрын
Thank you very much for your support! I wish you, your family and loved ones a happy new year!
@unclecode
8 ай бұрын
@@umarjamilai your welcome, I wish the same for you and your loved ones. Would you please let me know do you have any content focus on the transformer last step, where a linear layer picks up the next token based in the output of decoder. Basically the head MLP. Thx again.
@umarjamilai
8 ай бұрын
@@unclecode If you watch my video on how to code a transformer from scratch, you will learn all about the transformer, including the normalization and the last layer. I believe the best way to learn a model is to code is from scratch and see it in action.
@unclecode
8 ай бұрын
@@umarjamilai Roget that
@jman5447
4 ай бұрын
Thank you! Your clear explaination really make my life easier!
@kozer1986
8 ай бұрын
Amazing!!! Simply amazing! Haven't seen a channel with such explanation on those topics!!!
@snowflareai
8 ай бұрын
Thanks!
@umarjamilai
8 ай бұрын
Thank you very much for your support! Let's connect on LinkedIn
@wilsvenleong96
8 ай бұрын
Your content is god-given! I live for your content! Thank you so very much!
@karanjakhar
8 ай бұрын
Great content. Well explained. Loved it. Please keep up the great job. Thanks.
@jasonma3449
6 ай бұрын
exceptionally clear illustration on the SWA concept!
@justjeremiah4255
8 ай бұрын
Great video as usual, Umar! Thank you, sir.
@alessiocaffi5992
8 ай бұрын
watching your vids is worth the time even for ppl not too much into AI yet. got here from trying to understand Karpathy's vids, great Job. Would be nice if someone on yt would make a vid on how to create an attoGPT/ attoLM or call it bookGPT (bookLM) from any book, e.g DanteGPT🙂 , so to train, on consumer PC without advanced GPUs.
@rraviteja
7 ай бұрын
Super content & explanation thanks please upload videos regularly
@harshitkumar5147
3 ай бұрын
This is just awesome!
@snehotoshbanerjee1938
Ай бұрын
@umar, thank you for this video. As always, it is full of knowledge. I was wondering how much hard work went in creating this video. Referring to the code is not easy unless a person went though the code and extracted the relevant pieces. There is no doubt the content and concepts are very complex.
@Paluth
4 ай бұрын
Thank you very much, your videos are excellent as always. Keep up the good work, if you have the time!
@akashkumar-jg4oj
6 ай бұрын
This is literal gold!!!
@ryan-reynolds-q3u
5 ай бұрын
Thank you! I understood a lot from this.
@prasannaprabhakar1323
2 ай бұрын
Thank you!
@AndreasAlexandrou-to5pw
7 ай бұрын
Excellent as always. Thank you!
@raahuldutta
8 ай бұрын
Again another great video😊
@aam1819
7 ай бұрын
Fantastic explanation! Thank you!
@Itay12353
5 ай бұрын
You Are King!
@angelinakoval8360
8 ай бұрын
Thank you for the video, a lot of new information for me!
@utkarshjain3814
7 ай бұрын
bro is doing god's work. Keep it up!
@yukewang3164
7 ай бұрын
great explaination, very helpful, thanks!
@anshul.singhs
8 ай бұрын
Thanks! was waiting for it, can you do mamba and S4 next?
@hichamelkaissi7786
8 ай бұрын
Quality content.. Thank you immensely ❤
@islamtorky1762
8 ай бұрын
Great work! Can you do a video for flash attention? Thanks!
@michellem6685
4 ай бұрын
amazing explanation
@AndreasAlexandrou-to5pw
7 ай бұрын
A question on batching; As far as I understand, batching inputs together has minimal cost on inference. I.e. 100 forward passes through all the decoder layers take roughly the same amount of time irrespective of your batch size. The video mentions that compute is wasted whilst calculating attention for the padding tokens, and thus concludes that unrolling the batch is preferrable? I don't see how this makes sense from a performance standpoint. Compute is very underutilised during attention, so the "wasted attentions" do not really cost anything. On the other hand, unrolling the batch increases the number of forward passes by your batch size. For example; a batch of 5 inputs with a length of 100, takes 100 forward passes in the first case, but takes 500 passes after unrolling. Am I missing something here? Doesn't unrolling completely nullify the performance boost from the wasted attentions?? Edit: Tested this: - Sq length: 1024, batch size 1: takes ~ 38 seconds. - Sq length: 1024, batch size 4: takes ~ 39 seconds. - Sq length: 4096, batch size 1: takes ~ 155 seconds.
@Yassjams
5 ай бұрын
Amazing video ! can you do Falcon architecture explanation 🙏🙌
@haralc
4 ай бұрын
Thanks
@amitshukla1495
8 ай бұрын
Wohooo 🥳
@kenilshah-hb6fy
5 ай бұрын
I have one point! At 5:46, table is shown in which 2nd row 2nd column. You have written No. of Encoder Layers. My question: If the Mistral is Decoder layer, then why we are considering 32 as the No. of Encoder layer ?
@lukeskywalker7029
6 ай бұрын
Another great one! Any chance you'll take on "The Era of 1-bit LLMs" paper next? ;)
@baothach9259
6 ай бұрын
This video is so good!!!!
@trungquang1581
5 ай бұрын
great job, thanks a lot man
@ihitsuperhuman3227
4 ай бұрын
thanks
@gangs0846
6 ай бұрын
Helped alot thank you
@ml.9106
6 ай бұрын
Thanks!
@jatinarora6680
7 ай бұрын
Very detailed explanation! Thanks for the video. Could you also make a video on vision transformers like BEiT.
@cfalguiere
8 ай бұрын
Thanks for sharing
@random-ds
6 ай бұрын
Thank you for this great video. I have a question though. When mistral released the intruct-v2, do they follow the exact architecture and change only the data and way of training, or, they can also twist a little bit the classic architecture of mistral? Thanks in advance!
@XartakoNP
4 ай бұрын
Around min 14 - you explain that the sliding window attention will result in fewer dot products. From your explanation I derive that the sliding window mask is applied after the Q@Kt operation, where we perform all the dot products within the Q and K tensors. Is that operation fused in some way or is there a trick to achieve it the reduction in the number of dot products?
@aamir122a
8 ай бұрын
Open source Multi model modal models (MMLLM ) are also becoming main stream , please do an episode on them as well.
@waynelau3256
6 ай бұрын
Hey Umar, great video! I have some questions, how does SWA work at training? Because I am trying to wrap my head around how the previoius context is fed to the window. From my understanding in the mistral model, one of the tokens is catered to the previous attetntions. In this case, wouldn't this make it autoregressive and not parallelizable, because the previous attention needs to be computed?
@snehotoshbanerjee1938
Ай бұрын
@umar, one question... In the current setup, the current chunk has the attention scores w.r.t. previous chunk. But, is it not losing the attention score w.r.t. N - 2, N - 3 etc chunks? I mean is the attention score till the previous chunk is enough? Or are we saying that it has a rippling effect from left to right?
@GrifinsBrother
8 ай бұрын
Amazing job, keep going!
@zhenfutaofang2534
8 ай бұрын
Amazing Video !!! 加油
@umarjamilai
8 ай бұрын
谢谢你!我在中国有个微信小组关于AI和深度学习,你想交流在领英给我发消息,我Invite你参加。
@zhenfutaofang2534
8 ай бұрын
ok@@umarjamilai
@sahilc7750
6 ай бұрын
is there a way to learn different boiler plate codes and how they operated provided by Xformers ? There github is not very intuitive.
@subhamkundu5043
8 ай бұрын
Amazing content. Are you going to put some video on coding a MOE model from scratch?
@MrNathanShow
7 ай бұрын
Is the xformers part primarily used for training or more for just if we had a service and wanted to support the generation of the outputs. Also, for each expert are they trained independently? Or are they trained with the same dataset? From what I understand the MOE layer is just a feed forward lin layer that are weights. I think I might be wrong though... Thank you!
@MrNathanShow
7 ай бұрын
Ok, so each "expert" is technically just a feedforward output that is gate controlled by a linear series of weights. The top two are selected to post process the token at the end.
@MrNathanShow
7 ай бұрын
The whole data set is used to train each of these experts.
@pratyushrao7979
6 ай бұрын
I had a query regarding the rolling buffer cache. Why did they not use a Queue for storing the vectors instead of a rolling buffer cache? I know there's an issue with the implementation of a queue, but wouldn't that be time wise way less complex? Instead of O(n) you can roll back in O(1).
@umarjamilai
6 ай бұрын
You can implement it however you like, but you should always avoid shrinking and growing tensors, because it may move data around the GPU memory, which is slow.
@pratyushrao7979
6 ай бұрын
@@umarjamilaiOkay thank you. Your explanation was great!
@alexis91459
Ай бұрын
Awesome! So is it true that KV cache length should be of the same size as the sliding window attention?
@umarjamilai
Ай бұрын
Yes, because anyway the model cannot "attend" to anything beyond the sliding window size, so the KV Cache size is limited to it.
@vinc6966
8 ай бұрын
Great video, but I have two questions about sliding window attention: 1. How applying mask to tokens outside of sliding window attention makes it more efficient? Since we still have to perform calculations on NxN matrix, but with some zeros. Are floating point operations on zeros faster? 2. Receptive field increases as depth increases. Consequently, in mistral only last layer can attend to all tokens, so tokens have less time to communicate. If we have a task that requires N steps to be solved and ALL OF information from the tokens, will the model be able to solve it? Thanks
@umarjamilai
8 ай бұрын
Hi! 1. When you know that the two matrices you're multiplying will have many zeros in the output, you can use the "sparse attention", which basically represents matrices in a way very similar to Python dictionaries, so we only store the values of the non-zero indices. There are many deep learning frameworks that support sparse matrix multiplication, if I remember correctly DeepSpeed supports sparse attention calculation. 2. It is wrong to say that the last layer will attend to all tokens. One token only attends to W preceding tokens, where W is the size of the sliding window. But because of how the information gets "accumulated" in the embedding after each layer, we can claim that the information "flows" from one token to another even if they are outside the window. You're right in saying that the information that's carried this way is less "strong" (it's like you hear a news from a friend instead of reading it by yourself on the newspaper: every intermediate person will alter the real story). If a task requires the information of all the tokens, it MAY (we can't be sure) still able to perform it, but it all depends on how many layers you have and what's the size of the sliding window. Have a nice day!
@vinc6966
8 ай бұрын
@@umarjamilai Okay, I think that answers my questions ;) Thanks a lot!
@سودانتوك
8 ай бұрын
Great content as always. can you do a video about ControlNet?
@madhusudhanreddy9157
8 ай бұрын
Please create a one vecotr database with LLM RAG Implementation video sir
@Anson-rr6ej
7 ай бұрын
Great videos. Are the 8 experts and gating funtion in each layer are different ? So total there are 8 x 32 experts, is this correct?
@umarjamilai
7 ай бұрын
Yes, each layer has different experts: 8 per layer, so in total 8x32.
@Anson-rr6ej
7 ай бұрын
@@umarjamilai Thank you!
@elieelezra2734
8 ай бұрын
Hi Sir, great work as usual. However, I have a question regarding the gate in the 'Sparse Mixture of Experts' section. Is it a simple one layer network that produces 8 logits? Thanks! Keep up the good work !
@umarjamilai
8 ай бұрын
Yes, for every token in the sequence it produces 8 numbers. The two highest numbers indicate which FFN the token should run through.
@elieelezra2734
8 ай бұрын
Correct me if I'm wrong, it means that the behavior of this kind of block is not the same during training and during inference. During training token embedding goes through the 8 feed forward neural networks, then the output of the two best are selected according to the output of the gate, whereas during inference, the embedding token goes through the two best feed forward neural networks according to the gate. Again thanks a lot for your time and your explanation, I really appreciate it@@umarjamilai
@tryit-wv8ui
8 ай бұрын
Yep the same question here@@elieelezra2734 @umarjamilai
@tryit-wv8ui
8 ай бұрын
Is the next assertion from elie elezra below is correct@@umarjamilai ?
@umarjamilai
8 ай бұрын
@@tryit-wv8ui hi! The behavior during training and inference IS EXACTLY THE SAME: what I have shown for inference is exactly what happens during training. Because that's how the gate function is trained in producing logits and selecting the best feed forward networks for each token and that's also the reason why some feed forward networks will "specialize" in particular subsets of the tokens (for example some may specialize on Japanese tokens, others on English tokens etc..)
@siqb
8 ай бұрын
When we are training or even inference and use as input "[SOS] Love that", do we use the embedding of 'that' for passing to the softmax to predict 'can'?
@umarjamilai
8 ай бұрын
Only during inference. During training you just compare the entire output with the target to calculate the loss.
@ЖирайрАйрапетян-щ2у
8 ай бұрын
At 9:25, why are Q and K the same matrices in the case of self-attention? There are different linear layers for mapping the input sequence to queries and values, isn't there?
@umarjamilai
8 ай бұрын
I recommend you watch my previous video on the Transformer, where I explain the origin of the Q, K and V matrices.
@siqb
8 ай бұрын
Yup. Q, K, V are 3 different projections of the input. If they were literally the same, the QKt will be a symmetric matrix.
@umarjamilai
8 ай бұрын
@@siqb you're right. I should have mentioned that. Because I was talking about the "tokens" they "refer to" and not to the single values they are made up of, it may have caused confusion. Thanks for pointing out
@GrifinsBrother
8 ай бұрын
But your explanation about specialising of experts is wrong. Because it is stated in the paper, that knowledge of each expert is distributed equally and there is no any specialisation. Check "Routing analysis" block of the paper.
@umarjamilai
8 ай бұрын
The paper on the actual performance of the mixture of experts came AFTER I published my video. What I was talking about is not what happens actually (since I didn't have the data on the actual performance back then), but on what's the intuition behind creating a mixture of experts: the idea is that each model - hopefully - specializes in a subset of the data. It may also happen that each model does NOT specialize, like in the case of Mamba. I believe the authors of Mamba also hoped in some kind of specialization, but in reality it didn't happen.
@邱雨-e1u
6 ай бұрын
amazing
@farzinhaddadpour7192
8 ай бұрын
Thanks!
@umarjamilai
8 ай бұрын
Thank you very very very much!
@mihirrege206
7 ай бұрын
Thanks!
@avogadroarts4366
8 ай бұрын
Thanks
@reginoldlu
8 ай бұрын
Thanks!
@reginoldlu
8 ай бұрын
谢谢!Request the flashattention and falshattention2! keep working!!😀
Пікірлер: 115