You're really unique in this field! Using LORA for a specific layer allows us to create applications that quickly fine-tune a model for each user across multiple tasks. Why bother with RAG or complicated prompt engineering when you can achieve such rapid "over-fitting"? I learned a lot and expanded my ideas on turning fine-tuning into a service or product. Kudos
@alchemication
4 ай бұрын
I don’t think that is the case as explained at the end of the video, and the time difference isn’t that big. But I get your point, doing and maintaing both: rag and ft, especially at scale and where data chsnges a lot is very annoying
@AM-yk5yd
4 ай бұрын
* Llama 3 has tons of reserved tokens. Might as well use them for padding * Usually simplest way to expand vocab is to make wrapper module around input embeddings which is basically `return orig_emb(token_id) if token_id < N else my_emb(token_id-N)`: helps that you can train embeddings this way without touching original input_emb. * Anyway you need to zero-out new token in lm_head: we don't want it to ever be produced and having it non zero kinda changes a lot * Reminds of adapters. Even authors of the paper remembered them. * It seems pretty simple so training on multiple GPU is trivial to make with accelerate. * Simplest hack to merge seems to be to enable bias on eg MLP's down proj and inject it there. so `down_proj.bias = b` for (h+b). Which also means you can target QKV proj
@TrelisResearch
4 ай бұрын
thanks for the advanced tips! 1. good idea on using reserved tokens 2. yeah good idea, that avoids having to resave all embeds 3. hmm, yeah, I guess the model could accidentally generate the pad token in the wrong place. I haven't seen this though 4. Not sure what you mean by "reminds of adapters" 5. multi-GPU should be possible with accelerate yea 6. that's a cool hack on the bias. The authors are saying that h+b performs badly though...
@AM-yk5yd
4 ай бұрын
@@TrelisResearch 3) It's more about stability and reproducibility, as you said random collision in N-dimensional space is kind hard. The poitnt if you hard-code valeus of embeddings, model(x) is always the same. (Also instead of zeroing you can take value of some invisible token like BOS or UNK, though llama3 doesn't have unk token anymore) 4) Adapters are placed after MLP. Reft is placed after MLP. the main difference is adapters also have non-linearity before upscaling. so if their (best) idea is to use `B (A h + b)` Adaptes B use `B (ReLU(A h))` It also reminds of neftune, which is also based in bias. Only neft adds bias (noise) to input tokens instead of the middle.
@TrelisResearch
4 ай бұрын
@@AM-yk5yd thanks yeah on 4) I suppose ReFT is Lora applied to adapters, but no relu. (Btw ReFT is applied to the residual stream)
@MasterFX2000
4 ай бұрын
I'll try that. It sounds really interesting and sufficient for some usecases instead of complete finetuning
@TrelisResearch
4 ай бұрын
yeah 100%
@user-yu8sp2np2x
4 ай бұрын
Hey, I have a use case, where I want to refrain the LLM's from generating specific words. Can you suggest the possible approaches for this problem
@TrelisResearch
4 ай бұрын
yup, best approach is to use "outlines", check it out on github. I also use it in the ALL-SORT video. You can do regex matching
@user-yu8sp2np2x
3 ай бұрын
@@TrelisResearch Hey, Thanks for the response. But, Outlines library is used for Structured Text Generation. How will it help me in stop generating some words?
@TrelisResearch
3 ай бұрын
@@user-yu8sp2np2x I think that possibly, with regex, you can block certain words, I'd need to dig in more. search around for blocking words, I think this may even be possible with transformers generate
@josecorte-real4565
4 ай бұрын
Great video! I have a quick question, the ReFT repo says they use a fraction of the parameters used by reft techniques, but (from my understanding) that is a function of applying the intervention to a single layer. In this video you demonstrate that we can also do this with Lora, arriving at a similar parameter count. So I guess my question is, besides the ability to combine 2 orthogonal reft adapters, is there a major difference between using either technique? I know this is what you addressed at the end w.r.t. the intervention being wh+b and thus similar to a linear layer. It seems like that would mean that reft and Lora would be equivalent under these conditions.
@TrelisResearch
4 ай бұрын
Yeah, I think you're getting it right. The two approaches - when you target a specific layer are quite similar. ReFT goes even further by only applying it to a single input position (but this is also possible with LoRA, in principle). So the main difference is that LoRA targets weights and ReFT targets activations, but even this can become equivalent if you target the residual stream and set up your intervention... So, my gut feel is that there is close equivalence, although the pyreft authors perhaps combined the thought of going after a very narrow intervention.
@francycharuto
4 ай бұрын
Superb
@AdhilAseem
Ай бұрын
How do we prevent over-fitting to the data when using REFT?
@TrelisResearch
Ай бұрын
You can reduce overfitting by training for less epochs. Keep in mind though that the point of ReFT is mostly to overfit, but overfit a tiny representation/intervention. This engrains specific behaviour but shouldn't cause the model as a whole to overfit.
@adhilaseem2518
Ай бұрын
What metrics is used for the training loss? How can we use the same metric to measure the performance on a test data?
@adhilaseem2518
Ай бұрын
I am trying to teach the model a specific skill. I want the model to extract specific items from the given input. I want to asses the performance on a test set so I have an idea of the real world performance
@TrelisResearch
Ай бұрын
@@adhilaseem2518 reft overfitting will be fine. The overall model shouldn't overfit.
@TrelisResearch
Ай бұрын
@@adhilaseem2518 it's just cross entropy loss, and yes, you can use an eval dataset and do x-entropy on that too. You just need to pass an eval dataset to the trainer.
@maizizhamdo
4 ай бұрын
thinks bro for tut ,great, if possible fine tuning embeding for arabic lang , and fine tune llama 3 for arabic lang
Пікірлер: 23