In this video, I go over a simple implementation of LoRA for fine-tuning BLOOM 3b on the SQuADv2 dataset for extractive question answering!
LoRA learns low-rank matrix decompositions to slash the costs of training huge language models. It adapts only low-rank factors instead of entire weight matrices, achieving major memory and performance wins.
🔗 LoRA Paper: arxiv.org/pdf/2106.09685.pdf
🔗 Intrinsic Dimensionality Paper: arxiv.org/abs/2012.13255
🔗 Colab: colab.research.google.com/dri...
About me:
Follow me on LinkedIn: / csalexiuk
Check out what I'm working on: getox.ai/
Негізгі бет Ғылым және технология Low-rank Adaption of Large Language Models Part 2: Simple Fine-tuning with LoRA
Пікірлер: 81