"Textbooks Are All You Need" is a new paper from Microsoft research demonstrating the feasibility of training a powerful LLM for Python code with a relatively small training data set and number of model parameters.
Timestamps:
00:00 - Textbooks are all you need
01:46 - Headline results on HumanEval and MBPP
02:50 - Training & data details
05:05 - Filtering existing code datasets with a transformer
07:30 - The code exercises dataset
08:03 - Model architecture and training
08:30 - What does it cost to produce Phi-1?
10:16 - Data pruning experiments
11:16 - Implications for the commercial LLM ecosystem
11:42 - Limitations
13:20 - Closing thoughts
Paper link: arxiv.org/abs/2306.11644
Topics: #LLMs #ai #microsoft #phi-1
For related content:
- Twitter: / samuelalbanie
- Research lab: caml-lab.com/
- personal webpage: samuelalbanie.com/
- KZitem: / @samuelalbanie1
- TikTok: / samuelalbanie
- Instagram: / samuelalbanie
- LinkedIn: / samuel-albanie
- Discord server for filtir: / discord
(Optional) if you'd like to support the channel:
- www.buymeacoffee.com/samuelal...
Acknowledgements:
- Thanks to economist consultant Tom who pointed out the near-equivalence in pricing between Apple Vision Pros and the Phi-1 compute budget.
Негізгі бет Textbooks Are All You Need
Пікірлер: 330