This is exactly the kind of breakdown of this paper I was looking for as I am just getting into the deep end of neural audio generation. The explanation of RVQ is very helpful. Thank you for making these videos!!
@zhaosilas2494
4 ай бұрын
thank you so much! Your explanation is so clear, and it is excatly what I'm looking for.
@schnik24
Жыл бұрын
What an amazing video, finally I fully understood the paper! Thank you so much!
@audiocipher
Жыл бұрын
Fantastic overview Gabriel, thanks for sharing
@gabrielmongaras
Жыл бұрын
Glad you found it helpful!
@sedthh
Жыл бұрын
Thanks! Just what I needed.
@willwhite866
Жыл бұрын
Amazing overview
@ganeshsuryanarayanan4274
10 ай бұрын
Nicely Done!
@tellmebaby183
Жыл бұрын
nice illustration
@zoahmed8923
7 ай бұрын
Great video! Is the transformer essentially predicting the indices of each codebook?
@odysy5179
4 ай бұрын
Hi, what I am still wondering is why we cant just use teacher forcing on the codebook dimension? Thanks.
@crushedkeyz7681
Жыл бұрын
Great explanation on the paper and audio encoding and decoding. But where does the text prompt fit in to this?
@crushedkeyz7681
Жыл бұрын
Where do the text tokens fit in?
@gabrielmongaras
Жыл бұрын
MusicGen kind of experiments with three different conditioning methods. Generally, when using a transformer, you can just condition the transformer with a cross-attention block like in the original attention is all you need paper. MusicGen specifically experiments with three different encoders, but the method of conditioning the transformer is the same for all text encoding types.
@KwstaSRr
Жыл бұрын
@@gabrielmongaras The model is supposed to accept text as input right? When it generates music based on a text prompt what is the model's input?
@gabrielmongaras
Жыл бұрын
The text would be put through some sort of encoder transformer architecture and the output of that encoder would then be used to condition the main audio generation model through cross-attention. The audio is generated the same, just with this extra conditioning.
@KwstaSRr
Жыл бұрын
@@gabrielmongaras ok I understand the conditioning but without audio input at inference mode, to what do we add the text-conditioning embedding? maybe a silly question i dont know.
@hi_6546
Ай бұрын
Hey, if the frame rate is equal to 50*64, that mean that each second the model need to predict 50*64 values? (without counting the dim)
@hi_6546
Ай бұрын
the model works with the index of the codebook no?
@nickackerman8755
24 күн бұрын
Where do you get the 64? If we have e.g. 4 codebooks, and there are 50 time steps per one second of audio, we need to predict 4 * 50 values per second (4 codebook indices per time step), I believe.
Пікірлер: 23