A month ago, I was dreaming of creating a project like this one to automate video creation. But I was not able to do that. Now you give me a good starting point. ❤️❤️❤️❤️
@WawaSensei
Жыл бұрын
Wow, I'm so happy to help! Don't hesitate if you have questions 🙌
@burakylmaz9973
5 күн бұрын
This video has been on my list for a while and I truly appreciate the time you put into explaining these concept. I still haven’t been able to dive deep into the code yet but do you reckon this can be implemented also to create a lip-synced animation experience for speech-to-speech interactions with LLMs? The tech for low-latency text-to-speech & speech-to-text is already there but I wonder if you deem it possible to have a low latency solution also for outputting the required phoneme data within the browser. Again, I haven’t deep-dived yet but curious about your opinion on this.
@bernaferrari
8 ай бұрын
Best tutorial I've seen in a long time. Wish you knew some alternatives to ready player one, but not your fault. Amazing tutorial.
@WawaSensei
8 ай бұрын
Thanks a lot! I'm considering paying artists on Fiverr for some new content
@AaronAcheneje
11 күн бұрын
@@WawaSenseiI have headshot and cc4, I can create a character for you if you’ll show how to lipsync them
@mandymozart
9 ай бұрын
really like you didactics and calm when explaining.
@WawaSensei
9 ай бұрын
🙏 happy to read it, glad it resonates with you! As I do my best, such kind comments mean a lot 🤗
@AnishKumar-s4e
5 ай бұрын
Hey, its great, but i am trying to implement the same thing but its not working, how i can do it in threejs
@kushaltanna5569
Жыл бұрын
Wow another unique idea.! Really great video. Thanks.
@WawaSensei
Жыл бұрын
😻 Thanks, wish you'll give lip sync a try!
@hafizabdullah2375
2 ай бұрын
The transition between animations is not smooth as it hit the T-pose first. Any solution?
@pulkitgarg189
9 ай бұрын
@WawaSensei any other alternative we can choose for avatar other than Ready Player Me? I need more realistic avatars
@WawaSensei
8 ай бұрын
Free ones I don't know, but you can purchase any you like on the different 3D marketplaces
@habibahegazy333
6 ай бұрын
Can I apply this to blender or daz studio avatars?
@stamatiskourtis9653
6 ай бұрын
Fantastic tutorial! very well done!
@WawaSensei
6 ай бұрын
Thanks a lot! 🙌
@dindustack
Жыл бұрын
Thanks for your consistency
@WawaSensei
Жыл бұрын
Thanks for your support! 🙌
@sergiogonzalez2611
3 ай бұрын
thank you man, clear and precise tutorial, thnak you for shared
@WawaSensei
3 ай бұрын
Thanks for the motivation 🫶
@hamitaksln
Жыл бұрын
Great tutorial! Is there a way to smooth transition between visemes? It looks it's instant now and it feels a bit off. Thanks!
@WawaSensei
Жыл бұрын
Thank you 🙌 You're right, I didn't try to see what it would look like but should be very easy to do! I will try and update the code if it's better 😊 In the meantime if you want to try it on your own, instead of setting the value to 0 or 1, you can use THREE.MathUtils.lerp to smoothly transition from the current value of the viseme towards 1 or 0
@hamitaksln
Жыл бұрын
@@WawaSensei Thanks for the detailed answer. I would be glad if you update the code 😊I've tried something with lerp but I didn't get what I want.
@hamitaksln
Жыл бұрын
@@WawaSensei I've raised PR for it. I got some helps from chatGPT and the result looks better :)
@WawaSensei
Жыл бұрын
@@hamitaksln github.com/wass08/r3f-lipsync-tutorial/blob/main/src/components/Avatar.jsx Done! You were right the result is way better this way 🙌
@hamitaksln
Жыл бұрын
@@WawaSensei Ah I just saw your commit. Thanks for your time and work
@BrajBliss
9 ай бұрын
Hey, I followed your tutorial but for some reason, the shoulders are all tucked in weirdly when I am using the mixamo animations. They look just fine on the website but not when rendering them after downloading the file from mixamo. Any idea?
@WawaSensei
9 ай бұрын
Hey, you can check my second lipsync tutorial maybe I do it differently. Maybe it depends on the param used on how we export the fbx, let me know if that helps 🙌
@BrajBliss
9 ай бұрын
@@WawaSensei Is it this one? kzitem.info/news/bejne/0X2DsX2ven6UfmUsi=MGrvPgVROT9tHLk8
@WawaSensei
9 ай бұрын
Oh sorry, this one kzitem.info/news/bejne/prCwyXmCmGOgbKQ 🙌
@GordonMcIntyre-tw1rl
6 ай бұрын
Love it but a question: In the elevenlabs-nodejs module textToSpeech you specify 'responseType: "stream"',. I don't understand why not "ArrayBuffer" and use responseType: "stream", in the textToSpeechStream? Btw there's a bit of duplicate code in lines 93-96 in the index.js.
@ignacioarias2234
Ай бұрын
How did you create shape keys (o "morph targets") in the avatar?
@pulkitgarg189
10 ай бұрын
@wawa-sensei thanks a lot for shring this rare stuff on youtube. Learning a lot from this. Can you upload also please video for Metahuman avatar rather than Ready Player Me. Thanks!!
@WawaSensei
10 ай бұрын
Thank you! Are you sure it's usable with Threejs and not only within Unreal?
@pulkitgarg189
10 ай бұрын
@@WawaSensei honestly I am not sure, but I know we can download file into glb format thus technically this should be possible.
@bbg-u5w
Жыл бұрын
Thank you very much for the video you shared, but I encountered this problem when expanding the function. I tried to use eyeBlinkLeft in the model to control blinking, but in LeftEye, the animation did not take effect. Is there any good solution?
@WawaSensei
Жыл бұрын
You're welcome! Please join us on Discord and share your project to be able to help you 🙌
@kanandave-dh4js
2 ай бұрын
Hello Thank you for This Video ... But I m looking for other lip sync library than rhubarb, is there any library ?? please help ...
@jfluffydog2110
Жыл бұрын
thanks for making these videos, they help a lot
@WawaSensei
Жыл бұрын
My pleasure! 🙌 Thank you for your feedback!
@jfluffydog2110
Жыл бұрын
@@WawaSensei please can you make a video on gsap scroll trigger for full page scroll? In your portfolio tutorials the solution you use doesn't scale well for multiple sections
@pranitithape9934
Ай бұрын
My avatar somehow lost eyes, in mixamo its there but on UI its not appearing. what could be the possible cause ?
@martrom0
5 ай бұрын
Wonderful !! Thanks man !! Great Job !!🎉🚀
@splitspecs
Жыл бұрын
Once again great video ❤. Can you make a video on techniques for optimizing react three fiber projects
@WawaSensei
Жыл бұрын
Thank youuu! 🙏 I'm finishing writing my getting started with react three fiber course which include a lesson for optimizing projects 🙌 I might do it a light version for free in the future too (But basically it covers aspects you have in R3F documentation docs.pmnd.rs/react-three-fiber/advanced/scaling-performance and the performance pitfalls too)
@harimohan_choudhary_6
Жыл бұрын
This is really nice video. Thank you for sharing.
@WawaSensei
Жыл бұрын
Glad you enjoyed it, thaaaaanks 🙌
@ThETaBLaGiRL-h9t
2 ай бұрын
Hey! Wawa Sensei! I'm facing an error while yarn dev It is not working at all, there are only errors. Please help me! I'm trying hard from last 3 days
@pramodramdurg
7 ай бұрын
Thanks for this amazing tutorial. I am unable to play both animation and lipsync together, either one of them is playing at a time. do you know what could be the issue ?
@WawaSensei
7 ай бұрын
Did you download the model correctly with the query string parameters?
@khanhhq2044
3 ай бұрын
Does this lip sync work well for all languages?
@dounia-o7i
28 күн бұрын
can u please tell me how to use the aditional blend shapes of oculus visemes ? i want my avatar to close his eyes in a natural way
@WawaSensei
28 күн бұрын
Hey, please check this video, it's what I did: kzitem.info/news/bejne/prCwyXmCmGOgbKQ
@hoangsonit
Жыл бұрын
Hello Sensei, thanks for the video. Your video is very helpful for me. I have an idea to call the api to get the audio file, then automatically genarate the json file from Rhubarb LipSync. But I can't find any documentation that can automatically run Rhubarb LipSync in reactjs. Can you help me? I will be very grateful to you.
@WawaSensei
Жыл бұрын
Hello, Nice if you go this way! I can probably make a V2 of this video to show it in the near future. You shouldn't run Rhubarb client side as it's a binary your users won't have. It needs to be your server and then you return the JSON it generates. (So run the shell command using nodejs)
@boshen3953
8 ай бұрын
Hello Sensei, I followed your steps to export y as up and z as forward to Mixamo. However, the final animated character I run has the head facing downwards. I'm not sure where I went wrong in the process and would appreciate your assistance.
@WawaSensei
8 ай бұрын
Hey be careful about your threejs and r3f versions (compare with my package.json) Also check the process in my virtual gf video maybe I made it differently there
@udiiiiiit5020
Жыл бұрын
Hello Sensei, thanks for the video. But I'm having an issue after adding the animations. Console says: three.propertybinding: trying to update node for track: armature.quaternion but it wasn't found. And the model is rotated by 360 in z axis from its feet as pivot. I tried rotating and setting it's position but the result is not satisfactory
@WawaSensei
Жыл бұрын
Hello, you're welcome! About this error: three.propertybinding: trying to update node for track: armature.quaternion but it wasn't found. I also have it with mixamo + ready player me, it doesn't cause an issue to me, but I don't know where it comes from yet. About your model rotation, be sure to use the same threejs version / r3f / drei than me, and that you correctly exported the fbx with the right settings (shown in the video) to generate your mixamo animations 🙏
@udiiiiiit5020
Жыл бұрын
@@WawaSensei Yeah, I figured it out. Again thanks for the video😊
@nicolassuarez2933
10 ай бұрын
Outstanding! But your approauch with Rhubarb lipsync just runs on local, how to run it on server side, any clue? Thanks! 😄
@WawaSensei
10 ай бұрын
Hey, thank you very much! Well a backend is exactly like your local machine, you can have Rhubarb there and execute the command. I show it in this video how to create the backend kzitem.info/news/bejne/prCwyXmCmGOgbKQ
@pratikpatre8664
4 ай бұрын
Can I do this in React Native ? Please let me know
@ludoski68
5 ай бұрын
Thanks for your work!
@ShivamKumar-cu3lb
Жыл бұрын
Hi , this project was literally helpful for me. I was thinking, is there any possibility to add real time interaction using text to speech services and rhubarb oculus visemes?
@sharonthomas4010
Жыл бұрын
@ShivamKumar-cu3lb im working on a similar project, would love to have a discussion
@WawaSensei
Жыл бұрын
My friends, I think you will appreciate the next video, it will include ElevenLabs and ChatGPT 😊 Here's an early teaser of the facial expressions 👉 twitter.com/wawasensei/status/1711328416837029970
@ShivamKumar-cu3lb
10 ай бұрын
@@sharonthomas4010 Hi, sure let me know how we can connect?
@Venkatesh-vm4ll
Жыл бұрын
Wow this project is amazing thank you
@WawaSensei
Жыл бұрын
You are welcome 😊 Wish you'll give it a try!
@kilian5245
Ай бұрын
Hey, is it possible to make this character as an augmented reality project with three.js?
@WawaSensei
29 күн бұрын
Yes! I've helped on this project aivah.ai/ and that's what they are using!
@HamzaGabbar
Ай бұрын
how i apply voice recognition and voice synthesis
@NitinRawat-n1p
2 ай бұрын
Arigato Gojaaimasta Sensei ❤❤
@WawaSensei
2 ай бұрын
You’re welcome 🫶
@AnushkaJoshi-wu7pd
4 ай бұрын
in my case it is not working
@ryanchen1793
Жыл бұрын
great tutorial, but the morph changes too quick, would a lerp be helpful?
@WawaSensei
Жыл бұрын
Hey thank you! Yes I had changed the code and demo based another comment about it 🙌 Let me know what you think !
@tarunsukhu2614
Жыл бұрын
Another class act by our champion wawa
@WawaSensei
Жыл бұрын
Owww 🥰 Thanks a lot for your huge support everytime 🙌
@voloUA
3 ай бұрын
Hi, in my case avatar is rotated horizontally for some reason. And when I have uploaded it into Mixano, there was a half-sphere block in the bottm half of my avatar. Did somebody faced this?
@BhavinParekh-h6x
3 ай бұрын
use y forward when the time of export from blender
@Beata-yn9kn
Жыл бұрын
Hello, I really enjoying your videos! I am just wondering if you have any videos without started project to clone? So it's easier to understand everything from scratch! I would be really glad if you answer! Have a great day!
@WawaSensei
Жыл бұрын
Hello, thanks a lot! Sure, the first lessons of my course are free and include all the steps to create that starter project lessons.wawasensei.dev/courses/react-three-fiber Hit preview and you're ready to go 🙏
@Beata-yn9kn
Жыл бұрын
@@WawaSensei thank you so much! I am really enoying your videos, everything seems so easy, after your explanation!
@TA-du6bf
8 ай бұрын
Is this code compatible with other tools other than Ready Player Me for avatar creation as long as the file is in .glb format? I tried using a custom .glb avatar (very similar to the one in this video) but the page was frozen and eventually crashed.
@WawaSensei
8 ай бұрын
Hey it is but you must have a mixamo rig attached to your 3D model (If it’s not you can learn how to do it in this video kzitem.info/news/bejne/zpqgnY5piXWphZw)
@TA-du6bf
8 ай бұрын
@@WawaSensei Thank you!
@idkman8520
8 ай бұрын
i made a female avatar but for some reason i no matter how much i try to position her, her head is stuck to the ceiling of the page. I followed your code to the T. Any advice 😕
@WawaSensei
8 ай бұрын
Hey feel free to share a codesandbox on the discord
@ffff42
9 ай бұрын
I try to install step by step with your auto but I have the error : Avatar non defined in the Experience.jsx file
@WawaSensei
8 ай бұрын
Do you have a component named Avatar that you imported/exported correctly?
@freddyie-do4oy
Жыл бұрын
@WawaSensei Hello Sensei, thanks for the video, you are great. Can I use this avatar in a project for my clients?
@WawaSensei
Жыл бұрын
Yes, of course! Look at Ready Player Me terms of service part 7 readyplayer.me/terms
@jesusleguiza77
Жыл бұрын
Hello sir, how can I make it generate a video with the animations directly from js, and make an mp4? sending only the video to the website, thanks
@WawaSensei
Жыл бұрын
Hi, You can use this library github.com/spite/ccapture.js/, not sure it would work full backend (you can try) but you could pre-record them and use it in front too 🙏
@jesusleguiza77
Жыл бұрын
Thanks you! I have tried that library and it only generates the video on the client side@@WawaSensei do you know other option?
@BeenaSharma-v3x
10 ай бұрын
Hi sir, I want to use this same in my android app but by using kotlin ...can i use it if yes then how ??
@WawaSensei
10 ай бұрын
Hum.. You can use it! The simplest solution would be to embed in a webview, or rewrite for Kotlin following my logic
@j45t7
Жыл бұрын
Hey, Do you know why audio doesn't work on mobile?
@WawaSensei
Жыл бұрын
Hey, is it because you didn't do any interaction with the screen before playing the audio? Maybe add a button to be pressed before playing the audio
@pdrusr
6 ай бұрын
Perfect!
@WawaSensei
6 ай бұрын
You are perfect 😍
@soumyaranjansia3782
10 ай бұрын
for lipsync- write command like rhubarb instead of ./rhubarb (only for windows user)
@NanoGi-lt5fc
10 ай бұрын
Still not working hey did u add the rhubarb to path to env
@soumyaranjansia3782
10 ай бұрын
@@NanoGi-lt5fc no please download the rhubarb file and then extract the zip file then put that zip file in the root of the project and then go to the rhubarb path in your project and on that path , type the command for rhubarb but note that , you need to type rhubarb instead of ./rhubarb
@WawaSensei
10 ай бұрын
Thanks for helping 😍
@alijeblawi3438
Жыл бұрын
You're the best ❤❤
@WawaSensei
Жыл бұрын
No, you are! I'm only second 😍
@eduardoarhuata274
10 ай бұрын
my avatar appears behind the background how can i fix it?
@WawaSensei
10 ай бұрын
Hey, did you try adjusting the positions?
@eduardoarhuata274
10 ай бұрын
@@WawaSensei yeah i did, but got another error lol now the lipsync dont work lol
@kimiaiassistant
9 ай бұрын
is possible to make text to speech with lip sync?
@WawaSensei
9 ай бұрын
Hey, what do you mean?
@nagasai4981
Жыл бұрын
when i enter preset="sunset" then i am getting errors
@WawaSensei
Жыл бұрын
Check your package.json with the one from my repository here github.com/wass08/r3f-vite-starter Old drei versions have an issue with the url used for environment presets
@thimethimesha8495
Жыл бұрын
Amazing...
@WawaSensei
Жыл бұрын
Thanks a lot 😊 !
@NabenduInfusAI
Жыл бұрын
Anyway to connect it to a OpenAI API endpoint, so that we can ask it a question and it can answer.
@WawaSensei
Жыл бұрын
Of course yes! It would be a bit long to get the answer but you'd need to have the following in your backend: Call open AI (ChatGPT) with the question > generate audio with eleven lab API > have the Rhubarb Lip Sync on your server and run it on the audio > then play it in your frontend!
@inteligenciafutura
Жыл бұрын
Super mega excellet
@WawaSensei
Жыл бұрын
✌🥰
@AhmadRaza-mw7ux
Жыл бұрын
❤️❤️❤️❤️❤️❤️❤️
@WawaSensei
Жыл бұрын
😻
@sarkarsourav1
7 ай бұрын
thank you for such a video
@WawaSensei
7 ай бұрын
Glad you liked it! 🙏
@sumitranjan7005
Жыл бұрын
how to make it eye blinking?
@WawaSensei
Жыл бұрын
Hey, the logic is the same than the visemes, there are morphTargets you can play with to open/close the eyes and by animating them smoothly you can make the avatar blink. I'm considering making a second more advanced version of the lipsync, would it interest you? What would you like to see inside?
@sumitranjan7005
Жыл бұрын
@@WawaSensei more advance video great. I'm sure all your subscriber are early waiting for that. i want to see how emotion and more advance(multiple animation during a single dialog) gesture can be controlled. 😇
@priyanshiyadav3350
7 ай бұрын
I want video in avatar's background please help
@WawaSensei
7 ай бұрын
You can with useVideoTexture -> lessons.wawasensei.dev/courses/react-three-fiber/lessons/image-and-video-textures
Пікірлер: 140