Hard doubts about lips sync

[djames.suhanko]

Hello, everyone!
I saw a couple of videos showing two ways to make lips sync in Moho. The first one, I can use the script "Bone Audio Wiggle", and yeah, seems a good point to start. But the mouth doesn't seem to speak the words correctly - of course, so I will need to adjust the vector to make the mouth seem to speak "o", "e" etc. But, doing this, I will need to adjust each one of these movements and repeat the same movements to each one of "o", "e" and so on. So, my first doubt is: is this script to use as is, or really I will need to adjust all the words (sorry, I don't know how to say "silabas" in English)?

The second one, is to use the layer switch, but I'm not sure how this works in fact. How to grant that the mouths will be changed correctly?

And writing here, I have remembered another question: How do I use expressions if I'm using lips sync? I got deformation changing the expression while the mouth is speaking, using audio bone wiggle. And, if I use the layer switch, the problem seems to be bigger.

Some good video tutorials about? I couldn't find one.

[Greenlaw]

Over the years I've taken a lot of different approaches to animating lip sync in Moho for many TV shows on Netflixs and other streaming channels, and what I finally settled on is using a Switch layer controlled using the Switch Selection Window. This is the easiest and most efficient way to select mouth shapes, and it's fully compatible with Interpolate Sub-Layers. (Unlike when using any Smart Bone techniques for animating switch layers.)

Here's my workflow...

1. setup the visemes for the mouth shapes in your switch layer. Take care to give the mouth drawings sensible names because the drawing and the label you give it will appear in the Switch Selection window when you are animating with it. I like to make the first mouth (at the bottom of the stack) the neutral expression, and then stack the visemes above that.

The visemes you choose is up to you, but I like to use a slightly modified version of the Preston Blair mouth shapes.

You can add specials and other expressive mouths in layers above the viseme layers. Moho reads drawings from bottom to top in a Switch layer, so any new drawing should always be added at the top of the stack...especially if you've already started animating the project so you won't upset any existing keyframes.)

2. Import your audio file. Ideally this is an uncompressed .wav or .aif file. If you import a compressed audio file, like an .mp3 for example, frame accuracy cannot be guaranteed.

3. Scrub or step through the audio and mark your keywords in the timeline. When marking, the first line of your text will appear on the timeline, and the rest will appear when the cursor hovers over the marker. I like to place one word or a few words in the first line of the note, and the complete dialog in the next lines so as not to clutter the timeline with a bunch of text.

4. Now select the Switch layer and open the Switch Selection window. The Switch layer is automatically loaded into the window. Dragging the slider will advance to each drawing in the Switch and set a keyframe. Now you can scrub/step through the project and set your mouth shapes as you listen to the track. Remember to look at the waveform...it visual cues for exactly on which frames the sounds are being spoken.

With a little practice, this process goes very quickly. For a typical scene, I can get through my first pass for lip sync animation (i.e, just the mouth shapes) in a few minutes. My second pass is usually just squashing and stretching the Switch layer itself using the Transform Layer tool to get some smoothness between the transitions, and this stage goes even more quickly.

Once the mouth animation looks good to me, I'll animate the character's head and facial features and body motions to really sell the performance. You can see examples of mouth animations in my 2017 and 2019 demo reels (links below,) but to watch these animations with the dialog tracks you'll need to watch the tv shows they were created for on Netflix and elsewhere.

Oh, actually some of my current training videos have examples of Moho animation with dialog, so you can check those too. Speaking of which, I'm planning a tutorial specifically on this topic but I have other tutorials I want to get out first. When the lip sync one is ready, I'll announce it here in the forums.

Hope this helps.

[djames.suhanko]

WOW!!! Amazing!
It's a very rich and precious information. I thank you very much for sharing this information with me!!!

[djames.suhanko]

Have you animated all of these animations in your video to TV??? Man, I have a young daughter, I watched almost all of these animations!
I love it! Congratulations, master!!

[Greenlaw]

Have you animated all of these animations in your video to TV??? Man, I have a young daughter, I watched almost all of these animations!

Thanks for asking and I'm glad you enjoyed the animations!

The work situation can be different from show to show. Sometimes I'm working on a small team, and other times I'm working solo.

For example, in the case of Boss Baby: Back In Business (the first series,) I drew and rigged many of the Moho characters while other artists began animating my rigs. Along the way, I trained the team so they could start rigging characters too. Eventually, I was able to start animating on the show too. Background art and character sheets were usually provided by our art department. So this was largely a collaborative effort.

My work on Puss In Boots was usually just me rigging and animating my own shots. The painted art was often provided by the art department but I occasionally painted my own artwork for my shots.

When I was working on the interactive Puss-In-Book episode, all I was provided was the audio tracks and the framed 'choice point' paintings as PSD files. My task was to design and create the Puss puppet, and rig the provided paintings. I was pretty much allowed to interpret Puss' performance as I wanted, which was a lot of fun. So, some collaboration here but the animation part was mainly me working solo.

I enjoy both working on a team and working alone. When I'm working with a talented team, I learn so much from my team-mates knowledge and experience. When I'm working solo, I get to focus and explore new techniques, and come up with my own ideas. I guess so long as I'm learning new things, I'm happy.

[drumlug13]

The only thing I would add...

I have all the phonemes/visemes based on Preston Blair too with a few extra expressions added into a switch layer group. All shapes with corresponding point placement so they can interpolate smoothly. Plus I use a smart bone to control the emotions of the mouth shapes. They go from smaller for mumbling, downward drooping for sad, normal, happy and large exaggerated shouting shapes. And it works great.

But when it comes to the actual keyframing of the lip sync itself I used to painstakingly dissect every syllable and it looked absolutely horrible when I finally finished. Imagine watching a mom that's scolding her kid for acting up at the grocery store. You know that thing moms do when they lean in towards their kid and point to their lips and overenunciate something like "Are you listening to me? Am I getting through to you now?" That's what all my characters looked like lol. It just didn't look natural at all.

If you watch people talking there really isn't that much movement going on. So when you simplify it looks way more natural. I usually point to a show like Family Guy. I've never given the lip sync a second thought probably because they keep it super simple. But if you are focused on it you can catch some interpolation but the actually placement of keyframes doesn't look overdone. Plus I've seen some cartoons that don't use interpolation at all and they look great.

I think now I just try to make it look convincing and not try to make it look perfect.

[Daxel]

What if you know the animation will be dubbed in a few languages? Do you try to make the lipsync more subtle?

[slowtiger]

I repeat myself: it all depends on style - and budget.

The Preston Blair style is not realistic, it's cartoony and exaggerated. There's a place for this, but even in a very cartoony show you wouldn't use this all the time, only in moments where you need to emphasize the dialogue (think Mel Blank screaming). If you do this, it helps to do it exactly for the language used. To do this in several languages requires a budget.

Much of worldwide animation is just "illustrated radio", with lengthy dialogue not carrying much emotion. This can be done with very subdued and simplified lipsync, which looks surprisingly real - we don't move our lips that much in real world while talking.

[djames.suhanko]

I keep reading the thread. It's very interesting this topic.

[Greenlaw]

I don't do this so much now but when I was starting out, I kept a small stand-up makeup mirror on my desk and I would study my own mouth (as well as my facial expressions.) These days it's enough to just picture it in my head but I still perform the mouth shapes while I'm animating...IMO, it's the best reference tool I own.

As for overenunciating dialog, yes, that's a common newbie mistake. I think we call that 'peas and carrots', meaning it looks more like a character is eating with their mouth open than talking naturally. Anyway, when an animator is starting out, it's very easy to get focused on using all the mouth drawings for every possible sound you can make in speech, but natural dialog isn't spoken that way. When you're animating lip sync, study how your own mouth moves by reciting the dialog slowly and at actual speed and you should notice that you don't actually make every mouth shape for every word. A lot of speech sounds come across in fewer mouth shapes than you might think.

I usually work with about 11 mouth shapes but if the character isn't going to get too expressive, I can get away with about 4 shapes (closed, open, open with teeth, O shape,) and still be convincing.

As mentioned above, getting the mouth shapes animated is the easy part, and in Moho that shouldn't take more than a few minutes. The hard (and fun) part is animating the character's performance, which can be everything beside the mouth. Without the performance to go with the talking, all you have is mouth flapping.

[MeadowsR]

Over the years I've taken a lot of different approaches to animating lip sync in Moho for many TV shows on Netflix and other streaming channels, and what I finally settled on is using a Switch layer controlled using the Switch Selection Window. This is the easiest and most efficient way to select mouth shapes, and it's fully compatible with Interpolate Sub-Layers. (Unlike when using any Smart Bone techniques for animating switch layers.)

Here's my workflow...

1. setup the visemes for the mouth shapes in your switch layer. Take care to give the mouth drawings sensible names because the drawing and the label you give it will appear in the Switch Selection window when you are animating with it. I like to make the first mouth (at the bottom of the stack) the neutral expression, and then stack the visemes above that.

The visemes you choose is up to you, but I like to use a slightly modified version of the Preston Blair mouth shapes.

You can add specials and other expressive mouths in layers above the viseme layers. Moho reads drawings from bottom to top in a Switch layer, so any new drawing should always be added at the top of the stack...especially if you've already started animating the project so you won't upset any existing keyframes.)

2. Import your audio file. Ideally this is an uncompressed .wav or .aif file. If you import a compressed audio file, like an .mp3 for example, frame accuracy cannot be guaranteed.

3. Scrub or step through the audio and mark your keywords in the timeline. When marking, the first line of your text will appear on the timeline, and the rest will appear when the cursor hovers over the marker. I like to place one word or a few words in the first line of the note, and the complete dialog in the next lines so as not to clutter the timeline with a bunch of text.

4. Now select the Switch layer and open the Switch Selection window. The Switch layer is automatically loaded into the window. Dragging the slider will advance to each drawing in the Switch and set a keyframe. Now you can scrub/step through the project and set your mouth shapes as you listen to the track. Remember to look at the waveform...it visual cues for exactly on which frames the sounds are being spoken.

With a little practice, this process goes very quickly. For a typical scene, I can get through my first pass for lip sync animation (i.e, just the mouth shapes) in a few minutes. My second pass is usually just squashing and stretching the Switch layer itself using the Transform Layer tool to get some smoothness between the transitions, and this stage goes even more quickly.

Once the mouth animation looks good to me, I'll animate the character's head and facial features and body motions to really sell the performance. You can see examples of mouth animations in my 2017 and 2019 demo reels (links below,) but to watch these animations with the dialog tracks you'll need to watch the tv shows they were created for on Netflix and elsewhere.

Oh, actually some of my current training videos have examples of Moho animation with dialog, so you can check those too. Speaking of which, I'm planning a tutorial specifically on this topic but I have other tutorials I want to get out first. When the lip sync one is ready, I'll announce it here in the forums.

Hope this helps.

Very interesting topic