Masking problem with hidden edges

Wondering how to accomplish a certain animation task? Ask here.

Moderators: Víctor Paredes, Belgarath, slowtiger

Post Reply
User avatar
parseh
Posts: 36
Joined: Sat Aug 23, 2014 8:39 pm
Location: Iran
Contact:

Masking problem with hidden edges

Post by parseh »

This is my first time rigging in Anime Studio. I'd like to add a little shading to my character using masks. I have used the Hide Edge tool on a few places.
Problem: when I render the scene I get gaps on the places where I have hidden the edges. I know this is because I have checked the Exclude Strokes option. Is there any way to solve this or basically I'm doing this the wrong way? As I said I'm a newbie here :)

Image

anme file:
https://copy.com/NcVSjooET0EjdzxD
User avatar
parseh
Posts: 36
Joined: Sat Aug 23, 2014 8:39 pm
Location: Iran
Contact:

Re: Masking problem with hidden edges

Post by parseh »

Sorry for bumping this thread. This is very important to me. Does anyone have a solution for this?

Thanks
User avatar
heyvern
Posts: 7035
Joined: Fri Sep 02, 2005 4:49 am

Re: Masking problem with hidden edges

Post by heyvern »

I tried to set up a sample file similar to the original post to see what could be done to work around this antialising issue. That is what is causing the white lines. It has to do with the mask antialiasing and creating a "lighter" color at the edges. This is not a scientific exact explanation but sort of a theory.

What is happening is when you have "exclude strokes" checked in the mask the mask is only using the fill. The fill mask "antialiases" to the edge of the stroke which creates the white edge. The pixels right at the edge are "less than black" so it it leaves an edge showing. In theory that antialised edge should "blend" through to the layers below and not show... but... that doesn't happen for some reason.

At this time there is no solution for this except to try and workaround it with other techniques. This issue has been around for a long time but I've just sort of avoided it by using some other tricks that give the same result without needing to use "exclude paths" on the mask when it causes a problem.

For example in the original example image, instead of using masking of shapes for the relatively simple shading, the Shading fill effect could be used and gives almost the exact same result.

Another trick would be to mask the shading layer WITHOUT using exclude strokes. Instead of a "solid" opaque color for the shading, use a transparent darker color and set the shading layer's blending mode to "multiply". This will allow the strokes underneath to show through the shading layer without needing the "exclude strokes" checked on the mask. This trick eliminates the lines on the mask completely.

Another trick I've used is to create stroke shapes from just a few points where the lines show on the mask layer. Make this small shape the same color as the fill and expand the width of that stroke shape. This works pretty well as it hides the white edges. It does require a few extra shapes though.

In my opinion there are alternative ways to create the shading that avoids the antialiasing white edges that come from excluding strokes in the maks. Of course the best solution would be if exclude strokes worked "better". I'm not exactly 100% sure of why this happens and why it can't be fixed.

If you post an exact situation or image of the issue I may be able to show you how to work around it without a ton of effort.
User avatar
parseh
Posts: 36
Joined: Sat Aug 23, 2014 8:39 pm
Location: Iran
Contact:

Re: Masking problem with hidden edges

Post by parseh »

Hi heyvern,
Thank you for your detailed explanation.I was thinking I must be doing something wrong and didn't know this is a limitation by the software. The "multiply fill"solution did the trick.

Thanks again
User avatar
Víctor Paredes
Site Admin
Posts: 5658
Joined: Wed Jan 26, 2005 12:18 am
Location: Barcelona/Chile
Contact:

Re: Masking problem with hidden edges

Post by Víctor Paredes »

You also can create strokes with the same color of the fill for the parts with conflicts. That solves the problem and it's relatively easy to make.
Image Image Image Image
Moho Product Manager

www.mohoanimation.com
Rigged animation supervisor in My father's dragon - Lead Moho artist in Wolfwalkers - Cartoon Saloon - My personal Youtube Channel
User avatar
parseh
Posts: 36
Joined: Sat Aug 23, 2014 8:39 pm
Location: Iran
Contact:

Re: Masking problem with hidden edges

Post by parseh »

There's more than one way to skin a cat! Thanks guys.
Post Reply