Masking workflow should be overhauled

[JoelMayer]

Now i know that's easier said than done but for an otherwise super intuitive software like Moho, the Masking workflow is a headache time and time again. There's no visual representation what does what and basically, for more complex structures, it's all trial and error. I think it would definitely be awesome, if we wouldn't have to throw stuff into a group layer anymore to do any sort of masking. Maybe something more like After Effects "Set Mask" effect could work? You could do that in the layer settings in a drop down, similiar to how you add a Smart Warp layer and then have some options on how the mask is going to affect the layer.

I know most of you and even me probably got used to how masking works in Moho over time but, compared to pretty much any other software, it's needlessly complicated and non-descriptive. I really think you could simplify it greatly by either a method like i said or maybe by overhauling the whole layer window and having masks shown as sublayers of affected regular layers etc.

For new users, this is definitely a big hurdle. Anyway, just my opinion

[SimplSam]

One old post that helped me back-in-the-day was this:

Masking explained... I hope!

First things first:

Masking looks incorrect in the preview when you use it with nested layers. It must be rendered to see it properly. This is what made me think that it was confusing at first. If you don't trust the preview and only look at a render than masking should make more sense.

-------------

A mask is just like an alpha channel in Photoshop. Adding isn't the same as... "adding" as you might think of it. A mask is a black or white "alpha channel".

This has nothing to do with the color of the shapes used in the mask layer. Black and white is just a term used that is related to alpha channels. When a layer has a filled shape it is the "transparency" that determines its masking or whether that layer is "black" or "white" in the masking alpha channel.

The really confusing bit is Reveal all and Hide all. These are opposites. The mask "starts out" in the opposite way.

Think of it this way:

Black hides
White reveals

Adding to a mask creates white, subtracting from a mask creates black It isn't really like taking away or adding to a mask at all... it is based on black or white pixels. Subtracting only "adds" black. Adding only "adds" white.

This has its origins from the mathematical concept of "adding" and "subtracting" the grayscale or RGB values of pixels. Each pixel has a number value from 0 to 255. Adding those numbers makes a pixel "brighter" subtracting wil make it "darker".

In the version of photoshop I started with that was the only way to combine masks by using Add or Subtract or Multiply from one alpha channel to another.

The hide and reveal with white and black alpha channel concept relates to the following group layer masking properties:

Reveal all
is a totally "white" mask revealing everything. It is as if there is no mask. Adding a layer to it is adding "white" to "white" which means nothing happens... unless you use "clear the mask and add to it". This "erases" the mask and "starts over". If you "clear" the mask you are making it all black. If the group mask property is set to "reveal all" and you clear it you basically have turned it into "Hide all".

Subtracting from Reveal all creates a "blank" spot. A black hole through the mask. remember black hides white shows or reveals and Reveal All is all white, unless of course you have "cleared it" now it is reversed. if not then the mask is all white so subtracting creates a black or hidden spot.

I can see how this could get confusing...

Hide All
Now the mask is "turned on" or completely black and hides everything. Adding to the mask will create a "white hole" through the black. remember when you "add" you are creating a white spot that makes things visible. If you use subtract this is just "adding black" to the black mask... nothing happens.

clear the mask and add this layer to it works differently now. Clearing the mask makes it all black and then adds that layer... but.. how it looks depends on layer order. If this layer is above another mask layer it will override that layers masking (see below about layer order).

Layer Order!!!
The ORDER of the masking layer is also a factor. Layers higher in the stacking order take precedence over layers below it.

This is critical for making complex masks. It allows you to stack elements under mask layers with different mask layers... <sigh> a lot of this requires experimentation to see how it all works. Eventually it starts to make sense.

Making a mask layer invisible will also turn off its effect on the mask... obviously.

------

This is the idea you start with. It will get very complicated when you start to add multiple group layers with different masking to this setup. You can create a complex masking situation in a group layer with multiple layers all doing different things, then set the masking of the GROUP layer itself to mask layers in it's parent layer which has DIFFERENT masking set.

You can create amazing stuff if you get the hang of it.

In the sample below the character on the right pours a can of red paint over the character on the left. This done with several group layers and duplicated layers to create the perfect mask to produce this effect. This would be an example of a "complex" mask because the character is duplicated in "red" and a mask is used to reveal the red character... but also the strokes had to be duplicated on another layer since they are ALSO part of the mask... etc etc etc... <sigh> very complicated but it works and I know WHY it works.

[MISSING IMAGE]


-------------------------------------------------
Additional info moved here from later post
-------------------------------------------------

Sample file used below:
http://hubumedia.com/anime-studio/forum ... t_mask.zip
Masking in AS "in general" is based on the transparency of the layer or shapes on a layer NOT ON COLOR VALUE!

What this means is, if you have a soft edge fill... then you get a "fuzzy" edged mask based on the "transparency" or "falloff" of the soft edge of that fill.

BUT if you use a black to white gradient... that has no effect on masking. The shape is filled but it is filled with black to white. I think this is incorrect. I think it should be an option in masking to use the color values. I think you should be able to create a black to white gradient and use it as a gradient mask but for now this will not work.

Now on to that other gradient example in your file which I think should ALSO work but doesn't... the one with a 100% transparent color for one of the colors in the gradient... This ABSOLUTELY SHOULD WORK but it does not.

I think in both of these situations masking is based on the "filled-ness" of the shape. If the shape has a fill PERIOD it is 100% filled and any transparency other than soft edge effect is ignored. So even though you have a transparent "end" on the gradient... the shape is still "filled" so no gradient mask. I did EXTENSIVE testing of this and absolutely no combination of layers or effects even with separate styles will create the proper gradient mask.

I believe this is INCORRECT behavior but unfortunately this is how it works.

There is however a "trick" you can use to make this work:

Simple gradient mask:
[MISSING IMAGE]

Simple version:
Soft edge does work as a mask. The trick then is to use a soft edged fill to create a "pseudo" gradient mask on a separate layer. You create a soft edge fill that is very large with a large edge radius for the soft edge and use the "edge" as the gradient mask. You would then have 2 mask layers.

Complex Gradient mask:
[MISSING IMAGE]

Complex version with 3 mask layers:
There are problems with the simple version since the large shape will mask EVERYTHING not just a specific bound area like a box with a gradient. So to do anything more complex will require breaking this up into several layers and using a combination of masking settings.

I will try to explain how the complex version in the sample file works:

http://hubumedia.com/anime-studio/forum ... t_mask.mov

The main group is set to hide all. The "mask" is now "black" hiding everything.

Layer 1 ---> the color shape layer to be masked
Mask 1 ----> Add to mask but keep invisible
(Adding is "white" so now the shapes are revealed in the layer above it but the mask layer is invisible. We have added white areas to the original black mask)

Mask 2 ----> Subtract from mask
(this layer is soft edge shape to create the gradient. Subtracting removes from the the whole mask. The mask started out black, then we added white areas, now we add a black area which is the soft edge fill.)

Mask 3 ----> Add to mask but keep invisible
This layer "masks" the soft edge layer. The last mask confines the soft edge layer shape to a specific shape but has no effect on the first mask layer above it because it is ADDING. The first mask layer is also adding... adding is creating "white" so adding white to white = white... so no change in the masking of the first mask layer. Mask 2 however is SUBTRACTED which is black. So now the third mask layer is ADDING WHITE only to the second mask. All of this is working based on layer order.)

EDIT:
That last bit for the 3rd mask layer seems... strange. You would think that "adding" to the 2nd mask layer would eliminate it... what is really happening is the 3rd mask is MASKING the 2nd mask. The end result is the actual mask that is masking Layer 1 at the top. As I said it is confusing. Once you "understand it" it gets easier. Took me ages to get the hang of it.

You can get more sophisticated with this by putting these "complex" masks inside ANOTHER group layer to further modify the mask. Remember a group layer can have its own set of masking with in it that is completely separate from the parent group.

You can even have groups within groups within groups. Remember though.. if you do THAT you can't rely on the preview you must render to see the results.

Remember that none of this has anything to do with color values! It has to do with the "filled-ness" (made up term) of shapes in the layer.

Using black and white to refer to the values of a mask is just to put it into alpha channel terms.

-vern