Techniques & Craft7 min readLessonBy · Production Lead at Studio432

Masking and Compositing Basics for Video Editors

Masking and compositing are the skills that let you work on one part of a frame without touching the rest — and they open up nearly every advanced visual technique in editing.

Every time you see an editor remove an unwanted object from a shot, place a subject against a new background, or isolate a sky for a color adjustment, they're using masking and compositing. These aren't techniques reserved for VFX studios — they're practical tools built into every professional NLE and compositing application, and editors at every level use them on ordinary projects. Understanding how masks work and how layers combine gives you precise control over your image instead of relying on adjustments that hit the entire frame equally. Once you know the fundamentals, a wide range of visual fixes and effects become straightforward rather than mysterious.

What a Mask Is

A mask is a shape or a grayscale image that tells your software which part of a layer to show and which part to hide. Wherever the mask is white, the layer is fully visible. Wherever the mask is black, the layer is completely hidden. Gray values produce partial transparency — the closer to white, the more visible; the closer to black, the more transparent. This simple logic is the foundation of all compositing work, from basic cleanups to complex multi-layer effects.

In practical terms, a mask lets you apply any effect — a color correction, a blur, a brightness adjustment, a completely different piece of footage — to only the area you've defined. The rest of the image stays untouched. You might draw a mask around a subject's face to correct their skin tone separately from the background, or mask out a distracting logo in the corner of a frame, or define the area of a sky so you can replace it or grade it independently.

Masks are non-destructive in most modern applications. You can move them, resize them, invert them, feather their edges, and delete them without affecting the underlying media. That makes them safe to work with even when you're still figuring out what result you're after.

Shape Masks vs. Luma Masks

Shape masks — sometimes called geometric masks or spline masks — are masks you draw manually. You place points or anchor handles around the area you want to isolate, creating a closed shape that defines the visible region. Common shape mask tools include rectangles, ellipses, and bezier splines. Bezier splines are the most flexible: by adjusting the handles on each anchor point you can trace around complex curved edges like hair, shoulders, or an irregular background element.

Luma masks, also called luminance masks, are generated automatically from the brightness values in the image itself. The software analyzes the frame and creates a mask based on which pixels are bright, which are dark, and which fall somewhere in between. This makes luma masks extremely useful for isolating skies — which tend to be the brightest part of an outdoor frame — or for protecting shadows while you adjust highlights, or vice versa. Because a luma mask derives from the image data directly, it can trace edges that would take hours to draw by hand.

Choosing between a shape mask and a luma mask depends on the shot. If the area you want to isolate has a clear geometric boundary or you need precise control over an irregular edge, draw a shape mask. If the thing you want to isolate differs significantly in brightness from its surroundings, a luma mask will often get you there faster with a cleaner result. Many compositing tasks use both at once: a luma mask to rough out an area, and a drawn mask to refine the edges.

Tracking a Mask

A static mask works fine for a locked-off shot where nothing moves. The moment your subject, your camera, or both are in motion, a mask that stays fixed will drift off the area you intended to isolate within a few frames. Tracking is what keeps a mask attached to a moving element throughout the clip.

Most NLEs and compositing applications include a built-in mask tracker. You define your mask on one frame, engage the tracker, and the software analyzes the motion of the pixels inside the mask across every subsequent frame — updating the mask's position, rotation, and sometimes scale to follow the element. For simple translations, such as a subject walking left to right, tracking is usually automatic and accurate. For more complex motion — rotation, perspective changes, or partially obscured elements — you may need to adjust the tracker settings or manually correct keyframes where the tracker loses its reference.

In DaVinci Resolve, the tracker in the Color page's Window controls follows motion using optical flow analysis. In After Effects, the Mask Tracker works similarly and can also drive other properties beyond just the mask. When the built-in tracker struggles — particularly with fast motion or heavy motion blur — point tracking or planar tracking tools give you more control by analyzing specific reference points in the frame rather than the general motion of the mask region.

Combining Layers in a Composite

Compositing is the process of combining multiple visual layers into a single, coherent image. At its simplest, you stack two clips on top of each other in your timeline and use a mask to reveal parts of the upper clip while letting the lower clip show through in the unmasked areas. The result looks like a single image drawn from two different sources.

The order in which layers are stacked matters. In most applications, the layer at the top of the stack is rendered in front of all the layers below it. Masking the top layer exposes whatever is directly beneath it, and you can add as many layers as you need below that to build up a composite scene. Complex composites — backgrounds, mid-ground elements, subjects, foreground elements, and effects layers — follow this same stacking logic no matter how many layers are involved.

A clean composite depends on edge quality. Hard mask edges look unnatural; a slight feather softens the transition between the isolated element and the background so the layers appear to belong together. The correct amount of feather depends on the original footage — footage shot with shallow depth of field will have softer natural edges that need more feather to match, while hard-edged objects against a contrasting background may need less.

Simple Cleanup and Object Removal

One of the most common compositing tasks editors encounter is removing something from a frame — a microphone that dipped into frame, a crew member visible in a reflection, a piece of signage the client wants gone. For objects against a relatively static background, the simplest approach is to clone or sample the background from another frame where the object isn't present and composite that clean frame over the area you want to remove.

In practice this means identifying a frame where the background behind the offending object is visible — either a frame before the object appeared or from a different take — and masking that clean area onto a layer above your primary clip, positioned to cover the object. If the background is stationary, you may not even need to track the mask. If the camera moves, you'll need to match the motion of the clean background layer to the camera's movement, which typically involves motion tracking or corner pinning.

For more complex removals — objects in front of moving backgrounds, or objects that cover a detailed area that doesn't exist in any other frame — dedicated tools like DaVinci Resolve's Object Removal or After Effects Content-Aware Fill analyze surrounding pixels to reconstruct what should be there. These automated tools save significant time but often need manual refinement, especially around fine detail or motion.

Blend Modes

Blend modes control how a layer's pixels interact mathematically with the pixels of the layers below it, rather than simply covering them. Without a blend mode adjustment, the default behavior is that the top layer covers whatever is beneath it. Switch the blend mode and the two layers begin to interact based on their brightness, color, or contrast values in ways that range from subtle to dramatic.

The most practical blend modes for editors are Screen, Multiply, Overlay, and Add. Screen makes dark pixels transparent and bright pixels visible — making it ideal for compositing light elements like muzzle flashes, lens flares, or fire against a black background without needing to mask them at all. Multiply does the opposite: white pixels become transparent and dark pixels darken whatever is below, which is useful for compositing shadows or textures onto a surface. Overlay combines contrast and color in a way that boosts both, making it common for texture overlays or light leaks. Add is similar to Screen but more intense and useful for highly luminous elements.

Blend modes interact with the underlying image differently depending on the brightness and color of both layers, so the best way to understand them is to experiment. Stack a texture or light effect layer above your footage and cycle through the modes — you'll develop intuition for which mode to reach for in a given situation far faster than memorizing descriptions of each one.

  • Screen — makes black transparent, keeps bright areas; ideal for light effects and fire on black backgrounds
  • Multiply — makes white transparent, darkens the image; ideal for shadows and texture overlays
  • Overlay — boosts contrast and color simultaneously; useful for grain, texture, and light leaks
  • Add — intensifies brightness beyond Screen; useful for very bright light elements
  • Soft Light — subtle version of Overlay; useful for gentle texture or luminosity blending

Putting It All Together: A Simple Composite

To see how these ideas connect, consider a common task: placing a subject shot against a solid-colored background onto a different background plate. If the original shoot used a green screen, chroma keying removes the green and the subject appears isolated. But even without a clean key, you can often pull a workable isolation using a combination of a drawn spline mask around the subject's general outline, a luma mask to handle lighter edge areas, some edge feathering to soften the transition, and a color correction pass to match the subject's lighting to the new background.

Once the subject is composited onto the new background, blend modes and color adjustments unify the two layers. A slight color grade applied to both layers simultaneously — using a merged or pre-comp approach — helps the subject appear to exist in the same light environment as the background. Small details like a shadow layer placed below the subject and blended with Multiply, or a subtle vignette applied over the whole composite, complete the illusion.

This sequence — isolate, composite, match, refine — is the same basic workflow whether you're doing a simple sky replacement or a multi-layer effects shot. Mastering each step individually makes the full process manageable rather than overwhelming.

FAQ
What is the difference between a mask and a key?

A key — like a chroma key or luma key — is a mask generated automatically by the software based on a specific color or brightness value in the footage. You tell the software to make all pixels of a particular color (green, blue) transparent, and it calculates the mask for you. A drawn mask is one you create manually by placing points and drawing a shape around the area you want to isolate. Keys are faster when the footage was shot specifically to support them — clean green screen, consistent lighting, hard edges. Drawn masks give you more control in situations where the background varies too much for a key to work cleanly.

How do I keep my mask edges looking natural?

Edge quality is controlled primarily by feathering — spreading the transition between the masked and unmasked areas over a number of pixels so it softens from fully visible to fully transparent gradually rather than cutting off sharply. The correct feather amount depends on how the original subject was photographed. Shallow depth of field creates soft edges that need more feather; a sharp, hard-edged object against a flat background needs less. In addition to feathering, look at the edge contrast: if the subject's edge has color spill from the background lighting, a decontamination pass or a color correction targeted to the edge region helps integrate the composite.

Do I need dedicated compositing software, or can I do this in my NLE?

Most modern NLEs — DaVinci Resolve, Adobe Premiere Pro, Final Cut Pro — include masking, basic compositing, and blend mode tools that are sufficient for the majority of editorial compositing tasks: sky replacements, simple object removals, isolated color corrections, and basic layer combinations. For complex work involving many layers, motion graphics integration, particle effects, or detailed rotoscoping, a dedicated compositing application like After Effects or Fusion (built into DaVinci Resolve) gives you more precise tools and a more organized node- or layer-based workflow. Start in your NLE and move to a compositor when your project demands it.

How does mask tracking work when the subject moves quickly?

Standard mask trackers analyze the optical flow — the apparent motion of pixels between frames — inside or near the mask region. Fast motion creates motion blur that reduces the amount of clear pixel information available between frames, which can cause the tracker to drift or lose its reference. When this happens, reduce the tracking interval to analyze more frames at shorter gaps, manually set keyframes on frames where the tracker slips, or switch to a point tracker that follows specific high-contrast features on the moving element rather than the general mask region. For very fast or erratic motion, manual keyframing on every few frames is often more reliable than automation.

What is feathering and when should I use it?

Feathering is the control that spreads the edge of a mask outward over a defined number of pixels, creating a gradual transition rather than a hard cut between the visible and hidden areas. You should use feathering almost every time you draw a mask intended to blend into a composite — zero feather on a drawn mask produces an obviously artificial hard edge that reads as cut-and-paste. The exception is when you're masking out a clean geometric shape with a genuinely hard physical edge, such as a screen or a sign. In most real-world compositing situations involving organic subjects or natural environments, some degree of feathering is necessary for the result to look integrated.

F
Written & reviewed by
Faran@432
Production Lead & Consultant · Studio432

Faran is the production lead at Studio432 — the studio arm of Club432, the Karachi collective behind 100+ filmed live sessions and concert films. He plans and consults on shoots worldwide, and owns the gear, settings and craft standards behind everything published here.

Don’t want to learn it — want it done?

Studio432 edits concert footage, music videos, gaming content, vlogs and short-form for creators worldwide. Send the footage, get a quote by email.

Start a project →