Rotoscope Motion Blur & Defocus Edges
The Edge Extension Trick
If you have ever used the Magic Mask in DaVinci Resolve, you have likely encountered a specific problem: the rotoscoping result looks great until you look closely at the edges. When you composite the subject over a new background, you often see contaminated color details in the soft edges. This actually happens with traditional rotoscoping too!
This happens not because Magic Mask is bad, but because there is detail in the defocused or motion-blurred edges that contains color data from the original background. There’s a mix from the old background on the edge - we need to get rid of the background colors.
Today, I’m going to share an advanced technique to fix this. We are going to manipulate the RGB color foreground completely separately from its Alpha channel . This "Edge Extension" technique allows you to retain all the original blur and shape without carrying over the old background artifacts.
Denoise Your Footage
Before doing any heavy compositing or keying, it is crucial to denoise the footage. You can use the built-in tools or a plugin like Neat Video's Reduce Noise.
Pro Tip: After adding noise reduction, right-click the node and select Cache to Disk (enable "Lock Branch" and "Lock Cache"). This prevents the system from recalculating the noise reduction every frame, saving you massive processing power later.
Optimize Color Space for Magic Mask
Magic Mask is trained on Rec.709 or sRGB images. If you are working in a linear space (like ACES CG/AP1), you should use a Color Space Transform (CST) node to convert the footage to Rec.709 before plugging it into the Magic Mask . This yields significantly better tracking results.
Pulling the Magic Mask
Mode: Use the Better mode (not Faster) to ensure you get a nice soft matte that retains motion blur.
Selection: Click on the subject (e.g., face and hair) and track forward and backward.
Visualization: Temporarily deselect "Post Multiply Image" while selecting to see the overlay better, then re-enable it to apply the matte.
Tracking the Background Replacement
You need to track the object you are replacing (in this example, a sign). You can use the built-in Planar Tracker or Mocha Pro.
Handle Occlusions: If a person walks in front of the sign, create a garbage matte (occlusion mask) for the person first so the tracker doesn't get confused by the foreground movement .
Reference Frame: Pick a reference frame (often frame 0) to align your tracking data. Use the Expand Surface function to extend the tracking data to the corners of the frame, ensuring you don't mess up the aspect ratio of your replacement element . If you are using the built in Fusion planar tracker, the reference frame is the frame you started tracking from and is baked into the exported planar transform.
The Edge Extension Workflow
This is the core of the tutorial. We need to create a version of our subject that has "pure" color extended outward, and then cut it out with a fresh alpha channel.
Split the Magic Mask
We are going to split the Magic Mask output into two separate Bitmap Mask streams.
Stream A (Color/Core Matte): Used to grab pure colors.
Stream B (Alpha/Edge Matte): Used to define the transparency.
Create the "Core" Matte (Stream A)
We need a matte that sits inside the subject, away from the blurry edges.
Connect a Blur node to the first Bitmap mask and blur it enough to lose the fine details.
Connect a Brightness/Contrast node. We will use this to erode (shrink) the mask.
Set the Brightness/Contrast to operate mainly on the Alpha channel. Adjust the Gamma slider down (to the left) to shrink the mask until it sits well inside the solid color values of your subject.
Hardening the Matte: Adjust the Threshold (Low and High) toward the center (e.g., 0.5) to remove anti-aliasing and make the edge hard
Make sure to check the box to clip black and white to avoid alpha values under 0 and over 1.
Extend the Edges (The Clean Plate Trick)
Now we extend the pixels from that core matte outward.
Add a Clean Plate node.
Feed your original (denoised) image into the Clean Plate input .
Take the result of your "Core Matte" (Stream A) and plug it into the Garbage Matte input of the Clean Plate node.
In the Clean Plate settings, check Garbage Matte Invert.
Slide the grow edges control to the right. This creates a "blur/divide" effect that grows the edge pixels perpendicular to the edge, filling the area with the subject's pure color.
Refine the Alpha (Stream B)
Now we prepare the transparency.
Take the second Bitmap Mask stream.
Add a Blur node just enough to match the original motion blur without keeping the artifacts.
Combine with Matte Control
We now combine our new "Clean Plated" colors with our soft Alpha.
Add a Matte Control node after the Clean Plate.
Plug the Clean Plate image into the Background input.
Plug your refined Alpha (Stream B) into the Foreground input (green input).
Settings:
Combine: Combine Alpha
Operation: Copy Alpha.
Check Post Multiply Image.
Final Compositing
Now you have a "clean" foreground subject with edge-extended colors and a soft alpha.
The "Atop" Merge: This is critical. We don't want to paste this edge-extended character over the entire image, only where the sign replacement is happening.
Add a Merge node. Connect your background (with the new sign replaced) to the background input.
Connect your Edge Extended character to the foreground input.
Set the Merge Operation to Atop. This tells Fusion to put the edge-extended patch only where the replacement sign exists, leaving the rest of the original plate untouched.
Summary
By separating the color processing from the alpha processing, you eliminate the "halo" of the old background.
The Problem: Defocused edges contain the old background color.
The Fix: Erode the mask to find pure color, extend that color outward using Clean Plate, and then re-apply a soft alpha.
Don't forget to add grain back on top of your final comp to match the original footage texture!.
