I think your referring to stencils, which can mask Characters / backgrounds like this
Check out maybe Kevins sketch entry for portals. It’s unity where I think a stencil is used… but same shader principles apply in unreal. I believe there is a breakdown on Kevins youtube as well somewhere if I remember correctly.