Hello again!
Im working on a grid-based game for unity and I plan to have some form of fog of war for the different rooms within the map.
I made a script that automatically generates a plane for each room acording to its shape. Since im working with grids, all planes will be euclidean and composed of multiple 90º angles.
Below you can see an example of the kind of planes that will compose a map.
Im interested in creating a shader that will mask the edges of these planes so that they can have a fog effect on the side and remain opaque in the center. Of couse, solutions such as freshnel wont work because I dont have any kind of curvature here. How could I create such shader?
The short version is you can’t do this with “a shader”.
At least not without doing more work somewhere else. There’s a lot of ways to go about this though.
You could render the room shapes into a render texture and blur the result. Either all of the shapes for rooms that are fogged, maybe more cheaply all of the rooms you can see assuming that’s a few number. Then use the resulting blurred texture as a mask for your “fog of war”. Could be done in screen space, or as view of the entire play area that only updates when your visibility changes, etc. Really you could render the visible shapes into a render texture aligned to use 1 pixel per grid box and sample the resulting texture with bilinear filtering and you’d basically be getting the blur for free. Just need to rescale the values a little to get whatever fog thickness you’d want.
You could use vertex coloring and some inset cuts into your meshes to mark the outside edge of your shapes. Maybe use different colors to mark the borders with other rooms and use a shader to decide which edges to fade or not based on that by setting properties on the material from the CPU.
You could use a custom shader that gets a list of rectangles in UV or world space. Then use the SDFs of those rectangles to fade the edges out. Could be a list of the shapes that make the room, or likely more useful the shapes for the rooms that border it. Again, letting you decide which edges to fade or not from the CPU.
For a while I considered doing it via vertex coloring but I couldnt come up with a solution to procedurally color the specific vertex points that make the edges of these planes. I’ll revisit that idea tho, Im sure there’s a way.
Either way I think i like it over the other propositions as im more comfortable than working with a render textures.
Using vertex coloring in a naïve way you could store up to 4 “border room edges”. One per color channel. But you’d have to cut up the “plane” like this:
Then all you need to do is have the interior vertices set to black (0,0,0,0), and color each border vertex around the side a channel per room.
In the vertex shader you can check if a vertex has the color channel for a bordering room that’s “hidden”, then pass black to the fragment shader, otherwise pass white. If you look, there’s a corner that’s shared by both the blue and green rooms that’s teal, aka blue + green. The weird cuts are to ensure the resulting gradient looks nice.
If you need more than 4 bordering rooms, you can use the bits of the vertex color and extract the data in a vertex shader. Vertex colors are generally stored as a byte per channel, so you’d have 8 bits to work with, meaning 8 possible bordering rooms. Though you’ll want to read up on bitwise operators for setting and testing against the stored value, the same idea applies. If a vertex is bordering a hidden room, set it to black.
More than 8 rooms, use bitwise packing in multiple vertex color channels.
