Code Snippet: Edge Detector/Antialiasing Shader


Some days ago i was looking for a simple and effective edge detector idea to use in an AntiAliasing shader for Deferred Shading (ideas that effectively make use of the Normal and the Depth Buffers).

Two called my attention: The first one is from the game Stalker, this page from GPU Gems explains the whole idea (nothing very complitaded =P, they show lot of the code). The results are pretty cool, as you can see in this image for example. The second is the one used in Tabula Rasa game, this page from GPU Gems shows the whole idea.

Stalker Idea

The shader code in HLSL is: (added the full version. for those interested only in the edge detection part, see the shader comments).

void PassThroughVS(inout float4 position : POSITION0,
                   inout float2 texcoord : TEXCOORD0) {

				   position.x =  position.x - pixel_size;
				   position.y =  position.y + pixel_size;
}

float2 e_barrier = float2(0.8f,0.00001f);  // x=norm, y=depth

float2 e_weights= float2(1,1);  // x=norm, y=depth

float2 e_kernel = float2(1,1);   // x=norm, y=depth

const static float2 offs[7] = {
   float2( 0.0,  0.0), //Center       0
   float2(-1.0, -1.0), //Top Left     1
   float2( 1.0,  1.0), //Bottom Right 5
   float2( 1.0, -1.0), //Top Right    3
   float2(-1.0,  1.0), //Bottom Left  7
   float2(-1.0,  0.0),  //Left         8
   float2( 0.0, -1.0) //Top          2
};  

float4 PShader2(float2 uv : TEXCOORD0) : COLOR0

{

 // Normal discontinuity filter

 float3 nc = tex2D(NormalMap, uv);

 float4 nd;

 nd.x = abs(dot(nc, tex2D(NormalMap, uv + offs[1]* pixel_size).xyz));

 nd.y = abs(dot(nc, tex2D(NormalMap, uv + offs[2]* pixel_size).xyz));

 nd.z = abs(dot(nc, tex2D(NormalMap, uv + offs[3]* pixel_size).xyz));

 nd.w = abs(dot(nc, tex2D(NormalMap, uv + offs[4]* pixel_size).xyz));

 nd -= e_barrier.x;

 nd = step(0, nd);

 float ne = saturate(dot(nd, e_weights.x));

 // Opposite coords

 float2 tc5r = -offs[5];

 float2 tc6r = -offs[6];

 // Depth filter : compute gradiental difference:

 // (c-sample1)+(c-sample1_opposite)

 float dc = tex2D(depthMap, uv).r;

 float4 dd;

 dd.x = tex2D(depthMap, uv + offs[1] * pixel_size).r +

         tex2D(depthMap, uv + offs[2]* pixel_size).r;

 dd.y = tex2D(depthMap, uv + offs[3]* pixel_size).r +

        tex2D(depthMap, uv + offs[4]* pixel_size).r;

 dd.z = tex2D(depthMap, uv + offs[5]* pixel_size).r +

        tex2D(depthMap, uv + tc5r* pixel_size).r;

 dd.w = tex2D(depthMap,uv +  offs[6]* pixel_size).r +

        tex2D(depthMap,uv +  tc6r* pixel_size).r;

 dd = abs(2 * dc - dd)- e_barrier.y;

 dd = step(dd, 0);

 float de = saturate(dot(dd, e_weights.y));

 // Weight

 float w = (1 - de * ne) * e_kernel.x; // 0 - no aa, 1=full aa

 //return float4(w,w,w,1); ///stop here for an edge detector shader

// The AA part (in edges we mix close pixels, non edge pixel w = 0 => use the normal uv, on edge we use a mean of four deslocated uv texture reads )
// Smoothed color

 // (a-c)*w + c = a*w + c(1-w)

 float2 offset = (uv ) * (1-w);

 float4 s0 = tex2D(TextureSampler, offset + (uv + offs[1] * pixel_size) * w);

 float4 s1 = tex2D(TextureSampler, offset + (uv + offs[2]* pixel_size) * w);

 float4 s2 = tex2D(TextureSampler, offset + (uv + offs[3]* pixel_size) * w);

 float4 s3 = tex2D(TextureSampler, offset + (uv + offs[4]* pixel_size) * w);

 return (s0 + s1 + s2 + s3)/4.h;

}

technique AntiAliasingStalker
{
    pass Pass1
    {
		VertexShader = compile vs_3_0 PassThroughVS();
                PixelShader = compile ps_3_0 PShader2();
    }
}

The normal buffer is stored as a 32 bits RGBA texture. The Depth Buffer is stored in a 32 bits Single Texture (storing post-projection z/w). If you use another depth enconding, just adjust the barrier variable value.

The pixel_size variable (a float2 with 1/target width in x and 1/target height in y) is used in DirectX 9 Shaders (Opengl and DirectX > 9 does not need this). The need for this “correction” is very well explained here.

The ScreenShot below is just the edge detector part of the shader. (just uncomment the following line in the shader and you will have a powerfull edge detector shader =P)

Example using this good edge detector shader. Cool for toon shading =P

 

Idea From Tabula Rasa

This is not so good as the one before (dont looks good when the camera is far from the objects or when the edges has little pixels), but is a bit simpler. Fort those interested  only in the edge detection part,

just return the “factor” variable in the pixel shader, instead of using it to blend the near pixels =P

const float2 delta[8] =
 {
 float2(-1,1),float2(1,-1),float2(-1,1),float2(1,1),
 float2(-1,0),float2(1,0),float2(0,-1),float2(0,1)
 };

 ///////////////////////////////////////////////////////
 ///silly, dummy functions to recover normal/depth (i was doing something with the read values before and i did not erased the function ...)
 float4 DL_GetDepth(float2 uv)
 {
	return tex2D(NormalMap  ,uv);
 }
 float4 DL_GetNormal(float2 uv)
 {
	return tex2D(depthMap ,uv);
 }

float depthSensibility;
float normalSensibility;
  ////////////////////////////
   // Neighbor offset table
   ////////////////////////////
   const static float2 offsets[9] = {
  float2( 0.0,  0.0), //Center       0
   float2(-1.0, -1.0), //Top Left     1
   float2( 0.0, -1.0), //Top          2
   float2( 1.0, -1.0), //Top Right    3
   float2( 1.0,  0.0), //Right        4
   float2( 1.0,  1.0), //Bottom Right 5
   float2( 0.0,  1.0), //Bottom       6
   float2(-1.0,  1.0), //Bottom Left  7
   float2(-1.0,  0.0)  //Left         8
};
float DL_GetEdgeWeight(in float2 screenPos)
{
  float Depth[9];
  float3 Normal[9];
  //Retrieve normal and depth data for all neighbors.
   for (int i=0; i<9; ++i)
  {
    float2 uv = screenPos + offsets[i] * pixel_size;
    Depth[i] = DL_GetDepth(uv);  //Retrieves depth from MRTs
    Normal[i]= DL_GetNormal(uv); //Retrieves normal from MRTs
  }
  //Compute Deltas in Depth.
   float4 Deltas1;
  float4 Deltas2;
  Deltas1.x = Depth[1];
  Deltas1.y = Depth[2];
  Deltas1.z = Depth[3];
  Deltas1.w = Depth[4];
  Deltas2.x = Depth[5];
  Deltas2.y = Depth[6];
  Deltas2.z = Depth[7];
  Deltas2.w = Depth[8];
  //Compute absolute gradients from center.
  Deltas1 = abs(Deltas1 - Depth[0]);
  Deltas2 = abs(Depth[0] - Deltas2);
  //Find min and max gradient, ensuring min != 0
   float4 maxDeltas = max(Deltas1, Deltas2);
  float4 minDeltas = max(min(Deltas1, Deltas2), 0.00001);
  // Compare change in gradients, flagging ones that change
   // significantly.
   // How severe the change must be to get flagged is a function of the
   // minimum gradient. It is not resolution dependent. The constant
   // number here would change based on how the depth values are stored
   // and how sensitive the edge detection should be.
   float4 depthResults = step(minDeltas * depthSensibility, maxDeltas);
  //Compute change in the cosine of the angle between normals.
  Deltas1.x = dot(Normal[1], Normal[0]);
  Deltas1.y = dot(Normal[2], Normal[0]);
  Deltas1.z = dot(Normal[3], Normal[0]);
  Deltas1.w = dot(Normal[4], Normal[0]);
  Deltas2.x = dot(Normal[5], Normal[0]);
  Deltas2.y = dot(Normal[6], Normal[0]);
  Deltas2.z = dot(Normal[7], Normal[0]);
  Deltas2.w = dot(Normal[8], Normal[0]);
  Deltas1 = abs(Deltas1 - Deltas2);
  // Compare change in the cosine of the angles, flagging changes
   // above some constant threshold. The cosine of the angle is not a
   // linear function of the angle, so to have the flagging be
   // independent of the angles involved, an arccos function would be
   // required.
  float4 normalResults = step(0.4, Deltas1* normalSensibility);  

  normalResults = max(normalResults, depthResults);
  return (normalResults.x + normalResults.y +
          normalResults.z + normalResults.w) * 0.25;
}  

float4 PShader(float2 texCoord : TEXCOORD0) : COLOR0
 {

 float4 tex = tex2D(NormalMap,texCoord);
 float factor = 0.0f;

 for( int i=0;i<4;i++ )
 {
	 float4 t = tex2D(NormalMap,texCoord+ delta[i]*pixel_size);
	 t -= tex;
	 factor += dot(t,t);
 }

 factor = min(1.0,DL_GetEdgeWeight(texCoord))*weight; 

 float4 color = float4(0.0,0.0,0.0,0.0);

 for( int j=0;j<8;j++ )
 {
	color += tex2D(TextureSampler,texCoord + delta[j]*pixel_size*factor);
 }
 color += 2.0*tex2D(TextureSampler,texCoord);
 return color*(1.0/10.0);

 }

The Vertex Shader and the buffers used are the same from the Stalker idea.

The first one is more beautiful but is slower.

The PloobsEngine uses these AA shaders (and some others also) to perform an Antialiasing Pass.

This is the first code snippet, when possible i will post more =P

 

, , , , ,