Code Snippet: Edge Detector/Antialiasing Shader

Some days ago i was looking for a simple and effective edge detector idea to use in an AntiAliasing shader for Deferred Shading (ideas that effectively make use of the Normal and the Depth Buffers).

Two called my attention: The first one is from the game Stalker, this page from GPU Gems explains the whole idea (nothing very complitaded =P, they show lot of the code). The results are pretty cool, as you can see in this image for example. The second is the one used in Tabula Rasa game, this page from GPU Gems shows the whole idea.

Stalker Idea

The shader code in HLSL is: (added the full version. for those interested only in the edge detection part, see the shader comments).

void PassThroughVS(inout float4 position : POSITION0,
                   inout float2 texcoord : TEXCOORD0) {

				   position.x =  position.x - pixel_size;
				   position.y =  position.y + pixel_size;

float2 e_barrier = float2(0.8f,0.00001f);  // x=norm, y=depth

float2 e_weights= float2(1,1);  // x=norm, y=depth

float2 e_kernel = float2(1,1);   // x=norm, y=depth

const static float2 offs[7] = {
   float2( 0.0,  0.0), //Center       0
   float2(-1.0, -1.0), //Top Left     1
   float2( 1.0,  1.0), //Bottom Right 5
   float2( 1.0, -1.0), //Top Right    3
   float2(-1.0,  1.0), //Bottom Left  7
   float2(-1.0,  0.0),  //Left         8
   float2( 0.0, -1.0) //Top          2

float4 PShader2(float2 uv : TEXCOORD0) : COLOR0


 // Normal discontinuity filter

 float3 nc = tex2D(NormalMap, uv);

 float4 nd;

 nd.x = abs(dot(nc, tex2D(NormalMap, uv + offs[1]* pixel_size).xyz));

 nd.y = abs(dot(nc, tex2D(NormalMap, uv + offs[2]* pixel_size).xyz));

 nd.z = abs(dot(nc, tex2D(NormalMap, uv + offs[3]* pixel_size).xyz));

 nd.w = abs(dot(nc, tex2D(NormalMap, uv + offs[4]* pixel_size).xyz));

 nd -= e_barrier.x;

 nd = step(0, nd);

 float ne = saturate(dot(nd, e_weights.x));

 // Opposite coords

 float2 tc5r = -offs[5];

 float2 tc6r = -offs[6];

 // Depth filter : compute gradiental difference:

 // (c-sample1)+(c-sample1_opposite)

 float dc = tex2D(depthMap, uv).r;

 float4 dd;

 dd.x = tex2D(depthMap, uv + offs[1] * pixel_size).r +

         tex2D(depthMap, uv + offs[2]* pixel_size).r;

 dd.y = tex2D(depthMap, uv + offs[3]* pixel_size).r +

        tex2D(depthMap, uv + offs[4]* pixel_size).r;

 dd.z = tex2D(depthMap, uv + offs[5]* pixel_size).r +

        tex2D(depthMap, uv + tc5r* pixel_size).r;

 dd.w = tex2D(depthMap,uv +  offs[6]* pixel_size).r +

        tex2D(depthMap,uv +  tc6r* pixel_size).r;

 dd = abs(2 * dc - dd)- e_barrier.y;

 dd = step(dd, 0);

 float de = saturate(dot(dd, e_weights.y));

 // Weight

 float w = (1 - de * ne) * e_kernel.x; // 0 - no aa, 1=full aa

 //return float4(w,w,w,1); ///stop here for an edge detector shader

// The AA part (in edges we mix close pixels, non edge pixel w = 0 => use the normal uv, on edge we use a mean of four deslocated uv texture reads )
// Smoothed color

 // (a-c)*w + c = a*w + c(1-w)

 float2 offset = (uv ) * (1-w);

 float4 s0 = tex2D(TextureSampler, offset + (uv + offs[1] * pixel_size) * w);

 float4 s1 = tex2D(TextureSampler, offset + (uv + offs[2]* pixel_size) * w);

 float4 s2 = tex2D(TextureSampler, offset + (uv + offs[3]* pixel_size) * w);

 float4 s3 = tex2D(TextureSampler, offset + (uv + offs[4]* pixel_size) * w);

 return (s0 + s1 + s2 + s3)/4.h;


technique AntiAliasingStalker
    pass Pass1
		VertexShader = compile vs_3_0 PassThroughVS();
                PixelShader = compile ps_3_0 PShader2();

The normal buffer is stored as a 32 bits RGBA texture. The Depth Buffer is stored in a 32 bits Single Texture (storing post-projection z/w). If you use another depth enconding, just adjust the barrier variable value.

The pixel_size variable (a float2 with 1/target width in x and 1/target height in y) is used in DirectX 9 Shaders (Opengl and DirectX > 9 does not need this). The need for this “correction” is very well explained here.

The ScreenShot below is just the edge detector part of the shader. (just uncomment the following line in the shader and you will have a powerfull edge detector shader =P)

Example using this good edge detector shader. Cool for toon shading =P


Idea From Tabula Rasa

This is not so good as the one before (dont looks good when the camera is far from the objects or when the edges has little pixels), but is a bit simpler. Fort those interested  only in the edge detection part,

just return the “factor” variable in the pixel shader, instead of using it to blend the near pixels =P

const float2 delta[8] =

 ///silly, dummy functions to recover normal/depth (i was doing something with the read values before and i did not erased the function ...)
 float4 DL_GetDepth(float2 uv)
	return tex2D(NormalMap  ,uv);
 float4 DL_GetNormal(float2 uv)
	return tex2D(depthMap ,uv);

float depthSensibility;
float normalSensibility;
   // Neighbor offset table
   const static float2 offsets[9] = {
  float2( 0.0,  0.0), //Center       0
   float2(-1.0, -1.0), //Top Left     1
   float2( 0.0, -1.0), //Top          2
   float2( 1.0, -1.0), //Top Right    3
   float2( 1.0,  0.0), //Right        4
   float2( 1.0,  1.0), //Bottom Right 5
   float2( 0.0,  1.0), //Bottom       6
   float2(-1.0,  1.0), //Bottom Left  7
   float2(-1.0,  0.0)  //Left         8
float DL_GetEdgeWeight(in float2 screenPos)
  float Depth[9];
  float3 Normal[9];
  //Retrieve normal and depth data for all neighbors.
   for (int i=0; i<9; ++i)
    float2 uv = screenPos + offsets[i] * pixel_size;
    Depth[i] = DL_GetDepth(uv);  //Retrieves depth from MRTs
    Normal[i]= DL_GetNormal(uv); //Retrieves normal from MRTs
  //Compute Deltas in Depth.
   float4 Deltas1;
  float4 Deltas2;
  Deltas1.x = Depth[1];
  Deltas1.y = Depth[2];
  Deltas1.z = Depth[3];
  Deltas1.w = Depth[4];
  Deltas2.x = Depth[5];
  Deltas2.y = Depth[6];
  Deltas2.z = Depth[7];
  Deltas2.w = Depth[8];
  //Compute absolute gradients from center.
  Deltas1 = abs(Deltas1 - Depth[0]);
  Deltas2 = abs(Depth[0] - Deltas2);
  //Find min and max gradient, ensuring min != 0
   float4 maxDeltas = max(Deltas1, Deltas2);
  float4 minDeltas = max(min(Deltas1, Deltas2), 0.00001);
  // Compare change in gradients, flagging ones that change
   // significantly.
   // How severe the change must be to get flagged is a function of the
   // minimum gradient. It is not resolution dependent. The constant
   // number here would change based on how the depth values are stored
   // and how sensitive the edge detection should be.
   float4 depthResults = step(minDeltas * depthSensibility, maxDeltas);
  //Compute change in the cosine of the angle between normals.
  Deltas1.x = dot(Normal[1], Normal[0]);
  Deltas1.y = dot(Normal[2], Normal[0]);
  Deltas1.z = dot(Normal[3], Normal[0]);
  Deltas1.w = dot(Normal[4], Normal[0]);
  Deltas2.x = dot(Normal[5], Normal[0]);
  Deltas2.y = dot(Normal[6], Normal[0]);
  Deltas2.z = dot(Normal[7], Normal[0]);
  Deltas2.w = dot(Normal[8], Normal[0]);
  Deltas1 = abs(Deltas1 - Deltas2);
  // Compare change in the cosine of the angles, flagging changes
   // above some constant threshold. The cosine of the angle is not a
   // linear function of the angle, so to have the flagging be
   // independent of the angles involved, an arccos function would be
   // required.
  float4 normalResults = step(0.4, Deltas1* normalSensibility);  

  normalResults = max(normalResults, depthResults);
  return (normalResults.x + normalResults.y +
          normalResults.z + normalResults.w) * 0.25;

float4 PShader(float2 texCoord : TEXCOORD0) : COLOR0

 float4 tex = tex2D(NormalMap,texCoord);
 float factor = 0.0f;

 for( int i=0;i<4;i++ )
	 float4 t = tex2D(NormalMap,texCoord+ delta[i]*pixel_size);
	 t -= tex;
	 factor += dot(t,t);

 factor = min(1.0,DL_GetEdgeWeight(texCoord))*weight; 

 float4 color = float4(0.0,0.0,0.0,0.0);

 for( int j=0;j<8;j++ )
	color += tex2D(TextureSampler,texCoord + delta[j]*pixel_size*factor);
 color += 2.0*tex2D(TextureSampler,texCoord);
 return color*(1.0/10.0);


The Vertex Shader and the buffers used are the same from the Stalker idea.

The first one is more beautiful but is slower.

The PloobsEngine uses these AA shaders (and some others also) to perform an Antialiasing Pass.

This is the first code snippet, when possible i will post more =P


, , , , ,

  1. #1 by Online College Degrees on 16 de janeiro de 2017 - 4:13 pm

    I think other site proprietors should take this site as an model, very clean and great user friendly style and design, let alone the content. You are an expert in this topic!

  2. #2 by sobrevivendo a menopausa on 16 de janeiro de 2017 - 4:14 pm

    Em outras palavras, a menopausa nãߋ teria impacto
    ѕobre а sаúdᥱ psicológica Ԁa mulher.

  3. #3 by Online Colleges on 16 de janeiro de 2017 - 4:15 pm

    Hello very nice web site!! Guy .. Beautiful .. Amazing .. I’ll bookmark your web site and take the feeds additionally…I am satisfied to seek out numerous helpful info right here in the put up, we’d like work out extra strategies in this regard, thanks for sharing. . . . . .

  4. #4 by armani exchange outlet store on 16 de janeiro de 2017 - 4:34 pm

    Hi! Nice post. I learn something more challenging on different blogs everyday. It will always be stimulating to read content from other writers and practice a little something from their store. I’d prefer to use some with the content on my blog whether you don’t mind. Natually I’ll give you a link on your web blog. Thanks for sharing.
    armani exchange outlet store

  5. #5 by geico car insurance quote on 16 de janeiro de 2017 - 4:50 pm

    Great wordpress blog here.. It’s hard to find quality writing like yours these days. I really appreciate people like you! take care

  6. #6 by Online Graduate Schools on 16 de janeiro de 2017 - 4:50 pm

    What i don’t understood is in reality how you’re not really a lot more neatly-favored than you might be right now. You’re so intelligent. You realize thus significantly relating to this topic, made me personally believe it from a lot of various angles. Its like men and women aren’t fascinated except it is one thing to do with Lady gaga! Your personal stuffs great. Always take care of it up!

  7. #7 by iphone screen on 16 de janeiro de 2017 - 4:57 pm

    It as very easy to find out any topic on web as compared to textbooks, as I found this piece of writing at this website.

  8. #8 by Car Insurance Quotes Online on 16 de janeiro de 2017 - 5:23 pm

    How can i transfer my Blogger give food to readers to my WordPress blog?

  9. #9 by arcteryx alpha sv sale on 16 de janeiro de 2017 - 5:41 pm

    *Would you be interested in exchanging links?
    arcteryx alpha sv sale

  10. #10 by bird lovers nest on 16 de janeiro de 2017 - 5:49 pm

    Thank you for your blog article.Thanks Again. Great.

  11. #11 by arcteryx sale vancouver on 16 de janeiro de 2017 - 7:36 pm

    Im curious if you ever have problems with what people post? Recently it seems to have become an epidemic, although it seems to be changing for the better. What are your thoughts?
    arcteryx sale vancouver

  12. #12 by houzz house plans on 16 de janeiro de 2017 - 7:39 pm

    Im grateful for the article.Thanks Again. Fantastic.

  13. #13 by coach factory outlet online store sale on 16 de janeiro de 2017 - 8:42 pm

    I feel this condition might be treatable.
    coach factory outlet online store sale

  14. #14 by imitation pandora jewellery on 16 de janeiro de 2017 - 9:08 pm

    Very good post.Really looking forward to read more.

  15. #15 by Anal Vibrator on 16 de janeiro de 2017 - 9:12 pm

    usually posts some incredibly intriguing stuff like this. If you are new to this site

  16. #16 by insurance ratings on 16 de janeiro de 2017 - 9:12 pm

    Oh my goodness! a tremendous article dude. Thank you Nevertheless I’m experiencing issue with ur rss . Don’t know why Unable to subscribe to it. Is there anybody getting equivalent rss problem? Anyone who knows kindly respond. Thnkx

  17. #17 by amaOn on 16 de janeiro de 2017 - 9:19 pm

    Non riesco a risolvere.

  18. #18 by Vay Nhanh 60s on 16 de janeiro de 2017 - 9:21 pm

    Thanks for helping out, great information. аЂа‹аЂ The four stages of man are infancy, childhood, adolescence, and obsolescence.аЂ аЂа› by Bruce Barton.

  19. #19 by home designers in hattiesburg on 16 de janeiro de 2017 - 9:31 pm

    This is one awesome article post.Much thanks again. Really Cool.

  20. #20 by karen millen outlet store on 16 de janeiro de 2017 - 9:40 pm

    Hello. fantastic job. I did not anticipate this. This is a remarkable story. Thanks!
    karen millen outlet store

  21. #21 by patagonia jacket sale on 16 de janeiro de 2017 - 10:36 pm

    Wonderful solutions.Id like to suggest taking a look at such as something like cheeseburger. What do you think?
    patagonia jacket sale

  22. #22 by house plan gallery yelp on 16 de janeiro de 2017 - 11:22 pm

    Thanks-a-mundo for the blog post.Thanks Again. Much obliged.

  23. #23 by johnyOn on 16 de janeiro de 2017 - 11:24 pm

    vous topic lisiez ?

  24. #24 by skype ip finder on 16 de janeiro de 2017 - 11:37 pm

    This is a topic close to my heart cheers, where are your contact details though?

  25. #25 by balmain jacket men on 16 de janeiro de 2017 - 11:43 pm

    Excellent information, excellent articles, will add your weblog to my weblog for my visitors to view!
    balmain jacket men

  26. #26 by Car Insurance Quotes Online on 16 de janeiro de 2017 - 11:55 pm

    I actually is beginning a Virtual assistant business and would like to start building a website that can broaden with me. I actually is on a very limited income so I have to start this for next to necessary upfront. Please help..

  27. #27 by カルティエ 結婚指輪 アフターケア 評判 on 17 de janeiro de 2017 - 12:37 am

    ブランド財布 グッチ バッグ 腕時計
    ★ S/SS品質 シリアル付きも有り 付属品完備!
    高品質 安心 最低価格保証
    経営方針: 品質を重視、納期も厳守、信用第一は当社の方針です。
    ★ 税関の没収する商品は再度無料にして発送します
    ◆以上 宜しくお願い致します。(^0^)
    カルティエ 結婚指輪 アフターケア 評判

  28. #28 by for details on 17 de janeiro de 2017 - 1:50 am

    Thanks for another wonderful article. Where else could anyone get that type of information in such a perfect way of writing? I ave a presentation next week, and I am on the look for such information.

  29. #29 by free download for pc on 17 de janeiro de 2017 - 1:58 am

    Hi this is kind of of off topic but I was wanting to know if blogs use WYSIWYG editors or if you have to manually code with HTML. I’m starting a blog soon but have no coding know-how so I wanted to get guidance from someone with experience. Any help would be enormously appreciated!

  30. #30 by android games for pc download on 17 de janeiro de 2017 - 1:59 am

    Whats up very nice web site!! Guy .. Excellent .. Amazing .. I’ll bookmark your website and take the feeds additionally?KI am glad to search out a lot of helpful information right here within the publish, we’d like develop more strategies on this regard, thank you for sharing. . . . . .

  31. #31 by stuart weitzman booties sale on 17 de janeiro de 2017 - 2:34 am

    I am going to go ahead and bookmark this content for my sis for a coming up study project for school. This is a beautiful web site by the way. Where did you get a hold the theme for this website?
    stuart weitzman booties sale

  32. #32 by jewelers in hamden selling pandora bracelets on 17 de janeiro de 2017 - 2:40 am

    Very informative article.Really looking forward to read more. Fantastic.

  33. #33 by patagonia fleece sale on 17 de janeiro de 2017 - 2:42 am

    Very good write-up,reasonable advice. With thanks a lot =)
    patagonia fleece sale

  34. #34 by louis vuitton handbags online on 17 de janeiro de 2017 - 2:44 am

    advice. I appreciate you taking time to share such valuable information. I had no clue on some of the things you mentioned earlier, thanks!
    louis vuitton handbags online

  35. #35 by free download for pc on 17 de janeiro de 2017 - 2:45 am

    Hi there, simply became alert to your weblog thru Google, and located that it is truly informative. I am going to be careful for brussels. I will appreciate when you proceed this in future. Numerous folks shall be benefited out of your writing. Cheers!

  36. #36 by pediatric dentists in baton rouge on 17 de janeiro de 2017 - 3:07 am

    Major thanks for the article.Really thank you! Fantastic.

  37. #37 by Victorian front doors in north London on 17 de janeiro de 2017 - 4:02 am

    This website was how do you say it? Relevant!! Finally I ave found something which helped me. Thanks a lot!

  38. #39 by coach factory store online on 17 de janeiro de 2017 - 4:40 am

    I had this website saved some time in the past but my computer crashed. I have since gotten a new one and it took me a while to find this! I also really like the theme though.
    coach factory store online

1 346 347 348
(não será publicado)