Code Snippet: Edge Detector/Antialiasing Shader

Some days ago i was looking for a simple and effective edge detector idea to use in an AntiAliasing shader for Deferred Shading (ideas that effectively make use of the Normal and the Depth Buffers).

Two called my attention: The first one is from the game Stalker, this page from GPU Gems explains the whole idea (nothing very complitaded =P, they show lot of the code). The results are pretty cool, as you can see in this image for example. The second is the one used in Tabula Rasa game, this page from GPU Gems shows the whole idea.

Stalker Idea

The shader code in HLSL is: (added the full version. for those interested only in the edge detection part, see the shader comments).

void PassThroughVS(inout float4 position : POSITION0,
                   inout float2 texcoord : TEXCOORD0) {

				   position.x =  position.x - pixel_size;
				   position.y =  position.y + pixel_size;

float2 e_barrier = float2(0.8f,0.00001f);  // x=norm, y=depth

float2 e_weights= float2(1,1);  // x=norm, y=depth

float2 e_kernel = float2(1,1);   // x=norm, y=depth

const static float2 offs[7] = {
   float2( 0.0,  0.0), //Center       0
   float2(-1.0, -1.0), //Top Left     1
   float2( 1.0,  1.0), //Bottom Right 5
   float2( 1.0, -1.0), //Top Right    3
   float2(-1.0,  1.0), //Bottom Left  7
   float2(-1.0,  0.0),  //Left         8
   float2( 0.0, -1.0) //Top          2

float4 PShader2(float2 uv : TEXCOORD0) : COLOR0


 // Normal discontinuity filter

 float3 nc = tex2D(NormalMap, uv);

 float4 nd;

 nd.x = abs(dot(nc, tex2D(NormalMap, uv + offs[1]* pixel_size).xyz));

 nd.y = abs(dot(nc, tex2D(NormalMap, uv + offs[2]* pixel_size).xyz));

 nd.z = abs(dot(nc, tex2D(NormalMap, uv + offs[3]* pixel_size).xyz));

 nd.w = abs(dot(nc, tex2D(NormalMap, uv + offs[4]* pixel_size).xyz));

 nd -= e_barrier.x;

 nd = step(0, nd);

 float ne = saturate(dot(nd, e_weights.x));

 // Opposite coords

 float2 tc5r = -offs[5];

 float2 tc6r = -offs[6];

 // Depth filter : compute gradiental difference:

 // (c-sample1)+(c-sample1_opposite)

 float dc = tex2D(depthMap, uv).r;

 float4 dd;

 dd.x = tex2D(depthMap, uv + offs[1] * pixel_size).r +

         tex2D(depthMap, uv + offs[2]* pixel_size).r;

 dd.y = tex2D(depthMap, uv + offs[3]* pixel_size).r +

        tex2D(depthMap, uv + offs[4]* pixel_size).r;

 dd.z = tex2D(depthMap, uv + offs[5]* pixel_size).r +

        tex2D(depthMap, uv + tc5r* pixel_size).r;

 dd.w = tex2D(depthMap,uv +  offs[6]* pixel_size).r +

        tex2D(depthMap,uv +  tc6r* pixel_size).r;

 dd = abs(2 * dc - dd)- e_barrier.y;

 dd = step(dd, 0);

 float de = saturate(dot(dd, e_weights.y));

 // Weight

 float w = (1 - de * ne) * e_kernel.x; // 0 - no aa, 1=full aa

 //return float4(w,w,w,1); ///stop here for an edge detector shader

// The AA part (in edges we mix close pixels, non edge pixel w = 0 => use the normal uv, on edge we use a mean of four deslocated uv texture reads )
// Smoothed color

 // (a-c)*w + c = a*w + c(1-w)

 float2 offset = (uv ) * (1-w);

 float4 s0 = tex2D(TextureSampler, offset + (uv + offs[1] * pixel_size) * w);

 float4 s1 = tex2D(TextureSampler, offset + (uv + offs[2]* pixel_size) * w);

 float4 s2 = tex2D(TextureSampler, offset + (uv + offs[3]* pixel_size) * w);

 float4 s3 = tex2D(TextureSampler, offset + (uv + offs[4]* pixel_size) * w);

 return (s0 + s1 + s2 + s3)/4.h;


technique AntiAliasingStalker
    pass Pass1
		VertexShader = compile vs_3_0 PassThroughVS();
                PixelShader = compile ps_3_0 PShader2();

The normal buffer is stored as a 32 bits RGBA texture. The Depth Buffer is stored in a 32 bits Single Texture (storing post-projection z/w). If you use another depth enconding, just adjust the barrier variable value.

The pixel_size variable (a float2 with 1/target width in x and 1/target height in y) is used in DirectX 9 Shaders (Opengl and DirectX > 9 does not need this). The need for this “correction” is very well explained here.

The ScreenShot below is just the edge detector part of the shader. (just uncomment the following line in the shader and you will have a powerfull edge detector shader =P)

Example using this good edge detector shader. Cool for toon shading =P


Idea From Tabula Rasa

This is not so good as the one before (dont looks good when the camera is far from the objects or when the edges has little pixels), but is a bit simpler. Fort those interested  only in the edge detection part,

just return the “factor” variable in the pixel shader, instead of using it to blend the near pixels =P

const float2 delta[8] =

 ///silly, dummy functions to recover normal/depth (i was doing something with the read values before and i did not erased the function ...)
 float4 DL_GetDepth(float2 uv)
	return tex2D(NormalMap  ,uv);
 float4 DL_GetNormal(float2 uv)
	return tex2D(depthMap ,uv);

float depthSensibility;
float normalSensibility;
   // Neighbor offset table
   const static float2 offsets[9] = {
  float2( 0.0,  0.0), //Center       0
   float2(-1.0, -1.0), //Top Left     1
   float2( 0.0, -1.0), //Top          2
   float2( 1.0, -1.0), //Top Right    3
   float2( 1.0,  0.0), //Right        4
   float2( 1.0,  1.0), //Bottom Right 5
   float2( 0.0,  1.0), //Bottom       6
   float2(-1.0,  1.0), //Bottom Left  7
   float2(-1.0,  0.0)  //Left         8
float DL_GetEdgeWeight(in float2 screenPos)
  float Depth[9];
  float3 Normal[9];
  //Retrieve normal and depth data for all neighbors.
   for (int i=0; i<9; ++i)
    float2 uv = screenPos + offsets[i] * pixel_size;
    Depth[i] = DL_GetDepth(uv);  //Retrieves depth from MRTs
    Normal[i]= DL_GetNormal(uv); //Retrieves normal from MRTs
  //Compute Deltas in Depth.
   float4 Deltas1;
  float4 Deltas2;
  Deltas1.x = Depth[1];
  Deltas1.y = Depth[2];
  Deltas1.z = Depth[3];
  Deltas1.w = Depth[4];
  Deltas2.x = Depth[5];
  Deltas2.y = Depth[6];
  Deltas2.z = Depth[7];
  Deltas2.w = Depth[8];
  //Compute absolute gradients from center.
  Deltas1 = abs(Deltas1 - Depth[0]);
  Deltas2 = abs(Depth[0] - Deltas2);
  //Find min and max gradient, ensuring min != 0
   float4 maxDeltas = max(Deltas1, Deltas2);
  float4 minDeltas = max(min(Deltas1, Deltas2), 0.00001);
  // Compare change in gradients, flagging ones that change
   // significantly.
   // How severe the change must be to get flagged is a function of the
   // minimum gradient. It is not resolution dependent. The constant
   // number here would change based on how the depth values are stored
   // and how sensitive the edge detection should be.
   float4 depthResults = step(minDeltas * depthSensibility, maxDeltas);
  //Compute change in the cosine of the angle between normals.
  Deltas1.x = dot(Normal[1], Normal[0]);
  Deltas1.y = dot(Normal[2], Normal[0]);
  Deltas1.z = dot(Normal[3], Normal[0]);
  Deltas1.w = dot(Normal[4], Normal[0]);
  Deltas2.x = dot(Normal[5], Normal[0]);
  Deltas2.y = dot(Normal[6], Normal[0]);
  Deltas2.z = dot(Normal[7], Normal[0]);
  Deltas2.w = dot(Normal[8], Normal[0]);
  Deltas1 = abs(Deltas1 - Deltas2);
  // Compare change in the cosine of the angles, flagging changes
   // above some constant threshold. The cosine of the angle is not a
   // linear function of the angle, so to have the flagging be
   // independent of the angles involved, an arccos function would be
   // required.
  float4 normalResults = step(0.4, Deltas1* normalSensibility);  

  normalResults = max(normalResults, depthResults);
  return (normalResults.x + normalResults.y +
          normalResults.z + normalResults.w) * 0.25;

float4 PShader(float2 texCoord : TEXCOORD0) : COLOR0

 float4 tex = tex2D(NormalMap,texCoord);
 float factor = 0.0f;

 for( int i=0;i<4;i++ )
	 float4 t = tex2D(NormalMap,texCoord+ delta[i]*pixel_size);
	 t -= tex;
	 factor += dot(t,t);

 factor = min(1.0,DL_GetEdgeWeight(texCoord))*weight; 

 float4 color = float4(0.0,0.0,0.0,0.0);

 for( int j=0;j<8;j++ )
	color += tex2D(TextureSampler,texCoord + delta[j]*pixel_size*factor);
 color += 2.0*tex2D(TextureSampler,texCoord);
 return color*(1.0/10.0);


The Vertex Shader and the buffers used are the same from the Stalker idea.

The first one is more beautiful but is slower.

The PloobsEngine uses these AA shaders (and some others also) to perform an Antialiasing Pass.

This is the first code snippet, when possible i will post more =P


, , , , ,

  1. #1 by dog grooming malahide on 20 de outubro de 2016 - 3:43 am

    Hi, Neat post. There is a problem with your site in internet explorer, would check this… IE still is the market leader and a huge portion of people will miss your wonderful writing because of this problem.

  2. #2 by waggles dog grooming dublin on 20 de outubro de 2016 - 3:43 am

    Fantastic beat ! I wish to apprentice at the same time as you amend your website, how could i subscribe for a blog site? The account aided me a applicable deal. I were a little bit familiar of this your broadcast offered bright transparent idea

  3. #3 by dog grooming supplies ireland on 20 de outubro de 2016 - 4:05 am

    Yeah bookmaking this wasn’t a risky determination great post! .

  4. #4 by プーマ ランニング レディース on 20 de outubro de 2016 - 4:20 am


  5. #5 by financial planning on 20 de outubro de 2016 - 4:38 am

    Really enjoyed this post.Much thanks again. Want more.

  6. #6 by call girls in new delhi on 20 de outubro de 2016 - 6:11 am

    Hello there! This is my first visit to your blog! We are a group of volunteers and starting a new initiative in a community in the same niche. Your blog provided us valuable information to work on. You have done a marvellous job!

  7. #7 by BusinessMap on 20 de outubro de 2016 - 6:22 am

    This can work well in the sense that they have worked together

  8. #8 by marketing based on interest targeting on 20 de outubro de 2016 - 6:28 am

    Very neat article post.Much thanks again. Keep writing.

  9. #9 by professional translation company on 20 de outubro de 2016 - 8:10 am

    Great blog post.Really thank you! Really Cool.

  10. #10 by dog groomers in tallaght on 20 de outubro de 2016 - 8:37 am

    But wanna remark on few general things, The website design and style is perfect, the written content is real wonderful. “The enemy is anybody who’s going to get you killed, no matter which side he’s on.” by Joseph Heller.

  11. #11 by tallaght dog groomers on 20 de outubro de 2016 - 8:37 am

    You could certainly see your enthusiasm in the work you write. The world hopes for more passionate writers like you who aren’t afraid to say how they believe. Always go after your heart.

  12. #12 by Dog grooming tallaght on 20 de outubro de 2016 - 9:00 am

    Usually I don’t read article on blogs, but I wish to say that this write-up very forced me to try and do it! Your writing style has been surprised me. Thanks, very nice article.

  13. #13 by aromatherapy certification online class on 20 de outubro de 2016 - 9:05 am

    What as Happening i am new to this, I stumbled upon this I have found It absolutely helpful and it has aided me out loads. I hope to contribute & assist other users like its helped me. Good job.

  14. #14 by Oxboard Kopen Goedkoop on 20 de outubro de 2016 - 9:10 am

    Backyard, haul items as well as see other exercises that produce your chest implement far more function than usual.

  15. #16 by profile on 20 de outubro de 2016 - 10:08 am

    I cannot thank you enough for the blog post.Really thank you! Great.

  16. #17 by on 20 de outubro de 2016 - 10:32 am

    Excellent post. I was checking continuously this weblog and I am impressed! Extremely useful information particularly the final section :) I care for such info much. I was seeking this particular information for a very lengthy time. Thank you and good luck.

  17. #18 by Search Engine Optimization on 20 de outubro de 2016 - 10:50 am

    Wow, awesome blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your web site is great, as well as the content!. Thanks For Your article about sex.

  18. #19 by messekonzept erstellen muster on 20 de outubro de 2016 - 12:37 pm

    Really enjoyed this blog post.Much thanks again. Awesome.

  19. #20 by szalvétatechnika on 20 de outubro de 2016 - 1:19 pm

    Hi would you mind letting me know which web host you’re utilizing? I’ve loaded your blog in 3 different browsers and I must say this blog loads a lot faster then most. Can you suggest a good web hosting provider at a reasonable price? Many thanks, I appreciate it!

  20. #21 by dekor on 20 de outubro de 2016 - 1:19 pm

    Whats Going down i am new to this, I stumbled upon this I have found It positively helpful and it has aided me out loads. I’m hoping to give a contribution & aid different customers like its helped me. Good job.

  21. #23 by certificazioni ambientali on 20 de outubro de 2016 - 2:25 pm

    This site can be a stroll-by means of for all the information you needed about this and didn?t know who to ask. Glimpse right here, and also you?ll undoubtedly uncover it.

  22. #24 by kreatív on 20 de outubro de 2016 - 2:59 pm

    I am not sure where you’re getting your info, but good topic. I needs to spend some time learning much more or understanding more. Thanks for excellent information I was looking for this information for my mission.

  23. #25 by on 20 de outubro de 2016 - 4:12 pm

    It as difficult to find educated people in this particular subject, but you seem like you know what you are talking about! Thanks

  24. #26 by (Prescott video marketing|Prescott AZ marketing channel} on 20 de outubro de 2016 - 5:08 pm

    Howdy! This is my 1st comment here so I just wanted to give a quick shout out and tell you I truly enjoy reading your articles. Can you recommend any other blogs/websites/forums that deal with the same subjects? Thanks a lot!

  25. #27 by (Prescott video marketing|Prescott AZ marketing channel} on 20 de outubro de 2016 - 5:08 pm

    I have to express some appreciation to the writer just for rescuing me from this particular issue. Right after surfing around throughout the search engines and coming across basics which were not beneficial, I assumed my entire life was over. Being alive devoid of the answers to the problems you’ve fixed as a result of your article is a critical case, as well as ones that might have negatively affected my career if I had not noticed your website. That know-how and kindness in dealing with the whole thing was helpful. I don’t know what I would’ve done if I hadn’t come across such a stuff like this. It’s possible to at this point relish my future. Thanks a lot so much for the skilled and sensible guide. I will not hesitate to suggest the blog to any person who needs and wants assistance about this subject.

  26. #28 by (Prescott video marketing|Prescott AZ marketing channel} on 20 de outubro de 2016 - 5:30 pm

    Wow that was strange. I just wrote an very long comment but after I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again. Anyhow, just wanted to say great blog!

  27. #29 by apply pr in singapore on 20 de outubro de 2016 - 5:56 pm


  28. #30 by apply for pr on 20 de outubro de 2016 - 7:27 pm


  29. #31 by great game websites on 20 de outubro de 2016 - 9:55 pm

    Zune and iPod: Maximum These examine the Zune to the Touch, but soon after seeing how slender and surprisingly very low and light-weight it is, I choose it in the direction of be a fairly distinctive hybrid that combines attributes of either the Contact and the Nano. It is really extremely colourful and stunning OLED display screen is slightly lesser than the contact screen, but the player alone feels fairly a little bit scaled-down and lighter. It weighs in excess of 2/3 as much, and is appreciably smaller sized within just width and top, while being precisely a hair thicker.

  30. #32 by car cleaning on 20 de outubro de 2016 - 11:55 pm

    please stop by the web pages we adhere to, which includes this one, because it represents our picks from the web

  31. #33 by health and wellness articles from magazines on 21 de outubro de 2016 - 1:59 am

    If you are still on the fence: seize your beloved earphones, head down in direction of a Least difficult Buy and talk to toward plug them into a Zune then an iPod and check out which a single appears greater in direction of oneself, and which interface would make you smile further more. Then you may comprehend which is directly for on your own.

  32. #34 by games to play now on 21 de outubro de 2016 - 2:31 am

    Among me and my husband we have owned extra MP3 avid gamers over the years than I can depend, such as Sansas, iRivers, iPods (classic & touch), the Ibiza Rhapsody, and so forth. But, the very last couple several years I’ve settled down towards just one line of avid gamers. Why? Simply because I was delighted to obtain how well-designed and enjoyable in the direction of use the underappreciated (and broadly mocked) Zunes are.

  33. #36 by buisness articles on 21 de outubro de 2016 - 4:44 am

    Arms down, Apple’s app store wins by a mile. It is a significant option of all types of apps vs a in its place unhappy selection of a handful for Zune. Microsoft contains Options, especially within the realm of online games, nevertheless I’m not certainly I might have to have in direction of guess upon the future if this element is vital in direction of on your own. The iPod is a significantly far better conclusion inside that case.

  34. #37 by Travail à domicile Maroc on 21 de outubro de 2016 - 4:44 am

    A big thank you for your article post.Much thanks again. Great.

  35. #38 by Prescott AZ marketing channel on 21 de outubro de 2016 - 7:07 am

    Outstanding post, you have pointed out some superb points, I likewise believe this s a very great website.

  36. #39 by hoverboard for sale on 21 de outubro de 2016 - 7:11 am

    Likewise, in addition there are lots non-prescription remedies that could be u .

  37. #40 by Prescott AZ marketing channel on 21 de outubro de 2016 - 7:32 am

    Heya i am for the first time here. I found this board and I find It truly useful & it helped me out a lot. I hope to give something back and aid others like you helped me.

  38. #41 by hoverboard goedkoop on 21 de outubro de 2016 - 7:53 am

    Most of operators lead to preserving handle of its automobile and additionally getting around different cars or trucks whereas cruising.

  39. #42 by Prescott AZ marketing channel on 21 de outubro de 2016 - 10:01 am

    I precisely wanted to thank you very much yet again. I’m not certain the things I would have implemented in the absence of the actual secrets revealed by you concerning such a situation. Entirely was the hard setting for me personally, nevertheless viewing the very expert form you solved the issue forced me to leap with delight. I will be thankful for your guidance and thus sincerely hope you realize what a great job you have been carrying out educating many others through the use of a site. I am certain you have never met all of us.

  40. #43 by home loan AFFILIATE program on 21 de outubro de 2016 - 1:03 pm

    Together with every thing which seems to be developing throughout this particular area, your points of view are actually relatively radical. On the other hand, I appologize, because I can not give credence to your whole plan, all be it exciting none the less. It appears to me that your opinions are actually not entirely validated and in fact you are your self not really thoroughly confident of the assertion. In any event I did take pleasure in examining it.

  41. #44 by payday affiliate network on 21 de outubro de 2016 - 1:29 pm

    I’ve been absent for a while, but now I remember why I used to love this blog. Thanks, I?¦ll try and check back more frequently. How frequently you update your web site?

  42. #45 by payday loan referral program on 21 de outubro de 2016 - 2:09 pm

    This design is incredible! You obviously know how to keep a reader amused. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Great job. I really loved what you had to say, and more than that, how you presented it. Too cool!

  43. #46 by IP Lookup on 21 de outubro de 2016 - 2:48 pm

    Great, thanks for sharing this blog post.Really thank you! Awesome.

  44. #47 by zusätzliche Ressourcen on 21 de outubro de 2016 - 4:20 pm

    So können Sie vor allem Interessenten im näheren Umkreis bis zu 30 km erreichen.

  45. #48 by automotve news on 21 de outubro de 2016 - 4:50 pm

    Sorry for the massive evaluate, nonetheless I’m exceptionally loving the refreshing Zune, and anticipate this, as nicely as the good quality testimonials some other persons include composed, will help on your own decide if it really is the right alternative for oneself.

  46. #49 by apply permanent resident on 21 de outubro de 2016 - 5:02 pm

    Very nice post!!

  47. #50 by art work websites on 21 de outubro de 2016 - 5:07 pm

    Amongst me and my husband we’ve owned even more MP3 avid gamers more than the decades than I can depend, including Sansas, iRivers, iPods (classic & touch), the Ibiza Rhapsody, etcetera. But, the remaining handful of many years I’ve fixed down in the direction of one particular line of avid gamers. Why? Considering that I was happy toward come across how well-designed and entertaining to hire the underappreciated (and greatly mocked) Zunes are.

1 256 257 258
(não será publicado)