Tight Frame Normal Map Compression - Graphics Hardware

Tight Frame Normal MapCompressionJacob Munkberg Ola Olsson Tomas Akenine-MöllerJacob StrömLund UniversityEricsson Research

Overview• Introduction to normal mapping• Previous work• Tight Frame Compression• Evaluation

Normal Maps• Add geometric detail with texture maps• Store value of the local normal vector• Realistic, detailed appearance at low cost+ =

Normal Map Generation• Shoot rays• from the lo-ressurfaceto the hi-ressurface

Normal Map Generation• Store the normal vector from theintersection points in an RGB texture

Normal Map Generation• Render lo res surface + normal map20k triangles2 triangles + normal map

Motivation• We need compression!Increased texturebandwidth!!!20k triangles2 triangles + normalmap

Overview• Introduction to normal mapping• Previous work• Tight Frame Compression• Evaluation

Previous Work• Surface normal compression [Deering 95]• S3 Texture Compression/DXTC [Iourcha99]• 3Dc [ATI05]• Dedicated format for normal maps• e3Dc [Munkberg 06]• Enhanced 3Dc with rotations and diff-coding• Adaptive bit rate [Wong06,Yang06]• Vector Quantization [Yamakasi et. al 06]

Design choices• Fixed rate encoding at 8 bpp• Fast random access• Simple decompressor• 4x4 texel blocks• Use advantages from e3Dc• Rotation encoding• Differential encoding• Variable point distribution• Exploit coherence between channels

3Dc Overview• Divide the input file in 4x4 blocks of texels

3Dc - Projection• Project the normals on the xy plane andfind min/max values of the bounding box•Zn=(x,y,z)xminYxmaxYymax(x,y)XXymin

3Dc - Texel Quantization• Map each texel to a quantized (x,y) value• Eight levels in x & y; (3,3) bits to select (xi,yi)xminxmaxymax3bymin3b

3Dc - Compressed Block• Compressed form• 4x8 bits for xmin, xmax, ymin, ymax• 6x16 bits for per texel index• Total: 128 bits per block : 8 bits per texel6b 6b 6b 6b6b 6b 6b 6b6b 6b 6b 6b6b 6b 6b 6bxmin8bymin8bxmax8bymax8b

Problems with 3Dc• Difficult scenarios• Slow gradients, sharp edges, directed features3DcOriginal

3Dc can be improved - e3Dc• Observation (used in DXT1)• Swap min & max values→ same reconstruction levels• One bit unused per channel!• Use these to signal new modes!xminxmax’xmaxxmin’X Y modexmin < xmax ymin < ymax Standard 3Dcxmin ≥ xmax ymin < ymax Rotation 30xmin < xmax ymin ≥ ymax Rotation 60xmin ≥ xmax ymin ≥ ymax Differential mode

Rotation Compression• Rotate coordinate frame for amore compact bounding box• e3Dc uses three angles: 0, 30 and 60 degrees1100 100 1

Variable Point Distribution• 3Dc : points in a 8x8 grid• Our approach : use aspect ratio of bbox• BBox twice as wide -> 16x4 instead of 8x8• Automatic selection -> No extra cost2 x 32 4 x 16 8 x 8 16 x 4 32 x 2

Variable Point Distribution3DcVariable Point Distribution in e3Dc

Differential Encoding• Slowly varying normals are problematic:• Smallest interval is too wide (range/255)xminxmax• The interval cannot be placed accurately enoughxminxmax

Differential Encoding• Slowly varying normals are problematic:• Smallest interval is too wide (range/255)xminxmin*xmin*+∆xxmax• The interval cannot be placed accurately enoughxminxmin*xmin*+∆xxmaxReinterpret the bits differentially!(xmin,xmax) → (xmin*, ∆x)

Overview• Introduction to normal mapping• Previous work• Tight Frame Compression• Evaluation

Tight Frame Compression• Example: directed lines• coherence between channels!

Tight Frame Encoding3Dce3DcTight FramePSNR5040302010015 30 45 75

c○ Association for Computing M· n c ), (3)Tight Frame Compressiongle between the uncomessedone (n c ).color images of the errenderimages with enpedsurfaces. For these,ilarity [WBSS04] quality• OBB is tighter than AABB• Store two point p & q• Enough to define localcoordinate frame e1, e2rithm, as it is easier to find a compact OBB tAABB (3Dc), or fix-rotation AABB (e3DC)for an illustration of the benefits of VPD.The flexibility of the OBB combined withtion of sample points (VPD) makes for a simerful algorithm which gives high quality comng example. • arbitrary Imagine rotation wep1, consisting mainly ofheightwidth• Store box aspect ratiosaligned, 3Dc will handlet axis-aligned • aspect bounding ratio = height/widthFigure 2: The coordinate system of our tight frtails. • If Variable the lines are Point rowillbe more spreadDistribution ing algorithm. stillout.works!e₂ ^e₁^q

ors in the normals maps, and also render images with environmentmapped and bump mapped surfaces. For these,we will compute the structural similarity [WBSS04] qualitymeasure.Tight Frame Bit Layout4. New AlgorithmLet us start with a simple motivating example. Imagine wehave a normal map, as in Figure 1, consisting mainly ofparallel lines. If the lines are axis-aligned, 3Dc will handlethis example pretty well, as a tight axis-aligned bounding• Target: box (AABB) would 128 capture bits the details. per If theblock lines are ro-- 8 bpptated, however, the projected values will be more spread out.• Indices with 6 x 16 bits as before• 32 bits left for encoding OBB• p & q are encoded using 7+7 bits each• Four bits for the aspect ratio, aerful algorithm which gives high quality comp• sixteen levels as ai = 1/32 + hi/16, i = 0...15Ex:e₂ ^heightpse₁^qwidthFigure 2: The coordinate system of our tight fring algorithm.c○ Association for Computing Ma1 a7 a15

Additional Mode• Unused bitcombinations?• px>=qx & py>=qy• Use to flag amode!d bp qpp qq

Tight Frame Differential Mode• Trigger when px >= qx and py >= qy• Encode all normals in a small square• Limit the square side length• Compact representation but highresolution!• Lower left corner 2x11 bits• Side of the square: 8 bits• 8x8 grid over the square• 6x16 + 2x11 + 8 = 126 bitsside

ely small compared to that of the specna small angular error in a normal mayt texture access in the environment map.ortant to look at the direct angle differmpressedand original normal, as well asped images with environment mapping.Tight Frame Differential Modeg the angular deviation [ANRS05], deas:ad = arccos(n o · n c ), (3)• Differential mode resolution• corner 2x11 bits, side: 8 bits,difference in angle between the uncom-) and the compressed one (n c ).• Limit max square side to 1/4• min square size = 1/(4*2 8 )= 1/1024will show false color images of the ermaps,and also render images with enandbump mapped surfaces. For these,e structural similarity [WBSS04] quality• Standard mode resolution• p and q with (2x7) bits each• max size = 1simple motivating example. Imagine wep, as in Figure 1, consisting mainly ofe lines are axis-aligned, 3Dc will handlewell, as a tight axis-aligned bounding• min square size = 1/128be noted that a similar setup has been discutexture compression [MCHAM06]. Once wethis oriented bounding box (OBB), we distribformly in the box, using the aspect ratio to seber of divisions along the two axes. For excase of a very wide OBB, it makes more senspoints along the widest axis. This variable ption (VPD) [MAMS06] becomes more powerfrithm, as it is easier to find a compact OBB thAABB (3Dc), or fix-rotation side AABB (e3DC).for an illustration of the benefits of VPD.The flexibility of the OBB combined withtion of sample points (VPD) makes for a simerful algorithm which gives high quality come₂ ^heightpse₁^qwidth

Overview• Introduction to normal mapping• Previous work• Tight Frame Compression• Evaluation

Evaluation• PSNR• Max error• A few bad normals “cracks” a smooth surface• Angular deviation [Abate 05]• Angle between compressed and original normal• Motivation: Even a small error in the specularreflection is visible• Presented as a histogram

Test Images

deviation for the test suite. As can be seen, our alslightlybetter scores than e3Dc for the majorityal maps, PSNR and significantly better scores than 3Dc. For the “bumpy”-map, e3Dc is better due to thatJ. Munkberg, O. Olsson, J. Ström, & T. Akenine-Möller / Tightm uses 7+7 bits for the endpoints, while e3Dcrther, as all normals 80 in that image are essentially10 6izontal line, there is no gain from being able to3Dce3Dcoxes. In the taght,we present 30.87 32.74 33.5010 570 3Dc e3Dc TFTFes obtained by60ing the MSE values for all the normal maps.en computed on this accumulated MSE usingPSNR50Note that it is not correct to simply average thes of the individual images, since this is not a lin-. In the extreme — 40 if one image would get zerould get infinite PSNR and the aggregate PSNRd also be infinite, 30irrespectively of the errors inages. Averaging the MSE and then calculatingvoids this pitfall.20As can be seen, our algorithmores than both 3Dc and e3Dc.bumpycardot1mum angle error (bottom part of Figure 4) indiuralgorithm is more 25 robust than the other algo-dot2dot3dot4lumpymetalNormalMaponetileturtlevoronoislowMapbulge10multiBulgestarboxestorusskinbarrelNumber of pixels (log scale)10 410 310 20Figur

4030Max angular deviationMAX ANGULAR DEVIATION20bumpy2520151050cardot1dot2dot3dot4lumpymetalNormalMaponetileturtlevoronoislowMapbulge10multiBulgestarboxestorusskin3Dce3DcTFbarrelm9[AT2[BlP[Dec[FBtP

ne-Möller Error / Tight distributionFrame Normal Map Compression10 6 Angle error in degrees3Dce3DcTFNumber of pixels (log scale)10 510 410 310 20 2 4 6 8 10

False Color3Dc e3Dc Tight Frame

Original100 %3Dc93.3e3Dc95.7Tight Frame96.3%RenderedQualitySSIM:structural similarityimage metric

Pros & Cons• Increased hardware cost• Twice the number of “multiply and divide” unitscompared to e3Dc, but still lightweight• Unlike e3Dc, there is no backwardcompatibilitywith 3Dc• The format cannot be used for encoding two 1Dsignals• Not depending on 3Dc patent• More robust results!

Conclusions• Higher quality than 3Dc• Still at 8 bits per texels• More flexibility with OBB, VPD and diff-mode• Rather modest HW extensions• API support?• Potential candidate for OpenGL ES?

Thank you!• http://graphics.cs.lth.se