Compute Shader Example: CMAA2

This page walks through a complete multi-pass compute PostFX mod: a port of Intel’s CMAA2 (Conservative Morphological Anti-Aliasing 2) with an optional temporal pass and RCAS sharpening. It builds on the basics in Creating custom shader PostEffects and shows the patterns that a single-pass example cannot: chaining passes through named targets, keeping a history texture across frames, reading velocity and depth, switching passes on and off at runtime, and sharing one shader body between a compute and a pixel shader variant.

Mod layout

lua/ge/extensions/cmaa2.lua                       effect setup, UI, per-frame constants
scripts/cmaa2/modScript.lua                       loads the extension when the mod is mounted
shaders/common/postFx/cmaa2/cmaa2_common.h.hlsl   shared uniforms and helpers
shaders/common/postFx/cmaa2/cmaa2_edges.cs.hlsl   pass 1: edge detection
shaders/common/postFx/cmaa2/cmaa2_zshapes.cs.hlsl pass 2: shape detection
shaders/common/postFx/cmaa2/cmaa2_apply.cs.hlsl   pass 3: blend along the shapes
shaders/common/postFx/cmaa2/cmaa2_temporal.cs.hlsl optional temporal accumulation
shaders/common/postFx/cmaa2/cmaa2_rcas.h.hlsl     RCAS sharpening body
shaders/common/postFx/cmaa2/cmaa2_rcas.cs.hlsl    RCAS, compute variant
shaders/common/postFx/cmaa2/cmaa2_rcas.ps.hlsl    RCAS, pixel shader variant
shaders/common/postFx/cmaa2/cmaa2_easu.ps.hlsl    FSR 1 EASU upscale, pixel shader

modScript.lua keeps the extension loaded for the whole session:

setExtensionUnloadMode("cmaa2", "manual")
extensions.load("cmaa2")

The pass chain

The root sets the timing for the whole chain: PFXAfterBin on AfterPostFX, the same slot the shipped SMAA T2x resolve uses. At that point the image is tone mapped, and the overlay bins and motion blur have not run yet. Every other pass is added to the root as a child and runs in the order it was added.

Each intermediate result has its own named target, so no pass reads what it writes. The passes that write $backBuffer are safe as well: the engine writes them to the free half of the scene color ping-pong, so $backBuffer in texture is still the previous result. The small edge mask uses an 8-bit target instead of a full color format.

Registering the objects from Lua

The shaders, the state block, and the effects are described as tables and created in a loop. Files ending in .ps.hlsl become pixel shaders and everything else is compute:

local shaderDir = 'shaders/common/postFx/cmaa2/'

-- name, file, pixVersion
local shaderDefs = {
  { 'CMAA2_EdgesShaderData', 'cmaa2_edges.cs.hlsl', 5.0 },
  { 'CMAA2_ZShapesShaderData', 'cmaa2_zshapes.cs.hlsl', 5.0 },
  { 'CMAA2_ApplyShaderData', 'cmaa2_apply.cs.hlsl', 5.0 },
  { 'CMAA2_TemporalShaderData', 'cmaa2_temporal.cs.hlsl', 5.0 },
  { 'CMAA2_SharpenShaderData', 'cmaa2_rcas.cs.hlsl', 6.0 },
}

local effectDefs = {
  { name = 'CMAA2_PostEffect', shader = 'CMAA2_EdgesShaderData', renderTime = 'PFXAfterBin', renderBin = 'AfterPostFX',
    textures = { '$backBuffer' }, target = '#cmaa2Edges', format = 'GFXFormatR8' },
  { name = 'CMAA2_ZShapes', shader = 'CMAA2_ZShapesShaderData',
    textures = { '#cmaa2Edges' }, target = '#cmaa2ZShapes', format = 'GFXFormatR8G8B8A8' },
  { name = 'CMAA2_Apply', shader = 'CMAA2_ApplyShaderData',
    textures = { '$backBuffer', '#cmaa2Edges', '#cmaa2ZShapes' }, target = '$backBuffer' },
  { name = 'CMAA2_Temporal', shader = 'CMAA2_TemporalShaderData', stateBlock = 'CMAA2_TemporalStateBlock',
    textures = { '$backBuffer', '#cmaa2History', '#VelocityBuffer', '#prepass[Depth]' }, target = '$backBuffer', skip = true },
  { name = 'CMAA2_History', shader = 'PFX_PassthruComputeShader', renderTime = 'PFXTexGenOnDemand',
    textures = { '$backBuffer' }, target = '#cmaa2History', format = 'GFXFormatR16G16B16A16F', skip = true },
  { name = 'CMAA2_Sharpen', shader = 'CMAA2_SharpenShaderData', textures = { '$backBuffer' }, target = '$backBuffer', skip = true },
}

local function createObjects()
  for _, def in ipairs(shaderDefs) do
    local shader = createObject("ShaderData")
    local file = shaderDir .. def[2]
    if file:find('%.ps%.hlsl$') then
      shader.DXVertexShaderFile = file
      shader.DXPixelShaderFile = file
      shader:setField("samplerNames", 0, "$inputTex")
    else
      shader.DXComputShaderName = file
    end
    shader.pixVersion = def[3]
    shader:registerObject(def[1])
  end

  local block = createObject("GFXStateBlockData")
  block.zDefined = true
  block.zEnable = false
  block.zWriteEnable = false
  block.samplersDefined = true
  for slot, sampler in ipairs({ "SamplerClampLinear", "SamplerClampLinear", "SamplerClampPoint", "SamplerClampPoint" }) do
    block:setField("samplerStates", slot - 1, sampler)
  end
  block:registerObject("CMAA2_TemporalStateBlock")

  local root
  for _, def in ipairs(effectDefs) do
    local fx = createObject("PostEffect")
    if def.renderTime then fx:setField("renderTime", 0, def.renderTime) end
    if def.renderBin then fx:setField("renderBin", 0, def.renderBin) end
    fx.shader = def.shader
    fx.stateBlock = scenetree.findObject(def.stateBlock or "PFX_DefaultStateBlock")
    for slot, texture in ipairs(def.textures) do
      fx:setField("texture", slot - 1, texture)
    end
    fx.target = def.target
    if def.format then fx:setField("targetFormat", 0, def.format) end
    if def.skip then fx:setField("skipDraw", 0, "1") end
    fx:registerObject(def.name)
    if not root then root = fx else root:add(fx) end
  end
end

Things to note:

• Only the temporal pass needs its own state block, because it is the only pass that filters through samplers. The other passes read with Texture2D.Load and use PFX_DefaultStateBlock.
• pixVersion = 6.0 is used for the shaders that include the FidelityFX headers.
• The extension deletes its objects in reverse order before recreating them, and rebuilds the chain in onUpdate if the root effect no longer exists.

Pass 1: a minimal compute pass

The edge pass shows the full structure of a compute effect. The shared header declares the uniforms, and the entry point defines PFX_COMPUTE before its first include:

// cmaa2_common.h.hlsl
#include "shaders/common/postFx/postFx.h.hlsl"

cbuffer perDraw : register(b0)
{
    // x edge threshold, y local contrast adaptation, z simple shape bluriness, w max Z line length
    uniform float4 cmaaParams;
    uniform float4 cmaaDebug;
    POSTFX_UNIFORMS
};

int2 cmaa2Clamp(int2 p)
{
    return clamp(p, int2(dispatchRect.xy), int2(dispatchRect.xy + dispatchRect.zw) - 1);
}

float cmaa2Luma(float3 color)
{
    return dot(sqrt(max(color, 0.0)), float3(0.299, 0.587, 0.114));
}

// cmaa2_edges.cs.hlsl
#define PFX_COMPUTE 1

#include "cmaa2_common.h.hlsl"
#include "shaders/common/postFx/postFxCompute.h.hlsl"

Texture2D<float4> colorTex : register(t0);

PFX_NUMTHREADS
void main(uint3 dispatchThreadID : SV_DispatchThreadID)
{
    uint2 storePos;
    if (!postFxComputeThread(dispatchThreadID, storePos))
    {
        return;
    }
    const int2 p = int2(storePos);

    float l[3][3];
    [unroll]
    for (int y = -1; y <= 1; y++)
    {
        [unroll]
        for (int x = -1; x <= 1; x++)
        {
            l[y + 1][x + 1] = cmaa2Luma(colorTex.Load(int3(cmaa2Clamp(p + int2(x, y)), 0)).rgb);
        }
    }

    // ... compare neighbours against cmaaParams.x and pack four edge bits into "packed"

    pfxOut[storePos] = float4(packed / 255.0, 0.0, 0.0, 0.0);
}

• storePos is already offset by the viewport origin, so it can be used directly with Load and pfxOut.
• Neighbour reads are clamped to dispatchRect, not to the texture size. In split screen or multi-monitor views the target is shared, and reading outside the viewport would pull in the neighbouring view.
• The edge bits are written as packed / 255.0 into the R8 target and decoded with * 255.0 + 0.5 in the next pass.
• The shader constants are set from Lua only when a slider changes, not every frame:

local params = string.format("%f %f %f %f", edgeThreshold, localContrast, bluriness, maxLineLength)
for _, name in ipairs({ 'CMAA2_PostEffect', 'CMAA2_ZShapes', 'CMAA2_Apply' }) do
  local fx = scenetree.findObject(name)
  if fx then fx:setShaderConst("$cmaaParams", params) end
end

Temporal pass and history

The temporal pass blends the anti-aliased frame with the previous result. It needs three things the single-frame passes do not: a texture that survives between frames, motion vectors, and the camera jitter.

History. CMAA2_History uses renderTime = PFXTexGenOnDemand. Normal effect targets are frame-local and returned to a pool after use. A PFXTexGenOnDemand effect owns its target allocation, so the content is still there next frame. Because it is a child of the chain, it still runs every frame in its position after the temporal pass. The temporal pass reads #cmaa2History before the copy overwrites it, so it always sees the previous frame. The shipped depth of field uses the same pattern for #dofHistory.

Samplers. The history is sampled with filtering, so the temporal shader uses uniform_sampler2D and the state block assigns one sampler per texture slot: linear for the current frame and history, point for velocity and depth.

// cmaa2_temporal.cs.hlsl
#define PFX_COMPUTE 1

#include "shaders/common/postFx/postFx.h.hlsl"

cbuffer perDraw : register(b0)
{
    // x history weight, y neighborhood clip gamma, z velocity reject in pixels, w velocity soft range in pixels
    uniform float4 cmaaTemporal;
    uniform float2 sceneJitterUV;
    POSTFX_UNIFORMS
};

#include "shaders/common/postFx/postFxCompute.h.hlsl"

uniform_sampler2D(currentSampler, 0);
uniform_sampler2D(historySampler, 1);
uniform_sampler2D(velocitySampler, 2);
uniform_sampler2D(depthSampler, 3);

PFX_NUMTHREADS
void main(uint3 dispatchThreadID : SV_DispatchThreadID)
{
    uint2 storePos;
    if (!postFxComputeThread(dispatchThreadID, storePos))
    {
        return;
    }
    const float2 viewUV = postFxComputeUv(dispatchThreadID);

    // the scene was rasterized with the jitter; sampling at viewUV + jitter returns the pixel center
    const float2 currentUV = viewportCoordToRenderTarget(viewUV + sceneJitterUV, rtParams0);
    float4 source = currentSamplerTex.SampleLevel(currentSampler_sampler_, currentUV, 0);

    // velocity of the closest depth sample in a 5 tap cross, depth is reversed
    // ...
    const float2 velocity = velocitySamplerTex.Load(int3(bestP, 0)).rg;
    const float2 historyUV = viewUV + velocity;

    // clamp the history to the 3x3 neighbourhood of the current frame, then blend
    const float2 historyTexUV = viewportCoordToRenderTarget(historyUV, rtParams1);
    // ...

    pfxOut[storePos] = float4(resolved, source.a);
}

• sceneJitterUV is bound automatically when the shader declares it. Adding it to the current-frame UV removes the jitter before accumulation; see Jitter and jittered UVs .
• viewportCoordToRenderTarget(uv, rtParamsN) converts a viewport UV into the UV space of texture slot N. Use the matching rtParams for every texture slot.
• #VelocityBuffer and #prepass[Depth] are at render resolution. The shader reads their size with GetDimensions instead of assuming dispatchRect.
• In compute shaders, sample with SampleLevel (or tex2Dlod). Automatic mip selection is not available.

Resetting the history. Lua updates the history weight every frame in onPreRender and sends 0 when the history is invalid. That covers a skipped frame, a change of upscale mode or render scale, and the first frame after the temporal pass is switched on:

local function onPreRender()
  local temporal = scenetree.findObject("CMAA2_Temporal")
  if not temporal then return end

  local frameId = Engine.Render.getFrameId()
  local rv = RenderViewManagerInstance:getView('main')
  local mode = rv and rv.upscaleMode or -1
  local scale = rv and rv.upscaleRenderScale or -1
  if lastFrameId ~= frameId - 1 or lastMode ~= mode or lastScale ~= scale then
    historyReset = true
  end
  lastFrameId, lastMode, lastScale = frameId, mode, scale

  local feedback = historyReset and 0 or historyWeight
  historyReset = false
  temporal:setShaderConst("$cmaaTemporal", string.format("%f %f %f %f", feedback, 1.25, 48, 16))
end

Switching passes at runtime

Optional passes are created once and toggled with skipDraw instead of being created and deleted:

local function setSkipDraw(name, skip)
  local fx = scenetree.findObject(name)
  if fx then fx:setField("skipDraw", 0, skip and "1" or "0") end
end

setSkipDraw("CMAA2_Temporal", not temporal)
setSkipDraw("CMAA2_History", not temporal)
setSkipDraw("CMAA2_Sharpen", sharpness <= 0)

skipDraw skips only the effect’s own dispatch, keeps its registered target name, and still processes its children with the incoming texture. skip stops the effect and all of its children. The whole chain is switched with enable() and disable() on the root. Only change these fields when the value actually changes; the mod builds a small state string and returns early if it matches the previous frame.

One shader body for compute and pixel shaders

RCAS sharpening is needed in two places: as a compute pass in the chain, and as a pixel shader child of the final stretch when rendering below 100% scale. The body lives in cmaa2_rcas.h.hlsl, which declares a load function that each variant defines:

// cmaa2_rcas.h.hlsl (excerpt)
cbuffer perDraw : register(b0)
{
    uniform float4 cmaaSharpen;
    POSTFX_UNIFORMS
};

// defined by the including shader
float4 rcasLoad(int2 p);

float4 rcasFilter(int2 p)
{
    // FsrRcasF from ffx_fsr1.h, which calls rcasLoad
}

// cmaa2_rcas.cs.hlsl
#define PFX_COMPUTE 1

#include "cmaa2_rcas.h.hlsl"
#include "shaders/common/postFx/postFxCompute.h.hlsl"

Texture2D<float4> colorTex : register(t0);

float4 rcasLoad(int2 p)
{
    return colorTex.Load(int3(clamp(p, int2(dispatchRect.xy), int2(dispatchRect.xy + dispatchRect.zw) - 1), 0));
}

PFX_NUMTHREADS
void main(uint3 dispatchThreadID : SV_DispatchThreadID)
{
    uint2 storePos;
    if (!postFxComputeThread(dispatchThreadID, storePos))
    {
        return;
    }
    pfxOut[storePos] = rcasFilter(int2(storePos));
}

// cmaa2_rcas.ps.hlsl
#include "cmaa2_rcas.h.hlsl"

uniform_sampler2D(inputTex, 0);

#include "shaders/common/postFx/postFx.hlsl"

float4 rcasLoad(int2 p)
{
    uint w, h;
    inputTexTex.GetDimensions(w, h);
    return inputTexTex.Load(int3(clamp(p, int2(0, 0), int2(w, h) - 1), 0));
}

#ifdef SHADER_STAGE_VS
#define mainV main

PFXVertToPix mainV(PFXVert IN)
{
    return processPostFxVert(IN);
}
#else
#define mainP main

float4 mainP(PFXVertToPix IN) : SV_Target
{
    return rcasFilter(int2(IN.hpos.xy));
}
#endif

The pixel shader variant is a single file compiled twice: SHADER_STAGE_VS selects the vertex entry point, the pixel stage gets the other branch. The FSR 1 EASU upscaler in the same mod stays a pixel shader, because it derives the output size from ddx/ddy, which compute shaders do not have.

Working with the built-in anti-aliasing

An anti-aliasing mod has to keep the game’s own anti-aliasing out of the way. The CMAA2 extension:

• Disables FXAA_PostEffect, SMAA_PostEffect, and SMAA_T2X_PostEffect while it is active.
• Sets upscaleMode, upscaleRenderScale, and upscaleSharpness on the world render views (main, the split screen views, and the multi-monitor views) through RenderViewManagerInstance:getView(name).
• Re-applies GraphicAntialias, GraphicUpscalingQuality, and GraphicUpscalingSharpness through core_settings_graphic when it is disabled or unloaded, so the player’s settings come back.

Effects that only change the image, such as color grading, do not need any of this.

Related pages

• Post Process Effects , especially Creating custom shader PostEffects , Render targets and buffers , and Timing and chaining
• Temporal Upscaling
• Anti-aliasing and Upscaling