LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Denoiser block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Denoiser › 12 denoiser blocks

LayerNorm

×

learned

Center and scale each position’s features and apply a learned gain; there is no bias.

xˉmlp=γ2⊙xn′−μ(xn′)σ2(xn′)+ε{\htmlData{light=mlp-norm}{\bar x_{\mathrm{mlp}}}}=\gamma_{2}\odot\frac{{\htmlData{light=residual attention-add}{x'_n}}-\mu({\htmlData{light=residual attention-add}{x'_n}})}{\sqrt{\sigma^2({\htmlData{light=residual attention-add}{x'_n}})+\varepsilon}}
ε=10−5\varepsilon=10^{-5}
in
xn′x'_n [B batch, T positions, 768 d]
out
xˉmlp\bar x_{\mathrm{mlp}} [B batch, T positions, 768 d]
parameters
768 · × 12 blocks = 9,216

Computed in float32 with autocast off; ε is PyTorch’s default.

Parameters

768 parameters owned by this component

Exact per-instance accounting
ComponentCalculationParameters
gain only768768768
Per-instance total768

Across 12 independent blocks: 768×12=9,216768 \times 12 = 9{,}216 parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

One gain for each of the 768 features; no bias.

Codefunctional.py 2561–2577
functional.py 2561–2577
def layer_norm(
    input: Tensor,
    normalized_shape: List[int],
    weight: Optional[Tensor] = None,
    bias: Optional[Tensor] = None,
    eps: float = 1e-5,
) -> Tensor:
    r"""Apply Layer Normalization for last certain number of dimensions.

    See :class:`~torch.nn.LayerNorm` for details.
    """
    if has_torch_function_variadic(input, weight, bias):
        return handle_torch_function(
            layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
        )
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
Sources2

Direct evidence

The denoiser’s and the token decoder’s LayerNorm computes in float32 and owns a gain and no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

PyTorch 2.4.1 functional LayerNorm defaults to epsilon 1e-5.

torch-layer-norm · pytorch/pytorch / torch/nn/functional.py
Pinned revision ee1b6804381c ↗ · lines 2561–2577

Read code excerpt
def layer_norm(
    input: Tensor,
    normalized_shape: List[int],
    weight: Optional[Tensor] = None,
    bias: Optional[Tensor] = None,
    eps: float = 1e-5,
) -> Tensor:
    r"""Apply Layer Normalization for last certain number of dimensions.

    See :class:`~torch.nn.LayerNorm` for details.
    """
    if has_torch_function_variadic(input, weight, bias):
        return handle_torch_function(
            layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
        )
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)