l2norm divergence

[pwilkin]

Does anyone know why the l2norm implementation is different than in the typical PyTorch implementations? Ran into the divergence when testing Qwen3Next with some tiny models. Pytorch does soft clamping:

def l2norm(x: torch.FloatTensor, dim: int = -1, eps: float = 1e-6):
    """This function is intended to align with the l2norm implementation in the FLA library."""
    inv_norm = torch.rsqrt((x * x).sum(dim=dim, keepdim=True) + eps)
    return x * inv_norm

while GGML does hard clamping (with fmax(sum, eps)). Is there a specific rationale for it?

[ggerganov]

It was implemented like this in #12412 - cc @MollySophia

[MollySophia]

It was implemented according to: https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.normalize.html

[pwilkin]

Yeah, I see base PyTorch uses the hard clamp, but both Llama4 and Qwen3 Next use the soft clamp (with adding the epsilon instead of max). Maybe we should admit a flag to specify which sort of clamping is needed? I'm not sure this will really matter in real-world scenarios too much, just bringing it up since I did encounter a discrepancy with a mock model.

[MollySophia]

I guess yes? We can add a flag to decide whether to use hard clamp or soft clamp