OpenCV4.8.0 DNN inference speed reduced by 50%. #23911
Description
System Information
OpenCV version: 4.8.0
Operating System / Platform: win10
Compiler & compiler version: vs2019
Compare versions：
OpenCV version:OpenCV 4.5.5-openvino
and the same envirment
Detailed description
Display compilation information and statistical time for both versions. The time consumption has increased from 20ms to 30ms.
General configuration for OpenCV 4.5.5-openvino =====================================
Version control: c3d60a6cac5b5b3c452d766494d21b005221efe0
Platform:
Timestamp: 2022-03-11T02:31:28Z
Host: Windows 10.0.19044 AMD64
CMake: 3.14.5
CMake generator: Visual Studio 16 2019
CMake build tool: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/MSBuild/Current/Bin/MSBuild.exe
MSVC: 1929
Configuration: Debug Release
CPU/HW features:
Baseline: SSE SSE2 SSE3 SSSE3 SSE4_1 POPCNT SSE4_2
requested: SSE4_2
Dispatched code generation: FP16 AVX AVX2 AVX512_SKX
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
FP16 (1 files): + FP16 AVX
AVX (5 files): + AVX
AVX2 (33 files): + FP16 FMA3 AVX AVX2
AVX512_SKX (8 files): + FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX
C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30037/bin/Hostx64/x64/cl.exe (ver 19.29.30040.0)
C++ flags (Release): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /GS /sdl /guard:cf /w34018 /w34146 /w34244 /w34267 /w34302 /w34308 /w34509 /w34532 /w34533 /w34700 /w34789 /w34995 /w34996 /MP /MD /O2 /Ob2 /DNDEBUG
C++ flags (Debug): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /GS /sdl /guard:cf /w34018 /w34146 /w34244 /w34267 /w34302 /w34308 /w34509 /w34532 /w34533 /w34700 /w34789 /w34995 /w34996 /MP /MDd /Zi /Ob0 /Od /RTC1
C Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30037/bin/Hostx64/x64/cl.exe
C flags (Release): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /GS /sdl /guard:cf /w34018 /w34146 /w34244 /w34267 /w34302 /w34308 /w34509 /w34532 /w34533 /w34700 /w34789 /w34995 /w34996 /MP /MD /O2 /Ob2 /DNDEBUG
C flags (Debug): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /GS /sdl /guard:cf /w34018 /w34146 /w34244 /w34267 /w34302 /w34308 /w34509 /w34532 /w34533 /w34700 /w34789 /w34995 /w34996 /MP /MDd /Zi /Ob0 /Od /RTC1
Linker flags (Release): /machine:x64 /guard:cf /dynamicbase /INCREMENTAL:NO
Linker flags (Debug): /machine:x64 /guard:cf /dynamicbase /debug /INCREMENTAL
ccache: NO
Precompiled headers: NO
Extra dependencies:
3rdparty dependencies:
OpenCV modules:
To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python3 stitching ts video videoio
Disabled: world
Disabled by dependency: -
Unavailable: java python2
Applications: tests perf_tests apps
Documentation: NO
Non-free algorithms: NO
Windows RT support: NO
GUI: WIN32UI
Win32 UI: YES
Media I/O:
ZLib: build (ver 1.2.11)
JPEG: build-libjpeg-turbo (ver 2.1.2-62)
PNG: build (ver 1.6.37)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
FFMPEG: YES (prebuilt binaries)
avcodec: YES (58.134.100)
avformat: YES (58.76.100)
avutil: YES (56.70.100)
swscale: YES (5.9.100)
avresample: YES (4.0.0)
DirectShow: YES
Media Foundation: YES
DXVA: YES
Intel Media SDK: YES (VPL::dispatcher VPL::api)
Parallel framework: Concurrency
Trace: YES (with Intel ITT)
Other third-party libraries:
Intel IPP: 2020.0.0 Gold [2020.0.0]
at: C:/jenkins/workspace/windows9f26d2f1/build_release/3rdparty/ippicv/ippicv_win/icv
Intel IPP IW: sources (2020.0.0)
at: C:/jenkins/workspace/windows9f26d2f1/build_release/3rdparty/ippicv/ippicv_win/iw
Inference Engine: YES (2022010000 / 2022.1.0)
* libs: C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/lib/intel64/Release/openvino.lib / C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/lib/intel64/Debug/openvinod.lib C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Release/openvino.dll / C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Debug/openvinod.dll C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Release/openvino.dll C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Debug/openvinod.dll
* includes: C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/include C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/include/ie
nGraph: YES (2022.1.0)
* libs: C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/lib/intel64/Release/openvino.lib / C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/lib/intel64/Debug/openvinod.lib C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Release/openvino.dll / C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Debug/openvinod.dll C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Release/openvino.dll C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/bin/intel64/Debug/openvinod.dll
* includes: C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/include C:/jenkins/workspace/windows9f26d2f1/deployment_tools/runtime/include/ie
Custom HAL: NO
Protobuf: build (3.19.1)
OpenCL: YES (NVD3D11)
Include path: C:/jenkins/workspace/windows9f26d2f1/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python 3:
Interpreter: C:/miniconda/envs/py3_env/python.exe (ver 3.4.5)
Libraries: C:/miniconda/envs/py3_env/libs/python34.lib (ver 3.4.5)
numpy: C:/miniconda/envs/py3_env/lib/site-packages/numpy/core/include (ver 1.11.3)
install path: python/cv2/python-3
Python (for build): C:/miniconda/envs/py3_env/python.exe
Install to: C:/jenkins/workspace/windows9f26d2f1/build_release/install
-----------------------------------------------------------------
[yolov4]
init >> 39.592ms
inference >> min = 21.601ms, max = 32.751ms, mean = 21.846ms, stddev = 1.35271ms
D:\vcworkspaces\yolov4_tiny_dnn_demo\x64\Release\yolov4_tiny_dnn_demo.exe (进程 7704)已退出，代码为 0。
按任意键关闭此窗口. . .
opencv4.8.0
General configuration for OpenCV 4.8.0 =====================================
Version control: 4.8.0
Platform:
Timestamp: 2023-06-28T12:35:18Z
Host: Windows 10.0.19045 AMD64
CMake: 3.23.3
CMake generator: Visual Studio 16 2019
CMake build tool: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin/MSBuild.exe
MSVC: 1928
Configuration: Debug Release
CPU/HW features:
Baseline: SSE SSE2 SSE3
requested: SSE3
Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
SSE4_1 (16 files): + SSSE3 SSE4_1
SSE4_2 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2
FP16 (0 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
AVX (7 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
AVX2 (35 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
AVX512_SKX (5 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX
C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe (ver 19.28.29334.0)
C++ flags (Release): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /wd4819 /MP /MD /O2 /Ob2 /DNDEBUG
C++ flags (Debug): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /wd4819 /MP /MDd /Zi /Ob0 /Od /RTC1
C Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe
C flags (Release): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MD /O2 /Ob2 /DNDEBUG
C flags (Debug): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MDd /Zi /Ob0 /Od /RTC1
Linker flags (Release): /machine:x64 /INCREMENTAL:NO
Linker flags (Debug): /machine:x64 /debug /INCREMENTAL
ccache: NO
Precompiled headers: NO
Extra dependencies:
3rdparty dependencies:
OpenCV modules:
To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo stitching video videoio world
Disabled: python3
Disabled by dependency: -
Unavailable: java python2 ts
Applications: apps
Documentation: NO
Non-free algorithms: NO
Windows RT support: NO
GUI:
Win32 UI: YES
VTK support: NO
Media I/O:
ZLib: build (ver 1.2.13)
JPEG: build-libjpeg-turbo (ver 2.1.3-62)
SIMD Support Request: YES
SIMD Support: NO
WEBP: build (ver encoder: 0x020f)
PNG: build (ver 1.6.37)
TIFF: build (ver 42 - 4.2.0)
JPEG 2000: build (ver 2.5.0)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
DC1394: NO
FFMPEG: YES (prebuilt binaries)
avcodec: YES (58.134.100)
avformat: YES (58.76.100)
avutil: YES (56.70.100)
swscale: YES (5.9.100)
avresample: YES (4.0.0)
GStreamer: NO
DirectShow: YES
Media Foundation: YES
DXVA: YES
Parallel framework: Concurrency
Trace: YES (with Intel ITT)
Other third-party libraries:
Intel IPP: 2021.8 [2021.8.0]
at: C:/GHA-OCV-1/_work/ci-gha-workflow/ci-gha-workflow/build/3rdparty/ippicv/ippicv_win/icv
Intel IPP IW: sources (2021.8.0)
at: C:/GHA-OCV-1/_work/ci-gha-workflow/ci-gha-workflow/build/3rdparty/ippicv/ippicv_win/iw
Eigen: NO
Custom HAL: NO
Protobuf: build (3.19.1)
Flatbuffers: builtin/3rdparty (23.5.9)
OpenCL: YES (NVD3D11)
Include path: C:/GHA-OCV-1/_work/ci-gha-workflow/ci-gha-workflow/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python (for build): C:/Python-3.9/python.exe
Java:
ant: C:/apache-ant-1.9.15/bin/ant.bat (ver 1.9.15)
Java: NO
JNI: C:/Program Files/Java/jdk-11.0.9/include C:/Program Files/Java/jdk-11.0.9/include/win32 C:/Program Files/Java/jdk-11.0.9/include
Java wrappers: NO
Java tests: NO
Install to: C:/GHA-OCV-1/_work/ci-gha-workflow/ci-gha-workflow/install
-----------------------------------------------------------------
[yolov4]
init >> 92.686ms
inference >> min = 29.27ms, max = 38.598ms, mean = 32.5029ms, stddev = 1.07574ms
D:\vcworkspaces\yolov4_tiny_dnn_demo\x64\Release\yolov4_tiny_dnn_demo.exe (进程 11968)已退出，代码为 0。
按任意键关闭此窗口. . .
Steps to reproduce
#include <iostream>
#include <queue>
#include <iterator>
#include <sstream>
#include <fstream>
#include <iomanip>
#include <chrono>
#include <opencv2/core.hpp>
#include <opencv2/dnn.hpp>
#include <opencv2/dnn/all_layers.hpp>
#include <opencv2/opencv.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <numeric>
constexpr float CONFIDENCE_THRESHOLD = 0;
constexpr float NMS_THRESHOLD = 0.4;
constexpr int NUM_CLASSES = 80;
// colors for bounding boxes
const cv::Scalar colors[] = {
{0, 255, 255},
{255, 255, 0},
{0, 255, 0},
{255, 0, 0}
};
const auto NUM_COLORS = sizeof(colors) / sizeof(colors[0]);
int main()
{
std::cout << cv::getBuildInformation() << std::endl;
std::vector<std::string> class_names;
{
std::ifstream class_file("yolo/coco.names");
if (!class_file)
{
std::cerr << "failed to open classes.txt\n";
return 0;
}
std::string line;
while (std::getline(class_file, line))
class_names.push_back(line);
}
std::string b = "./yolo/yolo_test.mp4";
cv::VideoCapture source(b);
auto net = cv::dnn::readNetFromDarknet("yolo/yolov4-tiny.cfg", "yolo/yolov4-tiny.weights");
net.setPreferableBackend(cv::dnn::DNN_BACKEND_OPENCV);
net.setPreferableTarget(cv::dnn::DNN_TARGET_CPU);
auto output_names = net.getUnconnectedOutLayersNames();
cv::Mat frame(416, 416, CV_32FC3), blob; //Tiny
std::vector<cv::Mat> detections;
std::vector<float> runtimes;
auto init_start = std::chrono::steady_clock::now();
cv::dnn::blobFromImage(frame, blob, 1 / 255.0, cv::Size(416, 416), cv::Scalar(), true, false, CV_32F); //Tiny
net.setInput(blob);
net.forward(detections, output_names);
auto init_end = std::chrono::steady_clock::now();
while (cv::waitKey(1) < 1)
{
source >> frame;
if (frame.empty())
break;
auto total_start = std::chrono::steady_clock::now();
cv::dnn::blobFromImage(frame, blob, 1.0 / 255, cv::Size(416, 416), cv::Scalar(), true, false, CV_32F); //Tiny
net.setInput(blob);
auto dnn_start = std::chrono::steady_clock::now();
net.forward(detections, output_names);
auto dnn_end = std::chrono::steady_clock::now();
std::vector<int> indices[NUM_CLASSES];
std::vector<cv::Rect> boxes[NUM_CLASSES];
std::vector<float> scores[NUM_CLASSES];
for (auto& output : detections)
{
const auto num_boxes = output.rows;
for (int i = 0; i < num_boxes; i++)
{
auto x = output.at<float>(i, 0) * frame.cols;
auto y = output.at<float>(i, 1) * frame.rows;
auto width = output.at<float>(i, 2) * frame.cols;
auto height = output.at<float>(i, 3) * frame.rows;
cv::Rect rect(x - width / 2, y - height / 2, width, height);
for (int c = 0; c < NUM_CLASSES; c++)
{
auto confidence = *output.ptr<float>(i, 5 + c);
if (confidence >= CONFIDENCE_THRESHOLD)
{
boxes[c].push_back(rect);
scores[c].push_back(confidence);
}
}
}
}
for (int c = 0; c < NUM_CLASSES; c++)
cv::dnn::NMSBoxes(boxes[c], scores[c], 0.0, NMS_THRESHOLD, indices[c]);
for (int c = 0; c < NUM_CLASSES; c++)
{
for (size_t i = 0; i < indices[c].size(); ++i)
{
const auto color = colors[c % NUM_COLORS];
auto idx = indices[c][i];
const auto& rect = boxes[c][idx];
cv::rectangle(frame, cv::Point(rect.x, rect.y), cv::Point(rect.x + rect.width, rect.y + rect.height), color, 3);
std::ostringstream label_ss;
label_ss << class_names[c] << ": " << std::fixed << std::setprecision(2) << scores[c][idx];
auto label = label_ss.str();
int baseline;
auto label_bg_sz = cv::getTextSize(label.c_str(), cv::FONT_HERSHEY_COMPLEX_SMALL, 1, 1, &baseline);
cv::rectangle(frame, cv::Point(rect.x, rect.y - label_bg_sz.height - baseline - 10), cv::Point(rect.x + label_bg_sz.width, rect.y), color, cv::FILLED);
cv::putText(frame, label.c_str(), cv::Point(rect.x, rect.y - baseline - 5), cv::FONT_HERSHEY_COMPLEX_SMALL, 1, cv::Scalar(0, 0, 0));
}
}
auto total_end = std::chrono::steady_clock::now();
float inference_fps = std::chrono::duration_cast<std::chrono::microseconds>(dnn_end - dnn_start).count() / 1000.0;
//std::cout << "模型推理时间为：" << inference_fps << " ms" << std::endl;
float total_fps = std::chrono::duration_cast<std::chrono::microseconds>(total_end - total_start).count() / 1000.0;
//std::cout << "单帧总耗费时间为：" << total_fps << " ms" << std::endl;
std::ostringstream stats_ss;
stats_ss << std::fixed << std::setprecision(2);
stats_ss << "Inference FPS: " << 1000.0 / inference_fps << ", Total FPS: " << 1000.0 / total_fps;
runtimes.push_back(total_fps);
auto stats = stats_ss.str();
int baseline;
auto stats_bg_sz = cv::getTextSize(stats.c_str(), cv::FONT_HERSHEY_COMPLEX_SMALL, 1, 1, &baseline);
cv::rectangle(frame, cv::Point(0, 0), cv::Point(stats_bg_sz.width, stats_bg_sz.height + 10), cv::Scalar(0, 0, 0), cv::FILLED);
cv::putText(frame, stats.c_str(), cv::Point(0, stats_bg_sz.height + 5), cv::FONT_HERSHEY_COMPLEX_SMALL, 1, cv::Scalar(255, 255, 255));
//cv::namedWindow("output", cv::WindowFlags::WINDOW_AUTOSIZE);
//cv::imshow("output", frame);
}
auto sum = std::accumulate(std::begin(runtimes), std::end(runtimes), 0.0f);
auto squared_sum = std::inner_product(std::begin(runtimes), std::end(runtimes), std::begin(runtimes), 0.0f);
auto min = *std::min_element(std::begin(runtimes), std::end(runtimes));
auto max = *std::max_element(std::begin(runtimes), std::end(runtimes));
auto mean = sum / runtimes.size();
auto stddev = std::sqrt(squared_sum / runtimes.size() - mean * mean);
std::cout << '[' << "yolov4-tiny" << "]" << '\n'
<< "\tinit >> " << std::chrono::duration_cast<std::chrono::microseconds>(init_end - init_start).count() / 1000.0 << "ms" << '\n'
<< "\tinference >> " << "min = " << min << "ms, max = " << max << "ms, mean = " << mean << "ms, stddev = " << stddev << "ms" << std::endl;
//cv::destroyAllWindows();
return 0;
}
Related models and video downloads.
Download: https://pan.baidu.com/s/1wpXBbdtJMUrYULAelglIiw?pwd=x67k
Issue submission checklist
- I report the issue, it's not a question
- I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- I updated to the latest OpenCV version and the issue is still there
- There is reproducer code and related data files (videos, images, onnx, etc)