Skip to content

videoio: use oneVPL load mechanism, encoder bitrate estimation #20614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Oct 21, 2021
Merged

videoio: use oneVPL load mechanism, encoder bitrate estimation #20614

merged 5 commits into from
Oct 21, 2021

Conversation

mshabunin
Copy link
Contributor

@mshabunin mshabunin commented Aug 26, 2021 •
edited by alalek
Loading

  • use oneVPL MFXLoad function instead of deprecated MFXInit
  • advanced encoding bitrate estimation with different formulas for MPEG2, H264 and H265
  • minor modification to libva search process

Troubleshooting:

  • (Linux) crash with following message might be caused by outdated kernel (observed with Linux 4.4.0)

    libva info: Found init function __vaDriverInit_1_12
libva info: va_openDriver() returns 0
opencv_test_videoio: /opt/src/msdk/_studio/mfx_lib/shared/src/mfx_mpeg2_encode_vaapi.cpp:1420: mfxStatus 
MfxHwMpeg2Encode::VAAPIEncoder::Execute(MfxHwMpeg2Encode::ExecuteBuffers*, mfxU32, mfxU8*, mfxU32): Assertion 
`0x00000000 == vaSts' failed.
Process returned: -11

    need to upgrade Linux kernel

force_builders=Custom,Custom Win

Xbuildworker:Custom=linux-1
buildworker:Custom=linux-6
build_image:Custom=onevpl-2021.6.0
test_opencl:Custom=ON
test_modules:Custom=videoio


buildworker:Custom Win=windows-3
build_image:Custom Win=onevpl-2021.6.0
test_opencl:Custom Win=ON
test_modules:Custom Win=videoio

alalek
alalek reviewed Sep 1, 2021
View reviewed changes
alalek
alalek reviewed Sep 28, 2021
View reviewed changes
modules/videoio/src/cap_mfx_common.hpp
@@ -18,6 +18,7 @@
# include <vpl/mfxvideo++.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add version check into CMake scripts (for which versions we want to support).
mfxvideo++.h file is missing in case of old VPL_VERSION=2021.1.11

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the check. Verified it with oneVPL 2021.1.1-66, 2021.2.2-212, 2021.4.0-328 from oneAPI and 2021.4.0 from GPGPU release. Support of 2021.1 has been successfully disabled. Additionally I've found that oneVPL has changed versioning scheme and 2021.4.0 is now "2.4", thus slightly more complex condition.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've found that oneVPL has changed versioning scheme and 2021.4.0 is now "2.4"

🤦‍♂️
Confirmed.

VPL_CONFIG=C:/Program Files (x86)/Intel/oneAPI/vpl/2021.4.0/lib/cmake/vpl/VPLConfig.cmake
...
VPL_VERSION=2.4

(enabled "Custom Win" builder, vpl-inspect dump is available on "init" step)

alalek
alalek reviewed Oct 1, 2021
View reviewed changes
Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It make sense to eliminate/suppress deprecation build warnings.

modules/videoio/src/cap_mfx_writer.cpp Outdated
Comment on lines 26 to 29
else if (codecId == MFX_CODEC_HEVC)
{
bitrate = (mp * 63 + 41) * pow(fps, 0.60f);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know what is wrong with .h265 tests on "Custom Win" builder?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the problem is. In my tests on laptop with UHD 620 (27.20.100.9466) and oneVPL 2021.4.0 from oneAPI all MFX tests are green. Perhaps some combinations of HW / Driver / oneVPL are not working very well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've increased bitrate slightly for HEVC. Let's see, maybe it will help.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there is mess in frames order.
Attached captured frames and encoded videos of failed test cases: pr20614.zip

BTW, encoded videos are broken (check ffmpeg -i 1.265 -r 1/1 frame_%05d.png)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see suspicious messages in the ffmpeg output:

$ ffmpeg -i 1.265  -r 1/1 frame_%05d.png
ffmpeg version 4.3.2-0+deb11u1ubuntu1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Ubuntu 10.2.1-20ubuntu1)
  configuration: --prefix=/usr --extra-version=0+deb11u1ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, hevc, from '1.265':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 640x480 [SAR 1:1 DAR 4:3], 1 fps, 1 tbr, 1200k tbn, 1 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'frame_%05d.png':
  Metadata:
    encoder         : Lavf58.45.100
    Stream #0:0: Video: png, rgb24, 640x480 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 1 fps, 1 tbn, 1 tbc
    Metadata:
      encoder         : Lavc58.91.100 png
frame=   20 fps=0.0 q=-0.0 Lsize=N/A time=00:00:20.00 bitrate=N/A speed= 837x    
video:342kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

Perhaps the video is too short and have challenging contents. We can try to increase bitrate even more for this codec, what do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encoded videos are broken

I mean in this two stage tests of encoding + decoding, the first encoding part is broken and provides incorrect results (check frame.png images).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aah, yes. It is usually the encoding which breaks in these tests 😃, that's why I tried to increase the bitrate. I don't see what else we can do here, maybe upgrading the driver on the test machine would help.

alalek
alalek approved these changes Oct 20, 2021
View reviewed changes
Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you 👍

@alalek alalek merged commit 7febec4 into opencv:master Oct 21, 2021
@alalek alalek mentioned this pull request Dec 30, 2021
Merged
@alalek alalek mentioned this pull request Feb 22, 2022
Merged
a-sajjad72 pushed a commit to a-sajjad72/opencv that referenced this pull request Mar 30, 2023
@mshabunin @alalek
Merge pull request opencv#20614 from mshabunin:use-onevpl-load
e327f7f 
videoio: use oneVPL load mechanism, encoder bitrate estimation

* videoio: updated oneVPL support - use mfxLoad

* videoio: advanced bitrate estimation for MFX encoder

* videoio: improved MediaSDK/oneVPL/libva detection

* videoio(ffmpeg): don't try oneVPL

* videoio(test): tune checks of videoio_mfx.read_write_raw tests

Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>
@mshabunin mshabunin deleted the use-onevpl-load branch July 10, 2023 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alalek alalek alalek approved these changes

Assignees

@alalek alalek

Labels
category: videoio feature
Projects
None yet
Milestone
4.5.5
Development

Successfully merging this pull request may close these issues.
2 participants
@mshabunin @alalek