This branch/patch adds TensorFlow support for general dnn_processing and application-specific sr and derain filters, using brew's standard libtensorflow formula.

These filters can be used with conventional neural network (CNN) and deep neural network (DNN) techniques and can be used with independent models for model-based frame processing. A common application for dnn processing is Super Resolution using a trained model.

Background

There are very few public builds (static or docker) that currently support these filters with TensorFlow. One docker build https://github.com/MIR-MU/ffmpeg-tensorflow does support TensorFlow, with the following observations:

• The container is built with FFmpeg 5.x, so does not support alternate dnn_processing backends such as OpenVINO and PyTorch.
• The above container does not support Apple Silicon
• The above container requires Docker, whereas homebrew-ffmeg is native
• The above container, does however, kindly provide some pre-trained sample models {espcn.pb, srcnn.pb, vespcn.pb, vsrnet.pb} which can be extracted from the container.

This homebrew-ffmpeg branch does not include or plan to include any pre-trained models. A user is expected to train their own model or source pre-trained models from elsewhere (modelzoo, openmodeldb or similar). A user is expected to place their model in a easily accessible folder and reference it when invoking the FFmpeg filter, eg $HOME/tensorflow-models. This branch does not offer any opinion on the quality or suitability of models.

The awesome homebrew-ffmpeg already includes optional support for OpenVINO backend for the dnn_processing filter. This patch adds support for TensorFlow as an alternative backend for dnn_processing, and with the addition of TensorFlow, supports the application-specific sr and derain filters.

Assuming that this patch is accepted, homebrew-ffmpeg would support two out of the three active backends. The FFmpeg 'Native' backend was deprecated & removed in April 2023

• TensorFlow. This branch/patch.
• OpenVINO. Already supported.
• PyTorch.

Installation using the homebrew-ffmpeg option

$ brew install homebrew-ffmpeg/ffmpeg/ffmpeg --with-tensorflow

Validation

After compilation & installation, FFmpeg command-line help lists TensorFlow as an available dnn_backend for each of the following filters.

$ ffmpeg -h filter=sr
$ ffmpeg -h filter=dnn_processing
$ ffmpeg -h filter=dnn_detect
$ ffmpeg -h filter=derain

It is not expected that TensorFlow would be available in the dnn_clasify filter. [https://ffmpeg.org/ffmpeg-all.html#dnn_005fclassify](Formal FFmpeg documentation) for dnn_classify states that "This option accepts only openvino now, tensorflow backends will be added.".

✅ Compilation and installation was successful using this branch on macOS (13.7 Apple Silicon). Not tested on homebrew linux.

Tests

Wow! it actually JustWorked(TM) using brew's "standard" libtensorflow and borrowing some models for testing.

sr filter with Super Resolution Conventional Neural Network (srcnn) model

A pre-trained srcnn.pb model can be extracted from https://github.com/MIR-MU/ffmpeg-tensorflow Docker container

$ ffmpeg -f 'lavfi' -i mandelbrot=size='ntsc':rate='ntsc',trim=end_frame=30 -filter:v sr=dnn_backend='tensorflow':scale_factor=4:model="$HOME/tensorflow-models/srcnn.pb":input='x':output='y',format=pix_fmts='yuv420p' -f 'null' -

✅ The above FFmpeg command was successful on macOS (13.7 Apple Silicon) 2024-10-23. Not tested on homebrew linux.

At the time of writing, the HTML docs for the sr filter are outdated (and do not reference mandatory options such as input='x':output='y'). Users are advised to use command-line help $ ffmpeg -h filter=sr for up to date command-line options.

Derain filter with derain & dehaze models

$ curl --location https://github.com/XueweiMeng/derain_filter/raw/refs/heads/master/models_for_test/derain/derain_RESCAN.pb -o $HOME/tensorflow-models/derain_RESCAN.pb 
$ ffmpeg -f 'lavfi' -i mandelbrot=size='ntsc':rate='ntsc',trim=end_frame=30 -filter:v derain=dnn_backend='tensorflow':model="$HOME/tensorflow-models/derain_RESCAN.pb":filter_type='derain':input='x':output='y',format=pix_fmts='yuv420p' -f 'null' -

$ curl --location https://github.com/XueweiMeng/derain_filter/raw/refs/heads/master/models_for_test/dehaze/dehaze_RESCAN.pb -o $HOME/tensorflow-models/dehaze_RESCAN.pb
$ ffmpeg -f 'lavfi' -i mandelbrot=size='ntsc':rate='ntsc',trim=end_frame=30 -filter:v derain=dnn_backend='tensorflow':model="$HOME/tensorflow-models/dehaze_RESCAN.pb":filter_type='dehaze':input='x':output='y',format=pix_fmts='yuv420p' -f 'null' -

✅ The above FFmpeg command was successful on macOS (13.7 Apple Silicon) 2024-10-23. Not tested on homebrew linux.

At the time of writing, the HTML docs for the derain filter are outdated and do not reference mandatory options such as input='x':output='y'. Users are advised to use command-line help $ ffmpeg -h filter=derain for up to date command-line options.

General dnn_processing filter

A pre-trained espcn.pb model can be extracted from https://github.com/MIR-MU/ffmpeg-tensorflow Docker container

$ ffmpeg -f 'lavfi' -i mandelbrot=size='ntsc':rate='ntsc',trim=end_frame=30 -filter:v dnn_processing=dnn_backend='tensorflow':model="$HOME/tensorflow-models/espcn.pb":input='x':output='y',format=pix_fmts='yuv420p' -f 'null' -

✅ The above FFmpeg command was successful on macOS (13.7 Apple Silicon) 2024-10-23. Not tested on homebrew linux.

Caveat Emptor

I'm not a developer; I'm not very good with git branches or commits; I can barely write a homebrew formula. If I have broken any git or contribution conventions, please forgive me. I shall aim to learn and improve.