Fix: different frame size as inputs of the filter graph #303

valnoel · 2017-08-31T13:47:46Z

In order to mux audio channels, AvTranscoder uses the amerge filter.
If, as inputs of the filter graph containing this amerge filter, we put frames with different sizes, the resulting frame size seems to be the minimum of the input frame size. This means that one or more output channels can have cut frames, changing the reading speed and generating clicks into the final audio stream.

Using the internal filter buffer, the output frame is ensured not to lose audio samples, solving the problem of speed and clicks.
However, the filter internal buffer is quiet limited and quickly overflowed, making the process crashing before the end of the program. So, using custom frame buffers before the filter graph inputs fixes this other problem.

Let the filter managing the output frame size, using its internal frame buffer queue.

Used as frame buffers on the filter graph inputs. For the moment, this first version only handles audio frames.

Returning frames with the same size as inputs of the graph, avoiding to overflow the internal filter buffers.

If the input frames have the same size and the buffers are empty, bypass to avoid useless data copies.

cchampet · 2017-09-06T08:23:26Z

src/AvTranscoder/filter/FilterGraph.hpp

+private:
+    void popFrame();
+
+    const AudioFrameDesc _audioFrameDesc;


It seems that you have a compilation error because this attribute is const...

Yes, thanks! Now, it's fixed in 380f238

Add a new getAvailableFrameSize() private method

Add utility methods, documentation and logs into FilterGraph and FrameBuffer classes

Avoid filling the filter graph buffers with wrong data frames. Improve documentation into FilterGraph::FrameBuffer class.

Since video frames cannot be split, this filter graph buffers will be specialized to audio frames.

Replace it by a simple AudioFrameBuffer vector empty() test, since buffers are initialized only for audio frames (into the addInBuffer() method) And rename _inputAudioFramesBuffer attribute to _inputAudioFrameBuffers

coveralls · 2017-09-07T19:18:33Z

Coverage decreased (-1.2%) to 76.929% when pulling 969c6c3 on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

Comparing the frame sizes in samples (instead of bytes)

Audio channels are extracted from different file format sources, to stereo and 5.1 output audio streams.

coveralls · 2017-09-09T01:47:29Z

Coverage decreased (-1.5%) to 76.616% when pulling 2dcdc1c on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

coveralls · 2017-09-09T01:47:29Z

Coverage decreased (-1.5%) to 76.616% when pulling 2dcdc1c on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

coveralls · 2017-09-09T15:13:25Z

Coverage increased (+0.5%) to 78.68% when pulling 343ef4e on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

coveralls · 2017-09-09T15:13:25Z

Coverage increased (+0.5%) to 78.68% when pulling 343ef4e on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

coveralls · 2017-09-09T15:13:25Z

Coverage increased (+0.5%) to 78.68% when pulling 343ef4e on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

Export AudioFrameBuffer class symbol for DLL

coveralls · 2017-09-11T14:36:39Z

Coverage increased (+0.5%) to 78.68% when pulling a26bae9 on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

coveralls · 2017-09-11T14:36:39Z

Coverage increased (+0.5%) to 78.68% when pulling a26bae9 on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

cchampet · 2017-09-12T08:34:47Z

test/pyTest/testMuxAudioChannels.py

+    dst_inputFile = av.InputFile(outputFileName)
+    dst_audioProperties = dst_inputFile.getProperties().getAudioProperties()
+    assert_equals(1, len(dst_audioProperties))
+    assert_equals(2, dst_audioProperties[0].getNbChannels())


Could you check the duration of the output stream?

Actually, it was already done using the process statistics, but it is indeed better using the output file properties: 9abcb90

cchampet · 2017-09-12T08:34:52Z

test/pyTest/testMuxAudioChannels.py

+    dst_inputFile = av.InputFile(outputFileName)
+    dst_audioProperties = dst_inputFile.getProperties().getAudioProperties()
+    assert_equals(1, len(dst_audioProperties))
+    assert_equals(6, dst_audioProperties[0].getNbChannels())


Could you check the duration of the output stream?

Duration check using the output file properties: 9abcb90

cchampet · 2017-09-12T08:41:05Z

src/AvTranscoder/transcoder/StreamTranscoder.cpp

+            if(!_filterGraph->hasFilters() || !_filterGraph->hasBufferedFrames(index))
+            {
+                continueProcess = false;
+                continue;


Did you consider to user break instead of continue? Because in this case, do we need to continue to get the remaining buffers in the filter graph?

Sure, it is better! See 0e49751

Into audio channels muxing tests

…cess won't be continued

coveralls · 2017-09-20T14:43:23Z

Coverage increased (+0.5%) to 78.68% when pulling 0e49751 on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

coveralls · 2017-09-20T14:43:23Z

Coverage increased (+0.5%) to 78.68% when pulling 0e49751 on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

coveralls · 2017-09-20T14:43:23Z

Coverage increased (+0.5%) to 78.68% when pulling 0e49751 on valnoel:filter_inputMethod into 93961f4 on avTranscoder:develop.

valnoel · 2017-09-21T08:31:27Z

@cchampet RTM?

cchampet · 2017-09-21T10:33:52Z

src/AvTranscoder/filter/FilterGraph.cpp

+
+    // Concatenate frames data
+    size_t extractedDataSize = 0;
+    unsigned char* outputData = new unsigned char[size];


Is this type mean that the supported data type is only 1byte per sample?

No, it is actually the raw data in bytes, without considering the sample format or whatever.

cchampet · 2017-09-21T10:35:06Z

@valnoel I had only one question.
Except that, you can merge whenever you want ;)

valnoel · 2017-09-21T12:12:32Z

@cchampet OK then! :)

FilterGraph: push input frames into the filter buffer

2dc9d24

Let the filter managing the output frame size, using its internal frame buffer queue.

valnoel assigned cchampet Aug 31, 2017

valnoel changed the title ~~Fix: different frame size as inputs of the filter graph~~ [WIP] Fix: different frame size as inputs of the filter graph Sep 5, 2017

Valentin NOEL added 4 commits September 5, 2017 16:43

FilterGraph: add new internal FrameBuffer class

80017d6

Used as frame buffers on the filter graph inputs. For the moment, this first version only handles audio frames.

FilterGraph: use FrameBuffers for each filter graph input

cb8ae5d

Returning frames with the same size as inputs of the graph, avoiding to overflow the internal filter buffers.

FilterGraph: improve input frame size computing

3864d7e

FilterGraph: frame buffers can be bypassed if not necessary

053f9be

If the input frames have the same size and the buffers are empty, bypass to avoid useless data copies.

cchampet reviewed Sep 6, 2017

View reviewed changes

Valentin NOEL added 6 commits September 6, 2017 12:03

FilterGraph: minor clean into FrameBuffer class

380f238

FilterGraph: refactoring of getMinInputFrameSize() private method

27cf448

Add a new getAvailableFrameSize() private method

StreamTranscoder: flush filter graph buffers after decoding process

82382fc

Add utility methods, documentation and logs into FilterGraph and FrameBuffer classes

StreamTranscoder: reset frame that has not been decoded

cbd4700

Avoid filling the filter graph buffers with wrong data frames. Improve documentation into FilterGraph::FrameBuffer class.

FilterGraph: renames FrameBuffer to AudioFrameBuffer

06b8b45

Since video frames cannot be split, this filter graph buffers will be specialized to audio frames.

FilterGraph: remove useless areInputAudioFrames() method

969c6c3

Replace it by a simple AudioFrameBuffer vector empty() test, since buffers are initialized only for audio frames (into the addInBuffer() method) And rename _inputAudioFramesBuffer attribute to _inputAudioFrameBuffers

Valentin NOEL added 4 commits September 8, 2017 10:43

FilterGraph: fix AudioFrameBuffer class name case typo

c823ad6

FilterGraph: fix AudioFrameBuffers order

d2a7f54

FilterGraph: fix audio frames management with different sample formats

2dcdc1c

Comparing the frame sizes in samples (instead of bytes)

Test: add a new python test for audio channels muxing

343ef4e

Audio channels are extracted from different file format sources, to stereo and 5.1 output audio streams.

avTranscoder deleted a comment from coveralls Sep 8, 2017

valnoel changed the title ~~[WIP] Fix: different frame size as inputs of the filter graph~~ Fix: different frame size as inputs of the filter graph Sep 11, 2017

valnoel changed the title ~~Fix: different frame size as inputs of the filter graph~~ [WIP] Fix: different frame size as inputs of the filter graph Sep 11, 2017

FilterGraph: fixes Python binding compilation on Windows

a26bae9

Export AudioFrameBuffer class symbol for DLL

valnoel changed the title ~~[WIP] Fix: different frame size as inputs of the filter graph~~ Fix: different frame size as inputs of the filter graph Sep 11, 2017

cchampet reviewed Sep 12, 2017

View reviewed changes

Valentin Noel added 2 commits September 12, 2017 16:31

Test: check output files duration directly through file properties

9abcb90

Into audio channels muxing tests

StreamTranscoder: avoid checking every filter graph buffer if the pro…

0e49751

…cess won't be continued

cchampet reviewed Sep 21, 2017

View reviewed changes

valnoel merged commit d33ffd2 into avTranscoder:develop Sep 21, 2017

Fix: different frame size as inputs of the filter graph #303

Fix: different frame size as inputs of the filter graph #303

Uh oh!

Conversation

valnoel commented Aug 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Sep 7, 2017

Uh oh!

coveralls commented Sep 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 9, 2017

Uh oh!

coveralls commented Sep 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 9, 2017

Uh oh!

coveralls commented Sep 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Sep 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 20, 2017

Uh oh!

coveralls commented Sep 20, 2017

Uh oh!

valnoel commented Sep 21, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cchampet commented Sep 21, 2017

Uh oh!

valnoel commented Sep 21, 2017

Uh oh!

Uh oh!

valnoel commented Aug 31, 2017 •

edited

Loading

coveralls commented Sep 9, 2017 •

edited

Loading

coveralls commented Sep 9, 2017 •

edited

Loading

coveralls commented Sep 9, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 20, 2017 •

edited

Loading