Skip to content

Commit 996680d

Browse files
torvaldstorvalds
committed
Merge tag 'media/v4.20-7' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
Pull more media updates from Mauro Carvalho Chehab: "The Intel IPU3 camera driver" * tag 'media/v4.20-7' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media: (23 commits) media: staging/ipu3-imgu: Add MAINTAINERS entry media: staging/ipu3-imgu: Address documentation comments media: v4l: Add Intel IPU3 meta buffer formats media: doc-rst: Add Intel IPU3 documentation media: ipu3-imgu: Fix firmware binary location media: ipu3-imgu: Fix compiler warnings media: staging/intel-ipu3: Add dual pipe support media: staging/intel-ipu3: Add Intel IPU3 meta data uAPI media: staging/intel-ipu3: Add imgu top level pci device driver media: staging/intel-ipu3: Add v4l2 driver based on media framework media: staging/intel-ipu3: Add css pipeline programming media: staging/intel-ipu3: css: Initialize css hardware media: staging/intel-ipu3: css: Compute and program ccs media: staging/intel-ipu3: css: Add static settings for image pipeline media: staging/intel-ipu3: css: Add support for firmware management media: staging/intel-ipu3: css: Add dma buff pool utility functions media: staging/intel-ipu3: Implement DMA mapping functions media: staging/intel-ipu3: mmu: Implement driver media: staging/intel-ipu3: abi: Add structs media: staging/intel-ipu3: abi: Add register definitions and enum ...
2 parents 5813540 + 38b11be commit 996680d

40 files changed

+24661
-19
lines changed

‎Documentation/media/uapi/v4l/buffer.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -472,6 +472,9 @@ enum v4l2_buf_type
472472
* - ``V4L2_BUF_TYPE_META_CAPTURE``
473473
- 13
474474
- Buffer for metadata capture, see :ref:`metadata`.
475+
* - ``V4L2_BUF_TYPE_META_OUTPUT``
476+
- 14
477+
- Buffer for metadata output, see :ref:`metadata`.
475478

476479

477480

‎Documentation/media/uapi/v4l/dev-meta.rst

Lines changed: 20 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,21 +14,27 @@ Metadata Interface
1414
******************
1515

1616
Metadata refers to any non-image data that supplements video frames with
17-
additional information. This may include statistics computed over the image
18-
or frame capture parameters supplied by the image source. This interface is
19-
intended for transfer of metadata to userspace and control of that operation.
17+
additional information. This may include statistics computed over the image,
18+
frame capture parameters supplied by the image source or device specific
19+
parameters for specifying how the device processes images. This interface is
20+
intended for transfer of metadata between the userspace and the hardware and
21+
control of that operation.
2022

21-
The metadata interface is implemented on video capture device nodes. The device
22-
can be dedicated to metadata or can implement both video and metadata capture
23-
as specified in its reported capabilities.
23+
The metadata interface is implemented on video device nodes. The device can be
24+
dedicated to metadata or can support both video and metadata as specified in its
25+
reported capabilities.
2426

2527
Querying Capabilities
2628
=====================
2729

28-
Device nodes supporting the metadata interface set the ``V4L2_CAP_META_CAPTURE``
29-
flag in the ``device_caps`` field of the
30+
Device nodes supporting the metadata capture interface set the
31+
``V4L2_CAP_META_CAPTURE`` flag in the ``device_caps`` field of the
3032
:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP`
31-
ioctl. That flag means the device can capture metadata to memory.
33+
ioctl. That flag means the device can capture metadata to memory. Similarly,
34+
device nodes supporting metadata output interface set the
35+
``V4L2_CAP_META_OUTPUT`` flag in the ``device_caps`` field of
36+
:c:type:`v4l2_capability` structure. That flag means the device can read
37+
metadata from memory.
3238

3339
At least one of the read/write or streaming I/O methods must be supported.
3440

@@ -42,10 +48,11 @@ to the basic :ref:`format` ioctls, the :c:func:`VIDIOC_ENUM_FMT` ioctl must be
4248
supported as well.
4349

4450
To use the :ref:`format` ioctls applications set the ``type`` field of the
45-
:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_META_CAPTURE`` and use the
46-
:c:type:`v4l2_meta_format` ``meta`` member of the ``fmt`` union as needed per
47-
the desired operation. Both drivers and applications must set the remainder of
48-
the :c:type:`v4l2_format` structure to 0.
51+
:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_META_CAPTURE`` or to
52+
``V4L2_BUF_TYPE_META_OUTPUT`` and use the :c:type:`v4l2_meta_format` ``meta``
53+
member of the ``fmt`` union as needed per the desired operation. Both drivers
54+
and applications must set the remainder of the :c:type:`v4l2_format` structure
55+
to 0.
4956

5057
.. c:type:: v4l2_meta_format
5158

‎Documentation/media/uapi/v4l/meta-formats.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ These formats are used for the :ref:`metadata` interface only.
1919
.. toctree::
2020
:maxdepth: 1
2121

22+
pixfmt-meta-intel-ipu3
2223
pixfmt-meta-d4xx
2324
pixfmt-meta-uvc
2425
pixfmt-meta-vsp1-hgo

‎Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
.. -*- coding: utf-8; mode: rst -*-
2+
3+
.. _v4l2-meta-fmt-params:
4+
.. _v4l2-meta-fmt-stat-3a:
5+
6+
******************************************************************
7+
V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
8+
******************************************************************
9+
10+
.. c:type:: ipu3_uapi_stats_3a
11+
12+
3A statistics
13+
=============
14+
15+
For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
16+
an input bayer frame. Those statistics, defined in data struct :c:type:`ipu3_uapi_stats_3a`,
17+
are obtained from "ipu3-imgu 3a stat" metadata capture video node, which are then
18+
passed to user space for statistics analysis using :c:type:`v4l2_meta_format` interface.
19+
20+
The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and
21+
Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,
22+
and AE (Auto-exposure) histogram.
23+
24+
struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above.
25+
26+
.. code-block:: c
27+
28+
struct ipu3_uapi_stats_3a {
29+
struct ipu3_uapi_awb_raw_buffer awb_raw_buffer;
30+
struct ipu3_uapi_ae_raw_buffer_aligned ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
31+
struct ipu3_uapi_af_raw_buffer af_raw_buffer;
32+
struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
33+
struct ipu3_uapi_4a_config stats_4a_config;
34+
__u32 ae_join_buffers;
35+
__u8 padding[28];
36+
struct ipu3_uapi_stats_3a_bubble_info_per_stripe stats_3a_bubble_per_stripe;
37+
struct ipu3_uapi_ff_status stats_3a_status;
38+
};
39+
40+
.. c:type:: ipu3_uapi_params
41+
42+
Pipeline parameters
43+
===================
44+
45+
IPU3 pipeline has a number of image processing stages, each of which takes a
46+
set of parameters as input. The major stages of pipelines are shown here:
47+
48+
Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
49+
50+
Linearization -> Lens Shading Correction -> White Balance / Exposure /
51+
52+
Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
53+
54+
Correction Matrix -> Gamma correction -> Color Space Conversion ->
55+
56+
Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
57+
58+
Correction -> XNR3 -> TNR -> DDR
59+
60+
The table below presents a description of the above algorithms.
61+
62+
======================== =======================================================
63+
Name Description
64+
======================== =======================================================
65+
Optical Black Correction Optical Black Correction block subtracts a pre-defined
66+
value from the respective pixel values to obtain better
67+
image quality.
68+
Defined in :c:type:`ipu3_uapi_obgrid_param`.
69+
Linearization This algo block uses linearization parameters to
70+
address non-linearity sensor effects. The Lookup table
71+
table is defined in
72+
:c:type:`ipu3_uapi_isp_lin_vmem_params`.
73+
SHD Lens shading correction is used to correct spatial
74+
non-uniformity of the pixel response due to optical
75+
lens shading. This is done by applying a different gain
76+
for each pixel. The gain, black level etc are
77+
configured in :c:type:`ipu3_uapi_shd_config_static`.
78+
BNR Bayer noise reduction block removes image noise by
79+
applying a bilateral filter.
80+
See :c:type:`ipu3_uapi_bnr_static_config` for details.
81+
ANR Advanced Noise Reduction is a block based algorithm
82+
that performs noise reduction in the Bayer domain. The
83+
convolution matrix etc can be found in
84+
:c:type:`ipu3_uapi_anr_config`.
85+
Demosaicing Demosaicing converts raw sensor data in Bayer format
86+
into RGB (Red, Green, Blue) presentation. Then add
87+
outputs of estimation of Y channel for following stream
88+
processing by Firmware. The struct is defined as
89+
:c:type:`ipu3_uapi_dm_config`. (TODO)
90+
Color Correction Color Correction algo transforms sensor specific color
91+
space to the standard "sRGB" color space. This is done
92+
by applying 3x3 matrix defined in
93+
:c:type:`ipu3_uapi_ccm_mat_config`.
94+
Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
95+
basic non-linear tone mapping correction that is
96+
applied per pixel for each pixel component.
97+
CSC Color space conversion transforms each pixel from the
98+
RGB primary presentation to YUV (Y: brightness,
99+
UV: Luminance) presentation. This is done by applying
100+
a 3x3 matrix defined in
101+
:c:type:`ipu3_uapi_csc_mat_config`
102+
CDS Chroma down sampling
103+
After the CSC is performed, the Chroma Down Sampling
104+
is applied for a UV plane down sampling by a factor
105+
of 2 in each direction for YUV 4:2:0 using a 4x2
106+
configurable filter :c:type:`ipu3_uapi_cds_params`.
107+
CHNR Chroma noise reduction
108+
This block processes only the chrominance pixels and
109+
performs noise reduction by cleaning the high
110+
frequency noise.
111+
See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
112+
TCC Total color correction as defined in struct
113+
:c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
114+
XNR3 eXtreme Noise Reduction V3 is the third revision of
115+
noise reduction algorithm used to improve image
116+
quality. This removes the low frequency noise in the
117+
captured image. Two related structs are being defined,
118+
:c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
119+
and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
120+
memory.
121+
TNR Temporal Noise Reduction block compares successive
122+
frames in time to remove anomalies / noise in pixel
123+
values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
124+
:c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
125+
vector and data memory respectively.
126+
======================== =======================================================
127+
128+
A few stages of the pipeline will be executed by firmware running on the ISP
129+
processor, while many others will use a set of fixed hardware blocks also
130+
called accelerator cluster (ACC) to crunch pixel data and produce statistics.
131+
132+
ACC parameters of individual algorithms, as defined by
133+
:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
134+
space through struct :c:type:`ipu3_uapi_flags` embedded in
135+
:c:type:`ipu3_uapi_params` structure. For parameters that are configured as
136+
not enabled by the user space, the corresponding structs are ignored by the
137+
driver, in which case the existing configuration of the algorithm will be
138+
preserved.
139+
140+
Both 3A statistics and pipeline parameters described here are closely tied to
141+
the underlying camera sub-system (CSS) APIs. They are usually consumed and
142+
produced by dedicated user space libraries that comprise the important tuning
143+
tools, thus freeing the developers from being bothered with the low level
144+
hardware and algorithm details.
145+
146+
It should be noted that IPU3 DMA operations require the addresses of all data
147+
structures (that includes both input and output) to be aligned on 32 byte
148+
boundaries.
149+
150+
The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"
151+
video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
152+
153+
.. code-block:: c
154+
155+
struct ipu3_uapi_params {
156+
/* Flags which of the settings below are to be applied */
157+
struct ipu3_uapi_flags use;
158+
159+
/* Accelerator cluster parameters */
160+
struct ipu3_uapi_acc_param acc_param;
161+
162+
/* ISP vector address space parameters */
163+
struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
164+
struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
165+
struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
166+
167+
/* ISP data memory (DMEM) parameters */
168+
struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
169+
struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
170+
171+
/* Optical black level compensation */
172+
struct ipu3_uapi_obgrid_param obgrid_param;
173+
};
174+
175+
Intel IPU3 ImgU uAPI data types
176+
===============================
177+
178+
.. kernel-doc:: drivers/staging/media/ipu3/include/intel-ipu3.h

‎Documentation/media/uapi/v4l/vidioc-querycap.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,9 @@ specification the ioctl returns an ``EINVAL`` error code.
258258
* - ``V4L2_CAP_STREAMING``
259259
- 0x04000000
260260
- The device supports the :ref:`streaming <mmap>` I/O method.
261+
* - ``V4L2_CAP_META_OUTPUT``
262+
- 0x08000000
263+
- The device supports the :ref:`metadata` output interface.
261264
* - ``V4L2_CAP_TOUCH``
262265
- 0x10000000
263266
- This is a touch device.

‎Documentation/media/v4l-drivers/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ For more details see the file COPYING in the source distribution of Linux.
4444
davinci-vpbe
4545
fimc
4646
imx
47+
ipu3
4748
ivtv
4849
max2175
4950
meye

0 commit comments

Comments
 (0)