Skip to content

Commit 87160cb

Browse files
catreecatree
committed
Add Demo 5: Basic panorama stitching from a rotating camera in the homography tutorial.
1 parent dcdd6af commit 87160cb

File tree

8 files changed

+179
-15
lines changed

8 files changed

+179
-15
lines changed

‎doc/tutorials/features2d/homography/homography.markdown

Lines changed: 77 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
Basic concepts of the homography explained with code {#tutorial_homography}
2-
=============================
2+
====================================================
33

4-
Introduction
5-
----
4+
@tableofcontents
5+
6+
Introduction {#tutorial_homography_Introduction}
7+
============
68

79
This tutorial will demonstrate the basic concepts of the homography with some codes.
810
For detailed explanations about the theory, please refer to a computer vision course or a computer vision book, e.g.:
@@ -13,10 +15,10 @@ For detailed explanations about the theory, please refer to a computer vision co
1315
The tutorial code can be found [here](https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/features2D/Homography).
1416
The images used in this tutorial can be found [here](https://github.com/opencv/opencv/tree/master/samples/data) (`left*.jpg`).
1517

16-
Basic theory
17-
----
18+
Basic theory {#tutorial_homography_Basic_theory}
19+
------------
1820

19-
### What is the homography matrix?
21+
### What is the homography matrix? {#tutorial_homography_What_is_the_homography_matrix}
2022

2123
Briefly, the planar homography relates the transformation between two planes (up to a scale factor):
2224

@@ -61,7 +63,7 @@ The following examples show different kinds of transformation but all relate a t
6163

6264
![](images/homography_transformation_example3.jpg)
6365

64-
### How the homography transformation can be useful?
66+
### How the homography transformation can be useful? {#tutorial_homography_How_the_homography_transformation_can_be_useful}
6567

6668
* Camera pose estimation from coplanar points for augmented reality with marker for instance (see the previous first example)
6769

@@ -75,10 +77,10 @@ The following examples show different kinds of transformation but all relate a t
7577

7678
![](images/homography_panorama_stitching.jpg)
7779

78-
Demonstration codes
79-
----
80+
Demonstration codes {#tutorial_homography_Demonstration_codes}
81+
-------------------
8082

81-
### Demo 1: Pose estimation from coplanar points
83+
### Demo 1: Pose estimation from coplanar points {#tutorial_homography_Demo1}
8284

8385
\note Please note that the code to estimate the camera pose from the homography is an example and you should use instead @ref cv::solvePnP if you want to estimate the camera pose for a planar or an arbitrary object.
8486

@@ -151,11 +153,16 @@ A quick solution to retrieve the pose from the homography matrix is (see \ref po
151153

152154
This is a quick solution (see also \ref projective_transformations "2") as this does not ensure that the resulting rotation matrix will be orthogonal and the scale is estimated roughly by normalize the first column to 1.
153155

156+
A solution to have a proper rotation matrix (with the properties of a rotation matrix) consists to apply a polar decomposition
157+
(see \ref polar_decomposition "6" or \ref polar_decomposition_svd "7" for some information):
158+
159+
@snippet pose_from_homography.cpp polar-decomposition-of-the-rotation-matrix
160+
154161
To check the result, the object frame projected into the image with the estimated camera pose is displayed:
155162

156163
![](images/homography_pose.jpg)
157164

158-
### Demo 2: Perspective correction
165+
### Demo 2: Perspective correction {#tutorial_homography_Demo2}
159166

160167
In this example, a source image will be transformed into a desired perspective view by computing the homography that maps the source points into the desired points.
161168
The following image shows the source image (left) and the chessboard view that we want to transform into the desired chessboard view (right).
@@ -186,7 +193,7 @@ To check the correctness of the calculation, the matching lines are displayed:
186193

187194
![](images/homography_perspective_correction_chessboard_matches.jpg)
188195

189-
### Demo 3: Homography from the camera displacement
196+
### Demo 3: Homography from the camera displacement {#tutorial_homography_Demo3}
190197

191198
The homography relates the transformation between two planes and it is possible to retrieve the corresponding camera displacement that allows to go from the first to the second plane view (see @cite Malis for more information).
192199
Before going into the details that allow to compute the homography from the camera displacement, some recalls about camera pose and homogeneous transformation.
@@ -363,7 +370,7 @@ The homography matrices are similar. If we compare the image 1 warped using both
363370

364371
Visually, it is hard to distinguish a difference between the result image from the homography computed from the camera displacement and the one estimated with @ref cv::findHomography function.
365372

366-
### Demo 4: Decompose the homography matrix
373+
### Demo 4: Decompose the homography matrix {#tutorial_homography_Demo4}
367374

368375
OpenCV 3 contains the function @ref cv::decomposeHomographyMat which allows to decompose the homography matrix to a set of rotations, translations and plane normals.
369376
First we will decompose the homography matrix computed from the camera displacement:
@@ -457,11 +464,66 @@ plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.75248
457464

458465
Again, there is also a solution that matches with the computed camera displacement.
459466

460-
Additional references
461-
----
467+
### Demo 5: Basic panorama stitching from a rotating camera {#tutorial_homography_Demo5}
468+
469+
\note This example is made to illustrate the concept of image stitching based on a pure rotational motion of the camera and should not be used to stitch panorama images.
470+
The [stitching module](@ref stitching) provides a complete pipeline to stitch images.
471+
472+
The homography transformation applies only for planar structure. But in the case of a rotating camera (pure rotation around the camera axis of projection, no translation), an arbitrary world can be considered
473+
([see previously](@ref tutorial_homography_What_is_the_homography_matrix)).
474+
475+
The homography can then be computed using the rotation transformation and the camera intrinsic parameters as (see for instance \ref homography_course "8"):
476+
477+
\f[
478+
s
479+
\begin{bmatrix}
480+
x^{'} \\
481+
y^{'} \\
482+
1
483+
\end{bmatrix} =
484+
\bf{K} \hspace{0.1em} \bf{R} \hspace{0.1em} \bf{K}^{-1}
485+
\begin{bmatrix}
486+
x \\
487+
y \\
488+
1
489+
\end{bmatrix}
490+
\f]
491+
492+
To illustrate, we used Blender, a free and open-source 3D computer graphics software, to generate two camera views with only a rotation transformation between each other.
493+
More information about how to retrieve the camera intrinsic parameters and the `3x4` extrinsic matrix with respect to the world can be found in \ref answer_blender "9" (an additional transformation
494+
is needed to get the transformation between the camera and the object frames) with Blender.
495+
496+
The figure below shows the two generated views of the Suzanne model, with only a rotation transformation:
497+
498+
![](images/homography_stitch_compare.jpg)
499+
500+
With the known associated camera poses and the intrinsic parameters, the relative rotation between the two views can be computed:
501+
502+
@snippet panorama_stitching_rotating_camera.cpp extract-rotation
503+
504+
@snippet panorama_stitching_rotating_camera.cpp compute-rotation-displacement
505+
506+
Here, the second image will be stitched with respect to the first image. The homography can be calculated using the formula above:
507+
508+
@snippet panorama_stitching_rotating_camera.cpp compute-homography
509+
510+
The stitching is made simply with:
511+
512+
@snippet panorama_stitching_rotating_camera.cpp stitch
513+
514+
The resulting image is:
515+
516+
![](images/homography_stitch_Suzanne.jpg)
517+
518+
Additional references {#tutorial_homography_Additional_references}
519+
---------------------
462520

463521
* \anchor lecture_16 1. [Lecture 16: Planar Homographies](http://www.cse.psu.edu/~rtc12/CSE486/lecture16.pdf), Robert Collins
464522
* \anchor projective_transformations 2. [2D projective transformations (homographies)](https://ags.cs.uni-kl.de/fileadmin/inf_ags/3dcv-ws11-12/3DCV_WS11-12_lec04.pdf), Christiano Gava, Gabriele Bleser
465523
* \anchor szeliski 3. [Computer Vision: Algorithms and Applications](http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf), Richard Szeliski
466524
* \anchor answer_dsp 4. [Step by Step Camera Pose Estimation for Visual Tracking and Planar Markers](https://dsp.stackexchange.com/a/2737)
467525
* \anchor pose_ar 5. [Pose from homography estimation](https://team.inria.fr/lagadic/camera_localization/tutorial-pose-dlt-planar-opencv.html)
526+
* \anchor polar_decomposition 6. [Polar Decomposition (in Continuum Mechanics)](http://www.continuummechanics.org/polardecomposition.html)
527+
* \anchor polar_decomposition_svd 7. [A Personal Interview with the Singular Value Decomposition](https://web.stanford.edu/~gavish/documents/SVD_ans_you.pdf), Matan Gavish
528+
* \anchor homography_course 8. [Homography](http://people.scs.carleton.ca/~c_shu/Courses/comp4900d/notes/homography.pdf), Dr. Gerhard Roth
529+
* \anchor answer_blender 9. [3x4 camera matrix from blender camera](https://blender.stackexchange.com/a/38210)

‎doc/tutorials/features2d/homography/images/homography_stitch_Suzanne.jpg

40.6 KB
Loading

‎doc/tutorials/features2d/homography/images/homography_stitch_compare.jpg

50.5 KB
Loading

‎modules/calib3d/include/opencv2/calib3d.hpp

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1885,6 +1885,12 @@ CV_EXPORTS_W cv::Mat estimateAffinePartial2D(InputArray from, InputArray to, Out
18851885
size_t maxIters = 2000, double confidence = 0.99,
18861886
size_t refineIters = 10);
18871887

1888+
/** @example decompose_homography.cpp
1889+
An example program with homography decomposition.
1890+
1891+
Check @ref tutorial_homography "the corresponding tutorial" for more details.
1892+
*/
1893+
18881894
/** @brief Decompose a homography matrix to rotation(s), translation(s) and plane normal(s).
18891895
18901896
@param H The input homography matrix between two images.

‎samples/cpp/tutorial_code/features2D/Homography/panorama_stitching_rotating_camera.cpp

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
#include <iostream>
2+
#include <opencv2/opencv.hpp>
3+
4+
using namespace std;
5+
using namespace cv;
6+
7+
namespace
8+
{
9+
void basicPanoramaStitching(const string &img1Path, const string &img2Path)
10+
{
11+
Mat img1 = imread(img1Path);
12+
Mat img2 = imread(img2Path);
13+
14+
//! [camera-pose-from-Blender-at-location-1]
15+
Mat c1Mo = (Mat_<double>(4,4) << 0.9659258723258972, 0.2588190734386444, 0.0, 1.5529145002365112,
16+
0.08852133899927139, -0.3303661346435547, -0.9396926164627075, -0.10281121730804443,
17+
-0.24321036040782928, 0.9076734185218811, -0.342020183801651, 6.130080699920654,
18+
0, 0, 0, 1);
19+
//! [camera-pose-from-Blender-at-location-1]
20+
21+
//! [camera-pose-from-Blender-at-location-2]
22+
Mat c2Mo = (Mat_<double>(4,4) << 0.9659258723258972, -0.2588190734386444, 0.0, -1.5529145002365112,
23+
-0.08852133899927139, -0.3303661346435547, -0.9396926164627075, -0.10281121730804443,
24+
0.24321036040782928, 0.9076734185218811, -0.342020183801651, 6.130080699920654,
25+
0, 0, 0, 1);
26+
//! [camera-pose-from-Blender-at-location-2]
27+
28+
//! [camera-intrinsics-from-Blender]
29+
Mat cameraMatrix = (Mat_<double>(3,3) << 700.0, 0.0, 320.0,
30+
0.0, 700.0, 240.0,
31+
0, 0, 1);
32+
//! [camera-intrinsics-from-Blender]
33+
34+
//! [extract-rotation]
35+
Mat R1 = c1Mo(Range(0,3), Range(0,3));
36+
Mat R2 = c2Mo(Range(0,3), Range(0,3));
37+
//! [extract-rotation]
38+
39+
//! [compute-rotation-displacement]
40+
//c1Mo * oMc2
41+
Mat R_2to1 = R1*R2.t();
42+
//! [compute-rotation-displacement]
43+
44+
//! [compute-homography]
45+
Mat H = cameraMatrix * R_2to1 * cameraMatrix.inv();
46+
H /= H.at<double>(2,2);
47+
cout << "H:\n" << H << endl;
48+
//! [compute-homography]
49+
50+
//! [stitch]
51+
Mat img_stitch;
52+
warpPerspective(img2, img_stitch, H, Size(img2.cols*2, img2.rows));
53+
Mat half = img_stitch(Rect(0, 0, img1.cols, img1.rows));
54+
img1.copyTo(half);
55+
//! [stitch]
56+
57+
Mat img_compare;
58+
Mat img_space = Mat::zeros(Size(50, img1.rows), CV_8UC3);
59+
hconcat(img1, img_space, img_compare);
60+
hconcat(img_compare, img2, img_compare);
61+
imshow("Compare images", img_compare);
62+
63+
imshow("Panorama stitching", img_stitch);
64+
waitKey();
65+
}
66+
67+
const char* params
68+
= "{ help h | | print usage }"
69+
"{ image1 | ../data/Blender_Suzanne1.jpg | path to the first Blender image }"
70+
"{ image2 | ../data/Blender_Suzanne2.jpg | path to the second Blender image }";
71+
}
72+
73+
int main(int argc, char *argv[])
74+
{
75+
CommandLineParser parser(argc, argv, params);
76+
77+
if (parser.has("help"))
78+
{
79+
parser.about( "Code for homography tutorial.\n"
80+
"Example 5: basic panorama stitching from a rotating camera.\n" );
81+
parser.printMessage();
82+
return 0;
83+
}
84+
85+
basicPanoramaStitching(parser.get<String>("image1"), parser.get<String>("image2"));
86+
87+
return 0;
88+
}

‎samples/cpp/tutorial_code/features2D/Homography/pose_from_homography.cpp

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,14 @@ void poseEstimationFromCoplanarPoints(const string &imgPath, const string &intri
107107
}
108108
//! [pose-from-homography]
109109

110+
//! [polar-decomposition-of-the-rotation-matrix]
111+
cout << "R (before polar decomposition):\n" << R << "\ndet(R): " << determinant(R) << endl;
112+
Mat W, U, Vt;
113+
SVDecomp(R, W, U, Vt);
114+
R = U*Vt;
115+
cout << "R (after polar decomposition):\n" << R << "\ndet(R): " << determinant(R) << endl;
116+
//! [polar-decomposition-of-the-rotation-matrix]
117+
110118
//! [display-pose]
111119
Mat rvec;
112120
Rodrigues(R, rvec);

‎samples/data/Blender_Suzanne1.jpg

24.8 KB
Loading

‎samples/data/Blender_Suzanne2.jpg

24.8 KB
Loading

0 commit comments

Comments
 (0)