Skip to content

Use input rotation and translation in camera calibration #20219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: 4.x
Choose a base branch
from
Open

Use input rotation and translation in camera calibration #20219

wants to merge 2 commits into from
+314 −52

Conversation

thezane
Copy link
Contributor

@thezane thezane commented Jun 6, 2021 •
edited
Loading

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
  • The PR is proposed to proper branch
  • There is reference to original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

Description

Add a flag CALIB_FIX_EXTRINSIC to calibrateCamera that allows users to fix input rotation and translation parameters during calibration. Unit tests are added to verify calibration results and format of extrinsic arguments . This PR is a response to #15781.

FAQ

The requester in #15781 asked for separate flags to fix rotation and translation. Why does this PR only allow fixing rotation and translation together?

When testing calibration results with calib1.data and calib2.data, I noticed the LM solver struggles to arrive at a good calibration result when only rotation is fixed but not when rotation and translation are fixed.

The reason is opencv performs calibration with Zhang's method, which estimates camera matrix first and translation matrices from camera matrix after. This forces the LM solver to optimize both the camera and translation matrices, which it's unable to do as the initial guess for the camera and translation matrices are a poor fit for the input rotation matrix. Likewise when only translation is fixed.

In contrast, when fixing both rotation and translation, the LM solver only has to optimize the camera matrix and is able to arrive at a good calibration result.

force_builders_only=linux,docs

@thezane thezane force-pushed the use-input-rotation-translation-in-calibration branch 9 times, most recently from c87d30f to abe2206 Compare June 6, 2021 18:37
@thezane thezane marked this pull request as ready for review June 6, 2021 21:31
@thezane thezane force-pushed the use-input-rotation-translation-in-calibration branch 3 times, most recently from 32cbf34 to 4b1eaa1 Compare June 7, 2021 05:18
@thezane thezane force-pushed the use-input-rotation-translation-in-calibration branch from 4b1eaa1 to 4c57506 Compare June 9, 2021 01:28
alalek
alalek reviewed Jun 15, 2021
View reviewed changes
Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution!

modules/calib3d/include/opencv2/calib3d.hpp Outdated
space. Due to its duality, this tuple is equivalent to the position of the calibration pattern with
respect to the camera coordinate space. If @ref CALIB_FIX_EXTRINSIC is specified, rotation
matrices and translation vectors equal in size to the number of images must be initialized and will
be used for calibration.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will be used for calibration

This contradicts with OutputArrayOfArrays rvecs parameter declaration which is critical for bindings generators (Python/Java/etc code).

Perhaps we should make dedicated overload (at least in public headers).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alalek . I see 4 api versions of calibrate camera in calib3d.hpp:

  • calibrateCameraExtended
  • calibrateCamera
  • calibrateCameraROExtended
  • calibrateCameraRO

Here's an initial proposal for the overload:

  • calibrateCameraFEExtended: Based on calibrateCameraROExtended but with rvecs and tvecs as InputOutputArrayOfArrays
  • calibrateCameraFE: Based on calibrateCameraRO but with rvecs and tvecs as InputOutputArrayOfArrays

The "FE" stands for "FixedExtrinsics". This is similar to how calibrateCameraROExtended and calibrateCameraRO are declared and how "RO" stands for "ReleasingObject". I selected InputOutputArrayOfArrays instead of InputArrayOfArrays in case one day we want the user to be able to pass in an initial guess for extrinsics that will be optimized by the LM solver.

Basing the extension on calibrateCameraRO also allows the method of releasing object to be combined with fixed extrinsics.

What's your opinion?

@thewoz
Copy link
Contributor

thewoz commented Mar 10, 2022

Hi @alalek what about this?
Can I do something?

@asmorkalov
Copy link
Contributor

@thezane @alalek friendly reminder.

@thezane
Copy link
Contributor Author

thezane commented May 26, 2022

Thanks for the reminder. I'll propose a new api soon.

@thezane thezane force-pushed the use-input-rotation-translation-in-calibration branch 2 times, most recently from 14f3a00 to 32942b2 Compare June 6, 2022 03:54
@thezane thezane marked this pull request as draft June 6, 2022 03:55
@thezane thezane force-pushed the use-input-rotation-translation-in-calibration branch 3 times, most recently from cbf30bf to c452d97 Compare July 18, 2022 08:18
@thezane thezane force-pushed the use-input-rotation-translation-in-calibration branch 2 times, most recently from c7adc4c to 180fb9c Compare August 28, 2022 19:19
@thezane thezane force-pushed the use-input-rotation-translation-in-calibration branch from 180fb9c to eb913c5 Compare August 28, 2022 23:20
@thezane thezane marked this pull request as ready for review August 29, 2022 07:47
thezane
thezane commented Aug 29, 2022
View reviewed changes
modules/calib3d/include/opencv2/calib3d.hpp
InputOutputArray cameraMatrix, InputOutputArray distCoeffs,
InputOutputArrayOfArrays rvecs, InputOutputArrayOfArrays tvecs,
int flags = 0, TermCriteria criteria = TermCriteria(
TermCriteria::COUNT + TermCriteria::EPS, 30, DBL_EPSILON) );
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alalek @asmorkalov I propose a dedicated api to support passing in extrinsics. Can you see if you're agreeable with the api?

The extrinsics type is InputOutputArrayOfArrays, which allows OpenCV to modify it in case we want to support an initial guess for the extrinsics in the future.

@asmorkalov asmorkalov modified the milestones: 4.7.0, 4.8.0 Dec 12, 2022
@thewoz
Copy link
Contributor

thewoz commented Jan 30, 2023

Hi, any news about this?

@asmorkalov asmorkalov modified the milestones: 4.8.0, 4.9.0 May 5, 2023
@asmorkalov asmorkalov modified the milestones: 4.9.0, 4.10.0 Dec 1, 2023
@asmorkalov asmorkalov modified the milestones: 4.10.0, 4.11.0 May 16, 2024
@asmorkalov asmorkalov modified the milestones: 4.11.0, 4.12.0 Dec 18, 2024
@thewoz
Copy link
Contributor

thewoz commented May 28, 2025

Hi everyone is there anything I can do about this pull request?
@asmorkalov

@asmorkalov asmorkalov modified the milestones: 4.12.0, 4.13.0 Jun 3, 2025
asmorkalov
asmorkalov reviewed Jul 24, 2025
View reviewed changes
modules/calib3d/include/opencv2/calib3d.hpp
@@ -517,6 +517,7 @@ enum { CALIB_NINTRINSIC = 18,
CALIB_FIX_TAUX_TAUY = 0x80000,
CALIB_USE_QR = 0x100000, //!< use QR instead of SVD decomposition for solving. Faster but potentially less precise
CALIB_FIX_TANGENT_DIST = 0x200000,
CALIB_FIX_EXTRINSIC = (1 << 23),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is CALIB_RECOMPUTE_EXTRINSIC for fisheye. Also need to align with new flags in 5.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asmorkalov asmorkalov asmorkalov left review comments

@alalek alalek alalek left review comments

Assignees
No one assigned
Labels
category: calib3d feature pr: Discussion Required test
Projects
None yet
Milestone
4.13.0
Development

Successfully merging this pull request may close these issues.
4 participants
@thezane @thewoz @asmorkalov @alalek