US010666865B2

(12) United
Lyon et al.
States Patent (10 ) Patent No.: US 10,666,865 B2
(45 ) Date of Patent: May 26 , 2020
(54 ) PANORAMIC CAMERA WITH MULTIPLE (56 ) References Cited
IMAGE SENSORS USING TIMED SHUTTERS
U.S. PATENT DOCUMENTS
(71) Applicant: Google LLC , Mountain View , CA (US) 4,062,045 A 12/1977 Iwane
4,532,550 A 7/1985 Bendell et al .
(72 ) Inventors: Richard Francis Lyon , Los Altos , CA (Continued )
(US); Gary Embler , Redwood City ,
CA (US ); Iain Richard Tyrone FOREIGN PATENT DOCUMENTS
McClatchie, Los Altos , CA (US );
Jason Holt, Mountain View , CA (US ) JP 2006217222 A 8/2006
JP 2006350713 A 12/2006
(73 ) Assignee: Google LLC , Mountain View , CA (US) (Continued )
( * ) Notice: Subject to any disclaimer , the term of this OTHER PUBLICATIONS
patent is extended or adjusted under 35
U.S.C. 154(b ) by 0 days . Foote et al, FlyCam : Practical panoramic video and automatic
(21) Appl. No.: 16 /449,917 camera control ( Year: 2000).*
(Continued )
(22) Filed : Jun . 24 , 2019
Primary Examiner Shan E Elahi
(65) Prior Publication Data ( 74 ) Attorney , Agent, or Firm — Lerner, David ,
US 2019/0313022 A1 Oct. 10 , 2019 Littenberg , Krumholz & Mentlik , LLP
Related U.S. Application Data (57) ABSTRACT
(60 ) Continuation of application No. 15 /711,128, filed on The present invention relates to the field of panoramic still
Sep. 21, 2017 , now Pat. No. 10,397,476 , which is a and motion photography . In a first embodiment, a camera
(Continued )
apparatus for panoramic photography includes a first image
sensor positioned to capture a first image. The first image
(51) Int. Ci. sensor has a rolling -shutter readout arranged in portrait
H04N 7/18 ( 2006.01) orientation . The camera apparatus also includes second
H04N 5/232 (2006.01) image sensor positioned to capture a second image. The
H04N 5/235 (2006.01) second image sensor has a rolling-shutter readout arranged
(52 ) U.S. Cl. in portrait orientation . Finally , the camera apparatus includes
a controller configured to signal the second image sensor to
CPC H04N 5/23238 (2013.01 ); H04N 5/2353 start capturing the second image before the first image
(2013.01); H04N 5/23203 (2013.01 ) sensor finishes capturing the first image . At least a portion of
( 58 ) Field of Classification Search the first image is in front of the second image relative to a
CPC HO4N 5/23203 ; H04N 5/23238 ; HO4N forward direction of the camera apparatus.
5/2353
(Continued ) 20 Claims, 6 Drawing Sheets

*
R5 Cameras Horizontal Scan Sequence

23

We Scan axis of symmetry

AO*#9eo Bcam&HiFOV

FOVHCam4 .

1. Muerbeted arrows incicale scan drection

2. Numberon atrows indicates relativo 2007 mart inte
2.0 is camera horizontal Ro ¥ angle
 US 10,666,865 B2
Page 2

Related U.S. Application Data 2007/0081091 A1

2007/0109408 A1
4/2007 Pan et al.
5/2007 Watanabe et al.
continuation of application No. 15 /006,449 , filed on 2007/0116453 Al 5/2007 Uchiyama
Jan. 26 , 2016 , now Pat. No. 9,794,479, which is a 2007/0182812 A1 8/2007 Ritchey
continuation of application No. 13 /780,493 , filed on 2007/0206945 Al 9/2007 DeLorme et al.
Feb. 28 , 2013 , now abandoned , which is a division of 2007/0237364 A1 * 10/2007 Song G06K 9/00369
382/115
application No. 12 /368,014 , filed on Feb. 9 , 2009, 2007/0263117 A1 11/2007 Kera
now Pat. No. 8,493,436 . 2008/0002914 A1 * 1/2008 Vincent G06K 9/3258
382/299
(60 ) Provisional application No.61/027,237, filed on Feb. 2008/0018774
2008/0027599
A1
Al
1/2008 Nakajima et al.
1/2008 Logan et al.
8 , 2008. 2008/0046150 A1 2/2008 Breed
( 58 ) Field of Classification Search 2008/0117287 Al 5/2008 Park et al.
USPC 348/38 2008/0117288 Al 5/2008 Park
See application file for complete search history. 2008/0303894 Al 12/2008 Ernst et al.
2009/0179895 A1 * 7/2009 Zhu G06T 19/00
( 56 ) References Cited 345/424
2009/0262196 A1 10/2009 Salivar et al.
U.S. PATENT DOCUMENTS 2009/0295971 A1 12/2009 Tsuchiya
2009/0322856 A1 12/2009 Duparre
5,023,723 A 6/1991 Date et al. 2010/0002073 A1 1/2010 Robinson et al .
5,128,767 A 7/1992 Suzuki et al. 2010/0066897 Al 3/2010 Miyanari
5,473,364 A 12/1995 Burt 2010/0133424 A1 6/2010 Lindsay
5,495,576 A 2/1996 Ritchey 2010/0165160 A1 7/2010 Olmstead et al .
5,657,073 A 8/1997 Henley 2010/0328435 Al 12/2010 Puah et al.
5,703,961 A 12/1997 Rogina et al. 2011/0178756 A1 7/2011 Anderson
5,801,760 A 9/1998 Uomori 2011/0211040 A1 9/2011 Lindemann et al.
6,108,440 A 8/2000 Baba et al . 2011/0304746 Al 12/2011 Iijima et al .
6,118,475 A 9/2000 Iijima et al. 2012/0016536 A1 1/2012 Patel et al.
6,141,034 A 10/2000 McCutchen 2012/0019670 A1 1/2012 Chang et al.
6,327,381 B1 12/2001 Rogina et al. 2012/0075414 Al 3/2012 Park et al.
6,535,114 B1 3/2003 Suzuki et al . 2019/0004535 A1 1/2019 Huang et al.
6,593,960 B1 7/2003 Sugimoto et al.
6,669,346 B2 12/2003 Metcalf FOREIGN PATENT DOCUMENTS
6,778,207 B1 8/2004 Lee et al.
6,795,113 B1 9/2004 Jackson et al. JP 2007-214620 A 8/2007
6,839,081 B1 1/2005 Iijima et al. WO 2007045714 A1 4/2007
6,947,059 B2 9/2005 Pierce et al.
6,993,159 B1 1/2006 Ishii et al.
7,012,637 B1 3/2006 Blume et al. OTHER PUBLICATIONS
7,015,951 B1 3/2006 Yoshigahara et al.
7,277,118 B2 10/2007 Foote Niclass, et al., “ Design and Characterization of a CMOS 3 - D Image
7,307,655 B1 12/2007 Okamoto et al. Sensor Based on Single Photon Avalanche Diodes,” IEEE Journal of
7,388,598 B2 6/2008 Lee et al . Solid -StateCircuits, vol. 40, No. 9 , Sep. 2005; pp . 1847-1854 .
7,436,429 B2 10/2008 Tillotson
7,567,293 B2 7/2009 Perlman et al. Svoboda , et al., “ Panoramic Cameras for 3D Computation ,” Czech
7,697,028 B1 4/2010 Johnson Pattern Recognition Workshop 2000 , Feb. 2-4 , 2000 ; 8 Pages.
7,710,463 B2 5/2010 Foote Wilburn et al., “ High -Speed Videography Using a Dense Camera
7,990,394 B2 8/2011 Vincent et al. Array,” IEEE Society Conference on Pattern Recognition , 2004 ; 8
8,027,531 B2 9/2011 Wilburn et al. Pages.
8,094,182 B2 1/2012 Park et al. Invitation to Pay Additional Fees and , Where Applicable, Protest
8,493,436 B2 7/2013 Lyon et al. Fee Including Partial International Search in PCT Application No.
8,937,643 B1 1/2015 Esteban et al .
8,941,777 B1 1/2015 Ratner PCT/US2009/000804, dated May 6 , 2009; 4 Pages .
9,253,480 B2 2/2016 Ohba et al. The State Intellectual Property Office of the People's Republic of
10,244,226 B2 3/2019 Weaver et al. China , “ Notification of the First Office Action ,” Chinese Appln . No.
2001/0015751 A1 8/2001 Geng 200980110758.8, dated Jun . 21, 2012 ; with English - language trans
2001/0038413 Al 11/2001 Peleg et al. lation , 13 pages.
2002/0102101 A1 8/2002 Pelletier Japanese Patent Office ,“ Notice of Reasons for Rejection ,” Japanese
2002/0122113 Al 9/2002 Foote
2002/0124260 A1 9/2002 Rivera Patent Appln . No. 2010-545890 , dated Sep. 28 , 2012 ( dated Oct. 2 ,
2004/0001138 A1 1/2004 Weerashinghe et al. 2012 ): with English -language translation , 7 pages.
2004/0027451 A1 2/2004 Baker " Circle - Vision 360 degree.," Wikipedia , accessed at http :// en .
2004/0100443 A1 5/2004 Mandelbaum et al. wikipedia.org/wiki/Circle-Vision.sub.—360%C2%B0, accessed on
2004/0160512 Al 8/2004 Lee et al . Feb. 5 , 2008 , 4 pages.
2004/0202381 A1 10/2004 Kitajima Examination Report, European Application No. 09708701.9 , Euro
2004/0212723 A1 10/2004 Lin pean Patent Office, Netherlands, dated Jan. 10 , 2012 , 5 pages.
2005/0018253 A1 1/2005 Takeda International Preliminary Report on Patentability and Written Opin
2005/0035297 A1 2/2005 Crosetto ion for International Application No. PCT/US2009/ 000804 , Inter
2005/0110869 A1 5/2005 Tillotson national Bureau of WIPO , Switzerland, dated Aug. 19 , 2010 .
2006/0028550 A1 2/2006 Palmer et al.
2006/0033813 Al 2/2006 Provinsal et al. International Search Report and Written Opinion for International
2006/0033966 A1 2/2006 Yamamoto et al. Application No. PCT/US2009 /000804 , European Patent Office,
2006/0125921 A1 6/2006 Foote Netherlands, dated Jul. 13 , 2009 .
2006/0157760 A1 7/2006 Hayashi et al. Australian Patent Examination Report directed to related Australian
2006/0268103 A1 11/2006 Kweon et al. Application No. 2009210672 , dated Mar. 14 , 2013 ; 4 pages.
2007/0014551 A1 1/2007 Fujisawa et al. Second Office Action directed to related Chinese Patent Application
2007/0030342 A1 2/2007 Wilburn et al. No. 200980110758.8 , dated Mar. 25 , 2013 ; 4 pages.
 US 10,666,865 B2
Page 3

( 56 ) References Cited
OTHER PUBLICATIONS
JPO , Office Action , dated Jul. 2, 2013 , Japanese Patent Application
No. 2013-002323 , 2 pages (English -language translation provided ,
3 pages ).
Japanese Office Action for Application No.2013-002323 dated Mar.
12 , 2014.
English Machine translation of JP 2006-217222 .
Kaszubiak , et al., Real Time, 3 -D -multi Object Position Estimation
and Tracking , 2004 .
Yoon et. al, Non -intrusive Iris Image Capturing System Using Light
Stripe Projection and Pan - Tilt-Zoom Camera , 2007 .
Dinh et al., High Resolution Face Sequence from A PTZ Network
Camera , 2011.
Park et al., Face Tracking and Recognition at a Distance: A Coaxial
and Concentric PTZ Camera System , Oct. 2013.
Gandhi and Trivedi, “ Vehicle Surround Capture : Survey of Tech
niques and a Novel Omni- Video -Based Approach for Dynamic
Panoramic Surround Maps” , IEEE Transactions on Intelligent Trans
portation Systems, vol. 7 , No. 3 , Sep. 2006 , pp . 293-308.
Xiao et al., “Capture of Vehicle Surroundings Using a pair of
Panoramic Annular Lens Cameras,” State Key Laboratory of Mod
ern Optics and Instrument, 2006 .
* cited by examiner
U.S. Patent May 26 , 2020 Sheet 1 of 6 US 10,666,865 B2

100

r ?112

f
114

FIG . 1A

172

******

grande

FIG . 1B
U.S. Patent May 26 , 2020 Sheet 2 of 6 US 10,666,865 B2

mu
220

202
L•*$250
FIG . 2A

22 234

202
206

FIG . 2B

240
228

202
232

FIG . 2C
U.S. Patent May 26, 2020 Sheet 3 of 6 US 10,666,865 B2

Saofscyxmaienstry :2*0-45aFOVoh3.O'scvnreimgzolnraepl
tsria2.NnUedScilTaDometrwins
R5CHSoaercqmiuzantcsel ZOE a1.Mdsiufrnecboatriwensd 2.ischFOVaornimzgeonlrtae
S.

worden
M1.***.* .
****1111!

... FOVH8Cam
********}9} }{:)***
haway vam24-FOV.
Bici168 E-
115345

**

Ca 4 ..
A
FOVCom6H
.

VAKUL 3
.
FIG
U.S. Patent May 26, 2020 Sheet 4 of 6 US 10,666,865 B2

422

426
402

--

FIG.4A
424
450 ?
??.

428 414
416 MAmar

418
420

FIG.4B
U.S. Patent May 26, 2020 Sheet 5 of 6 US 10,666,865 B2

??
??

????

? ?? ? {
532

??

???

??{
????

??
???
??

???
???
???
??
502

1.5
U.S. Patent May 26 , 2020 Sheet 6 of 6 US 10,666,865 B2

o
604

FIG
6
.
 US 10,666,865 B2
1 2
PANORAMIC CAMERA WITH MULTIPLE sensor is over-exposed , causing light to spillover to adjacent
IMAGE SENSORS USING TIMED SHUTTERS pixels or into the readout CCD structure.
What is needed is an imaging apparatus that reduces
CROSS -REFERENCE TO RELATED distortion and spatial parallax , while avoiding the blooming
APPLICATIONS 5 and streaking issues associated with CCDs.
This application is a continuation of U.S. patent applica BRIEF SUMMARY
tion Ser . No. 15/ 711,128, filed Sep. 21, 2017 , which is a
continuation of U.S. patent application Ser. No. 15 /006,449 , The present invention relates to the field ofpanoramic still
filed Jan. 26 , 2016 , now U.S.pat. No.9,794,479 , issued Oct. 10 and motion photography. In a first embodiment, a camera
17 , 2017 , which is a continuation of U.S. patent application apparatus for panoramic photography includes a first image
Ser. No. 13 /780,493, filed Feb. 28 , 2013 , which is a divi sensor positioned to capture a first image . The first image
sional of U.S. patent application Ser. No. 12/ 368,014 , filed sensor has a rolling -shutter readout arranged in portrait
Feb. 9 , 2009, now U.S. Pat. No. 8,493,436 , issued Jul. 23, orientation . The camera apparatus also includes second
2013 , which claims the benefit of U.S. Provisional Appl. No. 15 image sensor positioned to capture a second image. The
61/027,237 , filed Feb. 8, 2008 , the disclosures of which are second image sensor has a rolling -shutter readout arranged
incorporated herein by reference. in portrait orientation . Finally , the camera apparatus includes
BACKGROUND a controller configured to signal the second image sensor to
20 start capturing the second image before the first image
Field sensor finishes capturing the first image. At least a portion of
the first image is in front of the second image relative to a
The present invention relates to the field of panoramic still forward direction of the camera apparatus .
and motion photography. In a second embodiment, a method for panoramic pho
25 tography includes the steps of: capturing a first image with
Related Art a first image sensor and starting to capture a second image
with a second image sensor prior to completion of the
Imaging systems exist that include more than one camera capturing of the first image. The first and second image
in a rosette formation attached to a vehicle . Those imaging sensors have a rolling-shutter readouts arranged in landscape
systems may be used to capture panoramic images, for 30 orientation . At least a portion of the first image is in frontof
example , along a street. Each camera includes an entrance the second image relative to a forward direction of a camera
pupil. Having multiple entrance pupils at different locations apparatus comprising the first and second image sensors.
can cause spatial parallax . Parallax refers to a perceived shift In a third embodiment, a camera apparatus for motion
of an imaged object against a background caused by the photography includes a first camera having a first entrance
different viewpoints of the entrance pupils of the cameras. 35 pupil . During forward motion of the camera apparatus, the
Parallax is a particular problem when stitching together first camera captures a first image . A camera apparatus also
images from multiple cameras , since it can cause ghosting of includes a second camera having a second entrance pupil .
foreground objects when background objects are aligned in During the forward motion of the camera apparatus, the
the region where adjacent images overlap . second camera captures second image . Motion of the
Mirrors have been used to avoid parallax in panoramic 40 camera apparatus results in motion parallax . The timing of
imaging. For example, Disney's Circle - Vision 360 ° system when the first camera captures a first image relative to when
uses mirrors to view the world through multiple co - located the second camera captures the second image uses the
entrance pupils . This can result in zero parallax . However, motion parallax to reduce the effect of spatial parallax
making and using the necessary large mirrors is difficult . between the first camera and the second camera .
Also , this technique limits the vertical field of view . 45 In a fourth embodiment, a method for motion photogra
Each camera in the rosette formation includes an image phy with a camera apparatus includes: capturing a first
sensor that converts an optical signal into an electrical signal image with a first camera in the camera apparatus at a first
to form an image. Two types of image sensors avoid the need time during forward motion of the camera apparatus, and
for a mechanical shutter - an interline -shutter charge capturing a second image with a second camera in the
coupled device (CCD ) or a complementary metal oxide 50 camera apparatus at a second time the during forward
semiconductor (CMOS ) sensor. motion of the camera apparatus . Motion of the camera
The CMOS sensor typically has a rolling shutter, which apparatus results in motion parallax . The timing of when the
exposes different lines of the scene at slightly different capturing (a ) occurs relative to when the capturing (b )
times, in a sequence that rolls across the image . The rolling occurs uses themotion parallax to reduce the effect of spatial
shutter can lead to image distortion , such as warping, when 55 parallax between the first camera and the second camera.
the camera is moving relative to the scene or subject. Some In this way, embodiments of the present invention reduce
work has been done to compensate for the image distortion . distortion and spatial parallax .
Wilburn et al. describes using a rolling shutter timing offset Further embodiments , features, and advantages of the
to process and correct distortions induced by motion of a invention , as well as the structure and operation of the
subject of an image. See Wilburn et al., “ High -Speed Vid- 60 various embodiments of the invention are described in detail
eography Using a Dense Camera Array ” , 2004 IEEE Com below with reference to accompanying drawings .
puter Society Conference on Computer Vision and Pattern
Recognition (CVPR'04 ), Vol. 2 , pp. 294-301 . BRIEF DESCRIPTION OF THE FIGURES
The interline- shutter CCD avoids the image distortion
issues of the rolling shutter CMOS sensor. However, the 65 The accompanying drawings , which are incorporated
interline -shutter CCD can suffer from blooming and streak herein and form a part of the specification , illustrate the
ing . Blooming and streaking occur when a portion of the present invention and , together with the description , further
 US 10,666,865 B2
3 4
serve to explain the principles of the invention and to enable The term " camera ” used herein refers to a device that
a person skilled in the pertinent art to make and use the captures a photographic image . A camera includes an image
invention . sensor, and a lens having an entrance pupil.
FIG . 1A includes a diagram illustrating a camera that The term “ image sensor ” used herein refers to a device
includes a rolling - shutter CMOS sensor. 5 that converts optical signals into electrical signals to capture
FIG . 1B includes a diagram illustrating a camera having an image when exposed .
a rolling-shutter CMOS sensor with groups of photodiodes FIG . 1A includes a diagram 100 illustrating a camera that
arranged in columns substantially perpendicular to the includes a rolling -shutter CMOS image sensor. Diagram 100
ground , according to an embodiment of the present inven shows a camera and its image sensor at two points in time.
tion . 10 At a first point in time, the camera is at a position 102 and
FIGS . 2A - C include diagrams illustrating a portion of a its image sensor is at a position 104. At a second point in
camera rosette which may be used in panoramic imaging, time, the camera is at a position 112 and its image sensor is
according to an embodiment of the present invention . at a position 114. The dot in the figure represents the
entrance pupil of the lens, and rays are shown projecting
FIG . 3 includes a diagram showing the operation of the
camera rosette shown in FIGS. 2A - C in further detail.
15 through it in the manner of a pinhole camera for simplicity.
In an example , the camera may be attached to a vehicle,
FIGS. 4A -B show how positioning of the entrance pupils and movement of the vehicle may have caused the camera
of a camera impacts parallax . to move. Asmentioned above , the image sensor may be a
FIG . 5 shows a diagram illustrating timing the exposure rolling shutter CMOS sensor. A rolling shutter CMOS sensor
of cameras in a camera rosette to reduce spatial parallax , 20 may include groups of photodiodes that capture light and
according to an embodiment of the present invention . convert the light into electrical signals during exposure. A
FIG . 6 shows a diagram of a camera rosette affixed to a rolling shutter CMOS sensormay expose different groups of
vehicle. photodiodes at different times. A rolling shutter CMOS
The drawing in which an element first appears is typically sensor has a readout time. The readout time is the time
indicated by the leftmost digit or digits in the corresponding 25 necessary to read out all the groups of photodiodes at the
reference number. In the drawings, like reference numbers ends of their respective exposure times. In the example
may indicate identical or functionally similar elements . shown in FIG . 1A , the CMOS sensor is in " landscape ”
orientation , meaning that the groups of photodiodes are
DETAILED DESCRIPTION OF EMBODIMENTS arranged as rows running parallel to the ground .
30 The rolling shutter sensor may be exposed while the
Embodiments of present invention reduce distortion and vehicle is in motion . In an example, a rolling shutter sensor
spatial parallax in panoramic images taken from a camera may require 100 ms of readout time to capture an image .
rosette affixed to a vehicle . Each camera in the camera While the image is captured , the vehicle and camera may
rosette includes an image sensor. The image sensor may have moved forward 1 m . The movement causes the objects
expose to capture an imagewhile the vehicle ismoving. This 35 captured by distortion
This causes the image insensor to changeimage
the resulting as .shown at 106dif.
Because
movement can cause distortion in the resulting image . In an ferent
embodiment of the present invention , the CMOS sensor is exposedrows of photodiodes in the CMOS sensors are
arranged in “ portrait ” orientation . As is described in detail e.g. vertical features times
at different , the image may appear warped ,
may be slanted . An image sensor that
below , this reduces the image distortion . 40 does not have a rolling shutter and is exposed in its entirety ,
Having multiple cameras in different locations on the such as an interline -shutter CCD sensor, may reduce or
camera rosette can cause spatial parallax . In a further avoid warping . However, as mentioned earlier , interline
embodiment of the present invention , the motion parallax of shutter CCD sensors may suffer from blooming and streak
the vehicle is employed to cancel out spatial parallax . In that ing.
embodiment, as the vehicle moves, exposure of the cameras 45 To dealwith the problem of warping due to rolling shutter
in the camera rosette is timed such that the entrance pupils sensors, embodiments of the present invention have rolling
of the cameras are in approximately the same location when shutter image sensors arranged in “ portrait " orientation as
each sensor is exposed . This embodiment is described in illustrated in FIG . 1B . FIG . 1B includes a diagram 150
detail below . illustrating a camera having a rolling- shutter sensor, such as
In the detailed description of the invention that follows, 50 a CMOS sensor, in portrait orientation . Portrait orientation
references to " one embodiment” , “ an embodiment” , “ an means that the groups of photodiodes are arranged in
example embodiment" , etc., indicate that the embodiment columns running perpendicular to the ground .
described may include a particular feature, structure, or Diagram 150 shows a camera at different positions at
characteristic , but every embodiment may not necessarily different points in time. As mentioned earlier, the camera
include the particular feature, structure , or characteristic . 55 may be affixed to a vehicle, and the vehicle may be in
Moreover , such phrases are not neces cessarily referring to the motion . In an embodiment of the invention , the columns of
same embodiment. Further , when a particular feature , struc photodiodes are exposed back - to - front in object space and
ture , or characteristic is described in connection with an front-to -back in image space .
embodiment, it is submitted that it is within the knowledge At a position 152, the camera exposes a forward -most
of one skilled in the art to effect such feature, structure , or 60 photodiode column 156 capture an image including a
characteristic in connection with other embodiments back -most field of view 154. Then , at a position 160 , the
whether or not explicitly described . camera exposes a photodiode column 158 , which is behind
The term " camera rosette ” used herein refers to two or photodiode column 156. This exposure captures a field of
more cameras arranged to capture a panoramic image . Each view 172 , which is to the front of field of view 154. As the
camera captures an image . The images may be stitched 65 camera moves to a position 162 and a position 164 , columns
together to form the panoramic image. This stitching may , continue to expose in a front-to -back manner, which cap
for example, be done by well-known software techniques. tures objects from back -to -front. Finally, when the camera is
 US 10,666,865 B2
5 6
at a position 166 , the camera exposes a back -most photo 202 has completed its exposure. In diagram 240 , a photo
diode column 168 to capture an image including a forward diode column 232 is exposed , capturing a field of view 228 .
most field of view 170 . As camera 210 continues its exposure , it may overlap with
Positioning the image sensor in portrait orientation avoids an field of view with another camera (not shown ) and the
the warping of vertical features that occurs in landscape 5 process may continue for an entire camera rosette as is
orientation . Instead , objects may appear stretched by an described with respect to FIG . 3 .
amount that depends on their distance. Stretching is not as The embodiment illustrated in FIGS. 2A - C is described
visually unappealing as warping . In fact, stretching may be with respect to an illustrative example . In an example ,
a positive feature . Stretching makes foreground objects overlap field of view 222 for distant objects between camera
thinner with respect to background objects , rather than fatter, 10 202 and camera 210 may be 10 % of total field of view 224
when the rolling shutter is arranged in the direction of camera 202. In that example , the delay between the time
described . As result , the foreground objects occlude less of when camera 202 starts exposure and the time when camera
the scene behind them . This may be a benefit in many 210 starts exposure may be 90 % of camera 202 readout time.
applications related to panoramic imaging . Camera 202 and 210 may ormay not have the same readout
Panoramic imaging often includes taking images from 15 time. In an example where the readout time is 100 ms, the
multiple cameras oriented in different directions. The mul delay between the time when camera 202 starts exposure
tiple images may then be stitched together into a panorama and the timewhen camera 210 starts exposure may be 90 ms.
using well -known software techniques. When multiple cam The camera rosette may have at least one controller that
eras are used in this way , an object may exist in the field of synchronizes the cameras to capture images at specified time
view of more than one camera . In motion photography, 20 offsets . So , the controller may be pre-programmed with the
objects may move or change over time relative to the delays indicating when each camera starts to capture an
cameras. Ifmultiple cameras expose the object at different image.
times , then the objectmay appear differently in the multiple In another embodiment, the delay time may be offset
images to stitch together into a single panorama. The result according to the vehicle's velocity . As is described below
ing panorama may have ghosting or other undesirable image 25 with respect to FIG . 5 , this has the effect of using motion
problems. To deal with this, exposure needs to be coordi parallax from movement of the vehicle to cancel out spatial
nated between the multiple cameras to capture objects at the parallax caused by the multiple cameras .
same time or close to the same time. This becomes even As mentioned above, the operation of the two cameras
more challenging with rolling shutter image sensors because described with respect to FIGS. 2A - C may continue through
different portions of each image sensor are exposed as 30 an entire camera rosette . In one embodiment, a camera
different times. To deal with this , embodiments of the rosette may have a total of between six and ten cameras,
present invention time exposure of multiple image sensors inclusive , positioned to capture a 360 degree panorama. In
as illustrated in FIGS . 2A - C . a preferred embodiment, the camera rosette may have at
FIGS . 2A - C include diagrams illustrating a portion of a least eight cameras , such as the rosette illustrated with
camera rosette , which may be used in panoramic imaging . 35 respect to FIG . 3 .
Each camera in the camera rosette includes a rolling -shutter FIG . 3 shows a diagram 300 illustrating the operation of
image sensor, such as a rolling - shutter CMOS sensor, in a camera rosette according to an embodiment of the present
portrait orientation as described with respect to FIG . 1B . invention in further detail. In a preferred embodiment, a
FIG . 2A includes a diagram 200. Diagram 200 shows a camera rosette may include nine cameras eight in a hori
camera rosette with cameras 202 and 210. For clarity, only 40 zontal ring , and one directed straight up . Diagram 300 shows
two cameras are shown . However, in practice the rosette the eight horizontal cameras. Each camera in the camera
may contain a different number of cameras . Camera 202 has rosette includes a rolling shutter sensor in portrait orienta
photodiode column 204 exposed to a field of view 220 . tion with columns of photodiodes.
FIG . 2B includes a diagram 230. Diagram 230 shows the The camera rosette begins capturing a panoramic image at
rosette shown in FIG . 2A at a later point in time. Camera 202 45 the rear-most cameras 305 and 304. Both of the cameras 305
has a total field of view 224 , and camera 210 has a total field and 304 start by exposing the back most photodiodes
of view 234. As camera 202 scans across field of view 224 columns ( in object -space ) and scan forward as described
back - to - front (in object space ), camera 202 begins to expose with respect to FIG . 1B . Cameras 306 , 303 start capturing
photodiode columns from a group of photodiode columns images at a time defined by a time delay as described with
206. Group of columns 206 captures objects within field of 50 respect to FIG . 2B .
view 222. Field of view 222 overlaps with camera 210's The rosette continues to scan forward until the forward
total field of view 234 for distant objects . most cameras 301, 308 have completed scanning Each
When camera 202 begins to expose photodiode columns camera starts at a time defined by the start of the adjacent
from group of columns 206 , camera 210 begins to expose camera to its rear plus a delay. For example , a camera 302
photodiode columns from a group of photodiode columns 55 starts at a time defined by the start of camera 303 plus a
208. Camera 210 exposes photodiode columns in group of delay. The delay may correspond to the size of the overlap
columns 208 back -to -front ( in object space ) to capture field in the field of view of adjacent cameras, as described with
of view 226. Field of view 226 overlaps with camera 202's respect to FIG . 2B . The delay may also be offset according
total field of view 224 for distant objects. Effectively, camera to the velocity of the vehicle, as is described with respect to
202 exposes photodiode columns in group of columns 206 60 FIG . 5 .
simultaneously with camera 210 exposing photodiode col Each of the eight cameras in FIG . 3 is at a different
umns in group of columns 208.When camera 202 completes location and has an entrance pupil at a different location .
exposure of group of columns 206 , camera 202 finishes its Having entrance pupils at different locations causes spatial
exposure . However , camera 210 continues to expose pho parallax as illustrated in FIGS. 4A - B .
todiode columns, as shown FIG . 2C . 65 FIGS. 4A - B show how the positioning of the entrance
FIG . 2C shows a diagram 240. Diagram 240 shows the pupils of two cameras impacts spatial parallax . FIG . 4A
rosette shown in FIG . 2B at a later point in timeafter camera shows an example imaging apparatus 400. Imaging appara
 US 10,666,865 B2
7 8
tus 400 includes a camera 402 and a camera 404. Camera Finally, at position 530 entrance pupil 512 of camera 506
402 has an entrance pupil 406 , and camera 404 has an reaches approximately location 524 , and camera 506 cap
entrance pupil 408. Entrance pupil 406 and entrance pupil tures an image.
408 are separated by a distance 426. Each camera 402 , 404 The approximation may be due to inaccuracies in the
captures an image including an object 410.However there is 5 vehicles velocity , or changes to the vehicle's direction . In an
a wide difference in the background of object 410 as it example, the approximation may be based on an average
appears in cameras 402 , 404. This is shown by the large vehicle velocity as opposed to the actual velocity over the
parallax angle 422. relevant period. Also , the approximation may be due to the
FIG . 4B shows an example imaging apparatus 450. Imag 10 fact that cameras are positioned two or three dimensionally
ing apparatus 450 includes a camera 416 and a camera 414 . on the camera rosette , whereas the movement of the rosette
Camera 416 has an entrance pupil 418 , and camera 414 has (e.g., on a vehicle) may be in only one direction . For
an entrance pupil 420. Entrance pupil 418 and entrance pupil example , in FIG . 5 entrance pupil 508 is not at precisely the
420 are separated by a distance 428. Distance 428 is shorter same location as entrance pupil 514 because entrance pupil
than distance 426 in FIG . 4A . Each camera 416 , 414 15 508 are entrance pupil 514 are at different locations on the
captures an image including an object 412. There is a small axis perpendicular to the axis of motion .
difference in the background of object 412 as it appears in In another embodiment, a virtual plane is perpendicular to
cameras 416 , 414, as compared to cameras 402 , 404 in FIG . the direction ofmotion of a camera rosette . Each camera in
4A . This is shown by the parallax angle 424. Therefore , a camera rosette may expose when an entrance pupil of the
parallax angle 422 in FIG . 4B is less than parallax angle 424 20 camera passes through the plane. In this way, each camera
in FIG . 4A . exposes when its entrance pupil is at approximately the same
Thus, two cameras being in different places can cause location , reducing parallax .
spatial parallax . Similarly, motion parallax is caused As noted earlier , the time to trigger each subsequent
camera being in different locations at different times . camera may be calculated according to vehicle velocity and
According to an embodiment of the present invention , 25 the a distance
examplebetween the delay
shown, the camerasfrom, sucha timeas awhen
distance 522.504In
camera
motion parallax caused by motion of a camera rosette by captures
cancel out or reduce the effect of spatial parallax caused by image is andistance
image and a timewhen camera 506 captures an
the entrance pupils of the different cameras on the camera Supposing distance 522 522
divided by the vehicle's velocity.
was 10 cm and the vehicle velocity
rosette being located atdifferent locations . This embodiment 30 was 10 m /s, then the delay between the time when camera
is illustrated in FIG . 5 .
FIG . 5 shows a diagram 500 illustrating timing the captures an image would be 10 ms. Inwhen
504 captures an image and the time camera 506
other words, the
exposure of cameras in a camera rosette to reduce parallax , further rear camera (camera 506 ) starts to capture an image
according to an embodiment of the present invention . As 10 ms after the more forward camera ( camera 504 ).
shown in FIG . 4A - B , as camera entrance pupils get closer 35 As mentioned earlier, a vehicle velocity used to time the
together , the spatialparallax between the cameras decreases. cameras may be an average vehicle velocity . Alternatively,
Diagram 500 showshow to time the exposure of the cameras the vehicle velocity may be determined in real time by a
during motion of the vehicle so that each of the cameras is speedometer, a wheel encoder, or a GPS sensor. The velocity
exposed when its entrance pupil is located at approximately may be signaled to controllers in the cameras, which adjust
the same location . As result, in diagram 500 the motion 40 the timing accordingly .
parallax caused by the moving vehicle reduces the spatial In a preferred embodiment, the features shown in diagram
parallax due to multiple cameras . 500 may be used in conjunction with the features described
Diagram 500 shows a portion of a camera rosette at three with respect to FIGS. 2A - C . FIG . 2B describes timing
positions 510, 520 , 530 at three different points in time. The camera exposure in a camera rosette where each camera
camera rosette shown has three cameras 502 , 504 , and 506 45 includes a rolling -shutter CMOS sensor in portrait orienta
tion . In that embodiment, each camera is timed to start
shown. Cameras 502, 504 , and 506 have entrance pupils exposure
508, 514 , and 512 respectively . For clarity , only three where theirwhen the previous camera begins to scan an region
fields of view overlap at a distance . Describing
cameras are shown . However, in practice a different number the embodiment
of cameras may be used (e.g. 8 horizontal cameras and 9 when camera 202withstarts respect to FIG . 2B , a delay between
exposure and when camera 210
total cameras in another embodiment). 50
In the embodiment in FIG . 5 , the image sensors may or starts exposure substantially satisfies the equation :
may not have rolling shutters. The embodiment may be
useful with rolling shutters. However, the embodiment may
also reduce parallax with image sensors having “ global” t = R1: F1 - D12 V
shutter, such as an interline CCD or an image sensor using 55
a mechanical shutter. The image capture is described as
happening at an instant, even though there will be some where t is the delay time, R1 is a rolling -shutter readout
exposure duration and a small motion blur as a result . At time of the first image sensor, F1 is a percentage ofthe field
position 510 , camera 502 captures an image . At position of view ofthe first image sensorwhich does not overlap with
510 , camera 502's entrance pupil 508 is at a location 524. 60 the field of view of the second image sensor, D12 is a
Each subsequent camera is timed to take an image when that distance between an entrance pupil of the first image sensor
camera's entrance pupil is approximately located at location and an entrance pupil of the second image sensor, and v is
524 . a speed of the camera apparatus. It would be recognized that
During a later point in time as the vehicle is moving , the small changes such as small offsets and coefficients may be
camera rosette reaches position 520. At position 520 , 65 added to this equation .
entrance pupil 514 of camera 504 is approximately at In an example given with respect to FIG . 2B , overlap field
location 524. At this point, camera 504 captures an image . of view 222 between camera 202 and camera 210 may be
 US 10,666,865 B2
9 10
10 % of the total field of view of camera 202. In that sensor with a variable rolling readout time may be used . In
example , 90 % of camera 202’s field of view does not that instance , the readout timemay be adjusted according to
overlap with camera 210's field of view . In an example a speed of the vehicle .
where camera 202's readout time is 100 ms, the delay FIG . 6 shows a camera rosette 604 affixed to a vehicle
between the time when camera 202 starts exposure and the 5 602. Camera rosette 604 includes numerous cameras, such
time when camera 210 starts exposure may be 90 ms. In used as a camera 606. In an example , camera rosette 604 may be
other words, the more forward camera (camera 210 ) starts to along toa street
capture panoramic images of buildings running
.
capture an image 90 ms after the further rear camera starts
to capture an image (camera 202 ). As a result, cameras with Each of the cameras in the camera rosette is coupled to a
overlapping fieldsof view capture images of objects in their 10 the
controller
timing 608.
of theAsexposure
mentionedofearlier
each ,camera
controller 608 controls
. Controller 608
overlap region at the same time, or from the same vehicle may receive an input indicating a speed of the vehicle .
position . This timing is a first-order correction for parallax ,
Alternatively, controller 608 may be preprogrammed with
in that it prevents the large parallax errors that would result timing offsets , such as offsets based on an average speed of
if the objects in the overlap area were captured at very 15 the vehicle. A single controller 608 may be used . Alterna
different times and very different vehicle positions , as would tively , each camera may have its own controller .
happen if the cameras started their rolling shutters simulta Controller 608 may be implemented in hardware , soft
neously .However, there remains a smaller parallax error due ware , firmware , or any combination thereof. Controller 608
to the entrance pupils of the cameras not being in the same may be , for example, a general purpose computer running
place . 20 software to control the cameras ' exposure. Controller 608
In an example of the embodiment incorporating both may have a processor and memory .
features in diagram 500 and FIGS. 2A - C , the delay time is The Summary and Abstract sections may set forth one or
offset to further reduce spatial parallax . In the example given more but not all exemplary embodiments of the present
with respect to FIG . 5 , the further rear camera (camera 506) invention as contemplated by the inventor(s ), and thus, are
starts to capture an image 10 ms after the more forward 25 not intended to limit the present invention and the appended
camera (camera 504). In the example in FIGS . 2A -C , the claims in any way.
more forward camera (camera 210 ) starts to capture an willTheso fully foregoing description of the specific embodiments
reveal the general nature of the invention that
image 90 msafter the further rear camera starts to capture an
image ( camera 202 ). In the combined embodiment, the 90 others can , by applying knowledge within the skill of the art,
ms delay is offset by 10 ms. Therefore, the more forward 30 specific readily modify and/or ,adapt
embodiments for various
without undue applications such,
experimentation
camera starts to capture an image 80 ms after the further rear without departing
camera starts to capture an image . This embodiment has invention . Thereforefrom ,
the general concept of the present
such adaptations and modifications are
reduced spatial parallax , accounting jointly for rolling -shut intended to be within the meaning and range of equivalents
ter readout
ration .
delay , vehicle velocity, and entrance pupil sepa- 35 of the disclosed embodiments , based on the teaching and
guidance presented herein . It is to
In another embodiment, the timing offsets for reduced phraseology or terminology herein is for the purpose of
be understood that the
spatial parallax may only be applied to some of the cameras description and not of limitation , such that the terminology
in a camera rosette . In an example, the offset may be applied or phraseology of the present specification is to be inter
to only those cameras pointed sideways from the vehicle 40 preted by the skilled artisan in light of the teachings and
motion. The cameras pointed sideways may be the cameras guidance .
where typical vehicle motion moves the location of the The breadth and scope of the present invention should not
entrance pupil of the more rear camera into the location be limited by any of the above -described exemplary embodi
where the camera took an image in , for example, 0.01 s or ments, but should be defined only in accordance with the
less . In this example , spatialparallax of forward - looking and 45 following claims and their equivalents .
backward - looking portions of the panorama may not be The invention claimed is :
cancelled by motion parallax , but objects in these directions 1. A system for capturing images, comprising :
tend to be distant enough to not have a large parallax a camera rosette ;
problem . a movable structure ; and
As an alternative embodiment, a rolling shutter with a 50 a controller in communication with the camera rosette
very fast readout time may be useful. As the shutter readout configured to :
time becomes faster, the delay computed to account for the start to expose a first image using a first camera of the
rolling shutter (as described in FIGS . 2A - C ) decreases. So , camera rosette, the first camera having a first field of
at a particular readout time, the delay for the rolling shutter view :
and the offset to account for spatial parallax may cancel each 55 start to expose a second image using a second camera
other out. This readout time is the time it takes for the of the camera rosette at a timeoffset from the start of
vehicle to move forward a distance equal to the distance the first image , the time offset based on a velocity of
between entrance pupils . In an example , a readout time of a the movable structure , the second camera having a
first image sensor may be 10 ms. Ninety percent of the first second field a view overlapping with the first field of
image sensor's field of view may not overlap with a second , 60 view ; and
next forward -most image sensor. If the vehicle is stationary, complete exposure of the first image at a time prior to
the delay time between exposure of the first and second the completion of the exposure of the second image .
image sensor would be 9 ms. However, if the vehicle is 2. The system for capturing images of claim 1 , wherein
moving at 9 m /s and the distance between the first and the controller is further configured to receive a velocity of
second image sensor is 10 cm , then the delay offset due to 65 the movable structure, the velocity of the movable structure
motion of the vehicle would also be 9 ms. Thus , the offsets corresponding to a movement of the camera rosette at a
would effectively cancel each other out. Similarly, an image given speed .
 US 10,666,865 B2
11 12
3. The system for capturing images of claim 2 , wherein a completing , by the controller in communication with the
start time to expose the second image is determined by the camera rosette, exposure of the first image at a time
controllerbased on a distance between entrance pupils of the prior to the completion of the exposure of the second
image .
first and second cameras of the camera rosette and an
average velocity of the system . 5 12. The method of claim 11, further comprising receiving,
4. The system for capturing images of claim 2 , wherein by the controller in communication with the camera rosette ,
the controller is further configured to determine a start time a velocity of the movable structure , the velocity of the
to expose the second image based on the given speed . movable structure corresponding to a movement of the
camera rosette at a given speed .
5. The system for capturing images of claim 2 , wherein a 10
13. The method of claim 12 , wherein starting to expose
start time to expose the second image is determined by the the second image using the second camera of the camera
controller based on a distance between entrance pupils of the rosette at the time offset comprises determining, by the
first and second cameras of the camera rosette and the controller in communication with the camera rosette , a start
velocity of the system . time to expose the second image based on a distance
6. The system for capturing images of claim 5 , wherein 15
between entrance pupils of the first and second cameras of
the camera rosette includes a set of cameras including the the camera rosette and an average velocity of the system .
first and second cameras, and while moving the camera 14. The method of claim 12 , further comprising deter
rosette, capturing additional images using the set ofcameras mining,by the controller in communication with the camera
where each additional image has an overlapping field of rosette, a start time to expose the second image based on the
view with at least one other of the additional images. 20
given speed .
7. The system for capturing images of claim 5 , wherein 15. The method of claim 12 , wherein starting to expose
the velocity of the system is determined in real-timebased the second image using the second camera of the camera
at least in part, on data from a GPS sensor affixed to the rosette at the time offset comprises determining, by the
controller in communication with the camera rosette , a start
system , a speedometer of the system , or a wheel encoder that time to expose the second image based on a distance
detects movement of a wheel of the system . 25
between entrance pupils of the first and second cameras of
8. The system for capturing images of claim 1 , wherein the camera rosette and the velocity of the movable structure .
the controller is further configured to complete the exposure 16. The method of claim 15 , wherein the camera rosette
of the first image in order to reduce a likelihood that an includes a set of cameras including the first and second
object in the overlapping fields hasmoved between the start cameras , and while moving the camera rosette , capturing
ofsecond
exposure
imageof. the first image and the startof exposure of the 30 additional images using the set of cameras where each
9. The system for capturing images of claim 1 , wherein additional image has an overlapping field of view with at
the camera rosette is configured to capture a panoramic least one other of the additional images .
image. 17. The method of claim 15 , further comprising deter
10. The system for capturing images of claim 1, wherein 35
mining the velocity of the movable structure in real-time
the first camera includes a first image sensor and the second based at least in part, on data from a GPS sensor affixed to
camera includes a second image sensor and different por the movable structure , a speedometer of the movable struc
tions of the first and second image sensors are exposed at ture , or a wheel encoder that detects movement of a wheel
different times in order to capture the first and second of the moveable structure.
images . 40
18. The method of claim 11, further comprising complet
11. A method of capturing images comprising , ing , by the controller in communication with the camera
moving a movable structure including a camera rosette ; rosette , the exposure of the first image in order to reduce a
starting, by a controller in communication with the cam likelihood that an object in the overlapping fields has moved
era rosette , to expose a first image using a first camera between the start of exposure of the first image and the start
of the camera rosette , the first camera having a first 45
of exposure of the second image.
field of view ; 19. The method of claim 11, wherein the camera rosette
starting , by the controller in communication with the is configured to capture a panoramic image.
camera rosette, to expose a second image using a 20. The method of claim 11 , wherein the first camera
second camera of the camera rosette at a time offset includes a first image sensor and the second camera includes
a second image sensor and different portions of the first and
from
on a velocity of the movable structure , the second second image sensors are exposed atdifferent times in order
the start of the first image ,the time offset based 50
camera having a second field a view overlapping with to capture the first and second images.
the first field of view ; and