1. No category

Today
Panoramas
• Introduction
Computational Photography
Matthias Zwicker
University of Bern
Fall 2009
• Image reprojection & homographies
• Panoramas
With material from: Rick Szeliski, Steven Seitz,
University of Washington; Alyosha Efros, Carnegie
Mellon University
Introduction
Why mosaics?
• Panorama based on image mosaic
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
[© Jeffrey Martin (jeffrey-martin.com)]
[Brown & Lowe]
Why Mosaic?
Why Mosaic?
• Are you getting the whole picture?
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
– Human FOV
= 200 x 135°
– Compact Camera FOV = 50 x 35°
– Human FOV
= 200 x 135°
– Panoramic Mosaic
= 360 x 180°
[Brown & Lowe]
[Brown & Lowe]
1
Mosaics
Basic procedure
• Take sequence of images from same position
– Rotate the camera about its optical center
– Align images by computing mapping
(transformation) between second image and first
– Transform second image to overlap with first
– Blend to create mosaic
– Repeat if more images
• Why does it work?
– What about the 3D geometry of the scene?
– Why aren’t we using it?
Virtual wide-angle camera
Why no geometry required?
Aligning images
• A pencil of rays contains all views from same
center of projection
• Can generate any virtual view, as long as same
center of projection
Real
camera
Synthetic
camera
left on top
right on top
Translations are not enough!
Image reprojection
Today
• Mosaic has natural interpretation in 3D
• Images are reprojected onto common plane
Panoramas
– Mosaic formed on this plane
– Synthetic wide-angle camera
• Introduction
• Image reprojection & homographies
• Panoramas
mosaic projection plane (PP)
2
Image reprojection
Back to image warping
• How to relate two images from same camera center?
Which transform can implement the reprojection?
– How to map pixel from PP1 to PP2
• Answer
(u0 , v 0)
– Cast ray through pixel in PP1
– Draw pixel where ray intersects in PP2
PP2
• Reprojection
p j
defines 2D-to-2D mapping
pp g
between images
(u0 , v 0 ) = M (x, y)
(x, y)
– Think of it as 2D image warp,
rather than 3D reprojection
Translation
Affine
Perspective
2 unknowns
6 unknowns
8 unknowns
PP1
Homography
1D illustration
• Projective transformation between any two
PPs with same center of projection
• Perspective projection
– Rectangle maps to arbitrary
quadrilateral
– Preserve straight lines, but not
parallel lines
p
((u0 , v 0)
– Assume z-axis perpendicular to image plane
– Image plane at z=1
PP2
• Mathematically
• Projected point
x
x0 =
z
z=1
– 3x3 matrix H
z
 wu '
* * *   x 
 wv '  * * *   y 
 

 
* * *   1 
 w 
p’
u0 =
H
wu0
w
v0 =
(x, y)
PP1
x
p
x0
wv 0
w
Reprojection
Conclusion
1. Interpret point on projection plane as 2D point
⎡
⎤
⎡
⎤
2. Rotate
wu0 ⎦
x0 ⎦ 0 wu0
⎣
⎣
=
R
,u =
3. Reproject
w
1
w
• For 2D reprojection, the 3x3 transformation
matrix H can be interpreted as a 3D
rotation
z=0
 wu '
* * *   x 
 wv '  * * *   y 
 

 
* * *   1 
 w 
p’
p
z
p
• Third component is called homogeneous
coordinate
Rotate &
reproject
w
x
x0
H
u0
• Any scalar multiple of a point in
homogeneous coordinates represents the
same 2D point
3
Image warping
Finding the homography
Image plane on floor
• Given two images from same camera
position, do we weed geometry of
projection planes to determine
homography?
Image plane on right wall
Image plane in front
Outside source image
Image rectification
Solving for homographies
• Determine homography using pairs of
corresponding points
• Correspondences (xj , yj ), (u0j , vj0 )
•⎡Each correspondence
j ⎤should fulfill
⎤
⎡
⎤⎡
– Can solve formatrix H
– How many correspondences needed?
⎢
⎢
⎢
⎣
wj u0j
wj vj0
wj
⎥
⎥
⎥
⎦
a b c
⎢
= ⎢⎢⎣ d e f
g h i
⎥⎢
⎥⎢
⎥⎢
⎦⎣
xj
yj
1
⎥
⎥
⎥
⎦
– Unknowns a, b, c, d, e, f, g, h, i
• Note: arbitrary scale factor
– Set i=1
– 8 remaining unknowns
Solving for homographies
Solving for homographies
• One correspondence (omit index)
• Stack up rows for each correspondence
⎡
⎢
⎢
⎢
⎣
wu0
wv 0
w
⎤
⎥
⎥
⎥
⎦
=
⎡
⎢
⎢
⎢
⎣
a b c
d e f
g h i
⎤⎡
⎥⎢
⎥⎢
⎥⎢
⎦⎣
x
y
1
wu0
ax + by + c
u =
=
w
gx + hy
h +1
wv 0
dx + ey + f
0
v =
=
w
gx + hy + 1
0
⎤
⎡
⎥
⎥
⎥
⎦
⎡
⎣
xj yj 1 0 0 0
0 0 0 xj yj 1
ax + by + c − gu0 x − hu0 y = u0
dx + ey + f − gv0 x − hv 0y = v 0
⎡
• Matrix formulation
⎡
⎣
x y 1 0 0 0 −u0 x −u0 y
0 0 0 x y 1 −v 0 x −v 0y
⎢
⎢
⎢
⎢
⎢
⎢
⎤⎢
⎢
⎢
⎦⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
a
b
c
d
e
f
g
h
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
⎡
=⎣
u0
v0
⎤
⎦
−u0j xj
−vj0 xj
⎢
⎢
⎢
⎢
⎢
⎢
⎤⎢
0
−uj yj ⎦ ⎢⎢⎢
0
−vj yj ⎢⎢⎢
⎢
⎢
⎢
⎢
⎢
⎣
A
a
b
c
d
e
f
g
h
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
x
⎡
=⎣
u0j
vj0
⎤
⎦
b
• Overconstrained if more than 4 correspondences
• Solve in least squares sense minx kAx − bk2
– Matlab „\“, „help lmdivide“
4
Least squares example
Fun with homographies
• Given set of data points (X1, X1’), (X2, X2’), (X3, X3’), etc.
What is the shape of the b/w floor pattern?
– E.g., person’s height and weight
• Want compact formula to predict X’s from Xs
– E.g., a line Xa + b = X’
• Want to find parameters a and b
• With two pairs (X, X’) X 1a  b  X 1'
X 2 a  b  X 2'
• What if many data points, noisy data?
 X1
X
 2
X3

 ...
1
1  a 


1   b 

...
 X 1' 
 '
X2
 X 3' 
 
 ... 
Overconstrained
system of equations
min Ax  B
 X1
X
 2
1  a 

1  b 
 X 1' 
 '
X2
2
Least-squares
solution
Minimize sum of squared
deviations from line
Fun with homographies
The floor (enlarged)
Rectified floor
http://research.microsoft.com/apps/pubs/default.aspx?id=67260
Fun with homographies
Rectifica
ation
What is the shape of floor pattern?
From Martin Kemp The Science of Art
(manual reconstruction)
2 patterns have been discovered !
Automatically rectified floor
St. Lucy Altarpiece, D. Veneziano
http://research.microsoft.com/apps/pubs/default.aspx?id=67260
http://research.microsoft.com/apps/pubs/default.aspx?id=67260
Analysing patterns and shapes
Julian Beever: manual homographies
Automatic
rectification
Martin Kemp, The Science of Art
(manual reconstruction)
http://research.microsoft.com/apps/pubs/default.aspx?id=67260
http://users.skynet.be/J.Beever/pave.htm
5
Today
Panoramas
Panoramas
• Introduction
• Image reprojection & homographies
• Panoramas
1. Pick one image (red)
2. Warp overlapping images towards it (usually, one by
one)
3. Blend
4. Continue with 2. until no more images
Image Blending
Feathering
+
1
0
1
0
=
Effect of window size
1
left
Effect of window size
1
1
1
0
0
0
right
0
6
Good window size
Image feathering
• What if you have more than two images?
• Generate weight map for each image
– Typically large weight at center, small weight at edge
• Each output pixel is a weighted average of inputs
– Divide by sum of weights at the end
1
0
“Optimal” window: smooth but not ghosted
Doesn’t always work...
Alpha blending
Changing camera center
• Encoding blend weights I(x,y) = (R, G, B, )
• Color =
• Two step implemementation
• Does it still work?
1.
2.
Synthetic PP
PP1
Accumulate: add up the ( premultiplied) RGB values at each
pixel
Normalize: divide each pixel’s accumulated RGB by its  value
I3
PP2
p
I1
I2
Planar scene (or far away)
Planar mosaic
PP3
PP1
PP2
• OK: PP3 is a projection plane of both
centers of projection
• Used for big aerial photographs
7
360 panoramas
Cylindrical panoramas
• Reprojection onto a plane doesn’t allow
for 360 panorama
• Assumptions
• Reproject onto sphere or cylinder
• Algorithm
– Camera is level
– Rotation only around vertical axis
– Reproject each image onto a cylinder
– Align & blend images one by one
– Output the resulting mosaic
Mosaic projection cylinder
Cylindrical panoramas
Determining the focal length
• Map image to cylindrical (or spherical)
coordinates
1. Initialize from homography H
– Need to know focal length, or equivalently,
field of view
–
See Szeliski & Shum, “Creating full view
panoramic image mosaics and texturemapped Models“
2.. Use ca
camera’s
e a s EXIF tags (app
(approx.)
o .)
3. Use a tape measure
4m
Image 384x300 f = 180 (pixels)
f = 280
1m
f = 380
Cylindrical projection
• Map 3D point (X,Y,Z) onto
cylinder
• Convert to cylindrical
coordinates
Cylindrical warping
• Project cylindrical image back to planar image
• Given focal length f and image center (xc,yc)
Y
Z
X
• Scale and shift
(X Y Z)
(X,Y,Z)
– s defines size of final image
Unit cylinder
(sin,h,cos)
Y
Z
X
Unwrapped cylinder
Final cylindrical image
8
Cylindrical panoramas
• If camera is level and rotating only around
vertical axis, alignment of cylindrical
images requires only horizontal translation
– Use a tripod!
Spherical projection
• Map 3D point (X,Y,Z) onto
sphere
• Convert to spherical
coordinates
Y
Z
X
Unit sphere
• Scale and shift
– s defines size of final image
Unwrapped sphere
Cylindrical images
related by translation
Cylindrical panorama
Spherical reprojection
Spherical image
Spherical reprojection
• Map point on sphere
back to image plane
Y
Z
X
• Use image projection
matrix
Side view
input
f = 200 (pixels)
f = 400
f = 800
• Map image to spherical coordinates
– Need to know the focal length (or field of view)
– Use focal length as scaling parameter s
Top-down view
Aligning spherical images
Aligning spherical images
• Suppose we rotate the camera by  about the
vertical axis
• Suppose we rotate the camera by  about the
vertical axis
– How does this change the spherical image?
– How does this change the spherical image?
• Translation by 
– Can align images using translations only!
9
Spherical 360 panorama
Outlook
• So far
+
+
+
+
– Image alignment uses manually specified
features
– Incremental one-by-one image alignment
• Disadvantages
– Tedious manual feature selection
– One-by-one alignment leads to drift
• Goal
Richard Szeliski
Image Stitching
55
[Szeliski & Shum, “Creating full view panoramic image mosaics and texture-mapped Models“]
Automatic panoramas
– Automatic feature selection & correspondence
– Align all images simultaneously (global
alignment)
Automatic panoramas
• Demos & applications
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
http://download.live.com/photogallery
Input images
Output panoramas, [Brown & Lowe, “Recognising panoramas”]
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
Next time
References
• Automatic feature selection &
correspondence
• Brown & Lowe, “Automatic Panoramic
Image Stitching using Invariant Features”
http://www.cs.ubc.ca/~mbrown/papers/ijcv2007.pdf
• Autostitch
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
• SSzeliski,
li ki “Computer
“C
t Vi
Vision:
i
Al
Algorithms
ith
and
d
Applications”, Section 9.1
http://research.microsoft.com/en-us/um/people/szeliski/Book/
10