Center for Machine Perception Department of Cybernetics, Faculty of Electrical Engineering Czech Technical University in Prague Solving Minimal Problems in Computer Vision Zuzana Kukelova, Tomas Pajdla, Martin Bujnak with contribution of M. Byrőd, K. Josephson, K. Åstrőm Lund University

2/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  Many problems in computer vision can be formulated using systems of polynomial equations  The classical minimal problem – relative camera pose determination  5 matches are necessary and sufficient => 5pt relative pose problem Motivation

3/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Motivation  Minimal problems  Relative camera pose determination for cameras with radial distortion => more difficult systems of polynomial equations  systems are not trivial => special algorithms have to be designed to achieve numerical robustness and computational efficiency

4/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague One polynomial equation in one unknown - Companion matrix 1 equation in 1 variable  companion matrix  eigenvalues... a simple rule Eigenvalues = solutions 

5/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague System of linear polynomial equations - Gauss elimination  System of n linear equations in n unknowns We can represent this system in a matrix form Perform Gauss-elimination  reduces the matrix to a triangular form backsubstitution

6/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague System of m polynomial equations in n unknowns  A system of m polynomial equations in n variables with coefficients from  Our goal is to solve this system  Many different methods  Numerical methods  Polynomial eigenvalue solution  Symbolic methods  Grőbner basis methods  Resultant methods

7/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague System of m nonlinear polynomial equations in n unknowns - Grőbner basis method  System of m linear equations in n unknowns  Idea  construct a new system of polynomial equations with the same solutions as the initial one, but with a simpler structure  Grőbner bases  have the same solutions but are easier to solve

8/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague System of m nonlinear polynomial equations in n unknowns - Grőbner basis method  The reality is much harder  Computation of Grőbner bases  is a NP-complete in m, n (takes very long time to compute)  is an EXPSPACE-complete problem (needs huge space to remember intermediate results)  Fortunately in some cases, Grőbner bases can be obtained easier Equations  Grőbner basis no simple way

9/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague System of m nonlinear polynomial equations in n unknowns - Grőbner basis method How to find coefficients? Bucheberger’s algorithm ~ generalized Gauss elimination

10/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Grőbner basis method w.r.t. lexicographic ordering ~ generalized G-J elimination  General approach for constructing the Grőbner basis Add new equations – monomial multiples of initial equationsPerform Gauss-Jordan elimination

11/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague System of m polynomial equations in n unknowns – An action matrix  m equations in n unknowns Buchberger’s algorithm Generalized companion matrix Eigenvectors Solutions Grőbner basis (grevlex ordering) Polynomial division Eigenvectors ~ solutions

12/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Grőbner basis method  Computing Grőbner basis (even in grevlex ordering) may be very hard  Example: 4 polynomials, 3 variables, degree  6, small integer coeffs  Have extremly simple Grőbner basis

13/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Grőbner basis method  However when computed by the Buchberger’s algorithm over the rational numbers w.r.t. the grevlex ordering x > y > z, the following polynomial appears during the computation: ~ 80,000 digits

14/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Grőbner basis method C o m p l e x i t y Input Grőbner basis General algorithms can construct all GBs but often generate many complicated polynomials G e n e r a l a l g o r i t h m S p e c i f i c a l g o r i t h m

15/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Specific Grőbner basis algorithm  Find a short path towards the Grőbner basis which is independent from the actual coefficients, implement it eficiently run long, coefficient growth numerical problems fast, no coefficient growth, no numerical problems

16/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Specific Grőbner basis algorithm  Use a computer algebra system (Macaulay2) to compute the Grőbner basis in a finite field (fast!) for random coefficients and remember the path  Implement (hard-code) the path efficiently in  Try to avoid numerical instability  J.C. Faugere’s F4 algorithm  Single G-J Elimination (often) makes algorithms more robust than repeating G-J Elimination & multiplication

17/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Minimal problems for cameras with radial distortion  Three new minimal problems for estimating one-parameter radial distortion model and epipolar geometry from point correspondences 1. Minimal algorithm for uncalibrated cameras with same radial distortion and 8 point correspondences (IEEE CVPR 2007) 2. Minimal algorithm for partially calibrated cameras with same radial distortion and 6 point correspondences (IEEE CVPR 2008) 3. Minimal algorithm for uncalibrated cameras with different radial distortions in each image and 9 point correspondences (IEEE CVPR 2008)

18/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Minimal problems for cameras with radial distortion  One parameter division model (Fitzgibbon)  Epipolar constraint  We use the constraint  we obtain systems of algebraic equations  first simplify by eliminating some of the variables  solve using a Grőbner based method  result in the systems of algebraic equations  simplified by eliminating some of the variables  solved using a Groebner basis technique. Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague - uncalibrated cases - calibrated case

19/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  Correct radial lens distortion for uncalibrated cameras with different radial distortions and in each image using the minimal number of point correspondences => 9 correspondences  10 equations in 10 variables  Simplifying these equations => 4 equations in 4 variables (2nd,two 3rd,5th degree)  24 solutions  one G-J elimination of 178x203 matrix (8th degree equations)  runs ~ 16ms Minimal algorithm for uncalibrated cameras with different radial distortions (CVPR 2008) Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague

20/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  Real images, minimal algorithm for cameras with different distortions  66% cutouts from omnidirectional images taken with two different cameras Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Results Before correction After correction Left image Right image

21/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Results Before correction After correction Left image Right image  Cutout from the omnidirectional image and the perspective image

22/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  Given four 2D-to-3D correspondences, estimate camera position, orientation and recover the camera focal length  Generalization of existing P3P and P4P problems for fully calibrated cameras  Planar + general 3D scenes P4P problem for camera with unknown focal length (CVPR 2008) f o X1X1 X2X2 X3X3 X4X4 x1x1

23/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  5 equations in 4 unknowns (focal length + 3 depths)  Solve these equations using the method based on Grőbner basis and generalized companion matrix  one G-J elimination of 78x88 matrix  runtime ~1ms P4P problem for camera with unknown focal length (CVPR 2008)

24/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague Polynomial eigenvalue method Rewrite in a matrix form Rewrite in a form ~Polynomial eigenvalue formulation

25/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  Given 6 point correspondences in two images, find relative position, orientation and focal length of partially calibrated camera  Solutions:  Stewenius et al. CVPR Grőbner basis solution 6 point focal length problem Polynomial eigenvalue solution (BMVC 2008)

26/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  Epipolar constraint  Parametrize the fundamental as a linear combination of a basis of the space of all compatible fundamental matrices  Singularity of F  Trace constraint of E  10 3 rd and 5 th order equations in 3 variables and with 30 monomials 6 point focal length problem Polynomial eigenvalue solution (BMVC 2008)

27/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  Equations can be rewriten where  are known square coefficient matrices   Quadratic polynomial eigenvalue problem  solve using standard efficient algorithms, for example MATLAB function 6 point focal length problem Polynomial eigenvalue solution (BMVC 2008)

28/28 Zuzana Kúkelová Tomáš Pajdla Center for Machine Perception, Czech Technical University in Prague  No general robust and efficient method for solving systems of polynomial equations  Special algorithms have to be designed to achieve numerical robustness and computational efficiency Conclusion THANK YOU FOR YOUR ATTENTION