Modelling and Interpretation of Architecture from Several Images

GoBack

Modelling and Interpretation of Architecture fromSeveral ImagesA.R. DICK P.H.S. TORR R. CIPOLLAPaper published: 20041 / 35

⊲ OverviewProblem StatementAlgorithm OverviewThe ArchitecturalModelParametersFormal ProblemFormulationPrior EvaluationrjMCMCOverviewModel AcquisitionAlgorithmResults2 / 35

Problem StatementOverview⊲ ProblemStatementAlgorithm OverviewThe ArchitecturalModelParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResults3 / 35

Algorithm OverviewOverviewProblem Statement⊲ AlgorithmOverviewThe ArchitecturalModelParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithm1. Calculate structure from motion (SFM) from the imagesequence.2. Initial estimation of wall planes.(a)Iterative refinement of wall planes (rjMCMC).3. Initial estimation of primitives on wall.(a)Iterative refinement of primitives on wall(rjMCMC).Results4 / 35

The Architectural ModelOverviewProblem StatementAlgorithm OverviewThe ArchitecturalModel⊲ParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResults□□Each wall is modelled as a rectangle and contains a setof volumetric primitives.An architectural model M contains parameters:θ = {n W ,θ W ,θ P ,θ G }, where:n Wθ Pθ Wis the number of walls in the model.is the position, orientation and boundary of thewalls.the parameters for each wall and θ G the globalparameters that apply to the entire model, i.e.Gothic, Roman, . . .5 / 35

Overview⊲ ParametersWall PlaneVolumetric ShapePrimitivesGlobal ModelParametersFormal ProblemFormulationPrior EvaluationrjMCMCParametersModel AcquisitionAlgorithmResults6 / 35

Wall PlaneOverviewParameters⊲ Wall PlaneVolumetric ShapePrimitivesGlobal ModelParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResultsEach wall plane has a width, height, normal direction andposition: θ i P = wi P ,hi P ,ni ,p i Each wall parameter is furthersubdivided into: θ i W = n P,θ i L ,θi S ,θi T , where:n P is the number of volumetric primitives on the wall.θL i is an identifier for each primitive on the wall, see slide 8.θS i is the shape parameters for each primitive, see slide 8.θTi is the texture parameter for the wall. It is a value in therange (0 255) for each point g on a regular 2D gridcovering the model surface belonging to the wall (notprimitives).7 / 35

Volumetric Shape PrimitivesOverviewParametersWall Plane⊲ VolumetricShape PrimitivesGlobal ModelParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResults8 / 35

Global Model ParametersOverviewParametersWall PlaneVolumetric ShapePrimitivesGlobal ModelParameters⊲Formal ProblemFormulationThe global model parameter has prior information aboutshape, texture and kind of objects composing a building:Prior EvaluationrjMCMCModel AcquisitionAlgorithmResults9 / 35

OverviewParameters⊲Formal ProblemFormulationPosterior - 1Posterior - 2Fully expandedposteriorPrior EvaluationrjMCMCModel AcquisitionAlgorithmFormal Problem FormulationResults10 / 35

Posterior - 1OverviewParametersFormal ProblemFormulation⊲ Posterior - 1Posterior - 2Fully expandedposteriorPrior EvaluationrjMCMCModel AcquisitionAlgorithmResultsWe want to maximise the posterior probability of the modeland parameters, given our data and prior information:P(Mθ|DI), which can be rearranged using Bayes rule anddecomposed via repeated product rule:P(Mθ|DI) ∝ P(D|MθI)P(MθI)=P(D|MθI)P(θ|MI))P(MI)=P(D|MθI)P(θ G θ P θ G |MI))P(MI)=P(D|MθI)P(θ W |θ P θ G MI)P(θ P θ G |MI)P(MI)=P(D|MθI)P(θ W |θ P θ G MI)P(θ P |θ G MI)P(θ G |MI)P(MI)11 / 35

Posterior - 2OverviewParametersFormal ProblemFormulationPosterior - 1⊲ Posterior - 2Fully expandedposteriorPrior EvaluationrjMCMCModel AcquisitionAlgorithmResultsIt is assumed that all the parameters for the primitives areindependent of each other (BAD ASSUMPTION), thus:P(θ W |θ i P θ GMI) =n W∏i=1P(θ i W |θ i Pθ G MI) The wall parametercan be decomposed into its sub-parameters:P(θW i |θi P θ GMI) = P(θL i θi S θi T |θi P θ GMI)=P(θT i |θLθ i Pθ i G MI)P(θS|θ i Lθ i Pθ i G MI)P(θL|θ i Pθ i G MI)12 / 35

Fully expanded posteriorOverviewParametersFormal ProblemFormulationPosterior - 1Posterior - 2Fully expandedposterior⊲Prior EvaluationrjMCMCModel AcquisitionAlgorithmResultsThe probability for the texture of the wall and the shape ofits primitives are independent to each other, thus theequation for the fully expanded posterior is:P(Mθ|DI) ∝ [P(D|MθI)P(θ P |θ G MI)[ nW∏P(θT i |θLθ i Pθ i G MI)] P(θT i |θLθ i Pθ i G MI)i=1]P(θ i S |θi L θi P θ GMI)P(θ i L |θi P θ GMI)Each of the factors has an interpretation, see paper.13 / 35

OverviewParametersFormal ProblemFormulation⊲ Prior EvaluationLikelihoodEvaluationWall Plane PriorEvaluationShape PriorEvaluationTexture PriorEvaluationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResults14 / 35

Likelihood EvaluationOverviewParametersFormal ProblemFormulationPrior Evaluation⊲ LikelihoodEvaluationWall Plane PriorEvaluationShape PriorEvaluationTexture PriorEvaluationrjMCMCModel AcquisitionAlgorithmResultsThe likelihood of the data given the Model (M), theparameters (θ) and the prior information (I) is evaluated as:P(D|MθI) = ∏ ∏ 1√ exp − 1 ( ) i(g j 2) − i(g),2πσǫ 2 σg jǫwhere i(g j ) is the intensity of the gth pixel (sampled atregular intervals) in the jth image.Surfaces are assumed to be Lambertian (colour and intensityemission independent of viewing angle), to make up for thisthe distribution U(0, 255) multiplied by a mixing constant λis added: P(D|MθI) = λ ∗ P(D|MθI) + (1 − λ) 125615 / 35

Wall Plane Prior EvaluationOverviewParametersFormal ProblemFormulationPrior EvaluationLikelihoodEvaluation⊲Wall Plane PriorEvaluationShape PriorEvaluationTexture PriorEvaluationrjMCMCModel AcquisitionAlgorithmResultsThe wall plane prior is based on the architectural heuristicsthat neighbouring walls intersect at right angles, they areperpendicular to the ground plane and the height-width ratioof walls is constraint. The terms composing the prior areindependent of each other:P(θ P |θ G MI) = P(n|θ G MI)P(p|θ G MI)P({h p ,w P }|θ G MI)Each of these terms is represented by a simple distribution:∑n−1( ) 2 φI − π/2P(n|θ G MI)α, where φ i is the interiori=1σ φangle between planes i and i + 1.P({h p ,w p }|θ G MI) = 1, if 0.25 ≤ h P /w P ≤ 4= 0, otherwiseThe distribution of the walls position is uniform.16 / 35

Shape Prior EvaluationOverviewParametersFormal ProblemFormulationPrior EvaluationLikelihoodEvaluationWall Plane PriorEvaluation⊲Shape PriorEvaluationTexture PriorEvaluationrjMCMCModel AcquisitionAlgorithmResultsThe architectural shape prior for primitives,P(θ S |θ L θ P θ G MI), is defined and assessed using a MarkovChain simulation. This prior encodes information about:□□□□The scale of each primitive (if absolute scale is known).The shape of each primitive.The alignment of primitives.Other spatial relations, such as symmetry about avertical axis.As it is not feasible to draw samples from this distributiondirectly MCMC has to be used. They use Reversible JumpMarkov Chain Monte Carlo sampling, because of the varyingdimension of the parameter space.17 / 35

Texture Prior EvaluationOverviewParametersFormal ProblemFormulationPrior EvaluationLikelihoodEvaluationWall Plane PriorEvaluationShape PriorEvaluationTexture PriorEvaluation⊲rjMCMCModel AcquisitionAlgorithmThe texture prior, P(θ T |θ L θ P θ G MI), is learnt from a settraining of images. To reduce the impacts such as lightningand scale of the textures, a wavelet decomposition is appliedto the images. The output of that decomposition gives 4transformed images: HH, HL, LL and HH, where the firstletter stands for the horizontal direction and the second forthe vertical direction; H indicates the high-pass and L thelow-pass filter, respectively. These outputs are quantised andput into bins.Results18 / 35

OverviewParametersFormal ProblemFormulationPrior Evaluation⊲ rjMCMCScoring functionJumpingDistributionList of JumpsModel AcquisitionAlgorithmrjMCMCResults19 / 35

Scoring functionOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMC⊲ Scoring functionJumpingDistributionList of JumpsModel AcquisitionAlgorithmResultsThe scoring function is used to evaluate the shape prior bycombining its individual terms:f prior (θ) = f scale (θ) + f shape (θ) + f align (θ)f sym (θ) The shapeand scale terms are used to disqualify implausible primitives. Thealignment and symmetry terms are evaluated as:R∑f align (θ) = [V ar(t r ) + V ar(b r ) + V ar(r r − l r )] +r=1C∑[V ar(l c ) + V ar(r c ) + V ar(t c − b c )], where t r , b r , l r , r r andc=1t c , b c , l c , r c are the top, bottom, left and right coordinatesbelonging to a row and column of shapes, respectively.R∑f sym (θ) = [(l r − l) − (r r − r)] 2 , where l r and r r are ther=1leftmost and rightmost point of row r, respectively. NB:Symmetry only defined for columns.20 / 35

Jumping DistributionOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCScoring function⊲ JumpingDistributionList of JumpsModel AcquisitionAlgorithmrjMCMC requires a jumping distribution J t (θ t |θ t−1 ). Therequirements for this distribution are:□□□When adding/removing walls, the building should stillremain closed.Should be easy to sample, with high acceptance, i.e.jumps increase probability of model.Jumps must be reversible.Results21 / 35

List of JumpsOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCScoring functionJumpingDistribution⊲ List of JumpsModel AcquisitionAlgorithmResults22 / 35

OverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModelAcquisition⊲ AlgorithmDetailed AlgorithmOverviewWall PlanesOptimising WallPlanesPrimitivesPrimitive LikelihoodsPrimitive parameterrefinementMAP ModelModel Acquisition AlgorithmResults23 / 35

Detailed Algorithm OverviewOverviewParametersFormal ProblemFormulationFull algorithm for obtaining the MAP solution:Prior EvaluationrjMCMCModel AcquisitionAlgorithmDetailedAlgorithm⊲ OverviewWall PlanesOptimising WallPlanesPrimitivesPrimitive LikelihoodsPrimitive parameterrefinementMAP ModelResults24 / 35

Wall PlanesOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmDetailed AlgorithmOverview⊲ Wall PlanesOptimising WallPlanesPrimitivesPrimitive LikelihoodsPrimitive parameterrefinementMAP ModelFrom the point cloud obtained from the SFM, the wallplanes need to be estimated. First the the dominant planesare extracted using a RANSAC technique, where the twomost commonly occurring orientations of lines are belongingto each plane are taken as the horizontal and vertical axis ofthe plane. Next the sides of the wall are aligned with thesedirections. The dimensions of the wall is the minimum spanof all points from that plane. The inliers for that plane areremoved form the point cloud and the same technique isrepeated, to find subsequent wall planes.Results25 / 35

Optimising Wall PlanesOverviewParametersFormal ProblemFormulationPrior EvaluationThe two main error in the initial wall plane estimation arethe ground plane normal and the bounding rectangle, so themodel is optimized using gradient descent.rjMCMCModel AcquisitionAlgorithmDetailed AlgorithmOverviewWall PlanesOptimising Wall⊲ PlanesPrimitivesPrimitive LikelihoodsPrimitive parameterrefinementMAP ModelResultsPlot in middle error surface is flat, because sky and groundare featureless.26 / 35

PrimitivesOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmDetailed AlgorithmOverviewWall PlanesOptimising WallPlanes⊲ PrimitivesPrimitive LikelihoodsPrimitive parameterrefinementMAP ModelResultsParameters θ S 1 are sampled at regular intervals for eachprimitive θ L . The likelihood is evaluated from the singlemost frontal image D 1 as:P(D 1 |θ T 1θ S 1θ L MI) =n W∏i=0∏x∈θ i T∏kf θi Lk (W k(x)), whereW k (X) denotes a 3x3 patch of subband k of the waveletdecomposition centred at x. f θi Lkdenotes frequency in thenormalised histogram for primitive type θL i of subband k.□Texture histograms are learnt for different subtypes ofprimitives (window patterns)27 / 35

Primitive LikelihoodsOverviewParametersFormal ProblemFormulationCollection of shape primitives and their likelihoods.Prior EvaluationrjMCMCModel AcquisitionAlgorithmDetailed AlgorithmOverviewWall PlanesOptimising WallPlanesPrimitives⊲ PrimitiveLikelihoodsPrimitive parameterrefinementMAP ModelResults28 / 35

Primitive parameter refinementOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmDetailed AlgorithmOverviewWall PlanesOptimising WallPlanesPrimitivesPrimitive LikelihoodsPrimitiveparameter⊲ refinementMAP ModelResults□□□□After the parameters {x,y,w,h} from the single viewhave been initialised, from multiple views the parametersthe remaining parameters for the primitives {d,a,b,dw}are estimated.Occam Factor is used to penalize model complexity andkeep the models as simple as possible, to avoidoverfitting.To evaluate whether a primitive should be kept in themodel, its full likelihood ration against the model withno primitives is compared: L i M = P(D|θi MI)P(D|θ 0 M 0 IThe maximum likelihood model would include allnon-overlapping shapes with ratio < 1.29 / 35

MAP ModelOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmDetailed AlgorithmOverviewWall PlanesOptimising WallPlanesPrimitivesPrimitive LikelihoodsPrimitive parameterrefinement⊲ MAP ModelResultsThe complete scoring function, including the imageinformation, is given by:□ f(θ) = λf prior (θ + ∑ m∑( ) i(g j 2) − i(g))σi(g) j=1□ Seed points from previous MAP estimations, with anacceptance threshold of a fixed range of primitives permodel.30 / 35

OverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithm⊲ ResultsComparison toground truthDowning LibraryAmbiguitiesNotesResults31 / 35

Comparison to ground truthOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResultsComparison to⊲ ground truthDowning LibraryAmbiguitiesNotes32 / 35

Downing LibraryOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResultsComparison toground truth⊲ Downing LibraryAmbiguitiesNotes33 / 35

AmbiguitiesOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResultsComparison toground truthDowning Library⊲ AmbiguitiesNotes34 / 35

NotesOverviewParametersFormal ProblemFormulationPrior EvaluationrjMCMCModel AcquisitionAlgorithmResultsComparison toground truthDowning LibraryAmbiguities⊲ Notes□□Model choice (prior) is not that important forreconstruction.Initial wall planes estimates need to be accurate,otherwise primitive detection becomes unreliable.35 / 35