MCTS with Refinement for Proposals Selection Games in Scene Understanding

Sinisa Stekovic*, Alireza Moradi, Mahdi Rad, Friedrich Fraundorfer, Vincent Lepetit

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


We propose a novel method applicable in many scene understanding problems that adapts the Monte Carlo Tree Search (MCTS) algorithm, originally designed to learn to play games of high-state complexity. From a generated pool of proposals, our method jointly selects and optimizes proposals that minimize the objective term. In our first application for floor plan reconstruction from
point clouds, our method selects and refines the room proposals, modelled as 2D polygons, by optimizing on an objective function combining the fitness as predicted by a deep network and regularizing terms on the room shapes. We also introduce a novel differentiable method for rendering the polygonal shapes of these proposals. Our evaluations on the recent and challenging Structured3D and Floor-SP datasets show significant improvements over the state-of-the-art both in speed and quality of reconstructions, without imposing hard constraints nor assumptions on the floor plan configurations. In our second application, we
extend our approach to reconstruct general 3D room layouts from a color image and obtain accurate room layouts. We also show that our differentiable renderer can easily be extended for rendering 3D planar polygons and polygon embeddings. Our method shows high performance on the Matterport3D-Layout dataset, without introducing hard constraints on room layout configurations.
Original languageEnglish
Number of pages14
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Publication statusE-pub ahead of print - Aug 2022


Dive into the research topics of 'MCTS with Refinement for Proposals Selection Games in Scene Understanding'. Together they form a unique fingerprint.

Cite this