Multi-View Stereo Based on Attention and Epipolar geometry

• 1、Computer Science and Technology School, Beijing Jiaotong University, Beijing 100044
• 2、Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing 100044
• 3、Key Laboratory of Beijing for Railway Engineering, Beijing Jiaotong University, Beijing 100044

Abstract：\justifying Multi-View Stereo (MVS) based on deep learning can be roughly divided into four stages: feature extraction, cost volume construction, cost volume regularization, and cost regression. In the feature extraction stage, Feature Pyramid Networks (FPN) are commonly used to extract features from reference images and sourceimages. However, FPN struggles to capture global information within images and relational information between images. In the cost volume construction stage, variance or the innercost volume, with the related feature vectors being assigned equal weights by default. Currently, there are still challenges in reconstructing non-Lambertian surfaces and weakly textured areas.To address this, this paper designs a Transformer-based multi-view stereomodel called AE2MVSNet (Attention and Epipolar Geometry to MVSNet). In the feature extraction stage, global information within images and relational information between images are extracted using self-attention and cross-attention mechanisms, respectively. In the cost volume construction stage, the attention mechanism is applied in conjunction with MVS principles to build the cost volume. Experimental results show that on the DTU dataset and the Tanks and Temples public dataset, AE2MVSNet outperformsexisting MVS models in various metrics such as accuracy, completeness, and F-score.

Keywords： multi-view stereo attention mechanism epipolar geometry deep learning