LIPIDS: learning-based illumination planning in discretized (light) space for photometric stereo

Abstract

Photometric stereo is a powerful technique for estimating per-pixel surface normals from images under varied il-lumination. Although several methods address photometric stereo with different image (or light) counts ranging from one to two to a hundred, very few focus on learning optimal lighting configuration. Finding an optimal configuration is challenging due to the large number of possible lighting di-rections. Moreover, exhaustive sampling of all possibilities is impractical due to time and resource constraints. Pho-tometric stereo methods have demonstrated promising per-formance on existing datasets, which feature limited light directions sparsely sampled from the light space. There-fore, can we optimally utilize these datasets for illumination planning? In this work, we introduce LIPIDS - Learning-based Illumination Planning In Discretized light Space to achieve minimal and optimal lighting configurations for photometric stereo under arbitrary light distribution. We propose a Light Sampling Network (LSNet) that optimizes the lighting direction for a fixed number of lights by min-imizing the normal loss through a normal regression net-work. The learned light configurations can directly estimate surface normals during inference, even using an off-the-shelf photometric stereo method. Extensive qualitative and quantitative analysis on synthetic and real-world datasets show that photometric stereo under learned lighting config-urations through LIPIDS either surpasses or is nearly com-parable to existing illumination planning methods across different photometric stereo backbones.