Smart Computational Imaging (SCI) Lab


【AlphaGalileo】The synergy of traditional techniques and deep learning enables single‐frame highpreci

Thesynergy of traditional techniques and deep learning enables single‐framehighprecision fringe pattern analysis

A new publication from Opto Electronic Advances, 10.29026/oea.2024.230034 discusses how the synergy of traditional techniques and deep learning enables single frame high-precision fringe pattern analysis.

Optical metrology, as a general-purpose metrology technique that uses light as information carriers for non-contact and non-destructive measurement, is fundamental to manufacturing, basic research,and engineering applications. With the invention of the laser and charge-coupled device (CCD), many optical metrology methods and instruments are employed in state-of-the-art manufacturing processes, precision positioning, and quality assessment because of their advantages in terms of accuracy, sensitivity, repeatability, and speed. For many optical metrology techniques such as interferometry, digital holography, and fringe projection profilometry (FPP), fringe pattern analysis is the primary focus of research for recovering the underlying phase distribution from the recorded fringe patterns. The accuracy and efficiency of phase retrieval from fringe patterns are essential to dynamically reconstruct various desired physical properties of the objects (the profile, distance,strain, etc.).

For structured light 3D imaging based on FPP, to minimize the number of fringes required for a single reconstruction, Prof. Qian Chen and Chao Zuo’s research group at Nanjing University of Science and Technology established the theoretical framework for phase shifting profilometry and temporal phase unwrapping and developed a series of composite phase shifting methods for fast 3D measurement, including bi-frequency phase shifting, 2+2 phase shifting, geometric constraints based composite phase shifting, and micro Fourier transform profilometry (μFTP). These composite phase shifting methods reduce the number of fringe patterns required per 3D reconstruction from ~10 to 5, 4, 3, or even 2, achieving high-speed 3D sensing at 10k frames per second. Nevertheless, high accuracy 3D reconstruction usingonly one single pattern has been the ultimate goal of structured light 3D imaging in perpetual pursuit. However, the key to the success of FTP is thatthe high-frequency fringe information modulated by the object surface can be well separated from the background intensity in the frequency domain. As a result, the FTP technique is limited to measuring smooth surfaces with limited height variations.

Recently, with the explosive growth of available data and computing resources, deep learning, as a “data-driven”machine learning technique, has achieved impressive success in numerous fields, such as computer vision and computational imaging. Deep learning, which pervades almost all aspects of optical metrology, provides solutions to many challenging problems, such as fringe denoising, fringe analysis, and digital holographic reconstruction.

However, different from traditional fringe analysis methods, these deep learning approaches focus mainly on training a DNN to accurately identify an image-toimage transform from massive input and output data pairs, as the physical laws governing the image formation or other domain expertise pertaining to the measurement have not yet been fully exploited in current deep learning practice. Consequently, the performance of deep learning approaches in solving complex physical problems relies heavily on the underlying statistical characteristics within the dataset. In order to push the limits of fringe pattern analysis in terms of speed, accuracy, repeatability, and generalization, the synergy of physics-based traditional methods and data-driven learning approaches become to represent the general trend [Fig.1(b)].

In a recent publication in Opto-Electronic Advances, Prof. Qian Chen and Prof. Chao Zuo’s research group at Nanjing University of Science and Technology reported a physics-informed deep learning method for fringe pattern analysis (PI-FPA), which integrates a light weight DNN with a learning-enhanced Fourier transform profilometry (LeFTP) module [Fig.2], enabling more accurate and computationally efficient single-shot phase retrieval. The lightweight network refines the initial phase to further improve the phase accuracy at a low computational cost, compared with universal end-to-end image transform networks (U-Net and its derivatives).

Dynamic 360-degree 3D reconstruction results of a workpiece model by different fringe analysis methods are shown in [Fig. 3].For the 3-step PS method, when dynamic scenes are measured, the relative motion between the object and the phase-shifting fringe patterns sequentially projected will cause motion artifacts and thus introduce non-negligible errors into 3D reconstruction results. For single-frame fringe analysis, FTP is suitable for dynamic 3D measurement, but yields coarse 3D results with low quality in terms of accuracy and resolution due to the spectrum overlapping. U-Net can further improve the quality of 3D reconstruction, but it cannot reliably retrieve the phase of the object with metal materials which is relatively rare in the training dataset, precluding the recovery of fine surfaces. This experiment demonstrates that the proposed PI-FPA can be applied for high-quality and efficient 3D modeling of complex structure parts.

The proposed physics-informed deep learning technique for fringe pattern analysis (PI-FPA) not only learns the inherent statistical characteristics within the dataset like traditional neural networks, but also masters the physical laws describing the image formation, realizing single-frame phase reconstruction with high precision and high computational efficiency, while exhibiting its good generalization to rare samples never seen by the network. In the future, we will investigate the phase recovery performance of PI-FPA for different types of fringe images, and explore related fringe analysis applications in the fields of interferometry and digital holography in optical metrology, further pushing the limits of fringe pattern analysis in speed, accuracy, repeatability, and generalization.

# # # # # #

Smart Computational Imaging Laboratory (, or SCILab for short, is part of the Spectral Imaging and Information Processing Innovative Team and the first batch of the Huang Danian-Style Teacher Team in China led by Chang Jiang Scholar Prof. Qian Chen,vice president of Nanjing University of Science and Technology. Prof. Chao Zuo, the principal investigator of SCILab, is the Fellow of SPIE and Optica and a recipient of the National Science Fund for Distinguished Young Scholars. Drivenby national demand, the lab is dedicated to exploring the mechanism of new-typeoptical imaging, the practice of engineering, and the development of advanced instruments. The Lab’s research results have been published in more than 180 SCI-indexed papers with over 12,000 citations. Among them, more than 32 papershave been featured on journal covers of Opto-Electronic Advances, Light: Science and Applications, Optica, and Advanced Photonics, etc. and 19 papers have been selected as ESI highly cited papers/hot papers. Academics aside, SCILab members have earned various accolades, such as the First Prize ofTechnical Invention of the China Optical Engineering Society, National Outstanding Doctoral Dissertation/Nomination in Optical Engineering and the Wang Daheng Optical Award of the Chinese Optical Society. Figure 6 is a group photo of the SCILab team at Nanjing University of Science and Technology.

# # # # # #

Opto-Electronic Advances (OEA) is ahigh-impact, open access, peer reviewed monthly SCI journal with an impactfactor of 14.1 (Journal Citation Reports for IF2022). OEA is indexed in SCI, EI, DOAJ, Scopus, CA and ICI databases.

The journal is published by The Institute of Optics and Electronics, Chinese Academy of Sciences, aiming at providing a platform for researchers, academicians, professionals, practitioners, andstudents to impart and share knowledge in the form of high quality empirical and theoretical research papers covering the topics of optics, photonics andoptoelectronics.

# # # # # #

More information:

Editorial Board:

All issues available in the online archive(

Submissions to OEA may be made using ScholarOne(

ISSN: 2096-4579

CN: 51-1781/TN

Contact Us:

Twitter: @OptoElectronAdv (

WeChat: OE_Journal

# # # # # #

Yin W, Che YX, Li XS, Li MY, Hu Y et al.Physics-informed deep learning for fringe pattern analysis. Opto-Electron Adv7, 230034 (2024). doi: 10.29026/oea.2024.230034

# # # # # #

来源 | Alphagalileo

复审 | 左超

终审 | 徐峰

SCILab 官方公众号
SCILab 官方B站