Smart Computational Imaging (SCI) Lab

A typical fringe projection system

Multi-frequency temporal phase unwrapping

Schematic of a typical DLP projector

Flowchart of the proposed method

Measurement system and 3-D reconstruction results

3D thermograms reconstruction result

Calibration process for the unknown coefficients

The result of ouliers optimization algorithm

Illustration of the influence of abnormal phase jumps on multi-wavelength temporal phase unwrapping algorithm

The 3-D reconstruction result of the high-contrast surface

Illustration of the geometry constraint

The schematic of the dual-camera fringe projecion system

Real-time 3-D Imaging and System

Fringe projection profilometry (FPP) is playing a significant role in both scientific and industrial areas. Real-time 3-D imaging technique is a hotspot in this research field, and we are dedicated to develop a new system to make FPP more suitable for dynamic scenes. Decreasing the number of projected patterns is one of the valid schemes and it is our main line. Our new technique has realized real-time 3-D reconstruction with four or three patterns. Besides the related algorithm, the hardware for fringe projection and sychronization between camera and projector is also presented. Accompanied by this system, some other technique details were also proposed including the primary algorithm to obtain the absolute phase map, the method for calibration, and how to realize high dynamic measurement.



Fringe projection profilometry (FPP) is a noncontact, optical, active, triangulation-based 3D reconstruction technique that is well known for its simple implementation, low cost, and high accuracy. FPP works on the same principle as stereo vision, where an object's coordinate in 3D space is derived by triangulating between pixels of two cameras. SLI avoids the computational complexities of matching pixels across camera views by replacing one of the two component cameras with a projector that generates a series of fringe patterns. By analyzing the change in the pattern at a particular point on the target object's surface (a process known as phase demodulation and unwrapping of the captured fringes), unique correspondences can be derived between the camera and projector pixels.  A typical fringe projection profilometry system is shown on the right. It consists of a projection unit, an image acquisition unit and a processing/analysis unit. Measurement of shape through fringe projection techniques involves (1) projecting a structured pattern (usually a sinusoidal fringe pattern) onto the object surface, (2) recording the image of the fringe pattern that is phase distorted by the object height distribution, (3) calculating the (wrapped) phase by analyzing the image with fringe analysis techniques (such as Fourier transform method and phase shiftting method) (4) using a suitable phase unwrapping algorithm to get continuous phase distribution, and finally (5) calibrating the system for mapping the unwrapped phase distribution to real world 3-D coordinates.


We are dedicated to innovate high-resolution, high speed , real-time 3-D imaging/sensing methods and systems for numerous applications such as human-to-computer interaction, computer graphics, healthcare, and the manufacturing industry. We have developed a real-time three-dimensional (3D) imaging system based on digital fringe projection, both hardware and software. The system achieves 120 fps real-time data acquisition, processing and visualization. This novel system employs high speed fringe projection [by modified a commercial DLP projector with self-developed Field-Programmable Gate Array (FPGA) circuit], pattern design, direct phase unwrapping (novel 4-pattern strategy to realize phase shifting & temporal phase unwrapping), gamma distortion correction (pre-coding the distorted pattern in the on-chip memory of the FPGA), noise reduction, motion artifacts suppression, filter design and other techniques (implemented in C++ with Visual Studio 2008).  By combining an infrared camera (self-developed 640×512 VOx Microbolometer) with this system, real-time three-dimensional thermal imaging can be realized.


However, traditional fringe projection profilometers are usually designed to measure static object shapes under non-time-critical measurement conditions, their measurement of moving objects is limited. Besides, it is difficult to retrieve the absolute phase for spatially isolated surfaces simultaneously and rapidly since in the phase unwrapping, fringe orders will be ambiguous ( the so-called 2π phase discontinuities problem), making the depth difference between spatially isolated surfaces indiscernible. One typical  phase unwrapping method called mult-frequency temporal phase unwrapping (MFTPU).  By varying fringe frequency on an object several times, the MFTPU algorithm can be used to determine the fringe order number and, based on this, resolve 2π phase discontinuities. However, the MFTPU generally requires quite a lot of fringes (at least 6 for two frequency TPU) that inevitably prolongs the acquisition time.


Another bottleneck for realizing high-speed fringe projection is the fringe projection speed. A typical DLP projector contains two important components: a DMD chip and a color wheel. The DMD chip modulates the projected light after it is reflected off an array of tiny mirrors. Any displayed intensity is made up of pulses of light created by these mirrors switching on and off. The DLP video processor system separates color images into their RGB components for display, employing a rotating color wheel synchronized to RGB image separations displayed by the DMD. The color wheel spins at 120 Hz (in some older types of DLP projector, the color wheel may operate at 60 or 80 Hz) and is divided into multiple sectors: the primary colors: red, green, and blue. The synchronization between the DMD and the color-wheel is controlled by a 120 Hz feedback trigger signal generated by a photodiode located inside the color-wheel carrier. Once the DMD is triggered, it will project the RGB channels of the input color image sequentially by binary temporal pulse-width-modulation.  To triple the speed of the dlp projector, we should first remove the color filter of the projector and then create a sequence of color encoded pattern images, sent them sequentially and repeatedly at 120 Hz (by using proper hardware). When the projector receives this pattern sequence, the gray-scale patterns can be decoded by the DLP video processor system and then projected them in a correct order (360Hz). We are trying to hack the the commercial off-the-shelf DLP projector to realize high-speed 3D imaging system at a very low cost.

Overview of Our Work
The main research work and contributions are summarized as the following aspects (click picture for details):


Real-time 3D Imaging System


Identification and Removal of Invalid Points


Calibration Considering Lens Distortion


Realtime 3D infrared Imaging

Speckle-embedded Fringe plus GPU

General solution for high dynamic range 3D imaging

Real-time 3-D Imaging System

Optical non-contact three-dimensional (3-D) shape measurement techniques have been employed for many years to measure 3-D profile of objects. The 3-D profilometry based on fringe pattern projection offers advantageous of the non-scanning nature and full-field performance, which is widely employed and tested in research or industrial inspection, quality control, machine vision, entertainment, and biomedicine etc. With the recent advancement in digital display devices such as liquid crystal display (LCD) and digital mirror device (DMD), 3-D shape measurement based on digital fringe projection (DFP) units has been rapidly expanding. However, traditional DFP profilometers are usually designed to measure static object shapes under non-time-critical measurement conditions, their measurement of moving objects is limited [3]. Besides, it is difficult to retrieve the absolute phase for spatially isolated surfaces simultaneously and rapidly since in the phase unwrapping, fringe orders will be ambiguous, making the depth difference between spatially isolated surfaces indiscernible.

Real-time 3-D measurement with 2+2 phase shifting algorithm
This paper describes an easy-to-implement three-dimensional (3-D) real-time shape measurement technique using our newly developed high-speed 3-D vision system. It employs only four projection fringes to realize full-field phase unwrapping in the presence of discontinuous or isolated objects. With our self-designed pattern generation hardware and a modified low-cost DLP projector, the four designed patterns can be generated and projected at a switching speed of 360 Hz. Using a properly synchronized high-speed camera, the high-speed fringe patterns distorted by measured objects can be acquired and processed in real-time. The resulting system can capture and display high-quality textured 3-D data at a speed of 120 frames per second, with the resolution of 640 × 480 points. The speed can be trebled if a camera with a higher frame rate is employed. We detail our shape measurement technique, including the four-pattern decoding algorithm as well as the hardware design. Some evaluation experiments have been carried out to demonstrate the validity and practicability of the proposed technique.


3-D measurement using speckle-embedded fringe

This paper presents a novel two-frame fringe projection technique for real-time, accurate, and unambiguous three-dimensional (3-D) measurement. One of the frames is a digital speckle pattern, and the other one is a composite image which is generated by fusing that speckle image with sinusoidal fringes. The contained sinusoidal component is used to obtain a wrapped phase map by Fourier transform profilometry, and the speckle image helps determine the fringe order for phase unwrapping. Compared with traditional methods, the proposed pattern scheme enables measurements of discontinuous surfaces with only two frames, greatly reducing the number of required patterns and thus reducing the sensitivity to movements. This merit makes the method very suitable for inspecting dynamic scenes. Moreover, it shows close performance in measurement accuracy compared with the phase shifting method from our experiments. To process data in real time, a Compute Unified Device Architecture–enabled graphics processing unit is adopted to accelerate some time-consuming computations. With our system, measurements can be performed at 21 frames per second with a resolution of 307,000 points per frame.


Real-time 3-D shape measurement with multi-view system

In recent years, fringe projection has become an established and essential method for dynamic three-dimensional (3-D) shape measurement in di erent fields such as online inspection and real-time quality control. Numerous high-speed 3-D shape measurement methods have been developed by either employing high speed hardware, minimizing the number of pattern projection, or both. However, dynamic 3-D shape measurement of arbitrarily-shaped objects with full sensor resolution without the necessity of additional pattern projections is still a big challenge. In this work, we introduce a high-speed 3-D shape measurement technique based on composite phase-shifting fringes and a multi-view system. The geometry constraint is adopted to search the corresponding points independently without additional images. Meanwhile, by analysing the 3-D position and the main wrapped phase of the corresponding point, pairs with an incorrect 3-D position or a considerable phase di erence are e ectively rejected. All of the qualified corresponding points are then corrected, and the unique one as well as the related period order is selected through the embedded triangular wave. Finally, considering that some points can only be captured by one of the cameras due to the occlusions, these points may have di erent fringe orders in the two views, so a left-right consistency check is employed to eliminate those erroneous period orders in this case. Several experiments on both static and dynamic scenes are performed, verifying that our method can achieve a speed of 120 frames per second (fps) with 25-period fringe patterns for fast, dense, and accurate 3-D measurement.


3D infrared imaging using fringe projection

Infrared thermography determines the surface temperature of an object or human body. It is a promising imaging technology for medical and biological observations due to its contactless and completely non-invasive properties. However, traditional two dimensional (2-D) infrared thermography cannot retain the spatial information, and thus provides only qualitative diagnosis information. A novel real-time three dimensional (3-D) infrared imaging system which takes full advantages of high-speed, high-quality, high-sensitivity, and low-cost in 3D thermograph is presented. We demonstrate the real time 3-D thermal imaging at the speed of 24 frames per second (fps), with resolution of 640 × 480 points. Experimental results demonstrate quantitatively measurement of temperature distribution of 3-D surfaces in real-time is realized with this system.


Algorithms and Optimization

Phase unwrapping is one of the most interest and challenging algorithm in three-dimensional (3-D) shape measurement techniques using structured light. The unwrapped phase map containing the depth information can be obtained by this algorithm. Once this phase map is given, 3-D information in world coordinate is known through calibration between the camera and the projector which is another primary algorithm in 3-D reconstruction. Certainly, the noise and its effects can not be ignored so in the edge of fringes, errors emerge, and some measures are introduced to eliminate these errors.

A comparative review for temporal phase unwrapping

In fringe projection profilometry (FPP), temporal phase unwrapping is an essential procedure to recover an unambiguous absolute phase even in the presence of large discontinuities or spatially isolated sur-faces. So far, the reare typically three groups of temporal phase unwrapping algorithms proposed in the literature: multi-frequency (hierarchical) approach, multi-wavelength (heterodyne) approach, and number theoretical approach. In this paper, the three methods are investigated and compared in detail by analytical, numerical, and experimental means. The basic principles and recent developments of the three kind of algorithms are firstly reviewed. Then, the reliability of different phase unwrapping algorithms is compared based on arigorous stochastic noise model. Furthermore, this noise model is used to predict the optimum fringe period for each unwrapping approach, which is a key factor governing the phase measurement accuracy in FPP. Simulations and experimental results verified the correctness and validity of the proposed noise model as well as the prediction scheme. The results show that the multi-frequency temporal phase unwrapping provides the best unwrapping reliability, while the multi-wavelength approach is the most susceptible to noise-induced unwrapping errors.


Calibration considering camera lens distortion

Optical three-dimensional (3-D) profilometry is gaining increasing attention for its simplicity, flexibility, high accuracy,and non-contact nature. Recent advances in imaging sensors and digital projection technology further its progress in high-speed, real-time applications, enabling 3-D shapes reconstruction of moving objects and dynamics cenes. However, the camera lens is never perfect and the lens distortion does influence the accuracy of the measurement result, which is often overlooked in the existing real-time 3-D shape measurement systems.To this end, here we present an ovel high-speed real-time3-D coordinates measuring technique based on fringe projection with the consideration of the camera lens distortion. A pixel mapping relation between a distorted image and a corrected one is pre-determined and stored in computer memory for real-time fringe correction. The out-of-plane height is obtained firstly and the acquisition for the two corresponding in-plane coordinates follows on the basis of the solved height. Besides, a method of look up table (LUT) is introduced as well for fast data processing.

Our experimental results reveal that the measurement error of the in-plane coordinates has been reduced by one order of magnitude and the accuracy of the out-plane coordinate been tripled after the distortions being eliminated. Moreover, owing to the generated LUTs, a 3-D reconstruction speed of 92.34 frames per second can be achieved.


Automatic identification and removal of outliers

Phase-shifting profilometry combining with the two-frequency temporal phase unwrapping is widely used for high-speed, real-time acquisition of three-dimensional (3-D) shapes. However, when the object is not motionless during the acquisition process, some unreliable results may emerge, especially around the contours of the measured object. The main reason for this is that the same point in the projected pattern sequence can map to different points within the camera images resulting from depth changes over time. We present a novel approach for identifying those invalid pixels affected by such an error. By carefully examining the captured fringe pattern, comparing two modulation maps, utilizing the phase relationship between two neighboring pixels, and employing a Gaussian filter to detect the protruding points, the bad measurement pixels can be detected and filtered out effectively. The whole procedure is of low computational complexity because of the introduced lookup table-based fast data processing method.


High-dynamic-range 3D imaging

Fringe projection profilometry (FPP) has been rapidly expanding due to its nature of non-contact, fast speed, meanwhile the optical property increase the difficulty in the measurement of high-contrast surface. In recent years, changing the exposure of the camera to handle this problem is the main stream, but it is not convenient to real-time system. Here we will introduce several schemes suitable for real-time system to measure the shiny surface.  

General solution for the measurement of high dynamic range

This paper presents a general solution for realizing high dynamic range three-dimensional (3-D) shape measurement based on fringe projection. Three concrete techniques are involved in the solution for measuring object with large range of reflectivity (LRR) or one with shiny specular surface. For the first technique, the measured surface reflectivities are sub-divided into several groups based on its histogram distribution, then the optimal exposure time for each group can be predicted adaptively so that the bright as well as dark areas on the measured surface are able to be handled without any compromise. Phase-shifted images are then captured at the calculated exposure times and a composite phase-shifted image is generated by extracting the optimally exposed pixels in the raw fringes images.For the second technique, it is proposed by introducing two orthogonal polarizers which are placed separately in front of the camera and projector into the first technique and the third one is developed by combining the second technique with the strategy of properly altering the angle between the transmission axes of the two polarizers.


Dynamic measurement system for shiny surfaces

This paper presents a novel fringe projection technique for fast three-dimensional (3-D) shape measurements of moving highly reflective objects. By combining the standard three-step phase-shifting fringe patterns with a digital speckle image, dynamic 3-D reconstructions of shiny surfaces can be efficiently achieved with only four projected patterns. The phase measurement is performed by three-step phase-shifting algorithm as it uses the theoretical minimum number of fringe patterns for phase-shifting profilometry. To avoid the camera saturation, a dual-camera fringe projection system is built to measure shiny objects from two different directions. The erroneous phase obtained from a saturated pixel is corrected by the phase of its corresponding pixel in the other view which is free from the saturation problem. To achieve high measurement accuracy, the corresponding high light intensity areas in cameras are found by sub-pixel matches of the speckle pattern in either view. Benefited from the trifocal tensor constraint, the corresponding points in the two wrapped phase maps can be directly established, and thus, the difficulties in determining the correct fringe order for the discontinuous or isolated surfaces can be effectively bypassed. Experimental results indicate that the proposed method is able to successfully measure highly reflective surfaces for both stationary and dynamic scenes.