Capturing the 3D shape of objects and scenes has been explored within the field of computer vision for many years. Whether for computer graphics, motion pictures, or sports medicine, being able to accurately resconstruct the motion of a scene or actor is highly desirable. We develop a system that can generate 3D video from arbitrary positions surrounding a central reconstruction volume. We accomplish this by using three structured light stations positioned around a central volume. Systems which simply use multiple cameras to capture the 3D models can have difficulty reconstructing the scene when there is little uniquely identifiable texture. By using structured light, rather than just passive cameras, each point in the scene is guaranteed to have have enough texture to be identified by capturing cameras. Our current system consists of three structured light stations as illustrated below.
Our current results are constructed using a single structured light station. Within each station, two grayscale PointGray Dragonfly Express cameras are used with an Optoma TX 780 projector to capture depth. Each station is also equipped with a PointGray Flea2 for eventual use to capture the color texture of the scene.
Publications
Below is a listing of publications with abstracts for our work within the field of structured light depth capture.
Projector Domain Phase Unwrapping in a Structured Light System with Stereo Cameras
Abstract
Phase-shifted sinusoids are commonly used as projection patterns in structured light systems consisting of projectors and cameras. They require few image captures per reconstruction and have low decoding complexity. Recently, structured light systems with a projector and a pair of stereo cameras have been used in order to overcome the traditional phase discontinuity problem and allow for the reconstruction of scenes with multiple objects. In this paper, we propose a new approach to the phase unwrapping process in such systems. Rather than iterating through all pixels in the two cameras to determine the global phase of each pixel, we iterate through the projector pixels to solve for correspondences between the two camera views. An energy minimization framework is applied to these initial estimated correspondences to enforce smoothness and to fill in missing pixels. Unlike existing approaches, our method allows simultaneous unwrapping of both camera images and enforces consistency across them. We demonstrate the effectiveness of our approach experimentally on a few scenes
R. Garcia, A. Zakhor, "Projector Domain Phase Unwrapping in a Structured Light System with Stereo Cameras," 3DTV 2011, Antalya, Turkey, May 16-18, 2011. [Adobe PDF]
Temporally-Consistent Phase Unwrapping for a Stereo-Assisted Structured Light System
Abstract
Phase shifted sinusoidal projection patterns are extensively used in structured light systems due to their low decoding complexity and projection of only three frames per reconstruction. However, they require correct unwrapping of phase images to reconstruct depth accurately. For time varying scenes, it is important for the phase unwrapped results to be temporally coherent. In this paper, we propose a spatio-temporal phase unwrapping algorithm for a structured light system made of a projector and a pair of stereo cameras. Unlike existing frame by frame spatial domain approaches, our proposed algorithm results in a temporally consistent threedimensional unwrapped phase volume for time varying scenes. We experimentally show the effectiveness of our approach in recovering absolute phase for scenes with multiple disjoint objects, significant motion, and large depth discontinuities.
R. Garcia, A. Zakhor, "Temporally-Consistent Phase Unwrapping for a Stereo-Assisted Structured Light System," 3DIMPVT 2011, Hangzhou, China, May 16-19, 2011. [Adobe PDF]
Comparison of our phase unwrapping method to T.Weise et al [1].
Click to play video
[1] T. Weise, B. Leibe, L. Van Gool, "Fast 3D Scanning with Automatic Motion Compensation," Computer Vision and Pattern Recognition, 2007. CVPR '07. IEEE Conference on , vol., no., pp.1-8, 17-22 June 2007.
Selection of Temporally Dithered Codes for Increasing Virtual Depth of Field in Structured Light Systems
Abstract
Temporally dithered codes have recently been used for depth reconstruction of fast dynamic scenes using off-theshelf DLP Projectors. Even though temporally dithered codes overcome the DLP projector's limited frame rate, limitations with the optics create challenges for using these codes in an actual structured light system. Specifically, to maximize the amount of light leaving the projector, projector lenses are designed to have large apertures resulting in projected patterns that appear in focus over a narrow depth of field. In this paper, we propose a method to design temporally dithered codes in order to extend the virtual depth of field of a structured light system. By simulating the PWM sequences of a DLP projector and the blurring process from the projector lens, we develop algorithms for designing and decoding projection patterns in the presence of out of focus blur. Our simulation results show a 47% improvement in the depth of field when compared against randomly selected codewords.
R. Garcia, A. Zakhor, "Selection of Temporally Dithered Codes for Increasing Virtual Depth of Field in Structured Light Systems," PROCAMS 2010, San Francisco, CA, June 2010. [Adobe PDF]
Capturing 2 1/2D Depth and Texture of Time-Varying Scenes Using Structured Infrared Light
Abstract
In this paper, we describe an approach to simultaneously capture visual appearance and depth of a time-varying scene. Our approach is based on projecting structured infrared (IR) light. Specifically, we project a combination of (a) a static vertical IR stripe pattern, and (b) a horizontal IR laser line sweeping up and down the scene; at the same time, the scene is captured with an IR-sensitive camera. Since IR light is invisible to the human eye, it does not disturb human subjects or interfere with human activities in the scene; in addition, it does not affect the scene’s visual appearance as recorded by a color video camera. Vertical lines in the IR frames are identified using the horizontal line, intra-frame tracking, and inter-frame tracking; depth along these lines is reconstructed via triangulation. Interpolating these sparse depth lines within the foreground silhouette of the recorded video sequence, we obtain a dense depth map for every frame in the video sequence. Experimental results corresponding to a dynamic scene with a human subject in motion are presented to demonstrate the effectiveness of our proposed approach.
C. Frueh, and A. Zakhor, "Capturing 2 1/2 D Depth and Texture of Time-Varying Scenes Using Structured Infrared Light", PROCAMS workshop, San Diego, CA 2005, June 2005. Also presented at 3DIM, Ottawa, Canada,June2005 pp. 318-325. [Adobe PDF]
Sample Videos
The videos below show some of our recent results for static scene capture as well as dynamic scenes.
Captured Point Cloud
Click to play video
Intensity Textured Point Cloud
Click to play video
Dynamic Intensity Textured Point Cloud
Click to play video
All Publications
- R. Garcia, A. Zakhor, "Projector Domain Phase Unwrapping in a Structured Light System with Stereo Cameras," 3DTV 2011, Antalya, Turkey, May 16-18, 2011. [Adobe PDF]
- R. Garcia, A. Zakhor, "Temporally-Consistent Phase Unwrapping for a Stereo-Assisted Structured Light System," 3DIMPVT 2011, Hangzhou, China, May 16-19, 2011. [Adobe PDF]
- R. Garcia, A. Zakhor, "Selection of Temporally Dithered Codes for Increasing Virtual Depth of Field in Structured Light Systems," PROCAMS 2010, San Francisco, CA, June 2010. [Adobe PDF]
- C. Frueh, A. Zakhor, "Capturing 2 1/2 D Depth and Texture of Time-Varying Scenes Using Structured Infrared Light", PROCAMS workshop, San Diego, CA 2005, June 2005. Also presented at 3DIM, Ottawa, Canada,June2005 pp. 318-325. [Adobe PDF]