Three-dimensional models of objects, sites, buildings, structures, and urban environments made of geometry and texture are important in a variety of civilian and military applications such as training and simulation for disaster management, counteracting terrorism, virtual heritage conservation, virtual museums, historical sites documentation, mapping of hazardous sites and underground tunnels, and modeling of industrial and power plants for design verification and manipulation.
While object modeling has received a great deal of attention in recent years, 3D site modeling, particularly for indoor environments, poses significant challenges. The main objective of this proposal is to design, analyze, and develop architecture and algorithms, as well as associated statistical and mathematical framework for a human operated, portable, 3D indoor/outdoor modeling system, capable of generating photo-realistic rendering of the internal structure of multi-story buildings as well as external structure of a collection of buildings in a campus.
Key technical challenges consist of system architecture, sensor selection, calibration, and error characterization, local and global localization algorithms, sensor integration, registration, and fusion, and complete and accurate coverage of all details. Using a human-operated backpack system equipped with 2D laser scanners, cameras, and inertial measurement units, we develop four scan-matching-based algorithms to localize the backpack and compare their performance and tradeoffs. In addition, we show how the scan-matching-based localization algorithms combined with an extra stage of image-based alignment can be used for generating textured, photorealistic 3D models.
Photos of the backpack at ARPA-E Summit - February 2014
Cheryl Martin, ARPA-E Acting Director, takes RAPMOD for a spin
Secretary of Energy Dr. Earnest Moniz discusses the backpack
U.S. Congressman Adam Schiff (California's 28th district) tries on the backpack
U.S. Senator Sheldon Whitehouse (RI - Dem) checks out the hardware
Sample 3D Models
Models with Arbitrary Geometry and Exposure Compensation - December 2013
Models shown below correspond to interior and exterior views of hotels, whose model geometries are comprised of millions of triangles at various orientations. These triangles are segmented into independent regions for texturing either in a shortest-path or a tiled method. An additional exposure compensation step is performed across each region's generated texture as well.
Planar Models with Shortest Path/Tile-Caching Texturing Approach - July 2012
The first two images correspond to a textured model of the hallways located on the 2nd, 3rd, and 4th floors of Cory Hall on the UC Berkeley campus. The third image is a textured model of the Swarm Lab on the 4th floor of Cory Hall. For both models, geometry is generated by fitting planes to represent the walls, ceilings, and floors. These planes are textured by using a shortest path texturing approach on walls, and a tile-caching texturing method on floors and ceilings.
Video Examples and Walkthroughs
Download Links:
Texture Comparisons
Cory Hall 4th Floor Walkthrough
Cory Hall 3 Story Walkthrough
Pier 15 Walkthrough
Two Story Path - Planar Model - August 2011
This is a model of the hallways located on the 3rd and 4th floor of Cory Hall on the UC Berkeley campus. This data was captured in a single run, using a stairwell to move between floors. The model shown uses detected planes to represent the walls, ceiling, and floors.
Two Story Path - Planar Model generated w/ Autodesk Revit Architecture - August 2011
This is a model of the hallways located on the 3rd and 4th floor of Cory Hall on the UC Berkeley campus. This data was captured in a single run, using a stairwell to move between floors. The model shown uses the Autodesk Revit Architecture software package to detect planes in a user-guided way.
Two Story Path - Planar Model, Mosaic Texture - August 2010
This is a model of the hallways located on the 3rd and 4th floor of Cory Hall on the UC Berkeley campus. This data was captured in a single run, using a stairwell to move between floors. The model shown uses detected planes to represent the walls, ceiling, and floors and uses a single image mosaic for texture on each wall.
Download this video: Video 1 (12MB WMV, playable with VLC)
Three Story Point Cloud - July 2011
This is a point cloud of the hallways located on the 2nd, 3rd and 4th floor of Cory Hall on the UC Berkeley campus. This data was captured in a single run, using stairwells to move between floors.
Reconstructed Path:
Two Story Point Cloud - July 2011
This is a point cloud of the hallways located on the 3rd and 4th floor of Cory Hall on the UC Berkeley campus. This data was captured in a single run, using a stairwell to move between floors.
Staircase Point Cloud
This is a point cloud of the southeast staircase between the 3rd and 4th floors of Cory Hall at UC Berkeley.
Download this video: Video 1 (3.3MB WMV, playable with VLC)
T-Shaped Path - Planar Model - June 2011
This is a model of the hallways located on the 4th floor of Cory Hall on the UC Berkeley campus.
Download this video: Video 1 (10.6MB WMV, playable with VLC)
T-Shaped Path - Triangulated Model - February 2010
This is a model of the hallways located on the 4th floor of Cory Hall on the UC Berkeley campus.
Download this video: Video 1 (3.6MB WMV, playable with VLC)
Image based rendering of the 4th floor of Cory Hall on the UC Berkeley campus.
Download this video: Video 1 (3.4MB AVI-MJPEG, playable with VLC)
Video of this dataset rendered by our image-based renderer: Video (1.1GB AVI-MPNG, playable with VLC)
Reconstructed Path:
This path is 68.73m long, with an average position error of 0.66m.
Download this model: T-Shaped Model (180MB zipped IVE, requires osgviewer - see instructions)
Instructions for Viewing the Model Files
The above models are provided as textured, triangular meshes in the IVE format, which is a native format of OpenSceneGraph. To view the models, download and install OpenSceneGraph, which includes the osgviewer viewer application. Once installed, open a command prompt, change to the directory in which the models are stored and run the viewer with the command: osgviewer <model filename>, where <model filename> is the name of the .ive model file.
Point Clouds Obtained with the Pushcart System
In addition to the backpack system, we have a pushcart system consisting of 3 laser scanners, 2 cameras, and 1 IMU.
Cory 3rd Floor NW Lounge
This is a point cloud captured of the northwest lounge located on the 3rd floor of Cory Hall on the UC Berkeley campus.
Cory 5th Floor
This is a point cloud captured of the hallways of the 5th floor of Cory Hall on the UC Berkeley campus.
Download this point cloud: Cory 5th Floor (550MB ASCII XYZ, requires Quick Terrain Modeler or similar point cloud viewer)
Related Publications
- J. Kua, N. Corso, A. Zakhor, "Automatic Loop Closure Detection Using Multiple Cameras for 3D Indoor Localization," to be presented at IS&T/SPIE Electronic Imaging 2012, Burlingame, California, January 22-26, 2012. [Adobe PDF]
- T. Liu, M. Carlberg, G. Chen, J. Chen, J. Kua, A. Zakhor, "Indoor Localization and Visualization Using a Human-Operated Backpack System," 2010 International Conference on Indoor Positioning and Indoor Navigation, Zurich, Switzerland, September, 2010. [Adobe PDF]
- G. Chen, J. Kua, S. Shum, N. Naikal, M. Carlberg, and A. Zakhor. "Indoor Localization Algorithms for a Human-Operated Backpack System," 3D Data Processing, Visualization, and Transmission 2010, Paris, France, May 2010. [Adobe PDF]
- N. Naikal, J. Kua, G. Chen, and A. Zakhor. "Image Augmented Laser Scan Matching for Indoor Dead Reckoning," IEEE/RSJ International Conference on Intelligent RObots and Systems (IROS), St. Louis, MO, October 2009. [Adobe PDF]
Press Coverage
- GeoDataPoint - A Backpack for BIM, reported by Christine Grahl, August 29, 2012.
- Chevrolet Cruze-arati Video Segment - Veronica Belmont Tests Laser Backpack at UC Berkeley, April 22, 2011.
- Photonics Spectra - 3-D Interior Modeling: A Virtual Walk on the Inside, January 2011.
- North Gate Radio (KALX) - Laser Backpack: One-on-One with Avideh Zakhor, interview with Alexa Vaughn, October 7, 2010.
Audio Segment 1, Segment 2 (MP3) - SIGNAL Magazine - Laser Backpack Prototype Maps Inside Buildings, reported by George I. Seffers, September 30, 2010.
- Slashdot - A Portable Laser Backpack For 3D Mapping, September 18, 2010.
- CNET - Military backpack maps out 3D interiors, reported by Tim Hornyak, September 9, 2010.
- United Press International - Laser backpack measures interiors, September 8, 2010.
- Air Force News - Portable Laser Backpack Revolutionizes 3D Mapping, reported by Maria Callier, September 7, 2010.
- Daily Californian - New Backpack Makes 3D Maps of Buildings, reported by Matt Burris, August 12, 2010.
- Engadget - Laser backpack creates instant 3D maps, reported by Tim Stevens, August 11, 2010.
- ABC 7 KGO - Laser backpack creates instant 3D models, reported by Richard Hart, August 8, 2010.
Related Work
- Fast automated outdoor modeling of urban environments
- 3D modeling using airborne LiDAR and EO sensors