Ali Khaloo

Hierarchical Dense Structure-from-Motion Reconstructions for Infrastructure Condition Assessment

AbstractAccurate condition assessment of in-service infrastructure systems is critical for system-wide prioritization decisions. Current protocols require lengthy inspections and expensive equipment to examine large infrastructure systems. Furthermore, changes in inspection protocols over time can create discontinuities in recording and understanding the time history of a structure. To address these challenges, a systematic and adaptive technique for converting two-dimensional (2D) digital images into three-dimensional (3D) models has been developed, with the goal of creating high-resolution and scale-accurate inspection records. The developed reconstruction technique utilizes multiscale imaging to reconstruct a structure with varying levels of details and geometric complexity. The captured images are then converted into photorealistic, accurate, and dense 3D scene reconstructions by utilizing a hierarchical adaptation of a dense structure-from-motion (DSfM) algorithm. The result of this approach is a vir...

Unmanned aerial vehicle inspection of the Placer River Trail Bridge through image-based 3D modelling

Abstract Unmanned aerial vehicles (UAV) are now a viable option for augmenting bridge inspections. Utilising an integrated combination of a UAV and computer vision can decrease costs, expedite inspections and facilitate bridge access. Any such inspection must consider the design of the UAV, the choice of cameras, data acquisition, geometrical resolution, safety regulations and pilot protocols. The Placer River Trail Bridge in Alaska recently served as a test bed for a UAV inspection methodology that integrates these considerations. The end goal was to produce a three-dimensional (3D) model of the bridge using UAV-captured images and a hierarchical Dense Structure-from-Motion algorithm. To maximise the quality of the model and its benefits to inspectors, this goal guided UAV design and mission planning. The resulting inspection methodology integrates UAV design, data capture and data analysis together to provide an optimised 3D model. This model provides inspection documentation while enabling the monitoring of defects. The developed methodology is presented herein, as well as analyses of the 3D models. The results are compared against models generated through laser scanning. The findings demonstrate that the UAV inspection methodology provided superior 3D models with the accuracy to resolve defects and support the needs of infrastructure managers.

Robust normal estimation and region growing segmentation of infrastructure 3D point cloud models

Modern remote sensing technologies such as three-dimensional (3D) laser scanners and image-based 3D scene reconstruction are in increasing demand for applications in civil infrastructure design, maintenance, operation, and as-built construction verification. The complex nature of the 3D point clouds these technologies generate, as well as the often massive scale of the 3D data, make it inefficient and time consuming to manually analyze and manipulate point clouds, and highlights the need for automated analysis techniques. This paper presents one such technique, a new region growing algorithm for the automated segmentation of both planar and non-planar surfaces in point clouds. A core component of the algorithm is a new point normal estimation method, an essential task for many point cloud processing algorithms. The newly developed estimation method utilizes robust multivariate statistical outlier analysis for reliable normal estimation in complex 3D models, considering that these models often contain regions of varying surface roughness, a mixture of high curvature and low curvature regions, and sharp features. An adaptation of Mahalanobis distance, in which the mean vector and covariance matrix are derived from a high-breakdown multivariate location and scale estimator called Deterministic MM-estimator (DetMM) is used to find and discard outlier points prior to estimating the best local tangent plane around any point in a cloud. This approach is capable of more accurately estimating point normals located in highly curved regions or near sharp features. Thereafter, the estimated point normals serve a region growing segmentation algorithm that only requires a single input parameter, an improvement over existing methods which typically require two control parameters. The reliability and robustness of the normal estimation subroutine was compared against well-known normal estimation methods including the Minimum Volume Ellipsoid (MVE) and Minimum Covariance Determinant (MCD) estimators, along with Maximum Likelihood Sample Consensus (MLESAC). The overall region growing segmentation algorithm was then experimentally validated on several challenging 3D point clouds of real-world infrastructure systems. The results indicate that the developed approach performs more accurately and robustly in comparison with conventional region growing methods, particularly in the presence of sharp features, outliers and noise.

Long-term monitoring of structures through point cloud analysis

Modern remote sensing technologies have enabled the creation of high-resolution 3D point clouds of infrastructure systems. In particular, photogrammetric reconstructions using Dense-Structure-from-Motion algorithm can now yield point clouds with the necessary resolution to capture small-strain displacements. By tracking changes in these point clouds over time, displacements can be measured, leading to strain and stress estimates for long-term structural evaluations. This study determines the accuracy of a comparative point cloud analysis technique for measuring deflections in high-resolution point clouds of structural elements. Utilizing a combination of a recently developed point cloud generation process and localized nearest-neighbors cloud comparisons, the analytical technique is designed for long-term field scenarios and requires no artificial tracking, targets, and camera calibrations. A series of flexural laboratory experiments were performed in order to test the approach. The results indicate sub-millimeter accuracy in measuring the vertical deflection, making it suitable for the small-displacement analysis of a variety of large-scale infrastructure systems. Ongoing work seeks to extend this technique for comparison with as-built and finite element models.

Extracting Structural Models through Computer Vision

The ability to accurately and rapidly assess structural integrity after a disaster is critical from both a safety and economic perspective. Existing post-disaster inspection methods are time-consuming and expensive, requiring highly trained inspectors to travel to target sites and manually collect data. Automated analysis of civil structures from visual data through computer vision can be used to improve the level of accuracy in the condition assessment procedure. This paper presents a method of automated and systematic computer vision-based structural analysis. It uses a set of digital photographs to produce a 3D model through Structure from Motion (SfM) algorithms, followed by fully automated recognition and assembly of structural elements (e.g., columns and beams) from the image-based 3D dense reconstruction of the structure. There are three key challenges in this work: (i) proper 3D mesh segmentation, (ii) robust computer vision algorithms for isolating different structural components, and (iii) classification and localization of damage that is present in the 3D model. As the part of the proposed system, extracted information from the dense 3D model is used to assemble the structural elements and create a Finite-Element Method (FEM) model. Lastly, a supervised machine learning scheme built upon a large and comprehensive data set is used to automatically update the model to account for damage. The proposed methodology has applications beyond post-disaster condition assessment, from routine inspection to infrastructure management applications.

Repurposing video recordings for structure motion estimations

Video monitoring of public spaces is becoming increasingly ubiquitous, particularly near essential structures and facilities. During any hazard event that dynamically excites a structure, such as an earthquake or hurricane, proximal video cameras may inadvertently capture the motion time-history of the structure during the event. If this dynamic time-history could be extracted from the repurposed video recording it would become a valuable forensic analysis tool for engineers performing post-disaster structural evaluations. The difficulty is that almost all potential video cameras are not installed to monitor structure motions, leading to camera perspective distortions and other associated challenges. This paper presents a method for extracting structure motions from videos using a combination of computer vision techniques. Images from a video recording are first reprojected into synthetic images that eliminate perspective distortion, using as-built knowledge of a structure for calibration. The motion of the camera itself during an event is also considered. Optical flow, a technique for tracking per-pixel motion, is then applied to these synthetic images to estimate the building motion. The developed method was validated using the experimental records of the NEESHub earthquake database. The results indicate that the technique is capable of estimating structural motions, particularly the frequency content of the response. Further work will evaluate variants and alternatives to the optical flow algorithm, as well as study the impact of video encoding artifacts on motion estimates.

A Hierarchical Computer Vision Approach to Infrastructure Inspection

Currently, most infrastructure inspection standards require inspectors to visually assess structural integrity and log findings for comparison during future inspections. This process is qualitative and often inconsistent. Furthermore, changes in inspection protocols over time can create discontinuities in understanding the time-history of a structure. This paper presents a systematic technique for capturing and representing inspection data, leveraging a newly developed hierarchical computer vision methodology. This technique can be used to improve the level of accuracy in condition assessment procedures by allowing inspectors to recreate the structural inspection scenario on a computer through a high-fidelity virtual environment. The methodology presented herein utilizes adaptations of the Dense Structure from Motion (DSfM) algorithm, which reconstructs three-dimensional (3D) scenes from two-dimensional (2D) digital images. In order to produce highly-accurate and photorealistic 3D reconstructions there are four core stages: (i) an image capture plan that covers all views of the structure and emphasizes critical details, (ii) reconstruction of an initial dense point cloud to generate a geometrically accurate 3D model, (iii) reconstruction of separate, higher density point clouds of critical details or suspected deficiencies, (iv) application of robust computer vision algorithms to hierarchically register and merge the point clouds. The result of this approach is a virtual 3D model of the structure with accurate geometry and high-fidelity representation of fine details. The accuracy and adaptability of the developed technique was compared to both conventional DSfM reconstruction methods and terrestrial 3D laser scanning (TLS). The experimental validation indicates that the hierarchical technique produces denser and more comprehensive models with an accuracy of one tenth of a millimeter, an order of magnitude improvement over either conventional DSfM or TLS.

Automatic Detection of Structural Deficiencies Using 4D Hue-Assisted Analysis of Color Point Clouds

Recent developments in the fields of robotics and remote sensing technologies such as 3D laser scanners and photogrammetric approaches have provided an unprecedented opportunity to collect a massive amount of data from infrastructure systems in a contactless and nondestructive manner, which can potentially improve the structural health monitoring process. However, the complex nature of these geometrically accurate and high-resolution 3D models makes it inefficient and time-consuming to manually analyze and manipulate them and automating this process continue to pose a challenge. Thus, procedures that automate the data processing in order to detect a variety of damages are desired to make full use of these modern inspection technologies as a tool for infrastructure integrity assessment and asset management. The aim of this paper is to present a new algorithm to automatically identify and evaluate structural deficiencies in massive 3D point clouds of complex infrastructure systems. This approach takes advantage of both local geometry and color data properties associated with each point to improve the damage detection capabilities in a variety of scenarios. Linear and non-linear transformations from the RGB color space to non-RGB spaces were performed to increase separability between the damage and the structure and to achieve robustness to changes in illumination. Recently, a complex and large-scale gravity dam in Maryland, USA has served as a test bed for the developed methodology. In this experiment, a multi-scale photogrammetric computer vision approach was utilized to generate accurate and highly detailed 3D models of the targeted dam. In order to maximize the accessibility and to overcome geometric constraints, different multi-rotor Unmanned Aerial Vehicle (UAV) platforms with varied payload and maneuverability capabilities, each equipped with different optical sensors were used in this study. Experimental results demonstrate that the presented 4D point cloud analysis method can accurately detect and quantify a variety of anomalies from spalling to moisture infiltration in exposed concrete structures.

Integrating 3D Computer Vision and Robotic Infrastructure Inspection

Over the past decade, there has been growing interest in the field of robotic inspection systems, in particular Unmanned Aerial Vehicles (UAV). These systems afford many advantages in terms of reducing the cost of condition assessments and expediting the data acquisition while improving the monitoring process efficiency. However, robotic technologies are a means of collecting data, and data collection protocols must be tightly linked with the desired spatial data resolution and final products, particularly if integration with more conventional structural health monitoring technologies is to be attained. This process involves planning a set of views, physically altering the relative structureimaging sensor pose, globally registering all the acquired imagery data, and finally integrating different sources of data into a nonredundant model. In this context, this paper demonstrates the capabilities of using one such integration at the Brighton Dam, a complex and large-scale gravity dam in Maryland, USA. The goal of this study was to evaluate a robotic inspection methodology along with utilizing computer vision-based data analytics on photogrammetrically generated three-dimensional (3D) models of the dam. A series of experiments involving the analysis of both simulated and real structural damage were designed and performed. During these experiments, digital image data was collected from a variety of sources, including both fixed-wing and multi-rotor UAVs, as well as more conventional imaging platforms. These data sources were integrated and merged together to form a detailed high-resolution 3D model of the dam and surrounding environs, with varying level of details in damage and deterioration. 3D data processing techniques were then used to analyze these models to automatically detect and quantify various damages. The results of these tests reflect the capabilities of these powerful new monitoring techniques and indicate that a suite of vision-based algorithms is necessary for comprehensive analyses of these complex 3D models to reliably, and systematically detect damages. The findings of this study also reinforce the critical need for an in advance detailed planning along with team coordination during field data collection.