Andrew Owens

Discrete-continuous optimization for large-scale structure from motion

Recent work in structure from motion (SfM) has successfully built 3D models from large unstructured collections of images downloaded from the Internet. Most approaches use incremental algorithms that solve progressively larger bundle adjustment problems. These incremental techniques scale poorly as the number of images grows, and can drift or fall into bad local minima. We present an alternative formulation for SfM based on finding a coarse initial solution using a hybrid discrete-continuous optimization, and then improving that solution using bundle adjustment. The initial optimization step uses a discrete Markov random field (MRF) formulation, coupled with a continuous Levenberg-Marquardt refinement. The formulation naturally incorporates various sources of information about both the cameras and the points, including noisy geotags and vanishing point estimates. We test our method on several large-scale photo collections, including one with measured camera positions, and show that it can produce models that are similar to or better than those produced with incremental bundle adjustment, but more robustly and in a fraction of the time.

SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels

Existing scene understanding datasets contain only a limited set of views of a place, and they lack representations of complete 3D spaces. In this paper, we introduce SUN3D, a large-scale RGB-D video database with camera pose and object labels, capturing the full 3D extent of many places. The tasks that go into constructing such a dataset are difficult in isolation -- hand-labeling videos is painstaking, and structure from motion (SfM) is unreliable for large spaces. But if we combine them together, we make the dataset construction task much easier. First, we introduce an intuitive labeling tool that uses a partial reconstruction to propagate labels from one frame to another. Then we use the object labels to fix errors in the reconstruction. For this, we introduce a generalization of bundle adjustment that incorporates object-to-object correspondences. This algorithm works by constraining points for the same object from different frames to lie inside a fixed-size bounding box, parameterized by its rotation and translation. The SUN3D database, the source code for the generalized bundle adjustment, and the web-based 3D annotation tool are all available at http://sun3d.cs.princeton.edu.

SfM with MRFs: Discrete-Continuous Optimization for Large-Scale Structure from Motion

Recent work in structure from motion (SfM) has successfully built 3D models from large unstructured collections of images downloaded from the Internet. Most approaches use incremental algorithms that solve progressively larger bundle adjustment problems. These incremental techniques scale poorly as the number of images grows, and can drift or fall into bad local minima. We present an alternative formulation for SfM based on finding a coarse initial solution using a hybrid discrete-continuous optimization, and then improving that solution using bundle adjustment. The initial optimization step uses a discrete Markov random field (MRF) formulation, coupled with a continuous Levenberg-Marquardt refinement. The formulation naturally incorporates various sources of information about both the cameras and the points, including noisy geotags and vanishing point estimates. We test our method on several large-scale photo collections, including one with measured camera positions, and show that it can produce models that are similar to or better than those produced with incremental bundle adjustment, but more robustly and in a fraction of the time.

Systems Analysis of In-Space Manufacturing Applications for the International Space Station and the Evolvable Mars Campaign

Maintenance logistics support is a significant challenge for extended human operations in space, especially for missions beyond Low Earth Orbit (LEO). For missions to Mars (such as NASAs Evolvable Mars Campaign (EMC)), where timely resupply or abort in the event of emergency will not be possible, maintenance logistics mass is directly linked to the Probability of Loss of Crew (P(LoC)), and the cost of driving down risk is an exponential increase in mass requirements. The logistics support strategies that have maintained human operations in LEO will not be effective for these deep space missions. In-Space Manufacturing (ISM) is a promising technological solution that could reduce logistics requirements, mitigate risks, and augment operational capabilities, enabling Earth- independent human spaceflight. This paper reviews maintenance logistics challenges for spaceflight operations in LEO and beyond, and presents a summary of selected results from a systems analysis of potential ISM applications for the ISS and EMC. A quantitative modeling framework and sample assessment of maintenance logistics and risk reduction potential of this new technology is also presented and discussed.

Camouflaging an Object from Many Viewpoints

We address the problem of camouflaging a 3D object from the many viewpoints that one might see it from. Given photographs of an objects surroundings, we produce a surface texture that will make the object difficult for a human to detect. To do this, we introduce several background matching algorithms that attempt to make the object look like whatever is behind it. Of course, it is impossible to exactly match the background from every possible viewpoint. Thus our models are forced to make trade-offs between different perceptual factors, such as the conspicuousness of the occlusion boundaries and the amount of texture distortion. We use experiments with human subjects to evaluate the effectiveness of these models for the task of camouflaging a cube, finding that they significantly outperform naïve strategies.

The Threat of Uncertainty: Why Using Traditional Approaches for Evaluating Spacecraft Reliability are Insufficient for Future Human Mars Missions

Through the Evolvable Mars Campaign (EMC) study, the National Aeronautics and Space Administration (NASA) continues to evaluate potential approaches for sending humans beyond low Earth orbit (LEO). A key aspect of these missions is the strategy that is employed to maintain and repair the spacecraft systems, ensuring that they continue to function and support the crew. Long duration missions beyond LEO present unique and severe maintainability challenges due to a variety of factors, including: limited to no opportunities for resupply, the distance from Earth, mass and volume constraints of spacecraft, high sensitivity of transportation element designs to variation in mass, the lack of abort opportunities to Earth, limited hardware heritage information, and the operation of human-rated systems in a radiation environment with little to no experience. The current approach to maintainability, as implemented on ISS, which includes a large number of spares pre-positioned on ISS, a larger supply sitting on Earth waiting to be flown to ISS, and an on demand delivery of logistics from Earth, is not feasible for future deep space human missions. For missions beyond LEO, significant modifications to the maintainability approach will be required.Through the EMC evaluations, several key findings related to the reliability and safety of the Mars spacecraft have been made. The nature of random and induced failures presents significant issues for deep space missions. Because spare parts cannot be flown as needed for Mars missions, all required spares must be flown with the mission or pre-positioned. These spares must cover all anticipated failure modes and provide a level of overall reliability and safety that is satisfactory for human missions. This will require a large amount of mass and volume be dedicated to storage and transport of spares for the mission. Further, there is, and will continue to be, a significant amount of uncertainty regarding failure rates for spacecraft components. This uncertainty makes it much more difficult to anticipate failures and will potentially require an even larger amount of spares to provide an acceptable level of safety. Ultimately, the approach to maintenance and repair applied to ISS, focusing on the supply of spare parts, may not be tenable for deep space missions. Other approaches, such as commonality of components, simplification of systems, and in-situ manufacturing will be required.

Limitations of Reliability for Long-Endurance Human Spaceflight

United States. National Aeronautics and Space Administration. Space Technology Research Fellowship (NNX14AM42H)

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning

The sound of crashing waves, the roar of fast-moving cars—sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds can be used as a supervisory signal for learning visual models. To demonstrate this, we train a convolutional neural network to predict a statistical summary of the sound associated with a video frame. We show that, through this process, the network learns a representation that conveys information about objects and scenes. We evaluate this representation on several recognition tasks, finding that its performance is comparable to that of other state-of-the-art unsupervised learning methods. Finally, we show through visualizations that the network learns units that are selective to objects that are often associated with characteristic sounds. This paper extends an earlier conference paper, Owens et al. (in: European conference on computer vision, 2016b), with additional experiments and discussion.

Feasibility Analysis of Commercial In-Space Manufacturing Applications

Though in its infancy, In-Space Manufacturing (ISM) has the potential to be a paradigm shifting technology allowing for increased access to and exploration of space. ISM yields several benefits over the traditional Earth-build-and-launch approach. Most importantly, it removes launch considerations – including schedule, risk and strenuous loads and vibrations – from the component or system design process. As any new space venture can be a costly and risky endeavor, it would be prudent to understand which applications of ISM are viable and promising candidates for the commercial sector to invest in. This paper investigates the commercial feasibility of several potential applications of ISM through a series of case studies: ISM of antenna reflectors, ISM of solar panel support structure, and ISM of spare parts for long duration space missions. The studies quantified the sensitivity of the business cases of these applications to a variety of factors including ISM capability development cost, technical parameters of the component or system being built, and reliability of economic forecasting of the space sector. This paper also provides recommendations for strategic investments by both NASA and private partners to maximize the future potential and impact of ISM.

Supportability Challenges, Metrics, and Key Decisions for Future Human Spaceflight

Future crewed missions beyond Low Earth Orbit (LEO) represent a logistical challenge that is unprecedented in human spaceflight. Astronauts will travel farther and stay in space for longer than any previous mission, far from timely abort or resupply from Earth. Under these conditions, supportability – defined as the set of system characteristics that influence the logistics and support required to enable safe and effective operations of systems – will be a much more significant driver of space system lifecycle properties than it has been in the past. This paper presents an overview of supportability for future human spaceflight. The particular challenges of future missions are discussed, with the differences between past, present, and future missions highlighted. The relationship between supportability metrics and mission cost, performance, schedule, and risk is also discussed. A set of proposed strategies for managing supportability is presented – including reliability growth, uncertainty reduction, level of repair, commonality, redundancy, In-Space Manufacturing (ISM) (including the use of material recycling and In-Situ Resource Utilization (ISRU) for spares and maintenance items), reduced complexity, and spares inventory decisions such as the use of predeployed or cached spares – along with a discussion of the potential impacts of each of those strategies. References are provided to various sources that describe these supportability metrics and strategies, as well as associated modeling and optimization techniques, in greater detail. Overall, supportability is an emergent system characteristic and a holistic challenge for future system development. System designers and mission planners must carefully consider and balance the supportability metrics and decisions described in this paper in order to enable safe and effective beyond-LEO human spaceflight.