William Hoff

Surfaces from stereo: integrating feature matching, disparity estimation, and contour detection

An approach is described that integrates the processes of feature matching, contour detection, and surface interpolation to determine the three-dimensional distance, or depth, of objects from a stereo pair of images. Integration is necessary to ensure that the detected surfaces are smooth. Surface interpolation takes into account detected occluding and ridge contours in the scene; interpolation is performed within regions enclosed by these contours. Planar and quadratic patches are used as local models of the surface. Occluded regions in the image are identified, and are not used for matching and interpolation. A coarse-to-fine algorithm is presented that generates a multiresolution hierarchy of surface maps, one at each level of resolution. Experimental results are given for a variety of stereo images. >

In vivo knee kinematics derived using an inverse perspective technique.

Sixty-four subjects having implanted and nonimplanted knees were studied using fluoroscopic videos. Each subject, flexing in the sagittal plane, performed successive deep knee bends under fluoroscopic surveillance. Femorotibial contact in the sagittal plane was then determined using image matching and discrete digitization. At full extension, the mean contact point of the normal and posterior stabilized implanted femurs was anterior to the tibial midpoint in the sagittal plane. The average position was 6.49 mm (+3 - +13 mm) for the normal knees and 0.30 mm (0 - +4 mm) for the posterior stabilized knees. The implanted posterior cruciate retaining and anterior cruciate ligament deficient knees differed from the other knee types. Their average initial contact was posterior. The average contact at full extension for the posterior cruciate retaining and anterior cruciate ligament deficient knees was -5.13 mm (-2 - -8 mm) and -5.45 mm (-2 - -14 mm), respectively. The femur of the normal knee contacts the tibia anterior to the midpoint in the sagittal plane in full extension and translates posteriorly during flexion. The femur of the posterior stabilized knee contacts the tibia anteriorly, slightly less than the normal knee, and rolls back posteriorly during flexion similar to normal knees. The femurs of the posterior cruciate retaining and anterior cruciate ligament deficient knees contact the tibia posterior in extension, but translate anteriorly during midflexion in a substantial number of cases, which is kinematically opposite of the normal knees. The abnormal anterior femoral translation observed in the posterior cruciate retaining knees may be a factor in the premature polyethylene wear seen in retrieval studies.

Fluoroscopic analysis of kinematics after posterior-cruciate-retaining knee arthroplasty

We used fluoroscopy to study the kinematics of the knee in 47 patients with total knee arthroplasty (TKA) and four control subjects with normal knees while performing a single-leg deep-knee bend. The videos were analysed using still photographs taken at 5 degrees increments of flexion. Femorotibial contact points, patellar ligament rotation, and patellar rotation were calculated from each image. Maximum weight-bearing flexion was determined for each knee. Compared with the control group, posterior-cruciate-retaining TKA did not reproduce normal knee kinematics in any case, but showed a starting point posterior to the tibial midline which translated anteriorly with flexion. The curves from successive knee bends could not be consistently reproduced. Under weight-bearing conditions, the maximum flexion for any PCR TKA was 98 degrees and several patients could not flex beyond 70 degrees.

Three-dimensional determination of femoral-tibial contact positions under in vivo conditions using fluoroscopy.

OBJECTIVE: A method has been developed to accurately measure three-dimensional (3-D) femoral-tibial contact positions of artificial knee implants in vivo from X-ray fluoroscopy images using interactive 3-D computer vision algorithms. DESIGN: A computerized graphical (CAD) model of an implant component is displayed as an overlay on the original X-ray image. An image matching algorithm matches the silhouette of the implant component against a library of images, in order to estimate the position and orientation (pose) of the component. The operator further adjusts the pose of the graphical model to improve the accuracy of the match. BACKGROUND: Previous methods for in vivo measurement of joint kinematics make only indirect measurements of joint kinematics, require invasive procedures such as markers or pins, or make simplifying assumptions about imaging geometry which can reduce the accuracy of the resulting measurements. METHODS: Fluoroscopic videos are taken of implanted knees in subjects performing weight-bearing motion. Images from the videos are digitized and stored on a computer workstation. Using computerized model matching, the relative pose of the two knee implant components can be determined in each image. The resulting information can be used to determine where the two components are contacting, the area of the contact region, liftoff angle, and other kinematic data. RESULTS: Accuracy tests done on simulated imagery and in vitro real imagery show that the pose estimation method is accurate to less than 0.5 mm of error (RMS) for translations parallel to the image plane. Orientation error is less than or equal to 0.35 degrees about any axis. Errors are larger for translations perpendicular to the image plane (up to 2.25 mm). In a clinical study, the method was used to measure in vivo contact points, and characterize the kinematic patterns of two different knee implant designs. CONCLUSIONS: The ability to accurately measure knee kinematics in vivo is critical for the understanding of the behavior of knee implant designs and the ultimate development of new, longer lasting implants. RELEVANCE: This work shows that it is possible to accurately measure the three-dimensional position and orientation (pose) of artificial knee implants in vivo from X-ray fluoroscopy images using interactive 3-D computer graphics. The method can be applied to any joint when accurate CAD models are available. The resulting data can be used to characterize the kinematics of current knee implant designs.

Analysis of head pose accuracy in augmented reality

A method is developed to analyze the accuracy of the relative head-to-object position and orientation (pose) in augmented reality systems with head-mounted displays. From probabilistic estimates of the errors in optical tracking sensors, the uncertainty in head-to-object pose can be computed in the form of a covariance matrix. The positional uncertainty can be visualized as a 3D ellipsoid. One useful benefit of having an explicit representation of uncertainty is that we can fuse sensor data from a combination of fixed and head-mounted sensors in order to improve the overall registration accuracy. The method was applied to the analysis of an experimental augmented reality system, incorporating an optical see-through head-mounted display, a head-mounted CCD camera, and a fixed optical tracking sensor. The uncertainty of the pose of a movable object with respect to the head-mounted display was analyzed. By using both fixed and head mounted sensors, we produced a pose estimate that is significantly more accurate than that produced by either sensor acting alone.

Registration of range data using a hybrid simulated annealing and iterative closest point algorithm

The need to register data is abundant in applications such as: world modeling, part inspection and manufacturing, object recognition, pose estimation, robotic navigation, and reverse engineering. Registration occurs by aligning the regions that are common to multiple images. The largest difficulty in performing this registration is dealing with outliers and local minima while remaining efficient. A commonly used technique, iterative closest point, is efficient but is unable to deal with outliers or avoid local minima. Another commonly used optimization algorithm, simulated annealing, is effective at dealing with local minima but is very slow. Therefore, the algorithm developed in this paper is a hybrid algorithm that combines the speed of iterative closest point with the robustness of simulated annealing. Additionally, a robust error function is incorporated to deal with outliers. This algorithm is incorporated into a complete modeling system that inputs two sets of range data, registers the sets, and outputs a composite model.

Space-time representation of people based on 3D skeletal data

First survey dedicated to human representations based on 3D skeleton data.Our survey is comprehensive and covers the most recent and advanced approaches.An insightful categorization and analysis of the 3D skeleton-based representations is provided. Spatiotemporal human representation based on 3D visual perception data is a rapidly growing research area. Representations can be broadly categorized into two groups, depending on whether they use RGB-D information or 3D skeleton data. Recently, skeleton-based human representations have been intensively studied and kept attracting an increasing attention, due to their robustness to variations of viewpoint, human body scale and motion speed as well as the realtime, online performance. This paper presents a comprehensive survey of existing space-time representations of people based on 3D skeletal data, and provides an informative categorization and analysis of these methods from the perspectives, including information modality, representation encoding, structure and transition, and feature engineering. We also provide a brief overview of skeleton acquisition devices and construction methods, enlist a number of benchmark datasets with skeleton data, and discuss potential future research directions.

Three-dimensional motion and structure estimation using inertial sensors and computer vision for augmented reality

A new method for registration in augmented reality (AR) was developed that simultaneously tracks the position, orientation, and motion of the users head, as well as estimating the three-dimensional (3D) structure of the scene. The method fuses data from head-mounted cameras and head-mounted inertial sensors. Two extended Kalman filters (EKFs) are used: one estimates the motion of the users head and the other estimates the 3D locations of points in the scene. A recursive loop is used between the two EKFs. The algorithm was tested using a combination of synthetic and real data, and in general was found to perform well. A further test showed that a system using two cameras performed much better than a system using a single camera, although improving the accuracy of the inertial sensors can partially compensate for the loss of one camera. The method is suitable for use in completely unstructured and unprepared environments. Unlike previous work in this area, this method requires no a priori knowledge about the scene, and can work in environments in which the objects of interest are close to the user.

In Vivo Assessment of the Kinematics in Normal and Anterior Cruciate Ligament-Deficient Knees

The in vivo measurement of dynamic knee kinematics is important for understanding the effects of joint injuries and diseases and for evaluating the outcome of surgical procedures. Researchers have used in vitro approaches (involving cadavers), noninvasive approaches (involving studies done at gait laboratories), and in vivo approaches (involving roentgen stereophotogrammetry and fluoroscopy) to assess human knee motion. To ensure that the loads that are encountered during typical movements are accurately incorporated, treatments that are aimed at improving knee function should be evaluated with use of data that are obtained with dynamic measurement methods. This requires that the six-degrees-of-freedom pose (position and orientation) of objects be measured during dynamic activities. The purpose of the current study was to accurately determine the three-dimensional kinematic patterns of normal and anterior cruciate ligament-deficient knees during in vivo weight-bearing activities with use of a novel intensity-based two-dimensional to three-dimensional image registration method, similar to that previously utilized to analyze the kinematics of total knee arthroplasty1. ### Model Creation Ten healthy normal volunteers with an average age of thirty-seven years (range, twenty-two to forty-four years) and an average body mass of 76 kg participated in the study. These volunteers exhibited no lower extremity abnormalities on T2-weighted fast-spin-echo magnetic resonance imaging and had no clinically measurable knee ligament instability on either the pivot-shift test or the Lachman test. In addition, five patients with a recent isolated anterior cruciate ligament tear (that is, a tear that had occurred within the previous four to six weeks), who had an average age of thirty-nine years (range, twenty-five to forty-seven years) and an average body mass of 65 kg, were also included in the study so that the kinematics of anterior cruciate ligament-deficient knees could be compared with those of normal knees. The five patients with an anterior cruciate ligament-deficient knee were …

Pose estimation of artificial knee implants in fluoroscopy images using a template matching technique

The paper describes an algorithm to estimate the position and orientation (pose) of artificial knee implants from X-ray fluoroscopy images using computer vision. The resulting information is used to determine the kinematics of bone motion in implanted knees. This determination can be used to support the development of improved prosthetic knee implants, which currently have a limited life span due to premature wear of the polyethylene material at the joint surface. The algorithm determines the full 6 degree of freedom translation and rotation of knee components. This is necessary for artificial knees which have shown significant rotation out of the sagittal plane, in particular internal/external rotations. By creating a library of images of components at known orientation and performing a template matching technique, the 3D pose of the femoral and tibial components are determined. The entire process, when used at certain knee angles, gives a representation of the positions in contact during normal knee motion.