Suren Kumar

Combining skeletal pose with local motion for human activity recognition

Recent work in human activity recognition has focused on bottom-up approaches that rely on spatiotemporal features, both dense and sparse. In contrast, articulated motion, which naturally incorporates explicit human action information, has not been heavily studied; a fact likely due to the inherent challenge in modeling and inferring articulated human motion from video. However, recent developments in data-driven human pose estimation have made it plausible. In this paper, we extend these developments with a new middle-level representation called dynamic pose that couples the local motion information directly and independently with human skeletal pose, and present an appropriate distance function on the dynamic poses. We demonstrate the representative power of dynamic pose over raw skeletal pose in an activity recognition setting, using simple codebook matching and support vector machines as the classifier. Our results conclusively demonstrate that dynamic pose is a more powerful representation of human action than skeletal pose.

Improving Person Tracking Using an Inexpensive Thermal Infrared Sensor

This paper proposes a person tracking framework using a scanning low-resolution thermal infrared (IR) sensor colocated with a wide-angle RGB camera. The low temporal and spatial resolution of the low-cost IR sensor make it unable to track moving people and prone to false detections of stationary people. Thus, IR-only tracking using only this sensor would be quite problematic. We demonstrate that despite the limited capabilities of this low-cost IR sensor, it can be used effectively to correct the errors of a real-time RGB camera-based tracker. We align the signals from the two sensors both spatially (by computing a pixel-to-pixel geometric correspondence between the two modalities) and temporally (by modeling the temporal dynamics of the scanning IR sensor), which enables multi-modal improvements based on judicious application of elementary reasoning. Our combined RGB+IR system improves upon the RGB camera-only tracking by: rejecting false positives, improving segmentation of tracked objects, and correcting false negatives (starting new tracks for people that were missed by the camera-only tracker). Since we combine RGB and thermal information at the level of RGB camera-based tracks, our method is not limited to the particular camera-based tracker that we used in our experiments. Our method could improve the results of any tracker that uses RGB camera input alone. We collect a new dataset and demonstrate the superiority of our method over RGB camera-only tracking.

Estimating Human Dynamics On-the-fly Using Monocular Video For Pose Estimation

Human pose estimation using uncalibrated monocular visual inputs alone is a challenging problem for both the computer vision and robotics communities. From the robotics perspective, the challenge here is one of pose estimation of a multiply-articulated system of bodies using a single nonspecialized environmental sensor (the camera) and thereby, creating low-order surrogate computational models for analysis and control. In this work, we propose a technique for estimating the lowerlimb dynamics of a human solely based on captured behavior using an uncalibrated monocular video camera. We leverage our previously developed framework for human pose estimation to (i) deduce the correct sequence of temporally coherent gap-filled pose estimates, (ii) estimate physical parameters, employing a dynamics model incorporating the anthropometric constraints, and (iii) filter out the optimized gap-filled pose estimates, using an Unscented Kalman Filter (UKF) with the estimated dynamicallyequivalent human dynamics model. We test the framework on videos from the publicly available DARPA Mind’s Eye Year 1 corpus [8]. The combined estimation and filtering framework not only results in more accurate physically plausible pose estimates, but also provides pose estimates for frames, where the original human pose estimation framework failed to provide one.

Surgical tool attributes from monocular video

HD Video from the (monocular or binocular) endoscopic camera provides a rich real-time sensing channel from surgical site to the surgeon console in various Minimally Invasive Surgery (MIS) procedures. However, a real-time framework for video understanding would be critical for tapping into the rich information-content provided by the non-invasive and well-established digital endoscopic video-streaming modality. While contemporary research focuses on enhancing aspects such as tool-tracking within the challenging visual scenes, we consider the associated problem of using that rich (but often compromised) streaming visual data to discover the underlying semantic attributes of the tools. Directly analyzing the surgical videos to extract more realistic attributes online can aid in the decision-making and feedback aspects. We propose a novel probabilistic attribute labelling framework with Bayesian filtering to identify associated semantics (open/closed, stained with blood etc.) to ultimately give semantic feedback to the surgeon. Our robust video-understanding framework overcomes many of the challenges (tissue deformations, image specularities, clutter, tool-occlusion due to blood and/or organs) under realistic in-vivo surgical conditions. Specifically, this manuscript performs rigorous experimental analysis of the resulting method with varying parameters and different visual features on a data-corpus consisting of real surgical procedures performed on patients with da Vinci Surgical System [9].

Surgical tool pose estimation from monocular endoscopic videos

Surgical tool pose estimation has been proven to be useful for high- and low- level feedback tasks including safety-enhancement, semantic feedback and surgical skill assessment. Tool pose estimation using monocular camera input is a well-studied research problem as the monocular camera is one of the ubiquitous sensor across the spectrum of robotic devices. Current state-of-the art methods for visual tool pose estimation are computationally expensive and require elaborate geometric and appearance models of surgical tools. We propose a visual tool pose estimation method that maps the visual bounding box to the 3D tool pose without any explicit knowledge of tool geometry using Gaussian process regression. The proposed approach can be generalized to any surgical tool and provides tool pose estimates with a variance estimate in real-time. We demonstrate rigorous evaluation of the method under various conditions that might effect the estimation process. In order to evaluate the algorithm, we have instrumented a standard box trainer kit with two laparoscopic tools to get simultaneous ground truth pose and a video feed.

Automation for individualization of Kinect-based quantitative progressive exercise regimen

The Smart Health paradigm has opened up immense possibilities for designing cyber-physical systems with integrated sensing and analysis for data-driven healthcare decision-making. Clinical motor-rehabilitation has traditionally tended to entail labor-intensive approaches with limited quantitative methods and numerous logistics deployment challenges. We believe such labor-intensive rehabilitation procedures offer a fertile application field for robotics and automation technologies. We seek to concretize this Smart Health paradigm in the context of alleviating knee osteoarthritis (OA). Our long-term goal is the creation, analysis and validation of a low-cost cyber-physical framework for individualized but quantitative motor-rehabilitation. We seek build upon parameterized exercise-protocols, low-cost data-acquisition capabilities of the Kinect sensor and appropriate statistical data-processing to aid individualized-assessment and close the quantitative feedback-loop. Specifically, in this paper, we focus our attention on quantitative evaluation of a clinically-relevant deep-squatting exercise. Data for multiple trials with multiple of squatting motions were captured by Kinect system and examined to aid our individualization goals. Principal Component Analysis (PCA) approaches facilitated both dimension-reduction and filtering of the noisy-data while the K-Nearest Neighbors (K-NN) method was adapted for subject classification. Our preliminary deployment of this approach with 5 subjects achieved 95.6% classification accuracy.

An Optimization Based Framework for Human Pose Estimation in Monocular Videos

Human pose estimation using monocular vision is a challenging problem in computer vision. Past work has focused on developing efficient inference algorithms and probabilistic prior models based on captured kinematic/dynamic measurements. However, such algorithms face challenges in generalization beyond the learned dataset.

Approximating Markov Chain Approach to Optimal Feedback Control of a Flexible Needle

We present a computationally efficient approach for the intra-operative update of the feedback control policy for the steerable needle in the presence of the motion uncertainty. The solution to dynamic programming (DP) equations, to obtain the optimal control policy, is difficult or intractable for nonlinear problems such as steering flexible needle in soft tissue. We use the method of approximating Markov chain to approximate the continuous (and controlled) process with its discrete and locally consistent counterpart. This provides the ground to examine the linear programming (LP) approach to solve the imposed DP problem that significantly reduces the computational demand. A concrete example of the two-dimensional (2D) needle steering is considered to investigate the effectiveness of the LP method for both deterministic and stochastic systems. We compare the performance of the LP-based policy with the results obtained through more computationally demanding algorithm, iterative policy space approximation. Finally, the reliability of the LP-based policy dealing with motion and parametric uncertainties as well as the effect of insertion point/angle on the probability of success is investigated. [DOI: 10.1115/1.4033834]

Optimal Feedback Control of a Flexible Needle Under Anatomical Motion Uncertainty

Bevel-tip flexible needles allow for reaching remote/inaccessible organs while avoiding the obstacles (sensitive organs, bones, etc.). Motion planning and control of such systems is a challenging problem due to the uncertainty induced by needle-tissue interactions, anatomical motions (respiratory and cardiac induced motions), imperfect actuation, etc. In this paper, we use an analogy where steering the needle in a soft tissue subject to the uncertain anatomical motions is compared to the Dubins vehicle traveling in the stochastic wind field. Achieving the optimal feedback control policy requires solution of a dynamic programming problem that is often computationally demanding. Efficiency is not central to many optimal control algorithms that often need to be computed only once for a given system/noise statistics. However, intraoperative policy updates may be required for adaptive or patient-specific models. We use the method of approximating Markov chain to approximate the continuous (and controlled) process with its discrete and locally consistent counterpart. We examine the linear programming method of solving the imposed dynamic programming problem that significantly improves the computational efficiency in comparison to the state-of-the-art approaches. In addition, the probability of success and failure are simply the variables of the linear optimization problem and can be directly used for different objective definitions. A numerical example of the 2D needle steering problem is considered to investigate the effectiveness of the proposed method.Copyright

Estimating Dynamics On-the-Fly Using Monocular Video For Vision-Based Robotics

Estimating the physical parameters of articulated multibody systems (AMBSs) using an uncalibrated monocular camera poses significant challenges for vision-based robotics. Articulated multibody models, especially ones including dynamics, have shown good performance for pose tracking, but require good estimates of system parameters. In this paper, we first propose a technique for estimating parameters of a dynamically equivalent model (kinematic/geometric lengths as well as mass, inertia, damping coefficients) given only the underlying articulated model topology. The estimated dynamically equivalent model is then employed to help predict/filter/gap-fill the raw pose estimates, using an unscented Kalman filter. The framework is tested initially on videos of a relatively simple AMBS (double pendulum in a structured laboratory environment). The double pendulum not only served as a surrogate model for the human lower limb in flight phase, but also helped evaluate the role of model fidelity. The treatment is then extended to realize physically plausible pose-estimates of human lower-limb motions, in more-complex uncalibrated monocular videos (from the publicly available DARPA Minds Eye Year 1 corpus). Beyond the immediate problem-at-hand, the presented work has applications in creation of low-order surrogate computational dynamics models for analysis, control, and tracking of many other articulated multibody robotic systems (e.g., manipulators, humanoids) using vision.