Real-time human hand following system based on multiple vision sensors

Abstract

This paper designs a real-time human hand following system based on multi-vision sensors. Through the KinectV2 camera, human hand behavior can be observed; the pose information of the human hand joint can also be obtained by using OpenNI2 and NITE2 interface; calculating the angle information of each joint of the six-degree-of-freedom robot can be achieved by stereo geometric correlation calculation. Through LeapMotion, accurate human joint information is obtained, then calculating the finger distance to control manipulator s tension, which is smoothed by Kalman filtering. Control, simulation and visualization for real-time follow-up of the robotic arm and the human hand are achieved under the Robot Operating System(ROS). The final experimental results are verified by Moveit s simulation environment and Kinova s Jaco arm.