Wai-keung Fung

Development of augmented reality system for AFM-based nanomanipulation

Using atomic force microscopy (AFM) as a nanomanipulation tool has been discussed for more than a decade. However, its lack of real-time visual feedback during manipulation has hindered its wide application. Fortunately, this problem has been overcome by our recently developed augmented reality system. By locally updating the AFM image based on real-time force information during manipulation, not only can this new system provide real-time force feedback but also real-time visual feedback. The real-time visual display combined with the real-time force feedback provides an augmented reality environment, in which the operator not only can feel the interaction forces but also observe the real-time changes of the nano-environment. This augmented reality system capable of nanolithography and manipulation of nano-particles helps the operator to perform several operations without the need of a new image scan, which makes AFM-based nano-assembly feasible and applicable.

Haptic information in Internet-based teleoperation

Many tasks can be done easily by humans turn out to be very difficult to accomplish with a teleoperated robot. The main reason for this is the lack of tactile sensing, which cannot be replaced by visual feedback alone. Once haptic devices are developed, their potential in many fields is obvious. Especially, in teleoperation systems, where haptic feedback can increase the efficiency and even render some tasks feasible. This paper studies Internet-based teleoperation systems that include haptic feedback, concentrating on the control of such systems and their performance. The potential of this technology and its advantages are explored. In addition, key issues, such as stability, synchronization, and transparency are analyzed and studied. Specifically, an event-based planning and control of Internet-based teleoperation systems is presented with experimental results of several implemented system scenarios in micro- and macro-scales.

Supermedia-enhanced Internet-based telerobotics

This paper introduces new planning and control methods for supermedia-enhanced real-time telerobotic operations via the Internet. Supermedia is the collection of video, audio, haptic information, temperature, and other sensory feedback. However, when the communication medium used, such as the Internet, introduces random communication time delay, several challenges and difficulties arise. Most importantly, random communication delay causes instability, loss of transparency, and desynchronization in real-time closed-loop telerobotic systems. Due to the complexity and diversity of such systems, the first challenge is to develop a general and efficient modeling and analysis tool. This paper proposes the use of Petri net modeling to capture the concurrency and complexity of Internet-based teleoperation. Combined with the event-based planning and control method, it also provides an efficient analysis and design tool to study the stability, transparency, and synchronization of such systems. In addition, the concepts of event transparency and event synchronization are introduced and analyzed. This modeling and control method has been applied to the design of several supermedia-enhanced Internet-based telerobotic systems, including the bilateral control of mobile robots and mobile manipulators. These systems have been experimentally implemented in three sites test bed consisting of robotic laboratories in the USA, Hong Kong, and Japan. The experimental results have verified the theoretical development and further demonstrated the stability, event transparency, and event synchronization of the systems.

Supermedia in Internet-Based Telerobotic Operations

In the past decade robotics and the Internet, fed by the great advances in computing and networking, matured into giant interdisciplinary scientific fields. Therefore, it is not surprising that many are trying to merge these two technologies and develop Internet-based robotic teleoperation. More interestingly, Internet-based bilateral teleoperation, where supermedia is fed back to the operator in order to increase efficiency and achieve telepresence. Supermedia is the collection of multimedia (video, audio, ...), haptic and other sensory information. This paper studies supermedia enhanced teleoperation via the Internet, concentrating on the real-time control of such systems and their performance. The potential of this technology and its advantages will be explored. In addition, key issues, such as stability, synchronization and transparency, will be analyzed and studied. Specifically, event-based planning and control of Internet-based teleoperation systems is presented with experimental results of several implemented system scenarios.

Multi-site Internet-based cooperative control of robotic operations

The e-world, also known as the Internet, has added a new dimension to many of the traditional concepts in industrial applications and everyday life. The use of robots has dramatically expanded the potential of e-services. Individuals with particular expertise can perform highly accurate and fairly complicated tasks remotely via the Internet. This increase in the human reachability is faced by several obstacles. Reliable and efficient robot facilitated services via the Internet face several challenges. These range from human-computer interfacing and overcoming random time delay to task synchronization and human-robot interaction. These limitations intensify when many operators in many sites are involved. This paper provides new theoretical and experimental results on these challenges. Specifically, multisite cooperative control of an Internet based mobile manipulator is presented. The two main characteristics of this system are Internet based real-time closed loop control and coordinated operation. In addition, it is shown that despite random time delay the stability and synchronization of the system were achieved using event-based control.

Augmented reality system for real-time nanomanipulation

By introducing haptic force feedback, nanomanipulation using the AFM becomes easier. However, this technique has been impeded by tip displacement due to the softness of AFM cantilever. In this paper, a novel nanomanipulation system assisted by augmented reality is presented. By analyzing the cantilever-tip interaction with the environment, the normal force and the lateral force are obtained. These forces are then fed to a haptic device to provide real-time force feedback for operator. The visual images are also updated based on real-time force and position information, which provides the operator with real-time visual feedback. New calibration methods and compensation algorithm are introduced to compensate for the tip displacement, therefore, the visual display and force feeling from the haptic device become reliable. The real-time force feedback and the real-time visual display provide the operator with an augmented reality environment. Experimental results have verified the effectiveness of the proposed novel augmented reality system.

Modeling and control of Internet based cooperative teleoperation

Robotic operations carried out via the Internet face several challenges and difficulties. These range from human-computer interfacing and human-robot interaction to overcoming random time delay and task synchronization. These limitations are intensified when multi-operators at multisites are collaboratively teleoperating multirobots to achieve a certain task. In this paper, a new modeling and control method for Internet-based cooperative teleoperation is developed. Combining Petri net model and event-based planning and control theory, the new method provides an efficient way to model the concurrence and complexity of the Internet-based cooperative teleoperation. It also provides an efficient analysis tool to study the stability, transparency and synchronization of the system. Furthermore, the new modeling and control method enables us to design an Internet-based cooperative telerobotic system that is reliable, safe and intelligent. This new method has been experimentally implemented in a three site test bed consisting of robotic laboratories in the USA, Hong Kong and Japan. The experimental results have verified the theoretical development and further demonstrated the advantages of the new modeling and control method.

Simulation of large-scale electrical power networks on graphics processing units

In this paper suitability of applying graphics processing unit (GPU) based computing for electromagnetic transient (EMT) simulation of large scale power systems is demonstrated. As the network size increases there is a corresponding increase in simulation time with conventional central processing unit (CPU) based simulation tools. The paper shows that with a hybrid environment consisting of CPUs and GPUs, simulation time is much less compared to the CPU-only implementations. In this scheme the GPU is mainly used to do the computationally intensive part of the simulation in parallel on its built-in multiple processing cores, and the CPU is assigned for updating history terms and flow control of the simulation. The GPU-based approach is used to simulate a network with 117 nodes, and it is shown that the CPU-GPU based implementation takes less than half of the time taken by the CPU-only implementation of the simulation.

Task driven dynamic QoS based bandwidth allocation for real-time teleoperation via the Internet

In real-time Internet based teleoperation, different robotic tasks have different dexterity requirements during task progress and thus different network resources are required for safe and reliable task accomplishment. In order to control remote manipulators efficiently and smoothly via the Internet, dynamic bandwidth allocation is crucial to successful accomplishment of robotic tasks controlled by remote operator. In this paper, a novel bandwidth allocation mechanism is developed based on the online measured task dexterity index of current dexterous tasks so that operators can control remote manipulators efficiently and smoothly even under poor network quality. Experiments have been conducted to demonstrate the effectiveness of the presented resource (bandwidth) allocation algorithm in Internet based teleoperation system.

Adaptive categorization of ART networks in robot behavior learning using game-theoretic formulation

Adaptive Resonance Theory (ART) networks are employed in robot behavior learning. Two of the difficulties in online robot behavior learning, namely, (1) exponential memory increases with time, (2) difficulty for operators to specify learning tasks accuracy and control learning attention before learning. In order to remedy the aforementioned difficulties, an adaptive categorization mechanism is introduced in ART networks for perceptual and action patterns categorization in this paper. A game-theoretic formulation of adaptive categorization for ART networks is proposed for vigilance parameter adaptation for category size control on the categories formed. The proposed vigilance parameter update rule can help improving categorization performance in the aspect of category number stability and solve the problem of selecting initial vigilance parameter prior to pattern categorization in traditional ART networks. Behavior learning using physical robot is conducted to demonstrate the effectiveness of the proposed adaptive categorization mechanism in ART networks.