Burak Cizmeci

Point Cloud-Based Model-Mediated Teleoperation With Dynamic and Perception-Based Model Updating

In this paper, we extend the concept of model-mediated teleoperation for complex environments and six degrees of freedom interaction using point cloud surface models. In our system, a time-of-flight camera is used to capture a high-resolution point cloud model of the object surface. The point cloud model and the physical properties of the object (stiffness and surface friction coefficient) are estimated at the slave side in real-time and transmitted to the master side using the modeling and updating algorithm proposed in this paper. The proposed algorithm adaptively controls the updating of the point cloud model and the object properties according to the slave movements and by exploiting known limitations of human haptic perception. As a result, perceptually irrelevant transmissions are avoided, and thus the packet rate in the communication channel is substantially reduced. In addition, a simple point cloud-based haptic rendering algorithm is adopted to generate the force feedback signals directly from the point cloud model without first converting it into a 3-D mesh. In the experimental evaluation, the system stability and transparency are verified in the presence of a round-trip communication delay of up to 1000 ms. Furthermore, by exploiting the limits of human haptic perception, the presented system allows for a significant haptic data reduction of about 90% for teleoperation systems with time delay.

Model-Mediated Teleoperation: Toward Stable and Transparent Teleoperation Systems

Bilateral teleoperation systems with haptic feedback allow human users to interact with objects or perform complex tasks in remote or inaccessible environments. Communication delays in teleoperation systems jeopardize system stability and transparency, leading to degraded system performance and poor user experience. In this paper, we provide a survey of the model-mediated teleoperation (MMT) approach, which has been developed to guarantee both system stability and transparency in the presence of arbitrary communication delays. This survey focuses on two major parts: 1) the historical development of the MMT approach from the late 1980s to the present and 2) the main challenges facing the design of a reliable MMT system. Along with the discussion of the MMT challenges and the proposed solutions, a series of experiments has been conducted to compare the performance between the existing techniques and to supply data that were missing in the previous studies on the MMT approach.

Low bitrate source-filter model based compression of vibrotactile texture signals in haptic teleoperation

Vibrotactile signals convey the touch-based characteristics of object surfaces felt through a tool. They particularly enhance the quality of human-machine interactions by providing realistic haptic perception of textures. In this paper, inspired by the similarities observed between vibrotactile texture signals and speech signals, we present a novel vibrotactile texture codec for bilateral teleoperation, based on well-known speech coding techniques. The proposed low bitrate, high quality codec preserves not only the spectral signature vital to the general feel of the texture, but also important temporal features of the texture signal. We report a compression ratio of 8:1 (12.5 %) with a constant output bitrate of 4 kbps, and we validate the perceptual transparency of the codec via rigorous subjective tests and analyses.

A Visual-Haptic Multiplexing Scheme for Teleoperation over Constant-Bitrate Communication Links

We propose a novel multiplexing scheme for teleoperation over constant bitrate (CBR) communication links. The proposed approach uniformly divides the channel into 1 ms resource buckets and controls the size of the transmitted video packets as a function of the irregular haptic transmission events generated by a perceptual haptic data reduction approach. The performance of the proposed multiplexing scheme is measured objectively in terms of delay-jitter and packet rates. The results show that acceptable multiplexing delays on both the visual and haptic streams are achieved. Our evaluation shows that the proposed approach can provide a guaranteed constant delay for the time-critical force signal, while introducing acceptable video delay.

Haptic data reduction for time-delayed teleoperation using the time domain passivity approach

We propose a perceptual haptic data reduction approach for teleoperation systems which use the time domain passivity approach (TDPA) as their control architecture for dealing with time-varying communication delay. Our goal is to reduce the packet rate over the communication network while preserving system stability in the presence of time-varying and unknown delays. Compared to the existing wave variable-based (WV-based) haptic data reduction approaches, our proposed scheme leads to smaller distortion in the force signals and robustly deals with time-varying delays. Experiments show that our proposed approach can reduce the average packet rate by up to 80%, without introducing significant distortion. In addition, the proposed approach outperforms the existing WV-based approaches in both packet rate reduction and subjective preference for the tested communication delays.

On the perceptual artifacts introduced by packet losses on the forward channel of haptic telemanipulation sessions

In this work we study the position and velocity signal reconstructions using predictive coding when packets are lost during telemanipulation sessions and classify the high-level haptic artifacts perceived by the users. The usage of packet-switched networks for bilateral telemanipulation systems is challenging due to several adversities such as low transmission rates, packet delays, jitter and losses. The previously proposed deadband-based haptic data reduction approaches selectively decrease the high transmission rate of the force-feedback and position/velocity samples on account of human perception limitations. Recently, an error-resilient perceptual haptic data reduction approach was proposed to address the packet losses in the feedback channel. However, the impact of faltered transmission on the forward channel and its subjective influence on the user are still an open issue and thus are treated in this paper.

A Multiplexing Scheme for Multimodal Teleoperation

This article proposes an application-layer multiplexing scheme for teleoperation systems with multimodal feedback (video, audio, and haptics). The available transmission resources are carefully allocated to avoid delay-jitter for the haptic signal potentially caused by the size and arrival time of the video and audio data. The multiplexing scheme gives high priority to the haptic signal and applies a preemptive-resume scheduling strategy to stream the audio and video data. The proposed approach estimates the available transmission rate in real time and adapts the video bitrate, data throughput, and force buffer size accordingly. Furthermore, the proposed scheme detects sudden transmission rate drops and applies congestion control to avoid abrupt delay increases and converge promptly to the altered transmission rate. The performance of the proposed scheme is measured objectively in terms of end-to-end signal latencies, packet rates, and peak signal-to-noise ratio (PSNR) for visual quality. Moreover, peak-delay and convergence time measurements are carried out to investigate the performance of the congestion control mode of the system.

Energy Prediction for Teleoperation Systems That Combine the Time Domain Passivity Approach with Perceptual Deadband-Based Haptic Data Reduction

We study the combination of the perceptual deadband (PD)-based haptic packet rate reduction scheme with the time domain passivity approach (TDPA) for time-delayed teleoperation and propose a novel energy prediction (EP) scheme that deals with the conservative behavior of the resulting controller. The PD approach leads to irregular packet transmission, resulting in degraded system transparency and reduced teleoperation quality when the PD approach is combined with the TDPA. The proposed method (PD+TDPA+EP) adaptively predicts the system energy during communication interruptions and allows for larger energy output. This achieves less conservative control and improves the teleoperation quality. Evaluation of the displayed impedance shows that the PD+TDPA+EP method achieves improved system transparency, both objectively and subjectively, when compared with related approaches from literature. According to a subjective user study, the PD+TDPA+EP method allows for a high packet rate reduction (up to 80 percent) without noticeably distorting the perceived interaction quality. We also show that the PD+TDPA+EP method is preferred over related approaches from literature in a direct comparison test. Thus, with the proposed PD+TDPA+EP method, a high data reduction and a high teleoperation quality are simultaneously achieved for time-delayed teleoperation.

Macroblock level rate control for low delay H.264/AVC based video communication

In this paper, we propose a macro-block (MB) level rate control algorithm for low delay H.264/AVC video communication based on the ρ domain rate model. In the proposed algorithm, an exponential model is used to characterize the relation between ρ and the quantization step (Qstep) at the MB level, with which the quantization parameter (QP) for a MB can be obtained. Furthermore, a switched QP calculation scheme is introduced to obtain the QP for each MB to avoid large deviation of the actual frame size from the target bit budget. Compared with the original ρ domain rate control, the proposed method can achieve better video quality and improved bit-rate accuracy. Meanwhile, the computational complexity is also significantly reduced.

A Stereoscopic Vision System with Delay Compensation for 360° Remote Reality

The rapid development of virtual reality systems and their increasing acceptance result in a high demand for 3D content, in particular, content that can be viewed in 360°. The acquisition of monoscopic 360° videos is straightforward and is typically done with a single camera in combination with panoramic optics or an arrangement of two 180° fisheye cameras. Stereoscopic 360° videos support the perception of depth but are considerably more challenging to capture. They require sophisticated multi-camera arrangements, which leads to heavy, bulky and expensive systems. Furthermore, they need computationally demanding post-processing. For telepresence applications, where the user wears VR glasses to get a 3D view of the scene in front of a mobile telepresence platform, such a vision system must be real-time capable. Another challenge is the network-induced delay, which leads to motion sickness. In this paper, we present a stereoscopic vision system that captures stereo video content for VR displays and consists of only two cameras, a pan-tilt-unit and a visual delay compensation algorithm. With our approach, the user is allowed to rotate his head by 360°. The proposed system compensates the perceived delay when the user rotates the head around the z-axis by streaming a larger field-of-view (FoV) than needed by the VR display. We provide an analytical solution for the required camera FoV as a function of the FoV of the HMD and the communication delay. Our experimental evaluation for typical head motions shows that the proposed system achieves a mean compensation rate of up to 95% for the tested communication delays of 0-500ms.