Wenjia Yuan

Computer vision methods for visual MIMO optical system

Cameras have become commonplace in phones, laptops, music-players and handheld games. Similarly, light emitting displays are prevalent in the form of electronic billboards, televisions, computer monitors, and hand-held devices. The prevalence of cameras and displays in our society creates a novel opportunity to build camera-based optical wireless communication systems based on a concept called visual MIMO. We extend the common term MIMO from the field of communications (“multiple-input multiple-output”) that is typically used to describe multiple antenna, multiple transmitter radio frequency communications channel. In the visual MIMO communications paradigm, the transmitters are light-emitting devices such as electronic displays and cameras are the receivers. In this paper we discuss and address several challenges in creating a visual MIMO channel. These challenges include: (1) electronic display detection, (2) embedding the transmission signal in the display video, and (3) system characterization for electronic display appearance.

Dynamic and invisible messaging for visual MIMO

The growing ubiquity of cameras in hand-held devices and the prevalence of electronic displays in signage creates a novel framework for wireless communications. Traditionally, the term MIMO is used for multiple-input multiple-output where the multiple-input component is a set of radio transmitters and the multiple-output component is a set of radio receivers. We employ the concept of visual MIMO where pixels are transmitters and cameras are receivers. In this manner, the techniques of computer vision can be combined with principles from wireless communications to create an optical line-of-sight communications channel. Two major challenges are addressed: (1) The message for transmission must be embedded in the observed display so that the message is hidden from the observer and the electronic display can simultaneously be used for its originally intended purpose (e.g. signage, advertisements, maps); (2) Photometric and geometric distortions during the imaging process corrupt the information channel between the transmitter display and the receiver camera. These distortions must be modeled and removed. In this paper, we present a real-time messaging paradigm and its implementation in an operational visual MIMO optical systems. As part of the system, we develop a novel algorithm for photographic message extraction which includes automatic display detection, message embedding and message retrieval. Experiments show that the system achieves an average accuracy of 94.6% at the bitrate of 6222.2 bps.

Capacity of pervasive camera based communication under perspective distortions

Cameras are ubiquitous and increasingly being used not just for capturing images but also for communicating information. For example, the pervasive QR codes can be viewed as communicating a short code to camera-equipped sensors and recent research has explored using screen-to-camera communications for larger data transfers. Such communications could be particularly attractive in pervasive camera based applications, where such camera communications can reuse the existing camera hardware and also leverage from the large pixel array structure for high data-rate communication. While several prototypes have been constructed, the fundamental capacity limits of this novel communication channel in all but the simplest scenarios remains unknown. The visual medium differs from RF in that the information capacity of this channel largely depends on the perspective distortions while multipath becomes negligible. In this paper, we create a model of this communication system to allow predicting the capacity based on receiver perspective (distance and angle to the transmitter). We calibrate and validate this model through lab experiments wherein information is transmitted from a screen and received with a tablet camera. Our capacity estimates indicate that tens of Mbps is possible using a smartphone camera even when the short code on the screen images onto only 15% of the camera frame. Our estimates also indicate that there is room for at least 2.5x improvement in throughput of existing screen - camera communication prototypes.

Rate adaptation in visual MIMO

We propose a rate adaptation scheme for visual MIMO camera-based communications, wherein parallel data transmissions from light emitting arrays are received by multiple receive elements of a CCD/CMOS camera image sensor. Unlike RF MIMO, multipath fading is negligible in the visual MIMO channel. Instead, the channel is largely dependent on receiver perspective (distance and angle) and visibility issues (partial line-of-sight availability and occlusions). This allows for slower adaptation but requires the adaptation algorithm to choose among a more complex set of modes. In this paper, we define a set of operating modes for visual MIMO transmitters and propose a rate adaptation scheme to switch between these modes. Our Visual MIMO Rate Adaptation (VMRA) is a packet based rate adaptation protocol that bases its rate selection decisions on the packet error rate feedback. Using trace-based simulation results for a vehicle-to-vehicle communication scenario, we illustrate how our VMRA algorithms can adapt over distance as well as visibility variations in an optical link and achieve a higher average throughput.

Demo: visual MIMO based LED - camera communication applied to automobile safety

The inherent limitations in RF spectrum availability and susceptibility to interference make it difficult to meet the reliability required for automotive safety applications. To address this challenge, this work explores an alternative communication system called Visual MIMO that uses light emitting arrays as transmitters and cameras as receivers. Visual MIMO applied to vehicular communication proposes to reuse existing LED rear and headlights as transmitters and existing cameras (e.g. those used for parking assistance, rear-view cameras) as receivers. In this work we show a proof of concept based demonstration of the Visual MIMO system consisting of an LED transmitter array and a high-speed camera.

BiFocus: using radio-optical beacons for an augmented reality search application

Augmented Reality (AR) applications benefit from accurate detection of the objects that are within a persons view. Typically, it is not only desirable to identify what is currently within view, but also to navigate the users view to the item of interest - for example, finding a misplaced object. In this paper we demonstrate a low-power hybrid radio-optical beaconing system, where objects of interest are tagged with battery-powered RFID-like tags equipped with infrared light emitting diodes (LED) that emit periodic infrared beacons. These beacons are used for accurately estimating the angle and distance from the object to the receiver so as to locate it. The beacons are synchronized using the radio link that is also used to convey the objects unique ID.

Spatially varying radiometric calibration for camera-display messaging

Modern society has ubiquitous electronic displays including billboards, signage and kiosks. The concurrent prevalence of handheld cameras creates a novel opportunity to use cameras and displays as communication channels. The electronic display in this channel serves a twofold purpose: to display an image to humans while simultaneously transmitting hidden bits for decoding by a camera. Unlike standard digital watermarking, the message recovery in camera-display systems requires physics-based modeling of image formation in order to optically communicate hidden messages in real world scenes. By modeling the photometry of the system using a camera-display transfer function (CDTF), we show that this function depends on camera pose and varies spatially over the display.We devise a radiometric calibration to handle the nonlinearities of both the display and the camera, and we use this method for recovering video messages hidden within display images. Results are for 9 different display-camera systems for messages with 4500 bits. Message accuracy improves significantly with calibration and we achieve accuracy near 99% in our experiments, independent of the type of camera or display used.

Optimal radiometric calibration for camera-display communication

We present a novel method for communicating between a moving camera and an electronic display by embedding and recovering hidden, dynamic information within an image. A small intensity pattern is added to alternate frames of a time-varying display. A handheld camera pointed at the display can receive not only the display image, but also an underlying message. Differencing the camera-captured alternate frames leaves the small intensity pattern, but results in errors due to photometric effects that depend on camera pose. Detecting and robustly decoding the message requires careful photometric modeling for message recovery. The key innovation of our approach is an algorithm that performs simultaneous radiometric calibration and message recovery in one convex optimization problem. By modeling the photometry of the system using a camera-display transfer function (CDTF), we derive an optimal online radiometric calibration (OORC) for robust computational messaging as demonstrated with nine different commercial cameras and displays. The online radiometric calibration algorithms described in this paper significantly reduces message recovery errors, especially for low intensity messages and oblique camera angles.

What Am I Looking At? Low-Power Radio-Optical Beacons for In-View Recognition on Smart-Glass

Applications on wearable personal imaging devices, or Smart-glasses as they are called, can largely benefit from accurate and energy-efficient recognition of objects that are within the users view. Existing solutions such as optical or computer vision approaches are too energy intensive, while low-power active radio tags suffer from imprecise orientation estimates. To address this challenge, this paper presents the design, implementation, and evaluation of a radio-optical hybrid system where a radio-optical transmitter, or tag, whose radio-optical beacons are used for accurate relative orientation tracking of tagged objects by a wearable radio-optical receiver. A low-power radio link that conveys identity is used to reduce the battery drain by synchronizing the radio-optical transmitter and receiver so that extremely short optical (infrared) pulses are sufficient for orientation (angle and distance) estimation. Through extensive experiments with our prototype we show that our system can achieve orientation estimates with 1-to-2 degree accuracy and within 40 cm ranging error, with a maximum range of 9 m in typical indoor use cases. With a tag and receiver battery power consumption of 81 μW and 90 mW, respectively, our radio-optical tags and receiver are at least 1.5x energy efficient than prior works in this space.

Low-Power Radio-Optical Beacons for In-View Recognition

Object recognition on wearable devices using computer vision is too energy intensive and challenging when objects are similar looking, while low-power active radio frequency identification (RFID) systems suffer from imprecise orientation (angle and distance) estimates. To address this challenge, this paper presents a novel radio-optical based recognition system where a radio-optical transmitter, or tag, that emits a beacon whose infra-red (IR) signal strength is used for accurate relative orientation tracking of tagged objects at a wearable radio-optical receiver. A low-power radio link that conveys identity is used to reduce the battery drain by synchronizing the radio- optical transmitter and receiver so that extremely short optical pulses are sufficient for precise orientation estimation. Through extensive experiments with our prototype we show that our system can achieve orientation estimates with 1-2° accuracy and within 40cm ranging error, with a maximum range of 9m in typical indoor use cases. With a tag battery power consumption of 86μW, the radio-optical tags show potential to achieve about half a decade lifetimes.