Injoon Hong

A 320 mW 342 GOPS Real-Time Dynamic Object Recognition Processor for HD 720p Video Streams

Moving object recognition in a video stream is crucial for applications such as unmanned aerial vehicles (UAVs) and mobile augmented reality that require robust and fast recognition in the presence of dynamic camera noise. Devices in such applications suffer from severe motion/camera blur noise in low-light conditions due to low-sensitivity CMOS image sensors, and therefore require higher computing power to obtain robust results vs. devices used in still image applications. Moreover, HD resolution has become so universal today that even smartphones support applications with HD resolution. However, many object recognition processors and accelerators reported for mobile applications only support SD resolution due to the computational complexity of object recognition algorithms. This paper presents a moving-target recognition processor for HD video streams. The processor is based on a context-aware visual attention model (CAVAM).

A 646GOPS/W multi-classifier many-core processor with cortex-like architecture for super-resolution recognition

Object recognition processors have been reported for the applications of autonomic vehicle navigation, smart surveillance and unmanned air vehicles (UAVs) [1-3]. Most of the processors adopt a single classifier rather than multiple classifiers even though multi-classifier systems (MCSs) offer more accurate recognition with higher robustness [4]. In addition, MCSs can incorporate the human vision system (HVS) recognition architecture to reduce computational requirements and enhance recognition accuracy. For example, HMAX models the exact hierarchical architecture of the HVS for improved recognition accuracy [5]. Compared with SIFT, known to have the best recognition accuracy based on local features extracted from the object [6], HMAX can recognize an object based on global features by template matching and a maximum-pooling operation without feature segmentation. In this paper we present a multi-classifier many-core processor combining the HMAX and SIFT approaches on a single chip. Through the combined approach, the system can: 1) pay attention to the target object directly with global context consideration, including complicated background or camouflaging obstacles, 2) utilize the super-resolution algorithm to recognize highly blurred or small size objects, and 3) recognize more than 200 objects in real-time by context-aware feature matching.

10.4 A 1.22TOPS and 1.52mW/MHz augmented reality multi-core processor with neural network NoC for HMD applications

Augmented reality (AR) is being investigated in advanced displays for the augmentation of images in a real-world environment. Wearable systems, such as head-mounted display (HMD) systems, have attempted to support real-time AR as a next generation UI/UX [1-2], but have failed, due to their limited computing power. In a prior work, a chip with limited AR functionality was reported that could perform AR with the help of markers placed in the environment (usually 1D or 2D bar codes) [3]. However, for a seamless visual experience, 3D objects should be rendered directly on the natural video image without any markers. Unlike marker-based AR, markerless AR requires natural feature extraction, general object recognition, 3D reconstruction, and camera-pose estimation to be performed in parallel. For instance, markerless AR for a VGA input-test video consumes ~1.3W power at 0.2fps throughput, with TIs OMAP4430, which exceeds power limits for wearable devices. Consequently, there is a need for a high-performance energy-efficient markerless AR processor to realize a real-time AR system, especially for HMD applications.

18.3 A 0.5V 54μW ultra-low-power recognition processor with 93.5% accuracy geometric vocabulary tree and 47.5% database compression

Microwatt object recognition is being considered for many applications, such as autonomous micro-air-vehicle (MAV) navigation, a vision-based wake-up or user authentication for the smartphones, and a gesture recognition-based natural UI for wearable devices in the Internet-of-Things (IoT) era. These applications require extremely low power consumption, while maintaining high recognition accuracy - constraints that arise because of the requirement for continuous heavy vision processing under limited battery capacity. Recently, a low-power feature-extraction accelerator operating at near-threshold voltage (NTV) was proposed, however, it did not support the object matching essential for the object recognition [1]. Even state-of-the-art object matching accelerators consume over 10mW, thereby making them unsuitable for an MAV [2, 3]. Therefore, an ultra-low-power high-accuracy recognition processor is necessary, especially for MAVs and IoT devices.

18.1 A 2.71nJ/pixel 3D-stacked gaze-activated object-recognition system for low-power mobile HMD applications

Smart eyeglasses or head-mounted displays (HMDs) have been gaining traction as next-generation mainstream wearable devices. However, previous HMD systems [1] have had limited application, primarily due to their lacking a smart user interface (Ul) and user experience (UX). Since HMD systems have a small compact wearable platform, their Ul requires new modalities, rather than a computer mouse or a 2D touch panel. Recent speech-recognition-based Uls require voice input to reveal the users intention to not only HMD users but also others, which raises privacy concerns in a public space. In addition, prior works [2-3] attempted to support object recognition (OR) or augmented reality (AR) in smart eyeglasses, but consumed considerable power, >381mW, resulting in <;6 hours operation time with a 2100mWh battery.

A 2.71 nJ/Pixel Gaze-Activated Object Recognition System for Low-Power Mobile Smart Glasses

A low-power object recognition (OR) system with intuitive gaze user interface (UI) is proposed for battery-powered smart glasses. For low-power gaze UI, we propose a low-power single-chip gaze estimation sensor, called gaze image sensor (GIS). In GIS, a novel column-parallel pupil edge detection circuit (PEDC) with new pupil edge detection algorithm XY pupil detection (XY-PD) is proposed which results in 2.9× power reduction with 16× larger resolution compared to previous work. Also, a logarithmic SIMD processor is proposed for robust pupil center estimation, (1 pixel error, with low-power floating-point implementation. For OR, low-power multicore OR processor (ORP) is implemented. In ORP, task-level pipeline with keypoint-level scoring is proposed to reduce the number of cores as well as the operating frequency of keypoint-matching processor (KMP) for low-power consumption. Also, dual-mode convolutional neural network processor (CNNP) is designed for fast tile selection without external memory accesses. In addition, a pipelined descriptor generation processor (DGP) with LUT-based nonlinear operation is newly proposed for low-power OR. Lastly, dynamic voltage and frequency scaling (DVFS) for dynamic power reduction in ORP is applied. Combining both of the GIS and ORP fabricated in 65 nm CMOS logic process, only 75 mW average power consumption is achieved with real-time OR performance, which is 1.2× and 4.4× lower power than the previously published work.

An 1.61mW mixed-signal column processor for BRISK feature extraction in CMOS image sensor

In mobile object recognition (OR) applications, the power consumption of image sensor and data communication between image sensor and digital OR processor becomes crucial as digital OR processor consumes less power in deep sub-micron process. To reduce the amount of data transaction from image sensor to digital OR processor, digital/analog mixed-signal focal-plane processing of Binary Robust Invariant Scalable Keypoints (BRISK) feature extraction in CMOS image sensor (CIS) is proposed. The proposed CIS processor sends BRISK feature vectors instead of the whole image pixel data, resulting in 79% reduction of data communication. In this work, mixed-signal processing of corner detection and successive approximation register (SAR)-based scoring are implemented for BRISK feature point detection. To achieve scale-invariance in object recognition, scale-space is generated and stored in analog line memory. In addition, noise reduction scheme is integrated in column processing chain to remove salt and pepper noise, which degrades recognition accuracy. In a post layout simulation, the proposed system achieves 0.70pW/pixel*frame*feature at 30fps in a 130nm CMOS technology, which is 13.6% lower than the state-of-the-art.

Intelligent Network-on-Chip With Online Reinforcement Learning for Portable HD Object Recognition Processor

An intelligent Reinforcement Learning (RL) Network-on-Chip (NoC) is proposed as a communication architecture of a heterogeneous many-core processor for portable HD object recognition. The proposed RL NoC automatically learns bandwidth adjustment and resource allocation in the heterogeneous many-core processor without explicit modeling. By regulating the bandwidth and reallocating cores, the throughput performances of feature detection and description are increased by 20.4% and 11.5%, respectively. As a result, the overall execution time of the object recognition is reduced by 38%. The proposed processor with RL NoC is implemented in a 65 nm CMOS process, and it successfully demonstrates the real-time object recognition for a 720 p HD video stream while consuming 235 mW peak power at 200 MHz, 1.2 V.

An Energy-Efficient Embedded Deep Neural Network Processor for High Speed Visual Attention in Mobile Vision Recognition SoC

An energy-efficient Deep Neural Network (DNN) processor is proposed for high-speed Visual Attention (VA) engine in a mobile vision recognition SoC. The proposed embedded DNN (E-DNN) processor realizes VA to rapidly find Region-Of-Interest (ROI) tiles of potential target objects to reduce ~ 70% of recognition workloads of a vision recognition SoC. Compared to the previous scale-invariant feature transform (SIFT) based VA models, the proposed E-DNN VA model reduces the execution time by ~ 90%, which results in 73.4% reduction of the overall object recognition (OR) processing time. Also, the proposed E-DNN VA model shows ~4% higher OR accuracy for 113-object database (13 laboratory object database + COIL-100 objects database) than the previous model shows. Highly-parallel 200-way PEs are implemented in the E-DNN processor with 2D-axis layer sliding architecture, and only ~3 ms of the E-DNN VA latency can be obtained. In addition, the dual-mode configurable PE architecture is proposed to support both Convolution Neural Network (CNN) and Multi-Layer Perceptron (MLP) by utilizing the same hardware resources for high energy efficiency. As a result, the implemented E-DNN processor achieves only 1.9 nJ/pixel energy efficiency which is 7.7× smaller than the state-of-the-art VA accelerator which showed 14.6 nJ/pixel energy efficiency.

A 27 mW Reconfigurable Marker-Less Logarithmic Camera Pose Estimation Engine for Mobile Augmented Reality Processor

A marker-less camera pose estimation engine (CPEE) with reconfigurable logarithmic SIMD processor is proposed for the view angle estimation used in mobile augmented reality (AR) applications. Compared to previous marker-based approach, marker-less camera pose estimation can overlay virtual images directly on natural world without the help of markers. However, they require 150× larger computing costs and large power consuming floating-point operations where such requirement severely restricts real-time operations and low-power implementations in mobile platform, respectively. To overcome the gap in computational costs between marker-based and marker-less method, speculative execution (SE) and reconfigurable data-arrangement layer (RDL) are proposed to reduce computing time by 17% and 27%, respectively. For low-power implementation of floating-point units, logarithmic processing element (LPE) is designed to reduce overall power consumption by 18%. The proposed marker-less CPEE is fabricated in 65 nm Logic CMOS technology, and successfully realizes real-time marker-less camera pose estimation with only 27 mW power consumption.