Pei-Kuei Tsung

A 212 MPixels/s 4096

To provide more vivid perception, TV resolution is increasing dramatically. In addition, 3D video is emerging because it can present immersive and complete scenes. Therefore, multiview video coding (MVC) is currently being developed as an extension of H.264/AVC [1]. Disparity estimation (DE), which effectively exploits the inter-view redundancy and reduces bit rates 20% to 30%, is the most significant feature. However, DE and motion estimation (ME) require ultra-high computation and memory access. To encode a 3-view 1080p video, 82.4TOPS computing power and 54.6TB/s memory access are required with a full search algorithm. Moreover, view scalability is a critical functionality to deal with various MVC structures.

\times

3-D video will become one of the most significant video technologies in the next-generation television. Due to the ultra high data bandwidth requirement for 3-D video, effective compression technology becomes an essential part in the infrastructure. Thus multiview video coding (MVC) plays a critical role. However, MVC systems require much more memory bandwidth and computational complexity relative to mono-view video coding systems. Therefore, an efficient prediction scheme is necessary for encoding. In this paper, a new fast prediction algorithm, content-aware prediction algorithm (CAPA) with inter-view mode decision, is proposed. By utilizing disparity estimation (DE) to find corresponding blocks between different views, the coding information, such as rate-distortion cost, coding modes, and motion vectors, can be effectively shared and reused from the coded view channel. Therefore, the computation for motion estimation (ME) in most view channels can be greatly reduced. Experimental results show that compared with the full search block matching algorithm (FSBMA) applied to both ME and DE, the proposed algorithm saves 98.4-99.1% computational complexity of ME in most view channels with negligible quality loss of only 0.03-0.06 dB in PSNR.

2160p Multiview Video Encoder Chip for 3D/Quad Full HDTV Applications

With advances in video coding technology, two main streams of multimedia applications are emerging. The first involves more vivid perceptual experience and is leading to the next generation of TV specifications - Quad Full HD (QFHD, 4096×2160p) and 3D/multi-view TV. The second is the scalable broadcasting and streaming of video.

Content-Aware Prediction Algorithm With Inter-View Mode Decision for Multiview Video Coding

To provide more vivid perception, more and more advanced features, like the 4k×2k resolution and the multiview functionality, are emerging for TV. For a multiview video coding (MVC) encoder, motion and disparity estimation (ME/DE) take at least half the hardware requirement. To solve these challenges, a cache-based integer ME/DE algorithm is proposed. With a cache memory as the search window buffer, a predictor-centered ME/DE algorithm is presented. The search range can be reduced to ±16 pixels with less than 0.1dB quality drop compared with full search algorithm. Based on this algorithm, an integer ME/DE chip design is realized. It can reduce 82% on-chip SRAM and 39% system bandwidth. Moreover, the search candidate requirement is also reduced by 79%. As the result, an ME/DE chip design for 4k×2k quad-HD H.264 and HDTV MVC is implemented.

A 59.5mW scalable/multi-view video decoder chip for Quad/3D Full HDTV and video streaming applications

External memory bandwidth is an important issue in system-on-chip (SoC) systems. Especially in high definition (HD) video coding, the bandwidth requirement of off-chip memory is critical in video processing. In recent researches, embedded compression shows high potential on off-chip memory bandwidth reduction. Works about embedded compression have been done for low power applications. However, there is no suitable efficient embedded compression with good rate-distortion performance for high throughput applications. In this paper, an algorithm and hardware architecture of high performance lossy embedded compression is proposed to ease the bus congestion problem while keeping the latency low. Using the proposed algorithm, not only the high throughput requirement of HD video encoder is met, but also the hardware cost is relatively low. From our simulation, about 70% memory bandwidth is reduced with only 0.1 dB PSNR degradation in 1080p HD video.

Cache-based integer motion/disparity estimation for quad-HD H.264/AVC and HD multiview video coding

3-D video will become one of the most important video technologies in the next generation of television. Due to ultra high data bandwidth requirement for 3-D video, effective compression technology becomes an essential part in the infrastructure. Thus stereo and multiview video coding (MVC) plays a critical role. However, MVC systems require much more computational complexity relative to mono-view video coding systems. Therefore, an efficient prediction scheme is necessary for encoding. In this paper, a new fast motion estimation (ME) algorithm is proposed. By utilizing disparity estimation (DE) to find corresponding blocks between different views, the coding information such as motion vectors can be effectively shared and reused from the coded view channel. Therefore, the computation for ME in most view channels can be greatly reduced. Experimental results show that compared with the full search block matching algorithm applied to both ME and DE, the proposed algorithm saves 95% computation with near-FSBMA quality.

Architecture design of high performance embedded compression for high definition video coding

Multiview video can provide users a 3D and virtual reality perception by its multiple viewing angles. To improve the quality of virtual view synthesized frame and remove the disocclusion region, hole filling technique is required. By classifying the different image artifact and applying proper hybrid motion/depth-oriented inpainting algorithm, the image quality will be closer to the real image. In addition, the simulation result and comparison with previous works on inpainting show that both objective evaluation estimates and subjective perceptual vision come to a better result by the system proposed in this paper.

Fast motion estimation with inter-view motion vector prediction for stereo and multiview video coding

Fractional motion estimation (FME) is widely used in video compression standards. In H.264/AVC, the precision of motion vector is down to quarter pixels to improve the coding efficiency. However, FME occupies over 45% of the computation complexity in an H.264 encoder and this high complexity limits the processing capability. In this paper, a single-iteration full search FME is proposed. By the algorithm and architecture co-optimization, the bandwidth to the frame buffer is reduced by 31%. Furthermore, 82% of circuit area for the Hadamard transformation and subtraction are saved from the direct implementation. Compared with prior arts, the proposed design supports 3.39 × higher throughput with only 0.02 dB PSNR drop. Thus, the specification of 4096 × 2160 quad full high definition H.264/AVC FME processing can be achieved.

Hybrid motion/depth-oriented inpainting for virtual view synthesis in multiview applications

Multiview video can bring the 3D and virtual reality perceptual experience to users by its multiple view point characteristic. To support the smooth and free view point switching, both matrix-based depth image based rendering and the complex virtual view interpolation schemes are required. In order to reduce the high computation complexity and avoid the iterative processing scheduling in conventional view interpolation flows, a single iteration view interpolation algorithm is proposed. By use of the proposed algorithm, the redundant warping operations are reduced by 86%. In addition, based on the proposed artifact detecting and removing algorithm, artifacts due to imperfect depth maps can be detected and eliminated at the same time. Therefore, no additional post-processing or iteration is required and the single iteration processing is achieved.

Single-iteration full-search fractional motion estimation for quad full HD H.264/AVC encoding

Advances in semiconductors and developments in machine learning [1] have led to versatile multimedia applications with semantic processing abilities. Real-time applications, such as face detection, facial-expression recognition, scene analysis [2] and object recognition [3], have become indispensable functionality for Consumer Electronic (CE) products. To deal with complicated video-processing algorithms for multimedia content analysis, many powerful processors have been reported [2–5]. Although these processors can speed up video-processing tasks with massively parallel processing elements, they only focus on the feature-extraction parts, and there is no specialized hardware to support different kinds of advanced machine-learning algorithms, which require extensive computations. In this paper, a Semantic Analysis SoC (SASoC) that accelerates video processing and machine learning simultaneously, is developed to meet the demands of the near future.