Wen-Jiin Tsai

Physics-based ball tracking and 3D trajectory reconstruction with applications to shooting location estimation in basketball video

The demand for computer-assisted game study in sports is growing dramatically. This paper presents a practical video analysis system to facilitate semantic content understanding. A physics-based algorithm is designed for ball tracking and 3D trajectory reconstruction in basketball videos and shooting location statistics can be obtained. The 2D-to-3D inference is intrinsically a challenging problem due to the loss of 3D information in projection to 2D frames. One significant contribution of the proposed system lies in the integrated scheme incorporating domain knowledge and physical characteristics of ball motion into object tracking to overcome the problem of 2D-to-3D inference. With the 2D trajectory extracted and the camera parameters calibrated, physical characteristics of ball motion are involved to reconstruct the 3D trajectories and estimate the shooting locations. Our experiments on broadcast basketball videos show promising results. We believe the proposed system will greatly assist intelligence collection and statistics analysis in basketball games.

A Trajectory-Based Ball Tracking Framework with Visual Enrichment for Broadcast Baseball Videos *

Pitching contents play the key role in the resultant victory or defeat in a baseball game. Utilizing the physical characteristic of ball motion, this paper presents a trajectory-based framework for automatic ball tracking and pitching evaluation in broadcast baseball videos. The task of ball detection and tracking in broadcast baseball videos is very challenging because in video frames, the noises may cause many ball-like objects, the ball size is small, and the ball may deform due to its high speed movement. To overcome these challenges, we first define a set of filters to prune most non-ball objects but retain the ball, even if it is deformed. In ball position prediction and trajectory extraction, we analyze the 2D distribution of ball candidates and exploit the characteristic that the ball trajectory presents in a near parabolic curve in video frames. Most of the non-qualified trajectories are pruned, which greatly improves the computational efficiency. The missed balls can also be recovered in the trajectory by applying the position prediction. The experiments of ball tracking on the testing sequences of JPB, MLB and CPBL captured from different TV channels show promising results. The ball tracking framework is able to extract the ball trajectory, superimposed on the video, and in near real-time provide visual enrichment before the next pitch coming up without specific cameras or equipments set up in the stadiums. It can also be utilized in strategy analysis and intelligence statistics for player training.

Ball tracking and 3D trajectory approximation with applications to tactics analysis from single-camera volleyball sequences

Providing computer-assisted tactics analysis in sports is a growing trend. This paper presents an automatic system for ball tracking and 3D trajectory approximation from single-camera volleyball sequences as well as demonstrates several applications to tactics analysis. Ball tracking in volleyball video has great complexity due to the high density of players on the court and the complicated overlapping of ball-player. The 2D-to-3D inference is intrinsically challenging due to the loss of 3D information in projection to 2D frames. To overcome these challenges, we propose a two-phase ball tracking algorithm in which we first detect ball candidates for each frame, and then use them to compute the ball trajectories. With the aid of camera calibration, we involve physical characteristics of ball motion to approximate the 3D ball trajectory from the 2D trajectory. The visualization of 3D trajectory and the applications to trajectory-based tactics analysis not only assist the coaches and players in game study but also make game watching a whole new experience. The experiments on international volleyball games show encouraging results. We believe that the proposed framework can be extended and applied to various kinds of sports games.

Hybrid Multiple Description Coding Based on H.264

Multiple description (MD) video coding is one of the approaches that can be used to reduce the detrimental effects caused by transmission over error-prone networks. A number of approaches have been proposed for MD coding, where each provides a different tradeoff between compression efficiency and error resilience. This paper first presents two basic MD coding methods; one segments the video in the spatial domain, while the other in the frequency domain. Then a hybrid MD coding method is proposed. The hybrid MD encoder segments the video in both the spatial and frequency domains. In the case of data loss, the hybrid MD decoder takes advantage of the residual-pixel correlations in the spatial domain, and the coefficient correlations in the frequency domain, for error concealment. As a result, better error resilience can be achieved at high compression efficiency. The advantages of the proposed hybrid MD method are demonstrated in the contexts of descriptor loss in ideal channels and in packet-loss networks.

Joint Temporal and Spatial Error Concealment for Multiple Description Video Coding

Transmission of compressed video signals over error-prone networks exposes the information to losses and errors. To reduce the effects of these losses and errors, this paper presents a joint spatial-temporal estimation method which takes advantages of data correlation in these two domains for better recovery of the lost information. The method is designed for the hybrid multiple description coding which splits video signals along spatial and temporal dimensions. In particular, the proposed method includes fixed and content-adaptive approaches for estimation method selection. The fixed approach selects the estimation method based on description loss cases, while the adaptive approach selects the method according to pixel gradients. The experimental results demonstrate that improved error resilience can be accomplished by the proposed estimation method.

Scene Change Aware Intra-Frame Rate Control for H.264/AVC

Most of rate-control research focuses on inter-coded frames, instead of intra-coded frames which are more possible to cause the problem of buffer overflow. This letter presents a rate control algorithm for intra-frame coding. We propose a Taylor series-based rate-QS model and a scene-change aware rate-QS model to determine quantization parameters for general intra frames and scene-change frames, respectively. Simulation results show that compared to competed approaches, the proposed method achieves better and stable quality with low buffer fullness.

Pitch-by-Pitch Extraction from Single View Baseball Video Sequences

This paper presents a novel method for reducing a baseball video segment from one batter to next batter into a more compact pitch-by-pitch video by pitching ball trajectory detection. The pitch-by-pitch video shows the complete pitching and batting process and largely reduces the source video data, making pitching analysis of broadcast baseball sequences an easier task. The proposed method has been tested for several long sequences, and promising results are reported.

A new unequal error protection scheme based on FMO

This paper presents a novel scheme for video transmission over error-prone networks. The proposed scheme exploits the error resilient features of H.264/AVC and employs an unequal error protection (UEP) approach to protect effectively the streams. A novel algorithm is proposed to classify macroblocks into slice groups using the explicit mode of Flexible Macroblock Ordering (FMO). A rate distortion optimized algorithm is used to assign unequal amounts of FEC protection to the resulting slice groups according to their importance. In addition, a Converged Motion Estimation (CME) is proposed to further improve the proposed UEP scheme. The idea behind the CME is to make the macroblocks be referenced in a skewed manner, such that highly important macroblocks are converged on only a few and the use of redundancy for error protection is efficient. Simulation results show the superiority of the proposed method over other approaches for transmission of H.264/AVC streams.

Screen-strategy analysis in broadcast basketball video using player tracking

In basketball games, screen is a blocking move performed by an offensive player, who stands beside or behind a defender, in order to free a teammate to shoot, to receive a pass, or to drive in to score. Screen is the fundamental essence that most offensive tactics are executed with. In this paper, a screen-strategy analysis system is designed, and through combining the identified screens, what tactics are executed in basketball games can be speculated. The proposed system is capable of court region detection, camera calibration and player extraction. Player trajectories are computed by a Kalman filter-based tracking method and mapped to the real-world court coordinates. The player position/trajectory information greatly assists professional-oriented applications such as screen-strategy analysis and tactic inference. The experiments on broadcast basketball videos show encouraging results.

Human action recognition based on layered-HMM

We address the problem of human action understanding of the upper human body from video sequences. Time-sequential images expressing human actions are transformed to sequences of feature vectors containing the configuration of the human body. A human is modeled as a collection of body parts, linked in a kinematic structure. The relation of the joints is used to estimate the human pose. A proposed layered HMM framework decomposes the human action recognition problem into two layers. The first layer models the actions of two arms individually from low-level features. The second layer models the interrelationship of two arms as an action. Experiments with a set of six types of human actions demonstrate the effectiveness of our proposed scheme, and the comparisons with other HMM systems show the robustness.