Multi-Scale Spatially Weighted Local Histograms in O(1)
Mahdieh Poostchi, Ali Shafiekhani, Kannappan Palaniappan, Guna Seetharaman
MULTI-SCALE SPATIALLY WEIGHTED LOCAL HISTOGRAMS IN O(1)
Mahdieh Poostchi , Ali Shafiekhani , Kannappan Palaniappan , Guna Seetharaman Electrical Engineering and Computer Science Department University of Missouri-Columbia, MO, USA US Naval Research Laboratory, Washington D.C., USA
ABSTRACT
Weighting pixel contribution considering its location is a key feature in many fundamental image processing tasks including filtering, object modeling and distance matching. Several techniques have been proposed that incorporate Spa- tial information to increase the accuracy and boost the per- formance of detection, tracking and recognition systems at the cost of speed. But, it is still not clear how to efficiently ex- tract weighted local histograms in constant time using integral histogram. This paper presents a novel algorithm to compute accurately multi-scale Spatially weighted local histograms in constant time using Weighted Integral Histogram (SWIH) for fast search. We applied our spatially weighted integral his- togram approach for fast tracking and obtained more accurate and robust target localization result in comparison with using plain histogram.
Index Terms — integral histogram, local histogram, spa- tial weights, fast matching, tracking INTRODUCTION
In many image processing applications, histograms are com- monly used to characterize and analyze the region of inter- est within the image. Histogram-based features are space ef- ficient, simple to compute, robust to translation and particu- larly invariant to orientation for color-based features. How- ever, when computing a plain histogram, spatial information are missed which makes it sensitive to noise and occlusion. Several techniques are proposed to preserve spatial informa- tion including color Correlograms [1], Spatiogram [2], Mul- tiresolution histogram [3], locality sensitive histogram [4, 5] and fragment-based approaches that exploit the spatial rela- tionships between patches [6]. Spatially weighted histograms boost the performance of many image processing tasks at the expense of speed. In [7], Porikli generalized the concept of in- tegral image and presented computationally very fast method to extract the plain histogram of any arbitrary region in con- stant time. Integral histogram provides an optimum and com- plete solution for the histogram-based search problem. Since then many novel approaches have been presented based on in- tegral histogram to accelerate the performance of image pro- cessing tasks and incorporate the spatial information includ- (a) Manhattan (b) Euclidean (c) Gaussian
Fig. 1 . Illustration of linear and non-linear distance kernels. ing filtering [8, 9, 10, 11], classification and recognition [3, 12], and detection and tracking [13, 14]. Despite all different techniques that have been proposed to adaptively weight the contribution of pixels when comput- ing local histograms by considering their distance from cen- ter pixel, the problem of how accurately extract the spatially weighted histogram of any arbitrary region within an image in constant time using integral histogram is still unsolved. Frag- track [6] proposes a discrete approximation scheme instead of the continuous kernel weighting approach to give higher weight to the contribution of inner rectangle compare to re- gion margins for fast search. In this paper we present a novel fast algorithm to accu- rately evaluate spatially weighted local histograms in O(1) time complexity using an extension of the integral histogram method (SWIH). The main idea is to (1) split local histogram kernel into multiple quadrants q i and decompose the spa- tial filter into independent weights w i subsequently, (2) for all w i compute candidate region weighted integral histogram IH w i , (3) then for every quadrant q i compute its weighted local histogram using the corresponding IH w i and consider- ing its translation from center pixel, (4) normalize the local histograms and (5) finally add them together to build the full kernel spatially weighted local histogram (Figure 2). In the following section, we discuss the details of our new approach to extract the spatially weighted local histograms in constant time using integral histogram particularly for fast histogram-based similarity matching. Then, we evaluate the performance of SWIH and compare it with the brute-force implementation and approximation scheme in terms of com- putational complexity and accuracy. SPATIALLY WEIGHTED LOCAL HISTOGRAMS
Weighting pixel contributions is a key feature in many funda- mental image processing tasks including filtering, modeling w h
ROI h High weighted by b W SE W SW W NE W NW counts h R R R R P P P Low w Accurate Spatially Weighted (a) (b)
Local Histogram
Fig. 2 . Computational flow of accurate spatially weighted local histograms computation using weighted integral his- togram for the region of interest (ROI) of size w × h . and matching to increase the accuracy of results in detection, tracking, recognition, etc.. The main idea is to assign lower weights to the pixels that most likely belong to background or occluding objects. One common technique is to define a weighting function that assigns weights to pixels with respect to their distance from target center (since undesirable pixels are usually con- sidered around the region contours) including Manhattan, Euclidean, Gaussian or exponential weighting distance func- tions (Figure 1). Having such kernels enables us to adaptively weight the contribution of pixels and diminish the presence of background information when computing weighted local histograms. Figure 3 shows the accuracy of intensity fea- ture likelihood maps based on sliding window histogram matching when using plain local histograms versus spatially weighted local histograms. As it can be seen, using spatially weighted local histograms generates more robust matching results. In the following sections, we describe the straightfor- ward convolution-based approach, the discrete approximation scheme and our proposed novel, fast and accurate algorithm based on weighted integral histogram to compute spatially weighted local histograms for fast search. Brute-force Approach : the computational complexity of the brute-force approach to compute the adaptively weighted local histograms at each candidate pixel location is linear in the kernel size and the number of candidate pixels. Assuming a search window of size w × h and a neighborhood of size k × k and b -dimensional histogram, the computational com- Target Template (a)
Region of Interest (b) Plain Histogram (c) Weighted His- togram
Fig. 3 . Performance evaluation of intensity feature likeli- hood maps using sliding window (b) plain versus (c) spatially weighted histogram distance matching.
Fig. 4 . (a) Wedding-Cake Approach: the discrete approx- imation scheme to obtain the spatially weighted local his- togram for the candidate region considering inner-nested win- dows and using integral histogram ( w i < w i − < ... < w i − k ). (b) Tiling the kernel into four quadrants and decom- posing the weights to accurately compute spatially weighted local histogram. plexity of finding the best matched pixel location is O ( b × k × w × h ) , which makes the system far away from real-time performance particularly when it comes to large scale high resolution image analysis. Wedding-Cake Approach : One solution to meet the de- mands of real-time implementation is to extract local his- tograms in constant time using integral histogram. However, as of our knowledge, there is still no solution to accurately and efficiently extract spatially weighted local histograms in O(1) using integral histogram but the discrete approxi- mate scheme presented in [6]. Frag-track proposed a simple approach to approximate the kernel function with different weights instead of pixel-level kernel weighting. Assuming that we want to calculate a spatially weighted local histogram in the rectangular region R centered at point P using integral histogram. Such counting can be approximated by consid- ering several inner-nested windows R i at multiple scales around P (Figure 4). The goal is to compute the counts of the rings between two adjacent windows R i and R i − by sub- tracting their local histograms that are obtained in constant time using integral histogram. Then, rings histograms will be weighted appropriately with respect to their distance from P and combined to provide an approximate spatially weighted local histogram on R . The accuracy of this approximation relies on the number of considered inner-nested windows. In this paper, we present a new approach to compute spa- tially weighted local histograms that is more accurate than the wedding-cake method and takes constant time using integral histograms. Multi-scale Spatially Weighted Local Histograms in Constant Time Using Integral Histogram (SWIH)
When using integral histogram, it is not clear how to weight pixel contributions when computing arbitrary rectangular re- gion histogram in O(1). We propose to address the pixel-level weighting problem by tiling the kernel into multiple quadrants as well as decomposing the weights (Figure 4(b)). In this sec- tion we describe our proposed algorithm in details when us- R i , w i R i-1 , w i-1 R i-2 , w i-2 R i-k ,w i-k P R L L
SE SW
H H H H
NE NW
L L P un T i m e ( s e cs ) (a) ROI (b) Brute Force (c) Plain Histogram (d) Wedding Cake (e) SWIH
Fig. 5 . Performance evaluation of intensity likelihood maps estimation using sliding window histogram matching. Weighting pixel contribution considering its location results in more accurate and robust target localization as shown in (b) and (e). ing Manhattan distance function to adaptively weight pixel contributions (Figure 1(a)) for fast matching. Assuming that we want to weight the contribution of each pixel within re- gion R centered at P c = ( x c , y c ) by its Manhattan distance from P when computing histogram of region R . Manhattan or city-block weighting function measures the sum of the ab- solute distance between two points along each axis. In our
0 Brute Froce Wedding Cake SWIH (a)
Image Size (640x480)
10 51x51 8 151x151 301x301 6 351x501
0 Brute Froce Wedding Cake SWIH (b)
Image Size (1k x 1k) case, Manhattan distance of any arbitrary point P i = ( x i , y i ) withing region R is Dist
Manhattan ( P i , P c ) = | x i − x c | + | y i − y c | (1) Since the filter is rectilinear and symmetric, we propose to de- compose it into four independent weighting functions in the shape of four quadrants: TopLeft(TL), TopRight(TR), Bot- tomLeft(BL) and BottomRight(BR) (Figure 4(b)). As it is shown, weights linearly increase from one corner to its diag- onally opposite corner in each of the quadrants covering four directions: { SE, SW, NE, NW } . We extend these weights for the region of interest and compute four differently weighted integral histogram. For each direction, we consider two cor- related images f and w dir to compute the weighted integral histogram up to point ( x, y ) : Fig. 6 . Performance comparison by increasing the local his- togram kernel size. Computational time of integral histogram- based methods are invariant of kernel size. PERFORMANCE EVALUATION AND EXPERIMENTAL RESULTS
In this section, we evaluate the performance of our approach and compare it with brute-force implementation and approx- imation scheme with respect to computational complexity and accuracy. Figure 5 evaluates the performance of the es- timated intensity likelihood maps for a sample image from the VOT2016 data set [16] using sliding-window histogram matching. We compared the intensity likelihood map com- puted by the brute-force implementation with the matching results of the plain histogram, approximation scheme and our IH w dir ( x, y, b i ) = i ≤ x,j ≤ y δ ( Q ( f ( i, j )) − b i ) w dir ( i, j ) proposed accurate fast spatially weighted histogram. Back- f contains image feature values, Q is the quantization (2) func- ground clutter is one of the main challenges in object detec- tion systems relied on matching. We have selected an image tion that determines which bin to increase, δ is the impulse function and w dir is the pixel-wise weighing function that determine the value to increase at that bin. Having four dif- ferently weighted integral histogram, each of the quadrants spatially weighted local histogram will be computed in O(1) using its corresponding weighted integral histogram and con- sidering its translation from the kernel center point. We will normalize the histograms and add them together to build the full region spatially weighted histogram. Figure 2 illustrates the flow of the computation. It is noteworthy to mention that due to weights rectilinear changes, their values are indepen- dent of the pixel location in the region of interests. This char- acteristic enables us to appropriately normalize the computed weighted local histogram and match it with the target spa- tially weighted histogram regardless of its location. This new method provides multi-scale accurate spatially weighted local histogram in constant time and can be utilized for other spa- tial weighting functions. It can be easily adapted to any fast computation of integral histogram on GPUs to accelerate the computation [15]. that contains background clutter to make the matching pro- cess very challenging. It is shown that our proposed method not only provides exact results as the brute-force approach but is much faster. SWIH is 4.5 times faster than brute-force implementation for a candidate region of size × and sliding window of size × ( 5(a)). Figure 6 (a) and (b) shows the computational complexity of each of the discussed methods for standard image × as well as large scale image of size k × k for different sliding window size from small scale to very large scale. It can be seen that the local histograms computational time of the brute-force implemen- tation increases dramatically by enlarging the kernel size but is invariant of kernel size for the approximation scheme and SWIH. SPATIALLY WEIGHTED INTEGRAL HISTOGRAM FOR FAST TRACKING (SWIFT)
Many of the generative region-based tracking algorithms rely on histograms for a fast and memory efficient appearance R un T i m e ( l og , s e cs ) (a) Likelihood map estimation using sliding window plain histograms matching (b)
Likelihood map estimation using sliding window spatially weighted histograms matching
Fig. 7 . Performance evaluation of intensity likelihood map estimation using sliding window (a) plain histogram versus (b) our proposed accurate spatially weighted histograms matching approach for vehicle tracking in large scale aerial imagery. modeling of target object as well as candidate region includ- ing Mean-Shift [17] and kernel-based [18, 19] approaches. However, since many of these trackers discard the spatial in- formation when computing the conventional histogram, they rapidly lose the accuracy and converge to false targets. There- fore, many techniques have been presented to incorporate spa- tial information and enhance the tracker robustness which are either more computationally intensive and far away from real- time performance or a combination of different techniques to compensate the lack of spatial information [2, 20, 4, 13, 21]. Our proposed approach that retain spatial information as well as chromatic, shape or texture features presents a novel so- lution based on integral histogram that builds a robust and efficient histogram-based appearance model and provides ac- curate and fast histogram-based matching for visual tracking, detection and recognition purposes.
We integrated our proposed technique into our tracking system named
LoFT [22, 23] to evaluate the performance of intensity histogram matching when using plain histogram versus spatially weighted histogram.
LoFT is an appearance- based Likelihood of Features Tracking (LoFT) system, spe- cialized for low resolution targets with large displacements caused by low frame rate sampling in Wide Area Motion Im- agery(WAMI). Matching likelihood maps for individual fea- tures are computed using sliding window histogram similarity operators. The integral histogram method is used to acceler- ate computation of the sliding window histograms for a pos- teriori likelihood estimation [15]. Figure 7(a) describes the flow of the likelihood map computation using plain intensity histogram of target object and candidate regions. Similar to the results obtained for the synthetic image shown in Fig- ure 3, using plain histograms results in less accurate local- ization of object. Hence, we applied our accurate spatially weighted integral histogram to estimate features likelihood maps instead of regular integral histogram for fast search. Fig- ure 7(b) illustrates the computational flow compared to plain histogram and presents the more accurate target localization results when using spatially weighted histograms. CONCLUSION
This paper present a novel fast algorithm to accurately evalu- ate spatially weighted local histograms in constant time using an extension of the integral histogram method (SWIH). We have shown that SWIH produces exact local histograms com- pared to brute-force approach and is much faster. Utilizing the integral histogram makes it to be fast, multi-scale and flexi- ble to different weighting functions. This technique can be applied to fragment-based approaches to adaptively weight object patches considering their location. SWIH can be in- tegrated into any detection or tracking system to provide an efficient exhaustive search and achieve more robust and accu- rate target localization. ACKNOWLEDGMENT
This research was partially supported by the U.S. Air Force Research Laboratory under agreement AFRL FA875014-2- 0072. The views and conclusions contained in this document are those of the authors and should not be interpreted as rep- resenting the official policies, either expressed or implied, of AFRL, NRL, or the U.S. Government.
Region of Interest and Histogram Weights Compute Weighted Integral Histogram Compute Likelihood Compute Spatially Weighted Map Using Spatially Target Histogram Weighted Histogram Matching - Spatially Weighted Local Histogram Region of Interest Compute Plain Integral Histogram Compute Likelihood Map Using Histogram Matching
Compute Plain Target Histogram - Plain Intensity Local Histogram . REFERENCES [1] Jing Huang, S Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, and Ramin Zabih, “Image indexing using color correlograms,” in
IEEE CVPR , 1997, pp. 762–768. [2] Stanley T Birchfield and Sriram Rangarajan, “Spa- tiograms versus histograms for region-based tracking,” in
IEEE CVPR , 2005, vol. 2, pp. 1158–1163. [3] Efstathios Hadjidemetriou, Michael D Grossberg, and Shree K Nayar, “Multiresolution histograms and their use for recognition,”
IEEE PAMI , vol. 26, no. 7, pp. 831–847, 2004. [4] Shengfeng He, Qingxiong Yang, Rynson W.H. Lau, Jiang Wang, and Ming-Hsuan Yang, “Visual tracking via locality sensitive histograms,” in
IEEE CVPR , 2013, pp. 2427–2434. [5] Shengfeng He, Qingxiong Yang, Rynson WH Lau, and Ming-Hsuan Yang, “Fast weighted histograms for bi- lateral filtering and nearest neighbor searching,”
IEEE Transactions on Circuits and Systems for Video Technol- ogy , vol. 26, no. 5, pp. 891–902, 2016. [6] Amit Adam, Ehud Rivlin, and Ilan Shimshoni, “Robust fragments-based tracking using the integral histogram,” in
IEEE CVPR , 2006, vol. 1, pp. 798–805. [7] Fatih Porikli, “Integral histogram: A fast way to extract histograms in cartesian spaces,” in
IEEE CVPR , 2005, vol. 1, pp. 829–836. [8] Mohamed Hussein, Fatih Porikli, and Larry Davis, “Kernel integral images: A framework for fast non- uniform filtering,” in
IEEE CVPR , 2008, pp. 1–8. [9] Fatih Porikli, “Constant time o (1) bilateral filtering,” in
IEEE CVPR , 2008, pp. 1–8. [10] Ke Zhang, Gauthier Lafruit, Rudy Lauwereins, and Luc Van Gool, “Joint integral histograms and its application in stereo matching,” in
IEEE ICIP , 2010, pp. 817–820. [11] Yonghuai Liu, Luigi De Dominicis, Ralph R Martin, and Baogang Wei, “Illumination normalization using weighted gradient integral images,”
Lecture Notes on Software Engineering , vol. 1, no. 4, pp. 329, 2013. [12] Brian Fulkerson, Andrea Vedaldi, and Stefano Soatto, “Localizing objects with smart dictionaries,” in
Euro- pean Conference on Computer Vision , 2008, pp. 179– 192. [13] Qi Zhao and Hai Tao, “Object tracking using color cor- relogram,” in
IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance , 2005, pp. 263–270. [14] Piotr Dolla´r, Zhuowen Tu, Pietro Perona, and Serge Be- longie, “Integral channel features,”
BMVC Press , 2009. [15] Mahdieh Poostchi, Kannappan Palaniappan, Filiz Bun- yak, Michela Becchi, and Guna Seetharaman, “Effi- cient GPU implementation of the integral histogram,” in
Asian Conference on Computer Vision . Springer, 2012, pp. 266–278. [16]
VOT 2016 Benchmark Data , . [17] Dejun Wang, Weiping Sun, Shengsheng Yu, Lin Li, and Wei Liu, “A novel background-weighted histogram scheme based on foreground saliency for mean-shift tracking,” Multimedia Tools and Applications , pp. 1– 19, 2015. [18] Jaideep Jeyakar, R Venkatesh Babu, and KR Ramakrish- nan, “Robust object tracking with background-weighted local kernels,”
Computer Vision and Image Understand- ing , vol. 112, no. 3, pp. 296–309, 2008. [19] Dorin Comaniciu, Visvanathan Ramesh, and Peter Meer, “Kernel-based object tracking,”
IEEE PAMI , vol. 25, no. 5, pp. 564–577, 2003. [20] Wangsheng Yu, Xiaohua Tian, Zhiqiang Hou, Anqi Huang, and Xiang Liu, “Region edge histogram: A new feature for region-based tracking,” in
IEEE 12th Inter- national Conference on Signal Processing (ICSP) , 2014, pp. 1180–1185. [21] Jwu-Sheng Hu, Chung-Wei Juan, and Jyun-Ji Wang, “A spatial-color mean-shift object tracking algorithm with scale and orientation estimation,”
Pattern Recognition Letters , vol. 29, no. 16, pp. 2165–2173, 2008. [22] R. Pelapur, S. Candemir, F. Bunyak, M. Poostchi, G. Seetharaman, and K. Palaniappan, “Persistent target tracking using likelihood fusion in wide-area and full motion video sequences,” in , 2012, pp. 2420–2427. [23] Mahdieh Poostchi, Filiz Bunyak, Kannappan Palaniap- pan, and Guna Seetharaman, “Feature selection for appearance-based vehicle tracking in geospatial video,” in