FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors
Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, Yao Wang
11 FastDeRain: A Novel Video Rain Streak RemovalMethod Using Directional Gradient Priors
Tai-Xiang Jiang, Ting-Zhu Huang ∗ , Xi-Le Zhao ∗ , Liang-Jian Deng and Yao Wang Abstract —Rain streaks removal is an important issue in out-door vision systems and has recently been investigated extensively.In this paper, we propose a novel video rain streak removalapproach FastDeRain, which fully considers the discriminativecharacteristics of rain streaks and the clean video in the gradientdomain. Specifically, on the one hand, rain streaks are sparseand smooth along the direction of the raindrops, whereas on theother hand, clean videos exhibit piecewise smoothness along therain-perpendicular direction and continuity along the temporaldirection. Theses smoothness and continuity results in the sparsedistribution in the different directional gradient domain, respec-tively. Thus, we minimize 1) the (cid:96) norm to enhance the sparsityof the underlying rain streaks, 2) two (cid:96) norm of unidirectionalTotal Variation (TV) regularizers to guarantee the anisotropicspatial smoothness, and 3) an (cid:96) norm of the time-directionaldifference operator to characterize the temporal continuity. Asplit augmented Lagrangian shrinkage algorithm (SALSA) basedalgorithm is designed to solve the proposed minimization model.Experiments conducted on synthetic and real data demonstratethe effectiveness and efficiency of the proposed method. Accord-ing to comprehensive quantitative performance measures, ourapproach outperforms other state-of-the-art methods, especiallyon account of the running time. The code of FastDeRain can bedownloaded at https://github.com/TaiXiangJiang/FastDeRain. Index Terms —video rain streak removal, unidirectional to-tal variation, split augmented Lagrangian shrinkage algorithm(SALSA) .
I. I
NTRODUCTION O UTDOOR vision systems are frequently affected by badweather conditions, one of which is the rain. Raindropsusually introduce bright streaks into the acquired imagesor videos, because of their scattering of light into comple-mentary metal–oxide–semiconductor cameras and their highvelocities. Moreover, rain streaks also interfere with nearbypixels because of their specular highlights, scattering, andblurring effects [1]. This undesirable interference will degradethe performance of various computer vision algorithms [2],such as event detection [3], object detection [4], tracking [5],recognition [6], and scene analysis [7]. Therefore, the removalof rain streaks is an essential task, which has recently receivedconsiderable attention.Numerous methods have been proposed to improve the vis-ibility of images/videos captured with rain streak interference ∗ Corresponding authors. Tel.: +86 28 61831016.T.-X Jiang, T.-Z. Huang, X.-L. Zhao and L.-J. Deng are with the School ofMathematical Sciences, University of Electronic Science and Technology ofChina, Chengdu, Sichuan 611731, P. R. China. Y. Wang is with the Schoolof Mathematics and Statistics, Xian Jiaotong University, Xian 710049, P.R. China. E-mails: { taixiangjiang, yao.s.wang } @gmail.com, { tingzhuhuang,liangjian1987112 } @126.com, [email protected]. Fig. 1. A frame of a rainy video (left), the rain streaks removal result by theproposed method FastDeRain (middle) and the extracted rain streaks (right).The pixel values of the rain streaks are scaled for better visualization. [8–42]. They can be classified into two categories: multiple-images/videos based techniques and single-image based ap-proaches. Fig. 1 exhibits an example of video rain streaksremoval. Without loss of generality, in this paper, we use“background” to denote the rain-free content of the data.For the single-image de-raining task, Kang et al . [8] de-composed a rainy image into low-frequency (LF) and high-frequency (HF) components using a bilateral filter and thenperformed morphological component analysis (MCA)-baseddictionary learning and sparse coding to separate the rainstreaks in the HF component. To alleviate the loss of thedetails when learning HF image bases, Sun et al . [9] tactfullyexploited the structural similarity of the derived HF imagebases. Chen et al . [10] considered the similar and repeatedpatterns of the rain streaks and the smoothness of the back-ground. Sparse coding and dictionary learning were adopted in[12–14]. In their results, the details of backgrounds were wellpreserved. The recent work by Li et al . [15] was the first toutilize Gaussian mixture model (GMM) patch priors for rainstreak removal, with the ability to account for rain streaksof different orientations and scales. Zhu et al . [16] proposeda joint bi-layer optimization method progressively separaterain streaks from background details, in which the gradientstatistics are analyzed. Meanwhile, the directional property ofrain streaks received a lot of attention in [19–21] and thesemethods achieved promising performances. Ren et al. [23]removed the rain streaks from the image recovery perspective.Wang et al. [22] took advantage the image decompositionand dictionary learning. The recently developed deep learningtechnique was also applied to the single image rain streaksremoval task, and excellent results were obtained [24–31].For the video rain streaks removal, Garg et al . [32] firstlyraised a video rain streaks removal method with comprehen-sive analysis of the visual effects of the rain on an imagingsystem. Since then, many approaches have been proposed forthe video rain streaks task and obtained good rain removingperformance in videos with different rain circumstances. Com- a r X i v : . [ c s . C V ] J u l prehensive early existing video-based methods are summarizedin [33]. Chen et al. [11] took account of the highly dynamicscenes. Whereafter, Kim et al . [34] considered the temporalcorrelation of rain streaks and the low-rank nature of cleanvideos. Santhaseelan et al. [35] detected and removed the rainstreaks based on phase congruency features. You et al. [36]dealt with the situations where the raindrops are adhered to thewindscreen or the window glass. In [37], a novel tensor-basedvideo rain streak removal approach was proposed consideringthe directional property. Ren et al. [38] handled the videodesnowing and deraining task based on matrix decomposition.The rain streaks and the clean background were stochasticallymodeled as a mixture of Gaussians by Wei et al. [39] while Li et al. [40] utilized the multiscale convolutional sparse coding.For the video rain streaks removal, the deep learning basedmethods also started to reveal their effectiveness [41, 42].In general, the observation model for a rainy image isformulated as O = B + R [1], which can be generalized to thevideo case as: O = B + R , where O , B , and R ∈ R m × n × t arethree 3-mode tensors representing the observed rainy video,the unknown rain-free video and the rain streaks, respectively.When considering the noise or error, the observation model ismodified as O = B + R + N , where N is the noise or errorterm. The goal of video rain streak removal is to distinguishthe clean video B and the rain streaks R from an input rainyvideo O . This is an ill-posed inverse problem, which canbe handled by imposing prior information. Therefore, fromthis point of view, the most significant issues are the rationalextraction and sufficient utilization of the prior knowledge,which is helpful to wipe off the rain streaks and reconstructthe rain-free video. In this paper, we mainly focus on thediscriminative characteristics of rain streaks and backgroundin different directional gradient domains.From the temporal perspective, the clean video is continuousalong the time direction, while the rain streaks do not sharethis property [34, 39, 43]. As observed in Fig. 2, the time-directional gradient of the rain-free video (a-2) exhibits adifferent histogram compared with those of the rainy video(a-1) and the rain streaks (a-3). The temporal gradient of theclean video is much sparser and it is corresponding to thetemporal continuity of the clean video. Therefore, we intendto minimize (cid:107)∇ t B(cid:107) , where ∇ t is the temporal differentialoperator.From the spatial perspective, it has been widely recognizedthat natural images are largely piecewise smooth and their gra-dient fields are typically sparse [44, 45]. Many aforementionedde-rain methods take the spatial gradient into considerationand use the total variation (TV) to depict the property of therain-free part [1, 10]. However, the effects of the rain streakson the vertical gradient and horizontal gradient are different.This phenomenon was likewise noticed in [19–21]. Initially,for the sake of convenience, we assume that rain streaks areapproximately vertical. The impact of the vertical rain streakson the vertical gradient is limited. The subfigures (b-1,2,3)in Fig. 2 reveal that the vertical gradient of rain streaks aremuch sparser than those of the clean video and the rainyvideo. Nonetheless, the vertical rain streaks severely disruptthe horizontal piecewise smoothness. As exhibited in Fig. 2 ( a-1 ) ( a-2 )( a-3 ) ( b-1 )( b-2 ) ( b-3 ) ( c-1 ) ( c-2 ) ( c-3 ) r a i n s t r ea k d i r ec ti on ( y - d i r ec ti on ) horizontal direction (x-direction) rainy video clean video rain streaks Fig. 2. From left to right: the histograms of temporal gradient of therainy video (a-1), the clean video (a-2) and the isolated rain streaks (a-3),respectively; several example frames from the rainy video, the clean videoand the isolated rain streaks; and the histograms of the vertical gradient (b-1,2,3) and the intensities along a row (c-1,2,3) in the rainy video, the cleanvideo and the isolated rain streaks, respectively. (c-1,2,3), the pixel intensity is piecewise smooth only in (c-2),whereas burrs frequently appear in (c-1) and (c-3). Therefore,we intend to minimize (cid:107)∇ R(cid:107) and (cid:107)∇ B(cid:107) , where ∇ and ∇ are respectively the vertical difference (or say verticalunidirectional TV [46–48]) operator and horizontal difference(or say horizontal unidirectional TV) operator.Given a real rainfall-affected scene, without the wind, theraindrops generally fall from top to bottom. Meanwhile, whennot very windy, the angles between rain streaks and the verticaldirection are usually not very large. Therefore, the rain streakdirection can be approximated as the vertical direction, i.e. themode-1 (column) direction of the video tensor. Actually, thisassumption is reasonable for parts of the rainy sceneries. Forthe rain streaks that are oblique (or say far from being vertical),directly utilizing the directional property is very difficult forthe digital video data, which are cubes of distinct numbers. Tocope with this difficulty, in Sec. III-E, we would design theshift strategy, based on our automatical rain streaks’ directiondetection method.The contributions of this paper include three aspects. • We propose a video rain streaks removal model, whichfully considers the discriminative prior knowledge of therain streaks and the clean video. • We design a split augmented Lagrangian shrinkage al- gorithm (SALSA) based algorithm to efficiently andeffectively solve the proposed minimization model. Theconvergence of our algorithm is theoretically guaranteed.Meanwhile, the implementation on the graphics process-ing unit (GPU) device further accelerates our method. • To demonstrate the efficacy and the superior performanceof the proposed algorithm in comparison with state-of-the-art alternatives, extensive experiments both on thesynthetic data and the real-world rainy videos are con-ducted.This work is an extension of the material published in [37].The new material is the following: a) the proposed rain streaksremoval model is improved and herein introduced in moretechnical details; b) we explicitly use the split augmentedLagrangian shrinkage algorithm to solve the proposed model;c) to make the proposed method more applicable, we designan automatical rain streaks’ direction detecting method andprovide the shift strategy to deal with oblique rain streaks;d) in our experiments, we re-simulate the rain streaks for thesynthetic data, using two different techniques and consideringthe rain streaks not very vertical; e) three recent state-of-the-artmethods [27, 39, 40] are brought into comparison.The paper organized as follows. Section II gives the prelim-inary on the tensor notations. In Section III, the formulationof our model is presented along with a SALSA solver. Exper-imental results are reported in Section IV. Finally, we drawsome conclusions in Section V.II. N
OTATION AND PRELIMINARIESTABLE IT
ENSOR NOTATIONS
Notation Explanation X , X , x , x Tensor, matrix, vector, scalar. x (: i i · · · i N ) A fiber of a tensor X , defined by fixing every indexbut one. X (:: i · · · i N ) A slice of a tensor X , defined by fixing all but twoindices. (cid:104)X , Y(cid:105)
The inner product of two same-sized tensors X and Y . (cid:107)X (cid:107) F The
Frobenius norm of a tensor X . Following [49–51], we use lower-case letters for vectors,e.g., a ; upper-case letters for matrices, e.g., A ; and calli-graphic letters for tensors, e.g., A . An N -mode tensor isdefined as X ∈ R I × I ×···× I N , and x i ,i , ··· ,i N denotes its ( i , i , · · · , i N ) -th component.A fiber of a tensor is defined by fixing every index but one.A third-order tensor has column, row, and tube fibers, denotedby x : jk , x i : k , and x ij : , respectively. When extracted from theirtensors, fibers are always assumed to be oriented as columnvectors.A slice is a two-dimensional section of a tensor, defined byfixing all but two indices. The horizontal, lateral, and frontalslides of a third-order tensor X are denoted by X i :: , X : j : ,and X :: k , respectively. Alternatively, the k -th frontal slice of a third-order tensor, X :: k , may be denoted more compactly by X k .The inner product of two same-sized tensors X and Y is defined as (cid:104)X , Y(cid:105) := (cid:80) i ,i , ··· ,i N x i i ··· i N · y i i ··· i N . Thecorresponding norm ( Frobenius norm ) is then defined as (cid:107)X (cid:107) F := (cid:112) (cid:104)X , X (cid:105) .Please refer to [52] for a more extensive overview.III. M
AIN RESULTS
A. Problem formulation
As mentioned before, a rainy video
O ∈ R m × n × t can bemodeled as a linear superposition: O = B + R + N , (1)where O , B , R and N ∈ R m × n × t are four 3-mode tensorsrepresenting the observed rainy video, the unknown rain-free video, the rain streaks and the noise (or error) term,respectively.Our goal is to decompose the rain-free video B and the rainstreaks R from an input rainy video O . To solve this ill-posedinverse problem, we need to analyze the prior information forboth B and R and then introduce corresponding regularizers,which will be discussed in the next subsection. B. Priors and regularizers
In this subsection, we continue the discussion on the priorknowledge with the assumption that rain streaks are approxi-mately vertical. a) Sparsity of rain streaks:
When the rain is light, therain streaks can naturally be considered as being sparse. Toboost the sparsity of rain streaks, minimizing the (cid:96) norm ofthe rain streaks R is an ideal option. When the rain is veryheavy, it seems that this regularization is not proper. However,when the rain is extremely heavy, it is very difficult or evenimpossible to recover the rain-free part because of the hugeloss of the reliable information. The rainy scenarios discussedin this paper are not that extreme, and we assume that therain streaks always maintain lower energy than the backgroundclean videos. Therefore, when the rain streaks are dense, the (cid:96) norm can be viewed as a role to restrain the magnitude ofthe rain streaks. Meanwhile, in our model, other regularizationterms would also contribute to distinguishing the rain streaks.Thus, we can tackle the heavy raining scenarios by tuning theparameter of the sparsity term so as to reduce its effect. b) The horizontal direction: In Fig. 2, (c-1,2,3) showthe pixel intensities along a fixed row of the rainy video, theclean video and the rain streaks, respectively. It is obviousthat the variation of the pixel intensity is piecewise smoothonly in (c-2), whereas burrs frequently appear in (c-1) and (c-3). Therefore, a horizontal unidirectional TV regularizer is asuitable candidate for B . c) The vertical direction: It can be seen from Fig. 2 that(b-3), which is the histogram of the intensity of the verticalgradient in a rain-streak frame, exhibits a distinct distributionwith respect to (c-1) and (c-2). The long-tailed distributions in(c-1) and (c-3) indicate that the minimization of the l normof ∇ R would help to distinguish the rain streaks. d) The temporal direction: From the first column of Fig.2, it can be observed that clean videos exhibit the continuityalong the time axis. Sub-figures (a-1,2,3), which present thehistograms of the magnitudes in the temporal directionalgradient, illustrate that the clean video’s temporal gradientsconsist of more zero values and smaller non-zero values,whereas those of the rainy video and rain streaks tend to belong-tailed. Therefore, it is natural to minimize the l norm ofthe temporal gradient of the clean video B . By the way, thelow-rank regularization used in [37] is discarded since that thelow-rank assumption is not reasonable for the videos capturedby dynamic cameras and the rain streaks, which always sharethe repetitive patterns, can occasionally be more low-rank thanthe background along the spatial directions. C. The proposed model
Generally, there is an angle between the vertical directionand the real falling direction of the raindrops. The rain streakspictured in Fig. 2 are not strictly vertical and there is a 5-degreeangle between the rain streaks and the y-axis. In other words,the prior knowledge discussed above are still valid when thisangle is small. Large-angle cases would be discussed in Sec.III-E). Therefore, the rain streak direction is referred to asthe vertical direction corresponding to the y-axis, whereas therain-perpendicular direction is referred to as the horizontaldirection corresponding to the x-axis. Thus, as a summary ofthe discussion of the priors and regularizers, our model canbe compactly formulated as follows: min B , R α (cid:107) ∇ R(cid:107) + α (cid:107)R(cid:107) + α (cid:107) ∇ B(cid:107) + α (cid:107) ∇ t B(cid:107) + 12 (cid:107)O − ( B + R ) (cid:107) F s.t. O (cid:62) B (cid:62) , O (cid:62) R (cid:62) , (2)where ∇ , ∇ and ∇ t are the vertical, horizontal and temporaldifferential operators, respectively. ∇ and ∇ are also writtenas ∇ y and ∇ x in [19, 37]. An efficient algorithm is proposedin the following subsection to solve (2). D. Optimization
Since the proposed model (2) is concise and convex, manystate-of-the-art solvers are available to solve it. Here, we applythe ADMM [53], which has been proved an effective strategyfor solving large scale optimization problems [54–56]. Morespecifically, we adopt SALSA [57].After introducing four auxiliary tensors the proposed model(2) is reformulated as the following equivalent constrainedproblem: min B , V i , D i α (cid:107)V (cid:107) + α (cid:107)V (cid:107) + α (cid:107)V (cid:107) + α (cid:107)V (cid:107) + (cid:107)O − ( B + R ) (cid:107) F s.t. V = ∇ ( R ) , V = R , V = ∇ ( B ) , V = ∇ t ( B ) , O (cid:62) B (cid:62) , O (cid:62) R (cid:62) (3)where V i ∈ R m × n × t ( i = 1 , , , . Then, the augmented Lagrangian function of (3) is L µ ( B , R , V i , D i ) = 12 (cid:107)O − B − R(cid:107) F + α (cid:107)V (cid:107) + α (cid:107)V (cid:107) + α (cid:107)V (cid:107) + α (cid:107)V (cid:107) + µ (cid:107) ∇ R − V − D (cid:107) F + µ (cid:107)R − V − D (cid:107) F + µ (cid:107) ∇ B − V − D (cid:107) F + µ (cid:107) ∇ t B − V − D (cid:107) F , where the D i s ( i = 1 , , , are the scaled Lagrangemultipliers and the µ is a positive scalar. a) V i sub-problems: For i = 1 , , , , the V i sub-problem can be written as a equivalent problem: V + i = arg min V i α i (cid:107)V i (cid:107) + µ (cid:107)A i − V i (cid:107) F . Such a problem has a closed-form solution, obtained throughsoft thresholding: V + i = S αiµ ( A i ) . Here, the tensor non-negative soft-thresholding operator S v ( · ) is defined as S v ( A ) = ¯ A with ¯ a i i ··· i N = (cid:40) a i i ··· i N − v, a i i ··· i N > v, , otherwise . Therefore, V i ( i = 1 , , , can respectively be updated asfollows: V ( t +1)1 = S α µ ( ∇ R − D ) , V ( t +1)2 = S α µ ( R − D ) , V ( t +1)3 = S α µ ( ∇ B − D ) , V ( t +1)4 = S α µ ( ∇ t B − D ) . (4)The time complexity of each sub-problem above is O ( mnt ) . b) B and R sub-problems: B and R sub-problems areleast-squares problems: B + = arg min O≤B≤ (cid:107)O − B − R(cid:107) F + µ (cid:107) ∇ B − V − D (cid:107) F + µ (cid:107) ∇ t B − V − D (cid:107) F , R + = arg min O≤R≤ (cid:107)O − B − R(cid:107) F + µ (cid:107) ∇ R − V − D (cid:107) F + µ (cid:107)R − V − D (cid:107) F . Then, we have B + = O − R + µ ∇ (cid:62) ( V − D ) + µ ∇ (cid:62) t ( V − D ) + µ ∇ (cid:62) ∇ + µ ∇ (cid:62) t ∇ t R + = O − B + µ ∇ (cid:62) ( V − D ) + µ ( V − D ) + µ ∇ (cid:62) ∇ + µ (5)We adopt the fast Fourier transform (FFT) for fast calculationwhen updating B and R . Meanwhile, the elements in B ( t +1) and R ( t +1) that are smaller than 0 or larger than the corre-sponding elements in O will be shrunk. The time complexityof updating B (or R ) is O ( mnt · log ( mnt )) . c) Multipliers updating: The Lagrange multipliers D i s( i = 1 , , , ) can be updated as follows: D = D + ∇ R − V D = D + R − V D = D + ∇ B − V D = D + ∇ t B − V (6)The proposed algorithm for video rain streak removal isdenoted as “FastDeRain” and summarized in Algorithm 1. Fora video with dimensions of m × n × t , the time complexity ofthe proposed algorithm is proportional to O ( mnt log( mnt )) . Algorithm 1
FastDeRain
Input:
The rainy video O ; Initialization: B (0) = O , O = while not converged do Update V i ( i = 1 , , , via Eq. (4); Update B and R via (5); Update D i ( i = 1 , , , via Eq. (6); end whileOutput: The estimates of the rain-free video B and the rainstreaks R . E. Discussion of the oblique rain streaks
As we know that, in a real rainfall-affected scene, the rainstreaks are not always vertical. Thus, the directional propertywe utilized in our model is a double-edged sword whendealing with digital videos. In this subsection, we design anautomatical rain streaks’ angle detection method, and basedon it, we propose the shift strategy to deal with rain streaksnot vertical. a) Rain streaks direction detection:
Before starting ourstrategy, one important issue is how to automatically detectthe direction of the rain streaks. Based on our analysis of theprior knowledge, it’s not difficult to come up with a simpleand effective method to detect the direction. In this subsection,we assume that the rain streaks are in the same direction andthe angle between rain streaks and the vertical direction aredenoted as θ . For a rainy video O ∈ R m × n × t , our methodconsists of three steps:1) Filter the horizontal slices of the rainy video with a × median filter, i.e. , for i = 1 , , · · · , m , (cid:98) O ( i, : , :) = med ( O ( i, : , :)) , and obtain R = O − (cid:98) O .2) Rotate each frame of R with θ i = i ◦ , and obtain R θ i ( i = 0 , , · · · , t ) .3) For each R θ i , denote y i = (cid:107)∇ R θ i (cid:107) , then the detectedrain streaks angle ˆ θ = arg min θ i y i .Fig. 3 shows an example of our detection method, wherethe rain streaks are simulated with angle ◦ and the detectionresult (labeled red) is exactly 45 ◦ . Actually, the y i s are verylow when θ i is close to ◦ , according with the discussion inIII-B. Generally, the angle between the rain streaks and thevertical direction distributes in ( − ◦ , ◦ ) . If the angle ˆ θ ∈ ( − ◦ , ◦ ) , we can restrict it to the range of (0 ◦ , ◦ ) by theleft-right flipping of each frame. If the angle ˆ θ ∈ (45 ◦ , ◦ ) , we can restrict it to the range of (0 ◦ , ◦ ) by transposing (i.e.interchanging the rows and columns of a given matrix) eachframe. To save space, we only discuss the situations where ˆ θ ∈ [0 ◦ , ◦ ] in the following. Fig. 3. The magnitude of y i s with respect to θ i s. Shift IIShift I Shift IIShift I
Fig. 4. Illustrations of the shift I and the shift II operations. For bettervisualization, the rain streaks in the left part are roughly labeled with thered color, while the pixel values of the rain streaks images in the right arescaled. b) The shift strategy:
When the detected angle ˆ θ ∈ [15 ◦ , ◦ ] , we apply the shift strategy, which consists of twoshifting operations, as shown in Fig. 4, for different situations.The two shift operations are detailed as follows: Shift I If ˆ θ ∈ [35 ◦ , ◦ ] , for each frame O :: k , we slidethe i -th row ( i − pixel(s) to the right. Shift II If ˆ θ ∈ [15 ◦ , ◦ ) , for each frame O :: k , we slidethe i -th row (cid:98) ( i − (cid:99) pixel(s) to the right.Different from the rotation strategy recommended in [37],the core idea of the shift strategy is to rationally slide therows of the rainy frames and make the rain streaks beingapproximately vertical without any degradation caused byinterpolation Meanwhile, it is notable that these shifting op-erations wouldn’t affect the prior knowledges mentioned inIII-B. After shifting, the rain streaks is close to being vertical,and we can apply the algorithm 1. Finally, the result would beshifted back. The flowchart of applying our FastDeRain withthe shift strategy is shown in Fig. 5.IV. E XPERIMENTAL RESULTS
In this section, we evaluate the performance of the proposedalgorithm on synthetic data and real-world rainy videos. a) Implementation details:
Throughout our experiments,color videos with dimensions of m × n × × t are trans-formed into the YUV format. YUV is a color space thatis often used as part of a color image pipeline. Y stands (cid:98) x (cid:99) denotes the rounding the x to the nearest integers towards minusinfinity. Y NAngle detectionInputrainy videos FastDeRainShift IIShift ITransposingLeft-right flippingUp-down flipping Y N
Fig. 5. The flowchart of the dealing with rainy videos with the rain streaks of different directions. for the luma component (the brightness), and U and V arethe chrominance (color) components . We apply our methodonly to the Y channel with the dimension of m × n × t . Theexhibited rain streaks are scaled for better visualization.Since that the graphics processing unit (GPU) device isable to speed up the large-scale computing, we implement ourmethod on the platform of Windows 10 and Matlab (R2017a)with an Intel(R) Core(TM) i5-4590 CPU at 3.30GHz, 16 GBRAM, and a GTX1080 GPU. The involved operations in algo-rithm 1 is convenient to be implemented on the GPU device[58]. If we conduct our algorithm on the CPU, the runningtime for dealing with a video of size × × × is about 23 seconds, while 7 seconds on the GPU device.Meanwhile, Fu et al .’s method [27] can also be accelerated bythe GPU device, from 38 seconds on the CPU to 24 secondson the GPU, dealing with a video of size × × × .Thus, we only report the GPU running time of FastDeRainand Fu et al. ’s method in this section. b) Compared methods: To validate the effectiveness andefficiency of the proposed method, we compare our method(denoted as “FastDeRain”) with recent state-of-the-art meth-ods, including one single image based method, i.e., Fu et al .’sdeep detail network (DDN) method [27]; and three video-based mehtods, i.e., Kim et al .’s method using temporal cor-relation and low-rankness (TCL) [34], Wei et al .’s stochasticencoding (SE) method [39], and Li et al .’s multiscale convo-lutional sparse coding (MS-CSC) method [40]. In fact, DDNis a single-image-based rain streak removal method, but theirperformance has already surpassed some video-based methods.The deep learning technique shows a great vitality and anextremely wide application prospect. Hence, the comparisonwith DNN is reasonable and challenging. A. Synthetic dataa) Rain streak generation:
Adding rain streaks to a videois indeed a complex problem since there is not an existingalgorithm nor a free software to accomplish it in one step.Meanwhile, as Starik et al. pointed out in [43] that therain streaks can be assumed temporal independent, thus wecan simulate rain streaks for each frame using the syntheticmethod mentioned in many recently developed single imagerain streaks removal approaches [8, 13, 26], i.e. , using thePhotoshop software with the tutorial documents [59]. The https://en.wikipedia.org/wiki/YUV http://smartdsp.xmu.edu.cn/xyfu.html http://mcl.korea.ac.kr/ ∼ jhkim/deraining/deraining code with example.zip http://gr.xjtu.edu.cn/web/dymeng https://github.com/MinghanLi/MS-CSC-Rain-Streak-Removal Rainy TCL DDN FastDeRain GTRainy TCL DDN FastDeRain GTRainy TCL DDN FastDeRain GTRainy TCL DDN SE MS-CSC FastDeRain GT
Fig. 6. The rainy frame, rain streaks removal results, extracted rain streaks andcorresponding error images by different methods with synthetic rain streaksin case 1 , respectively. The corresponding videos from top to bottom are the“’foreman”, ”bus”, ”waterfall” and ”highway”. From left to right are: the rainydata (or the color bar), results by TCL [34], DDN [27], (SE [39], MS-CSC[40],) FastDeRain, and the ground truth (GT), respectively.
TABLE IIQ
UANTITATIVE COMPARISONS OF THE RAIN STREAK REMOVAL RESULTSOF [34], [27], [39], [40]
AND THE PROPOSED METHOD ON SYNTHETICVIDEOS . T HE BEST
QUANTITATIVE VALUES ARE IN
BOLDFACE . Video Method PSNR SSIM FSIM VIF UIQI GMSD Time C a s e foreman Rainy 34.67 0.9541 0.9723 0.6787 0.8693 0.0524 —TCL [34] 33.86 0.9612 0.9716 0.6431 0.8917 0.0400 1696.4DDN [27] 34.25 0.9730 0.9804 0.7253 0.9151 0.0300 71.2SE [39] 21.95 0.6959 0.7994 0.3060 0.4125 0.1997 740.8MS-CSC [40] 26.61 0.7922 0.8772 0.3754 0.5895 0.1470 143.9FastDeRain “bus” Rainy 31.01 0.9146 0.9664 0.6269 0.8800 0.0725 —TCL [34] 33.06 0.9562 0.9744 0.6873 0.9329 0.0360 2429.2DDN [27] 31.08 0.9534 0.9714 0.6626 0.9254 0.0399 46.1FastDeRain “waterfall” Rainy 31.63 0.9097 0.9550 0.5956 0.8834 0.0617 —TCL [34] 35.57 0.9578 0.9726 0.7297 0.9426 0.0242 2338.7DDN [27] 32.70 0.9517 0.9677 0.6580 0.9287 0.0407 43.6FastDeRain “highway” Rainy 30.94 0.8592 0.9411 0.5279 0.7169 0.0974 —TCL [34] 34.58 0.9639 0.9728 0.7063 0.8840 0.0277 2127.3DDN [27] 29.59 0.9308 0.9521 0.6089 0.8074 0.0534 43.3SE [39] 35.09 0.9730 0.9818 0.7878 0.9041 0.0127 656.3MS-CSC [40] 37.46 0.9753 0.9818 0.8173 0.9193 0.0143 280.5FastDeRain C a s e “foreman” Rainy 28.87 0.8991 0.9410 0.5535 0.7902 0.0922 —TCL [34] 30.75 0.9234 0.9486 0.5078 0.8186 0.0584 2625.8DDN [27] 33.21 0.9526 0.9671 0.6252 0.8634 0.0494 66.6FastDeRain “bus” Rainy 26.15 0.8238 0.9300 0.4951 0.7808 0.1150 —TCL [34] 28.08 0.8669 0.9341 0.4557 0.8119 0.0838 3394.4DDN [27] 29.42 0.9171 0.9507 0.5468 0.8747 0.0644 44.9FastDeRain “waterfall” Rainy 26.11 0.7827 0.8986 0.4198 0.7382 0.1096 —TCL [34] 29.14 0.8457 0.9210 0.4217 0.8041 0.0796 2880.3DDN [27] 30.44 0.8929 0.9370 0.4882 0.8546 0.0699 42.0FastDeRain “highway” Rainy 28.73 0.8772 0.9427 0.5320 0.6963 0.1014 —TCL [34] 31.78 0.9333 0.9543 0.5481 0.7728 0.0472 2176.1DDN [27] 31.22 0.9407 0.9512 0.5861 0.7922 0.0569 43.9SE [39] 30.21 0.9681 0.9819 0.7851 0.8970 0.0137 750.1MS-CSC [40] 31.79 0.9686 0.9811 0.7959 0.8970 0.0141 317.3FastDeRain C a s e “foreman” Rainy 23.75 0.9301 0.9631 0.6409 0.8355 0.0740 —TCL [34] 25.13 0.9321 0.9559 0.5627 0.8430 0.0582 1991.6DDN [27] 26.62 0.9586 0.9735 0.6756 0.8753 0.0487 66.0FastDeRain “bus” Rainy 22.87 0.9101 0.9612 0.6597 0.8643 0.1067 —TCL [34] 25.84 0.8965 0.9485 0.5555 0.8373 0.0813 2969.7DDN [27] 25.73 0.9363 0.9640 0.6434 0.8896 0.0770 41.3FastDeRain “waterfall” Rainy 22.34 0.9235 0.9587 0.6525 0.9016 0.0682 —TCL [34] 24.21 0.9226 0.9518 0.6205 0.9063 0.0463 2483.6DDN [27] 24.75 0.9417 0.9634 0.6533 0.9198 0.0566 41.1FastDeRain “highway” Rainy 22.90 0.9212 0.9702 0.6611 0.7650 0.0683 —TCL [34] 24.10 0.9358 0.9658 0.6437 0.7889 0.0401 2012.8DDN [27] 25.06 0.9362 0.9566 0.6339 0.7890 0.0551 41.2SE [39] 23.78 0.9530 0.9805 0.7947 0.8891 0.0145 659.1MS-CSC [40] 24.19 0.9531 0.9797 0.8075 0.8903 0.0160 251.6FastDeRain density of the simulated rain streaks by this method is mainlydetermined by the ratio of the amounts of dots (in step 8 of[59]) to the number of all the pixels, and for convenience,the ratio is denoted as r . Another way to synthesize the rainstreaks was proposed in [39], adding rain streaks taken byphotographers under black background .Referring to [59] and [39], we generate 3 types of rainstreaks as follows: Case 1
Rain streaks simulated referring to [59] with r ≤ . . In a single frame, the rain streaks share the same angle. Rainy TCL DDN FastDeRain GTRainy TCL DDN FastDeRain GTRainy TCL DDN FastDeRain GTRainy TCL DDN SE MS-CSC FastDeRain GT
Fig. 7. The rainy frame, rain streaks removal results, extracted rain streaks andcorresponding error images by different methods with synthetic rain streaksin case 2 , respectively. The corresponding videos from top to bottom are the“’foreman”, ”bus”, ”waterfall” and ”highway”. From left to right are: the rainydata (or the color bar), results by TCL [34], DDN [26], (SE [39], MS-CSC[40],) FastDeRain, and the ground truth (GT), respectively.
The fixed angles for different frames increase from − ◦ to ◦ with time; Case 2
Rain streaks simulated referring to [59] with r ≥ . . In a single frame, the rain streaks are with differentangles. The angles uniformly distribute in a range [ − ◦ , ◦ ] ; Case 3
Rain streaks simulated referring to [39].Four videos are selected as the clean background. Threevideos , named “foreman” with the size of × × × , http://trace.eas.asu.edu/yuv/ Rainy TCL DDN FastDeRain GTRainy TCL DDN FastDeRain GTRainy TCL DDN FastDeRain GTRainy TCL DDN SE MS-CSC FastDeRain GT
Fig. 8. The rainy frame, rain streaks removal results, extracted rain streaks andcorresponding error images by different methods with synthetic rain streaksin case 3 , respectively. The corresponding videos from top to bottom are the“’foreman”, ”bus”, ”waterfall” and ”highway”. From left to right are: the rainydata (or the color bar), results by TCL [34], DDN [26], (SE [39], MS-CSC[40],) FastDeRain, and the ground truth (GT), respectively. “bus” and “waterfall” with the size of × × × , arecaptured by dynamic cameras, while the other one , named “highway” with the size of × × × , are recordedby a static camera.SE [39] and MS-CSC [40] are designed mainly for thevideos captured by static cameras, and directly applying themon the video captured by dynamic camera would result inpoor performances (see the gray values Table II). Therefore, for a fair comparison, the compared methods included DDN[26] and TCL [34] when dealing with the synthetic rainydata generated on the videos “foreman” “bus” and “waterfall”.When dealing with the rainy data simulated with the video“highway”, SE [39] and MS-CSC [40] would be brought intocomparison. b) Quantitative comparisons: For quantitative assess-ment, the peak signal-to-noise ratio (PSNR) of the wholevideo, and the structural similarity (SSIM) [60], the featuresimilarity (FSIM) [61], the visual information fidelity (VIF)[62], the universal image quality index (UIQI) [63], and thegradient magnitude similarity deviation (GMSD, smaller isbetter) [64] of each frame are calculated. The PSNR, thecorresponding mean values of SSIM FSIM VIF and UIQI,and the running time are reported in Table II, in which thebest quantitative values are in boldface.As observed in Table II, our method considerably out-performed the other four state-of-the-art methods in termsof all the selected quality assessment indexes. Notably, inmany cases, the performances of the single-image-based deeplearning method DNN [26] surpassed the those of the video-based method TCL [34]. This is in agreement with theaforementioned rationality of considering comparisons withthe single-image-based method.The running time of the our FastDeRain is extremely low. Inparticular, our method took less than 10 seconds when dealingwith all the synthetic data. Although a tensor system mightbe expected to be computationally expensive, our algorithm,with closed-form solutions to its sub-problems and a timecomplexity of approximately O ( mnt log ( mnt )) for an inputvideo with a resolution of m × n and t frames, is expected to beefficient. In the meantime, the aforementioned implementationon the GPU device also largely accelerated our algorithm. c) Visual comparisons: Fig. 6, 7 and 8 exhibit the resultsconducted on videos with synthetic rain streaks in case 1, case2 and case 3, respectively. In Fig. 6, since the angles of rainstreaks in case 1 increase with time, we display the frames atthe beginning or end. Meanwhile, only one frame is exhibitedin Fig. 7, Fig. 8 on account of that the rain streaks in everyframe are of various directions.In Fig. 6, all the methods removed almost all of therain streaks and the proposed method maintained the bestbackground. Many details in the background were incorrectlyextracted to the rain streaks by DDN and TCL. It can be foundin the 6-th row of Fig. 6, i.e., the error images of the resultson the video “bus”, that little vertical patterns were mistakenlyextracted as the rain streaks by the proposed method.For the rain streaks in case 2, the denser rain streaks implythat it is more difficult than rain streaks in case 1. For instance,the denser rain streaks visibly degraded the performance ofSE. From Fig. 7, we can find that our method preserved thebackgrounds well and other four methods erased the details ofthe backgrounds.In Fig. 8, the proposed method removed most of the rainstreaks and considerably preserves the background. Othermethods tended to obtain over de-rain or under de-rain results.Considering the similarity of the extract rains streaks to
Rainy α = 10 − α = 10 − α = 10 − α = 10 − FastDeRain Ground truthFig. 9. The top row shows the 80th frame of the rainy video, the results by FastDeRain and its degraded versions, in which the α i s in Eq. (3) are set as − in turn, and the ground truth clean video, respectively. The middle row presents the extracted rain streaks by FastDeRain and its degraded versionsand the ground truth rain streaks, while the color bar and corresponding error images are exhibited in the bottom row.Fig. 10. The mean SSIM FSIM and UIQI values with respect to different values of α , α , α , α and µ . The solid lines are corresponding to the resultsof FastDeRain while the dashed lines are related to the results obtained by our method without the N in Eq. (1). the ground truth rain streaks, our FastDeRain held obviousadvantages.In summary, for these different types of synthetic data, ourmethod can simultaneously remove almost all rain streakswhile commendably preserving the details of the underlyingclean videos. d) Discussion of each component: There are four com-ponents in our model (2). To elucidate their distinct effects, wedegrade our method by setting each α i ( i = 1 , , , ) equal to − , respectively. These degraded methods and FastDeRainare tested on the video “waterfall” with synthetic rain streaksin case 1. We present the quantitative assessments in Fig. 11and the visual results in Fig. 9. SSIM FSIM VIF UIQI
SSIM FSIM VIF UIQI
Fig. 11. The quantitative performances of the proposed method and itsdegraded versions, in which the α i s in Eq. (3) are set as − in turn. From Fig. 11 and Fig. 9, we can conclude that all the fourcomponents contribute to the removal of rain streaks. Specif- ically, (a) when setting α = 10 − , the rain streaks tend tobe intermittent along the vertical direction; (b) the rain streaksare fatter when the sparsity term contributes little; (c) somerain streaks remain in the background when the horizontalsmoothness of the background is not sufficiently enhanced;(d) the temporal continuity seems overwhelmingly importantsince that without this regularization term our method nearlyfailed. e) Parameters: To examine the performance of the pro-posed FastDeRain with respect to different parameters, weconduct a series of experiments on the rainy data on syntheticvideo “waterfall” with the synthetic rain streaks in case 1 andthe Gaussian noise with zero mean and standard deviation0.02. In Fig. 10, a parameter analysis is presented and theSSIM FSIM and MUIQI are selected. Based on guidancefrom Fig. 10, our tuning strategy is as following: (1) set α and α as − and other α i s to 0.01, and µ = 1 , (2)tune α and α until the results are barely satisfactory, (3)and then fix α and α and enlarge α and α to furtherimprove the performance. The tuning principle is as follows:when some of the texture or detail of the clean video isextracted into the estimated rain streaks, we increase α and α or decrease α and α , and we do the opposite whenrain streaks remain in the estimated rain-free content. Ourrecommended set of candidate values for α through α is { . , . , . , . , . , . , . } . TheLagrange parameter µ is suggested to be 1. In practice, thetime cost for the empirical tuning of the parameters is notmuch. (a) Video “waterfall” (case 1) Rainy [37] FastDeRain GT (b) Video “bus” (case 2)
Rainy [37] FastDeRain GT (c) Video “highway” (case 3)Fig. 12. The deraining results by the proposed FastDeRain and the method in [37]. f) Discussion of the noise term N in Eq. (1): In thispaper, the noise (or error) term ( N in Eq. (1)) is taken intoconsideration in the observation model. To illustrate its effects,we conduct a series of experiments, in which the Gaussiannoises of different standard deviations are respectively addedto the video “waterfall” with synthetic rain streaks in case1. The quantitative assessments of the results obtained by theproposed method with and without the noise (or error) term N taken into consideration (denoted as “with N ” and “without N ”, respectively ) are reported in Table III. In addition, wealso exhibit the effects of different parameters on the proposedmethod without N in Fig. 10. TABLE IIIQ
UANTITATIVE COMPARISONS OF THE RAIN STREAK REMOVAL RESULTSOF THE PROPOSED F AST D E R AIN WITH AND WITHOUT THE NOISE TERMTAKEN INTO CONSIDERATION ON SYNTHETIC VIDEO “ WATERFALL ” WITHTHE SYNTHETIC RAIN STREAKS IN CASE
1. T HE BEST
QUANTITATIVEVALUES ARE IN
BOLDFACE . σ Method PSNR SSIM FSIM VIF UIQI GMSD0 Rainy 31.63 0.9097 0.9550 0.5956 0.8834 0.0617with N N N without N N without N N without N N without N From Table III, we can conclude our method without N would acquire a better result when the rainy video is free fromthe noise. However, when the video is simultaneously affectedby the rain streaks and the noise, which is unavoidable in realdata, our method with N got better results. Therefore, weadopt the term N in Eq. (3) which enhances the robustnessof our method to the noise. Meanwhile, the solid lines andthe dashed lines in Fig. 10 also demonstrate that taking thenoise (or error) term N into account would contribute to therobustness of the proposed method to different parameters. g) Comparisons with the method in the conference ver-sion: To clarify the improvement of the proposed method fromour conference version [37], we compared the performances
TABLE IVQ
UANTITATIVE COMPARISONS OF THE RAIN STREAK REMOVAL RESULTSOF THE PROPOSED F AST D E R AIN AND THE METHOD IN THE PREVIOUSCONFERENCE PAPER [37]
ON THE SYNTHETIC DATA . T HE BEST
QUANTITATIVE VALUES ARE IN
BOLDFACE . Data Method PSNR SSIM FSIM VIF UIQI GMSD Time (s)“waterfall” Rainy 31.63 0.9097 0.9550 0.5956 0.8834 0.0617 —case 1 [37] 37.86 0.9864 0.8397 0.9763 0.9787 0.0164 19.9FastDeRain “bus” Rainy 26.15 0.8238 0.9300 0.4951 0.7808 0.1150 —case 2 [37] 30.07 0.9331 0.9574 0.6369 0.8986 0.0590 23.3FastDeRain “highway” Rainy 22.90 0.9212 0.9702 0.6611 0.7650 0.0683 —case 3 [37] 24.02 0.9487 0.9823 0.7384 0.8312 0.0362 17.7FastDeRain of our FastDeRain and the method in [37]. To save space,results on the part of the synthetic data, which are listedin the first column of Table IV, are reported. The derainingresults are exhibited in Fig. 12, and, to avoid repetition, thenumbers of the frames in Fig. 12 are different from thosein foregoing figures. From Table IV and Fig. 12, we canconclude our FastDeRain made substantial progress comparedwith the method in the conference version [37]. These resultsalso accord with the above discussion of the irrationality ofthe low-rank regularizer.
B. Real data
Four real-world rainy videos are chosen in this subsection.The first one (denoted as “wall”) of size × × × is download from the CAVE dataset and the secondvideo (denoted as “yard”) of size × × × wasrecorded by one of the authors on a rainy day in his backyard.The background of the video “wall” is consist of regularpatterns while the background of the video “yard” is morecomplex. The third video is clipped from the well-known film“the Matrix”. The scene in this clips changes fast so that it ismore difficult to deal with this video. The last video of size × × × is denoted as “crossing” , and it wascaptured in the crossing with complex traffic conditions.Fig. 13 shows two adjacent frames of the results obtained onthe video “wall”. There are many vertical line patterns in the https://github.com/TaiXiangJiang/FastDeRain/blob/master/yard.mp4 https://github.com/hotndy/SPAC-SupplementaryMaterials/blob/master/Dataset Testing RealRain/ra4 Rain.rar Rainy frames TCL [34] DDN [27] SE [39] MS-CSC [40] FastDeRain(5902.2s) (75.8s) (2840.9s) (495.4s) ( s)Fig. 13. Rain streak removal performance of different methods obtained on the video “wall”. From top to bottom, two adjacent frames of the deraining resultsand corresponding extracted rain streaks are illustrated. From left to right are: the rainy data (or the color bar), results by different methods, and the groundtruth.Rainy frame TCL [34] DDN [27] SE [39] MS-CSC [40] FastDeRain(2685.6s) (63.8s) (558.9s) (448.3s) ( s)Fig. 14. Rain streak removal results on the video “yard”. From left to right are frames of the rainy video, rain streaks removal results and correspondingextracted rain streaks by different methods, respectively. From left to right are: the rainy data, results by different methods, and the ground truth. background of this video. Thus, exhibiting two adjacent frameswould further help to distinguish the rain streaks from thebackground. It can be found in the zoomed in red blocks thatthis rain streak with high brightness is not handled properlyby DNN, SE and MS-CSC. Our method removes almost allthe rain streaks and preserves the background best comparedwith the results by other three methods.Since there is little texture or structure similar to rainstreaks in the video “yard”, only one frame is exhibited inFig. 14. DNN and SE didn’t distinguish most of the rainstreaks, especially in the zoomed in red blocks. Although TCLand MS-CSC separated the majority of rain streaks, somefine structures of the background were improperly extracted.Our FastDeRain removed most of the rain streaks and wellpreserved the background.In Fig. 15, two adjacent frames of the rainy video “theMatrix” and deraining results by different methods are shown. The two adjacent rainy frames reveal the rapidly changingof the scene, particularly the luminance. Once again, ourFastDeRain obtained the best result, especially when dealingwith the obvious rain streak on the face of Neo.The results on the rainy video “crossing” are exhibited inFig. 16. From the zoomed in areas, we can observe that allthe methods except MS-CSC entirely removed the rain streaks.TCL extracted some the structure of the curb line into the rainstreaks while DNN tended to remove all the textures with linepattern. SE erased many structural details. The extracted rainstreaks by the proposed FastDeRain were visually the bestamong all the results.The scenarios in these four videos are of large differences.Our method obtains the best results, both in removing rainstreaks and in retaining spatial details. In addition, the runningtime of our method is also obviously less than other methods,especially those three video-based methods. Rainy frames TCL [34] DDN [27] SE [39] MS-CSC [40] FastDeRain(8431.4s) (80.6s) (3852.5s) (484.9s) ( s)Fig. 15. Rain streak removal performance of different methods obtained on the clips of movie “the Matrix”. From top to bottom, 2 adjacent frames of therainy video/deraining results and corresponding extracted rain streaks are illustrated. From left to right are: the rainy data, results by different methods, andthe ground truth.Rainy frame TCL [34] DDN [27] SE [39] MS-CSC [40] FastDeRain(7246.0s) (54.33s) (2821.0s) (484.9s) ( s)Fig. 16. Rain streak removal performance of different methods obtained on the video “crossing”. From left to right are: the rainy data, deraining results orextracted rain streaks by different methods, and the ground truth.
C. Oblique rain streaks
In this subsection, we examine the performance of ourmethod with the shift strategy and other four methods, whenthe rain streaks are far away from being vertical. We simulatedtwo rainy videos: one is rain streaks with angles varyingin [15 ◦ , ◦ ] added to the video “waterfall” (captured by adynamic camera); another one is rain streaks with anglesvarying in [35 ◦ , ◦ ] added to the video “highway” (capturedby a static camera). As shown in Table V and Fig. 17, theshift strategy helped our method to obtains the best resultswhen dealing with the oblique rain streaks. The superior ofthe proposed FastDeRain is obvious both quantitatively andvisually. V. C ONCLUSION
We have proposed a novel video rain streaks removalapproach: FastDeRain. The proposed method, based on di-rectional gradient priors in combination with sparsity, outper-forms a series of state-of-the-art methods both visually and
TABLE VQ
UANTITATIVE COMPARISONS OF THE RAIN STREAK REMOVAL RESULTSOF [34], [26], [39], [40]
AND THE PROPOSED METHOD WITH THE SHIFTSTRATEGY WHEN RAIN STREAKS ARE FAR AWAY FROM BEING VERTICAL .T HE BEST
QUANTITATIVE VALUES ARE IN
BOLDFACE . Video: “waterfall” Angle: ◦ − ◦ Method PSNR SSIM FSIM VIF UIQI GMSD time (s)Rainy 29.14 0.8612 0.9323 0.5111 0.8228 0.0754 —TCL [34] 33.55 0.9336 0.9602 0.6362 0.9110 0.0363 2929.2DDN [26] 32.10 0.9283 0.9589 0.5984 0.8993 0.0448 43.8SE [39] 25.27 0.6219 0.7811 0.3137 0.3844 0.1732 1028.0MS-CSC [40] 28.44 0.7593 0.8900 0.3876 0.6679 0.1154 264.3FastDeRain
Video: “highway” Angle: ◦ − ◦ Method PSNR SSIM FSIM VIF UIQI GMSD time (s)Rainy 29.18 0.8162 0.9197 0.4865 0.6554 0.0957 —TCL [34] 30.26 0.8859 0.9399 0.5460 0.7038 0.0603 1277.7DDN [26] 28.91 0.8208 0.9126 0.4563 0.6510 0.0877 38.3SE [39] 33.22 0.9703 0.9809 0.7944 0.8974 0.0127 564.8MS-CSC [40] 36.99 0.9747 0.9812 0.8137 0.9177 0.0137 182.1FastDeRain Rainy TCL DDN SE MS-CSC FastDeRain GTRainy TCL DDN SE MS-CSC FastDeRain GTFig. 17. From top to bottom are the rain streaks removal results, extracted rain streaks and corresponding error images by different methods on the video“highway1” (top 3 row) and “highway2” (bottom 3 row), respectively. From left to right are: the rainy data, results by TCL [34], DDN [27], SE [39], MS-CSC[40], FastDeRain with shift strategy and the ground truth. quantitively. We attribute the outperforming of FastDeRainto our intensive analysis of the characteristic priors of rainyvideos, clean videos and rain streaks. Besides, it notable thatour method is markedly faster than the compared methods,even including a every fast single-image-based method. Ourmethod is not without limitation. The natural rainy scenario issometimes mixed with haze, and how to handle the residualrain artifacts remains an open problem. These issues will beaddressed in the future.A
CKNOWLEDGMENT
The authors would like to express their sincere thanks to theeditor and referees for giving us so many valuable commentsand suggestions for revising this paper. The authors wouldlike to thank Dr. Xueyang Fu, Dr. Wei Wei and Dr. MinghanLi for their generous sharing of their codes. This researchwas supported by the National Natural Science Foundationof China (61772003, 61702083), and the Fundamental Re-search Funds for the Central Universities (ZYGX2016J132,ZYGX2016J129, ZYGX2016KYQD142).R
EFERENCES[1] Y. Li, R. T. Tan, X. Guo, J. Lu, and M. S. Brown, “Rain streak removalusing layer priors,” in the IEEE Conference on Computer Vision andPattern Recognition (CVPR) , 2016, pp. 2736–2744.[2] T. Bouwmans, “Traditional and recent approaches in background model-ing for foreground detection: An overview,”
Computer Science Review ,vol. 11, pp. 31–66, 2014.[3] M. S. Shehata, J. Cai, W. M. Badawy, T. W. Burr, M. S. Pervez, R. J.Johannesson, and A. Radmanesh, “Video-based automatic incident de-tection for smart roads: the outdoor environmental challenges regarding false alarms,”
IEEE Transactions on Intelligent Transportation Systems ,vol. 9, no. 2, pp. 349–360, 2008.[4] X. Zhang, C. Zhu, S. Wang, Y. Liu, and M. Ye, “A bayesian approach tocamouflaged moving object detection,”
IEEE Transactions on Circuitsand Systems for Video Technology , vol. 27, no. 9, pp. 2001–2013, 2017.[5] C. Ma, Z. Miao, X.-P. Zhang, and M. Li, “A saliency prior contextmodel for real-time object tracking,”
IEEE Transactions on Multimedia ,vol. 19, no. 11, pp. 2415–2424, 2017.[6] K. Garg and S. K. Nayar, “Vision and rain,”
International Journal ofComputer Vision , vol. 75, no. 1, pp. 3–27, 2007.[7] L. Itti, C. Koch, E. Niebur et al. , “A model of saliency-based visual at-tention for rapid scene analysis,”
IEEE Transactions on Pattern Analysisand Machine Intelligence , vol. 20, no. 11, pp. 1254–1259, 1998.[8] L.-W. Kang, C.-W. Lin, and Y.-H. Fu, “Automatic single-image-basedrain streaks removal via image decomposition,”
IEEE Transactions onImage Processing , vol. 21, no. 4, pp. 1742–1755, 2012.[9] S.-H. Sun, S.-P. Fan, and Y.-C. F. Wang, “Exploiting image structuralsimilarity for single image rain removal,” in the IEEE InternationalConference on Image Processing (ICIP) , 2014, pp. 4482–4486.[10] Y.-L. Chen and C.-T. Hsu, “A generalized low-rank appearance modelfor spatio-temporally correlated rain streaks,” in the IEEE InternationalConference on Computer Vision (ICCV) , 2013, pp. 1968–1975.[11] J. Chen and L.-P. Chau, “A rain pixel recovery algorithm for videoswith highly dynamic scenes,”
IEEE Transactions on Image Processing ,vol. 23, no. 3, pp. 1097–1104, 2014.[12] D.-Y. Chen, C.-C. Chen, and L.-W. Kang, “Visual depth guided colorimage rain streaks removal using sparse coding,”
IEEE transactions oncircuits and systems for video technology , vol. 24, no. 8, pp. 1430–1455,2014.[13] Y. Luo, Y. Xu, and H. Ji, “Removing rain from a single image viadiscriminative sparse coding,” in the IEEE International Conference onComputer Vision (ICCV) , 2015, pp. 3397–3405.[14] C.-H. Son and X.-P. Zhang, “Rain removal via shrinkage of sparse codesand learned rain dictionary,” in the IEEE International Conference onMultimedia & Expo Workshops (ICMEW) , 2016, pp. 1–6.[15] Y. Li, R. T. Tan, X. Guo, J. Lu, and M. S. Brown, “Single image rainstreak decomposition using layer priors,”
IEEE Transactions on ImageProcessing , vol. 26, no. 8, pp. 3874–3885, 2017.[16] L. Zhu, C.-W. Fu, D. Lischinski, and P.-A. Heng, “Joint bi-layer opti- mization for single-image rain streak removal,” in the IEEE InternationalConference on Computer Vision (ICCV) , Oct 2017.[17] B.-H. Chen, S.-C. Huang, and S.-Y. Kuo, “Error-optimized sparserepresentation for single image rain removal,” IEEE Transactions onIndustrial Electronics , vol. 64, no. 8, pp. 6573–6581, 2017.[18] S. Gu, D. Meng, W. Zuo, and L. Zhang, “Joint convolutional analysisand synthesis sparse representation for single image layer separation,” in the IEEE International Conference on Computer Vision (ICCV) , 2017,pp. 1717–1725.[19] Y. Chang, L. Yan, and S. Zhong, “Transformed low-rank model for linepattern noise removal,” in the IEEE Conference on Computer Vision andPattern Recognition (CVPR) , 2017, pp. 1726–1734.[20] L.-J. Deng, T.-Z. Huang, X.-L. Zhao, and T.-X. Jiang, “A directionalglobal sparse model for single image rain removal,”
Applied Mathemat-ical Modelling , vol. 59, pp. 662–679, 2018.[21] S. Du, Y. Liu, M. Ye, Z. Xu, J. Li, and J. Liu, “Single image deraining viadecorrelating the rain streaks and background scene in gradient domain,”
Pattern Recognition , vol. 79, pp. 303–317, 2018.[22] Y. Wang, S. Liu, C. Chen, and B. Zeng, “A hierarchical approach forrain or snow removing in a single color image,”
IEEE Transactions onImage Processing , vol. 26, no. 8, pp. 3936–3950, 2017.[23] D. Ren, W. Zuo, D. Zhang, L. Zhang, and M.-H. Yang, “Simultaneousfidelity and regularization learning for image restoration,” arXiv preprintarXiv:1804.04522 , 2018.[24] D. Eigen, D. Krishnan, and R. Fergus, “Restoring an image takenthrough a window covered with dirt or rain,” in the IEEE InternationalConference on Computer Vision (ICCV) , 2013, pp. 633–640.[25] W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, and S. Yan, “Deep joint raindetection and removal from a single image,” in the IEEE Conference onComputer Vision and Pattern Recognition (CVPR) , July 2017.[26] X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley, “Clearing theskies: A deep network architecture for single-image rain removal,”
IEEETransactions on Image Processing , vol. 26, no. 6, pp. 2944–2956, 2017.[27] X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removingrain from single images via a deep detail network,” in the IEEEConference on Computer Vision and Pattern Recognition (CVPR) , 2017,pp. 3855–3863.[28] H. Zhang, V. Sindagi, and V. M. Patel, “Image de-raining using a condi-tional generative adversarial network,” arXiv preprint arXiv:1701.05957 ,2017.[29] R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, “Attentive generativeadversarial network for raindrop removal from a single image,” pp.2482–2491, 2018.[30] S. Li, W. Ren, J. Zhang, J. Yu, and X. Guo, “Fast single image rainremoval via a deep decomposition-composition network,” arXiv preprintarXiv:1804.02688 , 2018.[31] H. Zhang and V. M. Patel, “Density-aware single image de-raining usinga multi-stream dense network,” in the IEEE Conference on ComputerVision and Pattern Recognition (CVPR) , 2018, pp. 695–704.[32] K. Garg and S. K. Nayar, “Detection and removal of rain from videos,”in the IEEE Conference on Computer Vision and Pattern Recognition(CVPR) , pp. I–528–I–535.[33] A. K. Tripathi and S. Mukhopadhyay, “Removal of rain from videos: areview,”
Signal, Image and Video Processing , vol. 8, no. 8, pp. 1421–1430, 2014.[34] J.-H. Kim, J.-Y. Sim, and C.-S. Kim, “Video deraining and desnowingusing temporal correlation and low-rank matrix completion,”
IEEETransactions on Image Processing , vol. 24, no. 9, pp. 2658–2670, 2015.[35] V. Santhaseelan and V. K. Asari, “Utilizing local phase information toremove rain from video,”
International Journal of Computer Vision , vol.112, no. 1, pp. 71–89, 2015.[36] S. You, R. T. Tan, R. Kawakami, Y. Mukaigawa, and K. Ikeuchi,“Adherent raindrop modeling, detectionand removal in video,”
IEEETransactions on Pattern Analysis and Machine Intelligence , vol. 38,no. 9, pp. 1721–1733, 2016.[37] T.-X. Jiang, T.-Z. Huang, X.-L. Zhao, L.-J. Deng, and Y. Wang, “Anovel tensor-based video rain streaks removal approach via utilizingdiscriminatively intrinsic priors,” in the IEEE Conference on ComputerVision and Pattern Recognition (CVPR) , 2017, pp. 4057–4066.[38] W. Ren, J. Tian, Z. Han, A. Chan, and Y. Tang, “Video desnowingand deraining based on matrix decomposition,” in the IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 4210–4219.[39] W. Wei, L. Yi, Q. Xie, Q. Zhao, D. Meng, and Z. Xu, “Should weencode rain streaks in video as deterministic or stochastic?” in the IEEEInternational Conference on Computer Vision (ICCV) , 2017, pp. 2516–2525. [40] M. Li, Q. Xie, Q. Zhao, W. Wei, S. Gu, J. Tao, and D. Meng, “Videorain streak removal by multiscale convolutional sparse coding,” in theIEEE Conference on Computer Vision and Pattern Recognition (CVPR) ,2018, pp. 6644–6653.[41] J. Chen, C.-H. Tan, J. Hou, L.-P. Chau, and H. Li, “Robust video contentalignment and compensation for rain removal in a cnn framework,” in theIEEE Conference on Computer Vision and Pattern Recognition (CVPR) ,2018, pp. 6286–6295.[42] J. Liu, W. Yang, S. Yang, and Z. Guo, “Erase or fill? deep joint recurrentrain removal and reconstruction in videos,” in the IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 3233–3242.[43] S. Starik and M. Werman, “Simulation of rain in videos,” in the IEEEInternational Conference on Computer Vision (ICCV) Texture Workshop ,vol. 2, 2003, pp. 406–409.[44] X. Guo and Y. Ma, “Generalized tensor total variation minimization forvisual data recovery,” in the IEEE Conference on Computer Vision andPattern Recognition , 2015, pp. 3603–3611.[45] Y. Jiang, X. Jin, and Z. Wu, “Video inpainting based on joint gradientand noise minimization,” in
The Pacific Rim Conference on Multimedia .Springer, 2016, pp. 407–417.[46] Y. Chang, L. Yan, H. Fang, and H. Liu, “Simultaneous destriping anddenoising for remote sensing images with unidirectional total variationand sparse representation,”
IEEE Geoscience and Remote Sensing Let-ters , vol. 11, no. 6, pp. 1051–1055, 2014.[47] Y. Chang, L. Yan, T. Wu, and S. Zhong, “Remote sensing imagestripe noise removal: from image decomposition perspective,”
IEEETransactions on Geoscience and Remote Sensing , vol. 54, no. 12, pp.7018–7031, 2016.[48] H.-X. Dou, T.-Z. Huang, L.-J. Deng, X.-L. Zhao, and J. Huang,“Directional (cid:96) sparse modeling for image stripe noise removal,” RemoteSensing , vol. 10, no. 3, p. 361, 2018.[49] T.-X. Jiang, T.-Z. Huang, X.-L. Zhao, T.-Y. Ji, and L.-J. Deng, “Matrixfactorization for low-rank tensor completion using framelet prior,”
Information Sciences , vol. 436, pp. 403–417, 2018.[50] S. Li, R. Dian, L. Fang, and J. M. Bioucas-Dias, “Fusing hyperspectraland multispectral images via coupled sparse tensor factorization,”
IEEETransactions on Image Processing , vol. 27, no. 8, pp. 4118–4130, 2018.[51] T.-Y. Ji, N. Yokoya, X. X. Zhu, and T.-Z. Huang, “Nonlocal tensorcompletion for multitemporal remotely sensed images’ inpainting,”
IEEETransactions on Geoscience and Remote Sensing , vol. 56, no. 6, pp.3047–3061, 2018.[52] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,”
SIAM Review , vol. 51, no. 3, pp. 455–500, 2009.[53] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributedoptimization and statistical learning via the alternating direction methodof multipliers,”
Foundations and Trends R (cid:13) in Machine Learning , vol. 3,no. 1, pp. 1–122, 2011.[54] T.-X. Jiang, T.-Z. Huang, X.-L. Zhao, and L.-J. Deng, “A novel non-convex approach to recover the low-tubal-rank tensor data: when t-svdmeets pssv,” arXiv preprint arXiv:1712.05870 , 2017.[55] X.-L. Zhao, F. Wang, T.-Z. Huang, M. K. Ng, and R. J. Plemmons,“Deblurring and sparse unmixing for hyperspectral images,” IEEETransactions on Geoscience and Remote Sensing , vol. 51, no. 7, pp.4045–4058, 2013.[56] X.-L. Zhao, F. Wang, and M. K. Ng, “A new convex optimization modelfor multiplicative noise and blur removal,”
SIAM Journal on ImagingSciences , vol. 7, no. 1, pp. 456–475, 2014.[57] M. V. Afonso, J. M. Bioucas-Dias, and M. A. Figueiredo, “An aug-mented lagrangian approach to the constrained optimization formulationof imaging inverse problems,”
IEEE Transactions on Image Processing
IEEETransactions on Image Processing , vol. 13, no. 4, pp. 600–612, 2004.[61] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarityindex for image quality assessment,”
IEEE transactions on ImageProcessing , vol. 20, no. 8, pp. 2378–2386, 2011.[62] H. R. Sheikh and A. C. Bovik, “Image information and visual quality,”
IEEE Transactions on image processing , vol. 15, no. 2, pp. 430–444,2006.[63] Z. Wang and A. C. Bovik, “A universal image quality index,”