2658

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 9, SEPTEMBER 2015

Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion Jin-Hwan Kim, Jae-Young Sim, Member, IEEE, and Chang-Su Kim, Senior Member, IEEE

Abstract— A novel algorithm to remove rain or snow streaks from a video sequence using temporal correlation and low-rank matrix completion is proposed in this paper. Based on the observation that rain streaks are too small and move too fast to affect the optical flow estimation between consecutive frames, we obtain an initial rain map by subtracting temporally warped frames from a current frame. Then, we decompose the initial rain map into basis vectors based on the sparse representation, and classify those basis vectors into rain streak ones and outliers with a support vector machine. We then refine the rain map by excluding the outliers. Finally, we remove the detected rain streaks by employing a low-rank matrix completion technique. Furthermore, we extend the proposed algorithm to stereo video deraining. Experimental results demonstrate that the proposed algorithm detects and removes rain or snow streaks efficiently, outperforming conventional algorithms. Index Terms— Video deraining, desnowing, rain streak removal, low rank matrix completion, sparse representation.

I. I NTRODUCTION

V

IDEO capturing devices, such as smart phones and digital cameras, are relatively cheap and popular nowadays, which facilitate the widespread production and consumption of high quality video sequences. Outdoor video sequences, however, are often degraded due to various weather conditions, e.g. haze, fog, rain, and snow [1]–[3]. Image processing and computer vision systems, including tracking and surveillance, may not work properly on these degraded videos. Therefore, attempts have been made to compensate for the weatherdependent degradation and enhance outdoor video sequences. Early attention has been drawn to haze removal. Haze exhibits a static distribution of tiny particles in the air, which degrades image contrast [1]. Thus, most dehazing algorithms attempt to enhance the contrast. On the contrary, as illustrated

Manuscript received October 27, 2014; revised February 23, 2015; accepted April 19, 2015. Date of publication May 4, 2015; date of current version May 19, 2015. This work was supported in part by the National Research Foundation (NRF) of Korea through the Ministry of Science, ICT and Future Planning (MSIP) under Grant 2009-0083495, and in part by NRF through the Korean Government within the MSIP under Grant 2012-011031. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Jean-Francois Aujol. J.-H. Kim is with the Defense Agency for Technology and Quality, Gumi 660-031, Korea (e-mail: [email protected]). J.-Y. Sim is with the School of Electrical and Computer Engineering, Ulsan National Institute of Science and Technology, Ulsan 689-805, Korea (e-mail: [email protected]). C.-S. Kim is with the School of Electrical Engineering, Korea University, Seoul 136-701, Korea (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TIP.2015.2428933

Fig. 1. Desnowing of a frame in the film “Edward Scissorhands” (original image courtesy of 20th Century Fox – ©1990 20th Century Fox, Inc.): (a) input frame, (b) desnowing result, and (c) snow map.

in Fig. 1, rain or snow causes elongated streaks, the locations of which are randomly distributed within an image and time-varying dynamically in the case of a video sequence. Moreover, a rain or snow streak is larger than a haze element, hiding the colors of objects behind. Therefore, video deraining or desnowing, which recovers rain-free or snow-free video sequences, is more challenging than dehazing. In this work, we will use the term ‘rain’ to mean rain and snow together, since deraining techniques can be used to remove snow and vice versa. Several deraining algorithms have been proposed [2]–[12]. However, most deraining algorithms do not consider global motions, object motions, and/or various sizes of rain streaks, and thus they may fail to remove rain streaks clearly. We propose a novel video deraining algorithm using temporal correlation and low-rank matrix completion. Assuming that adjacent frames, warped by the optical flows, are almost identical with a current frame except for rain streak regions, we generate an initial rain map from the differences between the current frame and the warped adjacent frames. Then, we represent the initial rain map using sparse basis vectors, which are dichotomized into rain streak ones and outliers using a support vector machine (SVM). By removing the outliers, we refine the rain map and detect rain streaks. Finally, we replace the detected rainy pixels using a matrix completion algorithm, which performs the expectation maximization (EM) iterations for the low-rank approximation. Moreover, we extend the proposed video deraining algorithm to stereo video deraining. Experimental results demonstrate that the proposed algorithm outperforms conventional algorithms, by removing rain streaks efficiently and reconstructing scene contents faithfully.

1057-7149 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

KIM et al.: VIDEO DERAINING AND DESNOWING USING TEMPORAL CORRELATION AND LOW-RANK MATRIX COMPLETION

The paper is organized as follows: Section II surveys conventional deraining algorithms. Section III describes the proposed algorithm, and Section IV extends the proposed algorithm to stereo deraining. Section V discusses experimental results. Section VI concludes the paper. II. R ELATED W ORK Hase et al. [4] proposed an early video deraining algorithm, which used a temporal median filter to restore pixel values. The median filtering is effective for deraining static scenes, but it may cause blurring artifacts around dynamic objects. Garg and Nayar [5] detected rainy pixels in a frame, which exhibited brighter intensities than the corresponding pixels in adjacent frames. They reduced outliers, falsely estimated as rain streaks, based on the constraints that rain streaks should have similar directions and their colors should be related to the background colors. This approach is physically sound [2], but yields good results only when rain streaks are distinguishable from moving objects. Garg and Nayar [6] also proposed a hardware-based scheme, which suppresses rain streaks by increasing the exposure time of a camera or focusing on objects behind rain streaks. It implicitly performs low-pass filtering, resulting in spatio-temporal blurring, and it may not handle rainy sequences with dynamic objects or diverse scene depths effectively. Zhang et al. [7] removed rain streaks by enforcing the photometric constraints: a pixel color should be dominated by the background color, and the brightness changes of rainy pixels should be similar through an entire sequence. However, they replaced a rainy pixel with a temporal average of background pixels, yielding blurring artifacts. Barnum et al. [8] proposed a deraining algorithm based on frequency analysis. Their algorithm transforms rainy frames into the Fourier domain to detect rain streaks, which tend to have elliptical shapes and appear at similar locations in the frequency domain. It removes rain well even in the case of dynamic scenes, but it cannot remove thick rain streaks properly. Another approach to deraining is based on optical flow estimation [9], [10]. A current frame is first reconstructed by warping an adjacent frame according to the optical flow. In general, an overall scene structure is identical between the current frame and the warped frame, except for rain streak regions. Therefore, a rain-free image can be obtained by replacing pixel values, which are bigger in the current frame than in the warped frame, with those of the warped frame. However, [9], [10] do not consider outliers, such as occluded regions, where the optical flow estimation fails. Kang et al. [3] proposed a single image deraining algorithm, assuming that rain streaks can be represented by a selected set of basis vectors. They decomposed high frequency components in a rainy image into basis vectors via sparse representation. They then clustered the basis vectors into rainy components and structural components. They recovered a rain-free image by keeping the structural components only. Kang et al. [11] also extended the single image algorithm to video deraining. However, their algorithm computes low frequency components by averaging consecutive frames. The resulting high frequency components may not be accurate enough to detect rain streaks in dynamic scenes. Recently, Chen and Chau [12]

2659

Fig. 2. An overview of the proposed algorithm: We obtain an initial rain map from an image frame, which is then refined based on sparse representation and classification. Finally, we reconstruct a rain-free frame by exploiting the information in adjacent frames.

proposed a video deraining algorithm based on background subtraction. Their algorithm decomposes a rainy frame into dynamic objects and background regions using a Gaussian mixture model. Then, it applies spatio-temporal averaging to the objects, while only temporal averaging to the background. It is designed to take into account object motions, but it cannot handle a rainy sequence, captured with a moving camera, properly. III. P ROPOSED A LGORITHM Fig. 2 shows an overview of the proposed algorithm. First, we obtain an initial rain map by computing the difference between a current frame and an optimally warped frame. Second, we decompose the initial rain map using sparse basis vectors, and employ an SVM classifier to dichotomize those vectors into valid ones and outliers. We then reconstruct a refined rain map by employing the valid vectors only. Finally, we replace rainy pixels with rain-free values, by formulating the rain streak removal as a matrix completion problem. A. Initial Rain Detection Figs. 3(a) and (b) show a current frame Ik and its previous frame Ik−1 in the video sequence “Mailbox.” We see that rain streaks appear randomly, and each streak occupies a relatively small area in a frame and moves fast between consecutive frames. Also, when a rain streak passes across a pixel, the pixel value becomes brighter than its original color [5]. We can hence detect a rainy pixel, which has a larger value in a current frame than in adjacent frames [2]. This approach, however, is prone to false detection, since the same pixel coordinates in different frames may not represent the same scene point. A video sequence may contain dynamic objects or be captured with a moving camera. To compensate for these mismatches between consecutive frames, we warp the previous frame into the current frame by estimating the optical flow field. An optical flow estimation algorithm finds a dense motion field between two consecutive image frames [13]–[15]. For each pixel in a reference frame, a motion vector is determined to find the most similar pixel in a target frame, while maintaining the similarity of the motion vectors among neighboring pixels. The optical flow estimation is often formulated as a minimization problem, in which an energy function E is given by E(U) = E d (U) + λE s (U)

(1)

where U is a flow field and λ is a regularization parameter. The data term E d measures the similarity between corresponding

2660

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 9, SEPTEMBER 2015

Fig. 3. Initial rain detection on a frame in the “Mailbox” sequence: (a) current frame, (b) previous frame, (c) warped previous frame, and (d) the difference between (a) and (c).

pixels in the reference frame I1 and the target frame I2 ,    (2) E d (U) = ψ (I2 (x + u(x)) − I1 (x))2 dx where u(x) is the optical flow vector of pixel x, and ψ is a penalty function [13], [14]. The smoothness term E s constrains neighboring vectors to be similar,    (3) E s (U) = ψ ∇u(x)2 dx where ∇u(x) denotes the gradient of u(x). In this work, we discretize the energy function in (1) and estimate the optical flow field between color image frames using the two-frame version of the Liu’s algorithm [15], which yields comparable performance to a recent state-ofthe-art method [16]. We set λ =√1 in (1), and employ the same penalty function ψ(x) = x 2 +  2 with  = 0.001 in (2) and (3). Instead of the optical flow estimation, we may use a patch-based motion estimation scheme. However, a patch-based scheme may provide unreliable motion vectors, when patches contain rain streaks. We adopt the optical flow estimation algorithm [15], since the smoothness term E s in the energy function in (1) alleviates the effects of rain streaks on the motion (or optical flow) estimation. prev Fig. 3(c) shows the warped previous frame I˜k , which is obtained by prev prev I˜k (x) = Ik−1 (x + uk (x))

(4)

prev where uk (x) Ik to Ik−1 . Due

is the optical flow vector at pixel x from to the smoothness term in (3), the estimated optical flow field is not significantly affected by small rain streaks and can compensate for camera motions reliably. Fig. 3(d) is the difference image between the current frame and the warped previous frame, which estimates rain streak locations roughly. When an object appears or disappears between two consecutive frames, the warped frame may contain occluded areas, which cause false detection of rainy pixels. To obtain a more reliable result, we employ the next frame Ik+1 as well and synthesize the warped next frame I˜knext (x) = Ik+1 (x + uknext (x))

(5)

where uknext (x) is the optical flow vector at x from Ik to Ik+1 . We obtain an optimally warped pixel value I˜k (x) by prev selecting either the forwardly warped pixel I˜k (x) or the

backwardly warped pixel I˜knext (x). We attempt to select what is more similar to the original pixel value Ik (x) in the current frame. This selection can be represented as the labelling, which assigns a binary label l(x) to each pixel x such that  prev I˜ (x), if l(x) = 0, (6) I˜k (x) = knext I˜k (x), if l(x) = 1. Let L denote the label map, which is composed of the labels of all pixels in the current frame. We determine L by minimizing the labeling cost C(L) = Cd (L) + τ Cs (L)

(7)

where τ = 50 is a regularization parameter. Cd (L) is the data cost to measure the similarity between the hybrid warped frame and the current frame,  (Ik (x) − I˜k (x))2 . (8) Cd (L) = x

Cs (L) is the smoothness cost to constrain the labels of neighboring pixels to be same, which is defined as   Cs (L) = (9) (l(x) ⊕ l(y)) x y∈Nx

where Nx is the set of the 4-connected neighbors to x, and ⊕ is the exclusive-or (XOR) operator. We employ the graph cut technique [17], [18] to minimize the cost function in (7). Fig. 4 shows an example of a hybrid warped frame I˜k . The man’s eyes are not correctly reconstructed in the warped prev in Fig. 4(b), whereas his right side is previous frame I˜k distorted in the warped next frame I˜knext in Fig. 4(c). We obtain the hybrid frame I˜k with a higher quality in Fig. 4(d), by selecting pixels adaptively according to the label map L in Fig. 4(e). Finally, we obtain an initial rain map R as the difference image between the current frame Ik and the hybrid warped frame I˜k ,   R(x) = max Ik (x) − I˜k (x), 0 (10) where negative differences are truncated to 0, since rainy pixels are assumed to be brighter than their original colors. Note that we use only the luminance components of the initial rain map in the following steps of the rain map refinement and the rain mask generation.

KIM et al.: VIDEO DERAINING AND DESNOWING USING TEMPORAL CORRELATION AND LOW-RANK MATRIX COMPLETION

2661

from valid rain streaks. Given an initial rain map R with n pixels, we consider a patch of size m around each pixel, and construct a matrix R of size m × n, in which each column represents each patch. Then, we construct an over-complete dictionary D ∈ Rm× p ( p > m), composed of p basis vectors of dimension m. By representing each patch as a linear combination of the basis vectors in D, the matrix R can be reconstructed to minimize the distortion R − DA

(11)

where A is the coefficient matrix of size p × n. Assuming that A should be sparse, we find the optimal coefficient matrix A∗ by solving the optimization problem [21]

1 ∗ 2 R − DA2 + ρA1 . A = arg min (12) A∈R p×n 2

Fig. 4. Hybrid warping: (a) current frame Ik , (b) warped previous prev frame I˜k , (c) warped next frame I˜knext , (d) hybrid warped frame I˜k , and (e) label map L, where gray and blue pixels indicate the selection of the warped previous and next frames, respectively. Incorrectly warped regions are highlighted in red and blue in (b) and (c). (original images courtesy of Fuji Television Network – ©1995 Fuji Television Network).

B. Rain Map Refinement We can obtain a binary rain mask by thresholding an initial rain map. Fig. 5 shows an initial rain map and the resultant binary rain masks using different thresholds. Note that the binary rain masks contain falsely detected outliers, caused by warping errors or brightness changes between frames, or fail to detect some valid rain streaks. In general, as we decrease the threshold, many outliers are falsely detected as rain streaks. On the contrary, as we increase the threshold, we can reduce such outliers but also miss valid rain streaks. Moreover, outliers often overlap with valid rain streaks. Therefore, to detect valid rain streaks reliably while suppressing outliers, we refine an initial rain map before the thresholding. To refine an initial rain map, we exploit the directional property of rain streaks: rain streaks tend to have elliptical shapes, whose major axes deviate little from the vertical direction. In contrast, falsely detected outliers have arbitrary shapes or yield random directions of major axes. Therefore, we can find outliers by comparing the horizontal components with the vertical components of detected ellipses. However, eliminating elliptical regions with large horizontal components may miss actual rainy pixels, since rain streaks and outliers occur simultaneously and may overlap each other. The morphological component analysis (MCA) [19], [20] decomposes a given signal into basis vectors based on sparse representation, and then reconstructs the signal by employing selected basis vectors only. Note that this approach was adopted in [3] and [11] to decompose an edge map, which represents high frequency information in an image, into rain-free components and rainy components. In this work, we also adopt the sparse representation technique [21], which represents a signal with a linear combination of a small number of basis vectors, in order to erase overlapped outliers

We set ρ to 0.15 as suggested in [22]. We also use the learning algorithm in [22] to construct the over-complete dictionary D, which performs the stochastic approximation to find an optimal set of basis vectors in the dictionary. Then, we apply the orthogonal matching pursuit [23] to find the sparse coefficient A∗ , which chooses basis vectors repeatedly to yield the biggest inner product with the residual signal. Notice that we construct a dictionary for each initial rain map, which is associated with each frame in a rainy sequence. Fig. 6(a) shows the basis vectors in a dictionary, extracted from an initial rain map for a frame in the “Mailbox” sequence. The basis vectors have the dimension m = 256, and are visualized as 16 × 16 patches. They have diverse directional structures. We classify the basis vectors in the dictionary into rain streak ones and outlier ones, and use only the rain streak ones to reconstruct a rain-free image. We adopt the kernel regression method [24] to analyze patch structures, which finds the best matching kernel to the intensity distribution in each patch based on the singular value decomposition (SVD). In general, rain streak vectors have vertically elongated elliptical kernels, whereas outliers have noisy patterns. Therefore, we can distinguish rain streak vectors from outliers by employing the angular components and eigenvalues of kernels. Specifically, given a patch Wi corresponding to the i th basis vector in D, we compute the covariance matrix Ci using the gradients of pixel values, given by  1  gh2 (x) gh (x)gv (x) (13) Ci = gh (x)gv (x) gv2 (x) |Wi | x∈Wi

where gh (x) and gv (x) are the gradient values at pixel x along the horizontal and vertical directions, respectively, and |Wi | is the size of Wi . We analyze the structural shape and orientation of the kernel for Wi , by applying the SVD to the covariance matrix    T cos θi − sin θi μi 0 cos θi − sin θi Ci = 0 νi sin θi cos θi sin θi cos θi (14) where θi is the rotational angle, and μi and νi are the two eigenvalues representing the kernel scales along the major and minor axes, respectively.

2662

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 9, SEPTEMBER 2015

Fig. 5. An initial rain map and its binarized rain masks with three different thresholds on the “Mailbox” sequence: (a) input frame, (b) initial rain map, and binary rain masks using thresholds of (c) 3, (d) 5, and (e) 10, respectively. In (b), small values are depicted in dark blue, and large values in red.

Fig. 8. Training examples to construct negative samples of the SVM, selected from the Trace dataset [26]. Left images are exemplar frames and right images are the corresponding difference images. The difference images are shown in color: light yellow indicates a low value, whereas purple means a high value. Fig. 6. Visualization of the basis vectors, extracted from the initial rain map of the third frame in the “Mailbox” sequence: (a) the basis vectors of the initial rain map and (b) the selective basis vectors for valid rain streaks.

Fig. 7. Training examples to construct positive samples of the SVM, which are synthesized with ‘Adobe After Effect ’ [27]. The synthesized rain streak images are in gray-level, but shown in color for better visibility: light yellow indicates a low value, whereas purple means a high value.

To separate valid rain streak vectors from outliers, we employ an SVM classifier [25] by taking the rotational angle θi and the two eigenvalues μi and νi in (14) as a feature vector. An SVM determines the maximum-margin hyperplane to separate input data into two classes. For training the SVM, we use 3,072 positive samples of valid basis vectors from synthetic rainy frames. We also generate 3,072 negative samples of outlier vectors by taking the difference between a frame and its warped adjacent frame in a rain-free sequence, which is randomly selected from the Trace dataset [26]. Fig. 7 and Fig. 8 show some training rain maps for constructing positive and negative samples. Note that we may attempt to differentiate valid vectors from outliers by simply thresholding the rotational angle and the two eigenvalues. However, it is not trivial to find optimal

thresholds for these parameters, and the optimal thresholds vary according to input sequences. We confirmed experimentally that the SVM classification outperforms the simple thresholding with various choices of thresholds. Moreover, the computational complexity of the SVM classification is much lower than that of the SVM training. It is worthy to point out that the training is performed only once offline. After the SVM classification, we replace all outlier vectors in the original dictionary D with the zero vector to get a ˆ Fig. 6(b) shows the new dictionary, which new dictionary D. eliminates most irregular outliers. Next, we refine the initial ˆ To this end, we rain map R into a more accurate rain map R. ˆ ˆ reconstruct the matrix R by multiplying the new dictionary D ∗ with the coefficient matrix A in (12), ˆ = DA ˆ ∗. R

(15)

ˆ We overlay each patch, corresponding to a column vector in R, ˆ Then, we set each pixel around the corresponding pixel in R. value in Rˆ to be the average of the overlapped values. Finally, we generate a binary rain mask M from the refined ˆ given by rain map R,  ˆ 1 if R(x) > ξ, M(x) = (16) 0 otherwise, where the threshold ξ = 3 in the default setting, which was shown to yield reliable performance on most rainy sequences [2]. Fig. 9 compares the initial rain map and the refined rain map, where the refined map suppresses most outliers in the initial map effectively. From the binary rain mask M, we further remove outliers by erasing small connected components of size 5 or less. Then, we perform the dilation operation on the rain mask, since the refinement procedure may distort the boundaries of rain streaks.

KIM et al.: VIDEO DERAINING AND DESNOWING USING TEMPORAL CORRELATION AND LOW-RANK MATRIX COMPLETION

2663

Algorithm 1 EM-Based Rain Streak Removal

values should be smaller than rainy pixel values. Thus, we have the constrained matrix completion problem:

Fig. 9. Rain map refinement on the third frame in the “Mailbox” sequence: (a) initial rain map, (b) refined rain map, (c) binary mask of the initial rain map, and (d) binary mask of the refined rain map. In (a) and (b), small values are depicted in dark blue, and large values in red.

C. Rain Streak Removal We restore pixel values, detected as rainy, by exploiting temporal redundancies in adjacent frames. Specifically, we formulate the rain removal as a low-rank matrix completion problem. We first partition the current frame Ik into disjoint blocks. For each block b, we search for the l most similar blocks from each of the four adjacent frames: Ik−2 , Ik−1 , Ik+1 , Ik+2 . Notice that we do not find similar blocks from the current frame. This is because similar blocks in the current frame tend to be selected near the given block b and affected by the same rain streak, degrading the deraining performance. To measure the similarity between two blocks, we compute the sum of the squared differences between rain-free pixels only. Then, we define a matrix B, by concatenating the given block b in the current frame and its 4l most similar blocks bi ’s in the adjacent frames, B = [b, b1 , b2 , ..., b4l ]

(17)

where each block is represented by a column vector. Note that we subtract the mean value from each block to compensate for the differences in illumination. We also define the binary rain mask matrix M for B, given by M = [m, m1 , m2 , ..., m4l ],

(18)

which consists of the corresponding mask values in (16). We adopt the low-rank matrix completion techniques [28]–[30] to find a filled-in matrix X from the incomplete matrix B. The desired matrix X should minimize the nuclear norm X∗ subject to a constraint (1 − M) ◦ X = (1 − M) ◦ B

(19)

where 1 is the matrix whose all elements are 1, and ◦ denotes the element-wise multiplication of two matrices. The constraint in (19) means that X should retain the non-rainy pixel values in B. We also use another constraint that rain-free pixel

minimize X∗ subject to (1 − M) ◦ X = (1 − M) ◦ B,

(20) (21)

M ◦ X ≤ M ◦ B,

(22)

where ≤ denotes the element-wise inequality. To solve this constrained optimization problem, we use the expectation maximization (EM) algorithm in [30] and [31]. In the expectation step of the tth iteration in the EM framework, we compose a filled-in matrix X(t ) by taking the elements of the input matrix B at rain-free pixels and taking the elements of the current estimate Y(t ) at rainy pixels, i.e. X(t ) = (1 − M) ◦ B + M ◦ (Y(t ) ∧ B).

(23)

Here, ∧ denotes the element-wise minimum operator to return the minimum between the two elements of compared matrices. In (23), we use the elements of (Y(t ) ∧ B), instead of Y(t ), due to the constraint in (22) that rain streaks make the pixel values brighter. In the maximization step, we update the estimator Y(t ) to be a low-rank approximation of the filled-in matrix X(t ). This is because we attempt to minimize the nuclear norm X∗ in (20), which is the sum of the singular values of X. To this end, we first perform the SVD X(t ) = U VT

(24)

where U is a rotation matrix and is a diagonal matrix, composed of the singular values. Then, we update the estimator Y(t +1) as a low rank approximation of X(t ), given by Y(t +1) = UHκ ( )VT .

(25)

Here, the operator Hκ truncates each singular value in to 0, if it is smaller than a threshold κ = 2000. We repeat the expectation step in (23) and the maximization step in (25) iteratively, until satisfying the stopping criteria that Y(t ) − Y(t +1)M <  or t = tmax . Here,  · M denotes the mean absolute difference between pixel values. Also,  is set to 0.05, and the maximum number of iterations tmax is set to 150. Finally, we replace the rainy pixel values in the input block b with the corresponding elements of the resultant filledˆ The overall procedure of the proposed EM-based in matrix X. rain streak removal algorithm is summarized in Algorithm 1.

2664

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 9, SEPTEMBER 2015

Fig. 10. Iterative removal of rain streaks on a 32 × 32 block: (a) input block, (b) 1st iteration, (c) 2nd iteration, (d) 5th iteration, (e) 10th iteration, and (f) 70th iteration (converged).

Fig. 11. Iterative removal of rain streaks on the 3th frame in the “Highway” sequence: (a) input frame, (b) 1st iteration, (c) 5th iteration, (d) 10th iteration, and (e) convergence. Also, (f) shows the number of iterations that are required for the convergence of the algorithm on each block.

Fig. 10 illustrates how the proposed algorithm removes rain streaks on a block of size 32×32. The EM algorithm faithfully recovers rain-free pixel values, as the iteration goes on, even though one of the rain streaks is quite thick. In this test, the proposed algorithm converges after 70 iterations. Fig. 11 shows the iterative removal of rain streaks on an entire frame, which is decomposed into blocks of size 32×32. After several iterations, the proposed algorithm derains most rain streaks effectively. Fig. 11(f) shows the required number of iterations for the convergence of each block. More than 90% of blocks require less than 10 iterations for the convergence. In rare cases, Algorithm 1 does not converge and Y(t ) oscillates. In such cases, we enforce Algorithm 1 to stop using the condition of t = tmax . This mandatory stopping condition is met for about 0.19% of blocks only. D. Postprocessing Even after the rain streak removal, thin rain streaks, which are not detected in the rain map in (16), may still remain in a derained frame. Various denoising techniques, such as the total variation based denoising in [32], can be applied to remove such thin rain streaks. However, in this work, we employ a simple blurring scheme. The human visual system is more sensitive to thin rain streaks in smooth regions with little texture than to those in highly textured regions. Thus, we blur thin rain streaks in smooth blocks only. We first select smooth blocks in a derained frame, whose standard deviations of pixel values are less than a pre-defined threshold δ. Let bk be a smooth block in the kth derained frame. We blur it to obtain

3 i=0 wi bk−i bˆ k = (26) 3 i=0 wi where bk−i is the most similar block to bk in the previously derained (k − i )th frame. The weight wi is determined by   bk − bk−i 2 wi = exp − (27) σ 2q

Fig. 12. Disparities of rain streaks and background objects [33]: the projected pixels pleft and pright of a rain streak point P exhibit a larger disparity than those q left and qright of a background point Q.

where q is the number of pixels in bk and σ 2 is a Gaussian parameter. In this work, δ = σ 2 = 15. We perform the blurring at the block level, rather than at the pixel level, to reduce the complexity, but we overlap blocks partially to alleviate blocking artifacts. IV. S TEREOSCOPIC V IDEO D ERAINING The proposed algorithm exploits temporal correlation of adjacent image frames to detect and remove rain streaks. Since a pair of stereo images contain spatial correlation as well, we extend the proposed monoscopic video deraining algorithm to stereo video deraining straightforwardly. In [33], we proposed a preliminary work on stereo video deraining. For simplicity, we describe the algorithm to derain a left-view frame only. A right-view frame is derained similarly.

KIM et al.: VIDEO DERAINING AND DESNOWING USING TEMPORAL CORRELATION AND LOW-RANK MATRIX COMPLETION

2665

Fig. 13. Result of the proposed deraining algorithm on the 5th frame in the “University” sequence: (a) input frame, (b) derained frame, and (c) refined rain map. The refined rain map is shown in color for better visibility.

A pair of stereo images, i.e. left-view and right-view frames, are captured at the same time but from different viewpoints. When a rain streak in front of an object is captured by stereo cameras, its projected pixels on the two image planes have a greater disparity than those of the object. In Fig. 12, point P on a rain streak is closer to the cameras than point Q on an object. These points are projected onto two image planes, and the disparity between the projected pixels pleft and pright of P is bigger than that between qleft and qright of Q. Similar to the temporal property that rain streaks do not affect the optical flow estimation significantly because of the smoothness constraint, we assume that rain streaks are too small to hinder the disparity estimation for stereo video sequences. Hence we also apply the optical flow estimation algorithm [15] for the disparity estimation. Let Ik and Jk denote the kth frames in the left-view and the right-view, respectively, of a stereo sequence. The synthesized left-view right frame I˜k , warped from the right-view frame Jk , is given by right I˜k (x)

= Jk (x

right + uk (x))

(28)

right

where uk (x) is the disparity vector at pixel x from Ik to Jk . right I˜k is reconstructed mainly by the disparities of dominant background objects, rather than by those of small rain streaks. Consequently, in the stereo video deraining, we generate a hybrid warped frame I˜k using the three spatio-temporally right warped frames: warped right frame I˜k , warped previous prev frame I˜k , and warped next frame I˜knext . We add a new label l(x) = 2 to denote the warped right frame, and obtain a ternary label map L that minimizes the energy in (7). Then the hybrid warped frame is given by ⎧ prev ⎪ ⎨I˜k (x) if l(x) = 0, (29) I˜k (x) = I˜knext (x) if l(x) = 1, ⎪ ⎩ ˜right Ik (x) if l(x) = 2. The remaining procedures of the rain streak detection and the rain map refinement are identical with the monoscopic video deraining algorithm. However, for the rain streak removal, we exploit blocks not only in the temporally adjacent frames but also in the right-view frame in the matrix completion problem. V. E XPERIMENTAL R ESULTS We evaluate the performance of the proposed algorithm on various video sequences. More results are available

Fig. 14. Results of the proposed deraining algorithm on the (a) 55th and (b) 85th frames in the “Sing in the Rain” sequence (original images courtesy of MGM – ©1952 MGM, Inc.). The top row shows the input frames, and the bottom row shows the derained frames.

as supplemental materials in our website.1 We use the Liu’s algorithm [15] for the optical flow estimation and the Mairal et al.’s algorithm [22] to learn the over-complete dictionary. For the SVM, we use the LIBSVM library [25] and set a linear kernel to discriminate outliers. We exploit the PROPACK toolkit [34] to reduce the computational complexity of the SVD in the rain removal process. We fix the threshold ξ in (16) to 3 for monocular video deraining. However, for stereo deraining, we set ξ = 12 since those scenes are captured in heavy snow. In (17), we set the block size to 32 ×32, since smaller blocks cannot handle thick rain streaks efficiently, and set and l to 5. In (26), we fix the block size to 20 × 20. The search range to find similar blocks for the rain streak removal and the post processing is fixed to [−12, 12] ×[−12, 12]. Also, in the post processing, we set the overlap size to the half of the block size. A. Performance on Natural Video Sequences Fig. 13 shows the deraining result on the 5th frame in the “University” sequence. Rain streaks of various sizes are detected reliably with few outliers and removed faithfully by the proposed algorithm. Fig. 14 shows the deraining results on 1 http://mcl.korea.ac.kr/deraining

2666

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 9, SEPTEMBER 2015

Fig. 15. Deraining and desnowing results on the films (a) “Four Weddings and a Funeral” (original image courtesy of PolyGram Filmed Entertainment – ©1994 PolyGram Filmed Entertainment, Inc.) and (b) “Love Letter” (original image courtesy of Fuji Television Network – ©1995 Fuji Television Network). The top row shows input frames, and the bottom row shows their deraining or desnowing results.

Fig. 17. Comparison of the desnowing performance on the 35th frame in the “Windowed Building” sequence. The input snowy frame in (a) is processed by (b) Zhang et al. [7], (c) Kang et al. [11], (d) Garg and Nayar [2], (e) Barnum et al. [8], and (f) the proposed algorithm, respectively.

Fig. 16. Deraining result on the “Australia” sequence with heavy rain (original image courtesy of 20th Century Fox – ©2008 20th Century Fox,Inc.): (a) input frame, (b) rain mask, and (c) derained frame.

two selected frames in the “Sing in the Rain” sequence. This sequence is challenging since the camera zooms in gradually toward the man. However, we see that the proposed algorithm successfully removes most rain streaks. Fig. 15 shows more results. Fig. 16 shows the deraining result on the “Australia” sequence with heavier rain. We observe that some rain streaks are not removed well in the derained frame. This is because similar patches found in adjacent frames also contain heavy rain streaks.

Note that the sequences in Figs. 1, 4, 14, 15, and 16 are from films. Except for “Australia” (2008) in Fig. 16, these films were released before 1995. We hence assume that the rain or snow streaks in these test sequences are natural, even though computer graphics techniques are used to render realistic rain or snow streaks in recent films or animations. We also compare the proposed algorithm with conventional algorithms: Zhang et al. [7], Kang et al. [11], Garg and Nayar [2], and Barnum et al. [8]. Fig. 17 compares the desnowing results on the “Windowed building” sequence. The Zhang et al.’s algorithm only considers static scenes without camera motions, and thus requires the stabilization to align frames. However, the stabilization may not warp the frames properly, which causes black regions in the left side in Fig. 17(b). Furthermore, their algorithm fails to distinguish snow streaks from the window frames due to the misalignment of warped frames. The Kang et al.’s algorithm obtains low frequency components by averaging consecutive frames. Therefore, it cannot remove thick snow streaks effectively in Fig. 17(c). The Garg and Nayar’s algorithm also assumes static scenes and thus detects the window frames as snow streaks, which are blurred with the background in Fig. 17(d). The Barnum et al.’s algorithm is based on the frequency analysis of consecutive frames to estimate snow streaks. However, in this example, snow streaks have various sizes according to their distances from the camera, and thus some small streaks

KIM et al.: VIDEO DERAINING AND DESNOWING USING TEMPORAL CORRELATION AND LOW-RANK MATRIX COMPLETION

2667

Fig. 20. Comparison of the rain removal results on the “Highway” sequence using different matrix completion schemes. An input frame in (a) is derained using (b) the soft shrinkage algorithm [29], (c) the damped Wiberg algorithm [28], and (d) the proposed EM-based algorithm.

Fig. 18. Comparison of the desnowing performance on the 34th frame in the “Mailbox” sequence. The input snowy frame in (a) is processed by (b) Zhang et al. [7], (c) Kang et al. [11], (d) Garg and Nayar [2], (e) Barnum et al. [8], and (f) the proposed algorithm, respectively.

the Barnum et al.’s algorithm provides a more plausible result than the previous algorithms, but still retains some thick streaks on the upper region of the mail box. In Fig. 18(f), the proposed algorithm achieves the most reliable desnowing performance. B. Performance on Synthetic Video Sequences

Fig. 19. Results of the proposed deraining algorithm on the synthetic rainy video sequences (a) “Port” and (b) “Golf.” The top row shows input frames, and the second row shows the corresponding derained frames.

are not removed completely in Fig. 17(e). On the other hand, in Fig. 17(f), the proposed algorithm effectively removes both thick and thin streaks faithfully, while preserving the overall scene structure including the window frames. Fig. 18 shows the comparison results on the “Mailbox” sequence, which exhibits dynamic motions. In Fig. 18(b), the Zhang et al.’s algorithm removes snow streaks well, but it removes important scene structures as well, especially the details of the mailbox. In Fig. 18(c), The Kang et al.’s algorithm fails to estimate low frequency components, and produces a severely distorted image. In Fig. 18(d), the Garg and Nayar’s algorithm also fails to remove snow streaks due to the dynamic motions. In Fig. 18(e),

Next, we quantitatively evaluate the performance of the proposed algorithm using nine synthetic rainy video sequences: “City,” “Coastguard,” and “Stefan” are captured with moving cameras, whereas “Pedestrian,” “Port,” “Niagara,” “Highway,” “Golf,” and “Tennis” are captured with stationary ones. We use ‘Adobe After Effect ’ [27] to synthesize rain streaks. Then, we perform the deraining algorithm on the synthesized rainy sequences, and measure the peak signal to noise ratio (PSNR) between original and derained images. Table I shows the results, where each PSNR value is the average over the first 100 frames in each test sequence. On all the test sequences, the proposed algorithm provides significantly higher PSNR’s than the conventional algorithms. On the “City,” “Coastguard,” and “Stefan” sequences, the Garg and Nayar’s algorithm, which does not consider dynamic motions, yields even lower PSNR’s than those of input rainy sequences. Moreover, the PSNR’s of the Kang et al.’s algorithm are always less than those of input rainy sequences. It is because Kang et al. averages temporally adjacent frames to compute the low frequency component of a current frame, which is not effective for sequences with dynamic objects or dynamic camera motions. Table I also compares the deraining performance of the proposed algorithm without the rain map refinement. An initial rain map detects larger rainy regions, due to outliers, than the corresponding refined rain map. These outliers degrade the deraining performance by about 2.8dB on average. In other words, the rain map refinement is an essential step for providing high quality deraining results.

2668

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 9, SEPTEMBER 2015

TABLE I PSNR C OMPARISON OF THE G ARG AND N AYAR ’ S A LGORITHM [2], THE K ANG et al.’ S A LGORITHM [11], AND THE P ROPOSED A LGORITHM ON S YNTHETIC R AINY S EQUENCES . T HE U NIT I S D ECIBEL . T HE H IGHEST PSNR FOR E ACH S EQUENCE I S H IGHLIGHTED IN B OLDFACE

Fig. 21. Rain removal performance comparison of the proposed algorithm and the inpainting algorithms on the “City” and “Highway” sequences. Each input frame in (a) is derained using (b) the total variation inpainting [32], (c) the exemplar-based inpainting [35], and (d) the proposed algorithm.

Fig. 19 shows sample frames of the synthetic rainy sequences and their derained results. We see that the proposed algorithm detects and removes rain streaks effectively, and reconstructs the original scene contents faithfully. C. Comparison of Matrix Completion Techniques We compare the rain removal performance of the proposed EM-based matrix completion algorithm with those of different matrix completion algorithms: the soft shrinkage algorithm [29] and the damped Wiberg algorithm [28]. These algorithms can fill in randomly distributed missing entries in an incomplete matrix efficiently, but their results often degrade when missing entries are locally clustered as in the rain removal problem. Fig. 20 compares rain removal results on the synthetic rainy sequence “Highway.” In Fig. 20(b), the soft shrinkage algorithm fails to recover some rainy pixels. In Fig. 20(c), the damped Wiberg algorithm yields a better result, but requires too much memory space to complete matrices. In this test, we use the rank-1 approximation for the damped Wiberg algorithm due to the memory constraint. In Fig. 20(f), the proposed EM-based algorithm achieves better performance than the conventional algorithms.

Fig. 22. Desnowing results on the (a) “Wall” and (b) “Glasswall” sequences. Input left-view frames (top) and their desnowing results (bottom).

D. Comparison With Inpainting Techniques In addition, we compare the rain removal performance of the proposed algorithm with those of image inpainting techniques: total variation inpainting [32] and exemplar-based inpainting [35]. As an image inpainting technique, [35] fills

KIM et al.: VIDEO DERAINING AND DESNOWING USING TEMPORAL CORRELATION AND LOW-RANK MATRIX COMPLETION

2669

Fig. 23. Comparison of the desnowing results on the left-view frames of the stereo video sequences “Car” and “Tree.” Input frames in (a) are desnowed by employing (b) temporal frames only, (c) right-view frames only, and (d) temporal and right-view frames together. TABLE II T HE AVERAGE E XECUTION T IME OF THE P ROPOSED A LGORITHM FOR D ERAINING A CIF (352 × 288) OR nHD (640 × 360) F RAME . T HE U NIT I S S ECOND

in rain streak regions by finding replacing patches within the same image. However, we straightforwardly extend it to find patches in four adjacent frames, as well as in a current frame. Fig. 21 compares rain removal results on the “City” and “Highway” sequences. In Fig. 21(b), the total variation inpainting fails to remove large rain streaks properly, e.g. the right side fence in the “Highway” sequence, and yields blurring artifacts. In Fig. 21(c), the exemplar-based inpainting causes artifacts around textured regions, since it uses only a single patch to replace a rainy pixel. In Fig. 21(d), the proposed algorithm provides higher quality results than these inpainting algorithms, based on the matrix completion. E. Stereo Deraining While there are several deraining algorithms for monocular video sequences, little attempt has been made to remove rain streaks from stereo scenes. We evaluate the performance of the proposed algorithm on snowy stereo video sequences, which are captured with a commercial stereo digital camera, ‘Fujifilm Real-3D .’ Fig. 22 shows only the desnowed left-view frames on the “Wall” and “Glasswall” sequences, since the right-view frames are desnowed similarly as well. Most streaks, including even thin ones, are effectively removed. However, some snow streaks still remain, which are regarded as geometrical structures in the optical flow estimation. Next, Fig. 23 compares the stereo desnowing performances on the left-view frames in the “Cars” and “Tree” sequences, when we use temporal frames only, right-view frames only, and temporal and right-view frames together for the snow

streak detection and removal. Using either temporal or right-view frames provides similar performances. However, the best results are achieved by using both temporal and right-view frames together. F. Execution Time We analyze the execution time of the proposed algorithm in Table II. We use a PC with a 3.4 GHz CPU and 8 GB RAM. The proposed algorithm is implemented in Matlab without code optimization. Note that the sparse representation for the rain map refinement and the block matching for the rain streak removal spend most of the execution time. Such high complexity could be reduced by approximating the sparse representation and employing a more efficient block matching algorithm. This is a future research issue. VI. C ONCLUSIONS We proposed a video deraining algorithm, which exploits temporal correlation in a video sequence based the low-rank matrix completion. The proposed algorithm obtains an initial rain map, by warping previous and next frames and comparing those warped frame with a current frame. It then refines the initial rain map by removing outliers based on the sparse representation and the classification. Finally, the proposed algorithm fills in rainy pixels using the EM-based low-rank matrix completion. We also extended the proposed algorithm to stereo video deraining. Extensive experimental results demonstrated that the proposed algorithm removes rain and snow streaks more efficiently, while preserving scene structures more faithfully, than the conventional algorithms.

2670

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 9, SEPTEMBER 2015

R EFERENCES [1] S. G. Narasimhan and S. K. Nayar, “Vision and the atmosphere,” Int. J. Comput. Vis., vol. 48, no. 3, pp. 233–254, Jul. 2002. [2] K. Garg and S. K. Nayar, “Vision and rain,” Int. J. Comput. Vis., vol. 75, no. 1, pp. 3–27, Oct. 2007. [3] L.-W. Kang, C.-W. Lin, and Y.-H. Fu, “Automatic single-image-based rain streaks removal via image decomposition,” IEEE Trans. Image Process., vol. 21, no. 4, pp. 1742–1755, Apr. 2012. [4] H. Hase, K. Miyake, and M. Yoneda, “Real-time snowfall noise elimination,” in Proc. IEEE ICIP, Oct. 1999, pp. 406–409. [5] K. Garg and S. K. Nayar, “Detection and removal of rain from videos,” in Proc. IEEE Comput. Soc. Conf. CVPR, Jun./Jul. 2004, pp. I-528–I-535. [6] K. Garg and S. K. Nayar, “When does a camera see rain?” in Proc. 10th IEEE ICCV, Oct. 2005, pp. 1067–1074. [7] X. Zhang, H. Li, Y. Qi, W. K. Leow, and T. K. Ng, “Rain removal in video by combining temporal and chromatic properties,” in Proc. IEEE ICME, Jul. 2006, pp. 461–464. [8] P. C. Barnum, S. Narasimhan, and T. Kanade, “Analysis of rain and snow in frequency space,” Int. J. Comput. Vis., vol. 86, nos. 2–3, pp. 256–274, Jan. 2010. [9] M. Shen and P. Xue, “A fast algorithm for rain detection and removal from videos,” in Proc. IEEE ICME, Jul. 2011, pp. 1–6. [10] V. Santhaseelan and V. K. Asari, “A phase space approach for detection and removal of rain in video,” Proc. SPIE, vol. 8301, p. 830114, Jan. 2012. [11] L.-W. Kang, C.-W. Lin, C.-T. Lin, and Y.-C. Lin, “Self-learning-based rain streak removal for image/video,” in Proc. IEEE ISCAS, May 2012, pp. 1871–1874. [12] J. Chen and L.-P. Chau, “A rain pixel recovery algorithm for videos with highly dynamic scenes,” IEEE Trans. Image Process., vol. 23, no. 3, pp. 1097–1104, Mar. 2014. [13] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” in Proc. 8th ECCV, May 2004, pp. 25–36. [14] A. Bruhn, J. Weickert, and C. Schnörr, “Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods,” Int. J. Comput. Vis., vol. 61, no. 3, pp. 211–231, Feb. 2005. [15] C. Liu, “Beyond pixels: Exploring new representations and applications for motion analysis,” Ph.D. dissertation, Dept. Elect. Eng. Comput. Sci., Massachusetts Inst. Technol., Cambridge, MA, USA, May 2009. [16] L. Xu, J. Jia, and Y. Matsushita, “Motion detail preserving optical flow estimation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 9, pp. 1744–1757, Sep. 2012. [17] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 11, pp. 1222–1239, Nov. 2001. [18] V. Kolmogorov and R. Zabin, “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 2, pp. 147–159, Feb. 2004. [19] J.-L. Starck, M. Elad, and D. L. Donoho, “Image decomposition via the combination of sparse representations and a variational approach,” IEEE Trans. Image Process., vol. 14, no. 10, pp. 1570–1582, Oct. 2005. [20] M. J. Fadili, J.-L. Starck, J. Bobin, and Y. Moudden, “Image decomposition and separation using sparse representations: An overview,” Proc. IEEE, vol. 98, no. 6, pp. 983–994, Jun. 2010. [21] M. Elad, Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. New York, NY, USA: Springer-Verlag, 2010. [22] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Mach. Learn. Res., vol. 11, pp. 19–60, Mar. 2010. [23] S. G. Mallat and Z. Zhang, “Matching pursuits with timefrequency dictionaries,” IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397–3415, Dec. 1993. [24] H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image processing and reconstruction,” IEEE Trans. Image Process., vol. 16, no. 2, pp. 349–366, Feb. 2007. [25] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, Apr. 2011, Art. ID 27. [26] Video Trace Library. [Online]. Available: http://trace.eas.asu.edu/, accessed May 7, 2015. [27] After Effects CS6, Adobe Systems Inc., San Jose, CA, USA, 2012.

[28] T. Okatani, T. Yoshida, and K. Deguchi, “Efficient algorithm for low-rank matrix factorization with missing components and performance comparison of latest algorithms,” in Proc. IEEE ICCV, Nov. 2011, pp. 842–849. [Online]. Available: http://www.vision.is. tohoku.ac.jp/us/download/ [29] J.-F. Cai, E. J. Candès, and Z. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM J. Optim., vol. 20, no. 4, pp. 1956–1982, 2010. [Online]. Available: http://svt.stanford.edu/ [30] N. Srebro and T. Jaakkola, “Weighted low-rank approximations,” in Proc. 10th Int. Conf. Mach. Learn., Aug. 2003, pp. 720–727. [31] S. Zhang, W. Wang, F. Makedon, and J. Ford, “Learning from incomplete ratings using non-negative matrix factorization,” in Proc. SIAM Int. Conf. Data Mining, Apr. 2006, pp. 549–553. [32] A. Chambolle and T. Pock, “A first-order primal-dual algorithm for convex problems with applications to imaging,” J. Math. Imag. Vis., vol. 40, no. 1, pp. 120–145, May 2011. [33] J.-H. Kim, J.-Y. Sim, and C.-S. Kim, “Stereo video deraining and desnowing based on spatiotemporal frame warping,” in Proc. IEEE ICIP, Oct. 2014, pp. 5432–5436. [34] PROPACK. [Online]. Available: http://sun.stanford.edu/~rmunk/PROPACK/, accessed May 7, 2015. [35] A. Criminisi, P. Pérez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200–1212, Sep. 2004.

Jin-Hwan Kim (S’12) received the B.S., M.S., and Ph.D. degrees in electrical engineering from Korea University, Seoul, Korea, in 2008, 2010, and 2015, respectively. He is currently with the Defense Agency for Technology and Quality. His research interests include image enhancement, contrast enhancement, and deweathering.

Jae-Young Sim (S’02–M’06) received the B.S. degree in electrical engineering and the M.S. and Ph.D. degrees in electrical engineering and computer science from Seoul National University, Seoul, Korea, in 1999, 2001, and 2005, respectively. From 2005 to 2009, he was a Research Staff Member with the Samsung Advanced Institute of Technology, Samsung Electronics Company, Ltd., Yongin, Korea. In 2009, he joined the School of Electrical and Computer Engineering, Ulsan National Institute of Science and Technology, Ulsan, Korea, where he is currently an Associate Professor. His research interests include image and 3D visual signal processing, multimedia data compression, and computer vision.

Chang-Su Kim (S’95–M’01–SM’05) received the Ph.D. degree in electrical engineering from Seoul National University (SNU), Seoul, Korea, with a Distinguished Dissertation Award in 2000. From 2000 to 2001, he was a Visiting Scholar with the Signal and Image Processing Institute, University of Southern California, Los Angeles, CA, USA. From 2001 to 2003, he co-ordinated the 3D Data Compression Group, National Research Laboratory for 3D Visual Information Processing, SNU. From 2003 to 2005, he was an Assistant Professor with the Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong. In 2005, he joined the School of Electrical Engineering, Korea University, Seoul, Korea, where he is currently a Professor. He has authored over 220 technical papers in international journals and conferences. His current research interests include image processing and multimedia communications. In 2009, he received the IEEK/IEEE Joint Award for Young IT Engineer of the Year. He is also an Editorial Board Member of the Journal of Visual Communication and Image Representation and an Associate Editor of the IEEE T RANSACTIONS ON I MAGE P ROCESSING.