A Reduced Codebook and Re-Interpolation Approach for Enhancing Quality in Chroma Subsampling
11 A Reduced Codebook and Re-InterpolationApproach for Enhancing Quality in ChromaSubsampling
Kuo-Liang ChungDepartment of Computer Science and Information EngineeringNational Taiwan University of Science and TechnologyNo. 43, Section 4, Keelung Road, Taipei, 10672, Taiwan, R.O.C. [email protected]
Chen-Wei KaoDepartment of Computer Science and Information EngineeringNational Taiwan University of Science and TechnologyNo. 43, Section 4, Keelung Road, Taipei, 10672, Taiwan, R.O.C.
Abstract —Prior to encoding RGB full-color images or Bayercolor filter array (CFA) images, chroma subsampling is anecessary and crucial step at the server side. In this pa-per, we first propose a flow diagram approach to analyzethe coordinate-inconsistency (CI) problem and the upsamplingprocess-inconsistency (UPI) problem existing in the traditionaland state-of-the-art chroma subsampling methods under thecurrent coding environment. In addition, we explain why thetwo problems degrade the quality of the reconstructed images.Next, we propose a reduced codebook and re-interpolation(RCRI) approach to solve the two problems for enhancing thequality of the reconstructed images. Based on the testing RGBfull-color images and Bayer CFA images, the comprehensiveexperimental results demonstrated at least 1.4 dB and 2.4 dBquality improvement effects, respectively, of our RCRI approachagainst the CI and UPI problems for the traditional and state-of-the-art chroma subsampling methods.
Index Terms —Bayer color filter array (CFA) image, Chromasubsampling, Chroma upsampling, Codebook, Quality enhance-ment, Re-interpolation, RGB full-color image.
I. I
NTRODUCTION A S shown at the server side of Fig. 1, in our study, theinput image could be a RGB full-color image 𝐼 𝑅𝐺𝐵 ora demosaiced RGB full-color image which has been demo-saicked from the input Bayer color filter array (CFA) image 𝐼 𝐵𝑎𝑦𝑒𝑟 [2]. To demosaick 𝐼 𝐵𝑎𝑦𝑒𝑟 to a RGB full-color image,several demosaicking methods [18], [9], [41], [15], [29], [38],[25], [26] can be used; here, the demosaicking method in [15]is used. For easy exposition, we take the Bayer CFA patternin Fig. 3(a) as the representative, but our discussion is alsoapplicable to the other three patterns in Figs. 3(b)-(d). Priorto compression, according to BT.601-5 [13], the RGB full-color image is converted to a YCbCr image 𝐼 𝑌 𝐶𝑏𝐶𝑟 using thefollowing RGB-to-YCbCr color transformation: 𝑌 𝑖 𝐶𝑏 𝑖 𝐶𝑟 𝑖 = .
257 0 .
504 0 . − . − .
291 0 . . − . − . 𝑅 𝑖 𝐺 𝑖 𝐵 𝑖 + (1) where for each 2 × 𝐵 𝑌 𝐶𝑏𝐶𝑟 , ( 𝑌 𝑖 , 𝐶𝑏 𝑖 , 𝐶𝑟 𝑖 ) , ≤ 𝑖 ≤ , denotes the YCbCr triple-value in zigzag order; ( 𝑅 𝑖 , 𝐺 𝑖 , 𝐵 𝑖 ) denotes the collocated RGB triple-value of the2 × 𝐵 𝑅𝐺𝐵 .Chroma subsampling has two formats, namely 4:2:0 and4:2:2. 4:2:0 subsamples the ( 𝐶𝑏, 𝐶𝑟 ) -pair for each 2 × 𝐵 𝐶𝑏𝐶𝑟 and 4:2:2 subsamples the ( 𝐶𝑏, 𝐶𝑟 ) -pair for eachrow of 𝐵 𝐶𝑏𝐶𝑟 . Throughout this paper, we focus on 4:2:0,although the approach is also is applicable to 4:2:2. 4:2:0 hasbeen used in Bluray discs (BDs) and digital versatile discs(DVDs) for storing movies, sports, TV shows, etc.After decompressing the encoded subsampled YCbCr imageby the decoder, as depicted at the client side of Fig. 1, a chromaupsampling process is performed on the subsampled CbCrimage. Further, the upsampled YCbCr image is convertedto a reconstructed RGB full-color image using the followingYCbCr-to-RGB color conversion: (cid:34) 𝑅 𝑖 𝐺 𝑖 𝐵 𝑖 (cid:35) = (cid:34) .
164 0 1 . . − . − . .
164 2 .
018 0 (cid:35) (cid:34) 𝑌 𝑖 − 𝐶𝑏 𝑖 − 𝐶𝑟 𝑖 − (cid:35) (2)Suppose the input image is the demosaiced RGB full-colorimage. By Eq. (2), the upsampled YCbCr image can bedirectly converted to the reconstructed Bayer CFA image asthe output.In Subsection I.A, we introduce the related chroma subsam-pling works for 𝐼 𝑅𝐺𝐵 . Then, in Subsection I.B, we introducethe related chroma subsampling works for 𝐼 𝐵𝑎𝑦𝑒𝑟 . A. Related Works for 𝐼 𝑅𝐺𝐵
At the server side, suppose the input image is a RGB full-color image. We first introduce the five traditional chromasubsampling methods, namely 4:2:0(A), 4:2:0(L), 4:2:0(R),4:2:0(DIRECT), and 4:2:0(MPEG-B) [28]. Then, we introducefive state-of-the-art chroma subsampling combinations [43],[33], [4], [22]. a r X i v : . [ ee ss . I V ] J a n Fig. 1. The current coding system.
Among the five traditional chroma subsampling methods,4:2:0(DIRECT) subsamples the top-left chroma pair of 𝐵 𝐶𝑏𝐶𝑟 as the subsampled ( 𝐶𝑏, 𝐶𝑟 ) -pair of 𝐵 𝐶𝑏𝐶𝑟 . For simplicity,4:2:0(DIRECT) is abbreviated as 4:2:0(D). 4:2:0(MPEG-B)determines the subsampled ( 𝐶𝑏, 𝐶𝑟 ) -pair by performing the13-tap filter with mask [2, 0, -4, -3, 5, 19, 26, 19, 5, -3, -4, 0,2]/64 on the top-left location of 𝐵 𝐶𝑏𝐶𝑟 . 4:2:0(L) and 4:2:0(R)subsample the ( 𝐶𝑏, 𝐶𝑟 ) -pairs by averaging the chroma com-ponents in the left and right columns of 𝐵 𝐶𝑏𝐶𝑟 , respectively.4:2:0(A) subsamples the ( 𝐶𝑏, 𝐶𝑟 ) -pair of 𝐵 𝐶𝑏𝐶𝑟 by averagingthe four ( 𝐶𝑏, 𝐶𝑟 ) -pairs of 𝐵 𝐶𝑏𝐶𝑟 . The subsampled chromapositions of 4:2:0(D) and 4:2:0(MPEG-B) are located at (0,1); the subsampled chroma positions of 4:2:0(L), 4:2:0(A), and4:2:0(R) are located at (0, ), ( , ), and (1, ), respectively,and the four subsampled chroma positions are marked by thefour red bullets in Figs. 2(a)-(d). (a) (b) (c) (d)Fig. 2. The subsampled chroma positions of the five traditional chromasubsampling methods. (a) For 4:2:0(D) and 4:2:0(MPEG-B). (b) For 4:2:0(L).(c) For 4:2:0(A). (d) For 4:2:0(R). According to the new edge-directed interpolation (NEDI)[19] based chroma upsampling process which improvedthe previous method [1], Zhang et al. [43] proposed aninterpolation-dependent image downsampling (IDID) basedchroma subsampling method. Their combination is expressedas IDID-NEDI. To improve IDID-NEDI, Wang et al. [33]deployed the palette mode [27] in their JCDU (joint chromadownsampling and upsampling) based chroma subsamplingmethod, and their best combination is expressed as JCDU-BICU, where BICU denotes the bicubic interpolation basedchroma upsampling process. The experimental data demon-strated that JCDU-BICU outperforms IDID-NEDI and JCDU-BILI, in which BILI denotes the bilinear interpolation basedchroma upsampling process, particularly for screen content images (SCIs) [23].Following the COPY-based chroma upsampling process andthe differentiation technique used in [20], but consideringthe demosaiced RGB full-color block-distortion as the crite-rion, Lin et al. [22] proposed a modified 4:2:0(A) chromasubsampling method which selects the best case among thefour subsampled ( 𝐶𝑏, 𝐶𝑟 ) -pairs of 𝐵 𝐶𝑏𝐶𝑟 by consideringthe ceiling operation-based 4:2:0(A) and the floor operation-based 4:2:0(A). Naturally, Lin et al. ’s chroma subsamplingmethod is suitable for the input RGB full-color image. Atthe client side, they improved the chroma upsampling process[22] by considering the distance between each missing chromavalue and its three neighboring known (TN) pixels to achievegood quality performance. Their combination is expressed as“modified 4:2:0(A)-TN”.Differing from the the chroma subsampling-first luma mod-ification method [5], in [4], based on the subsampled chromaparameter-pair, at the server side, a BILI-based chroma esti-mation of 𝐵 𝐶𝑏𝐶𝑟 is deployed in the block-distortion functionwith two chroma parameters and four luma parameters. Next,using a multiple linear regression technique, a joint chromasubsampling and luma modification (CSLM) method [4] wasproposed to determine the subsampled ( 𝐶𝑏, 𝐶𝑟 ) -pair of 𝐵 𝐶𝑏𝐶𝑟 and the four modified luma values of 𝐵 𝑌 simultaneously. Theircombination is expressed as CSLM-BILI. Experimental dataindicated the quality superiority of CSLM-BILI relative toIDID-NEDI [43], JCDU-BICU [33], and modified 4:2:0(A)-TN [22]. (a) (b) (c) (d)Fig. 3. The four Bayer CFA patterns. (a) [ 𝐺 , 𝑅 , 𝐵 , 𝐺 ] . (b) [ 𝑅 , 𝐺 , 𝐺 , 𝐵 ] . (c) [ 𝐵 , 𝐺 , 𝐺 , 𝑅 ] . (d) [ 𝐺 , 𝐵 , 𝑅 , 𝐺 ] . B. Related Works for 𝐼 𝐵𝑎𝑦𝑒𝑟
For 𝐼 𝐵𝑎𝑦𝑒𝑟 , we mainly introduce the four state-of-the-artchroma subsampling methods [20], [7], [22], [8].Chen et al. [3] observed that in Eq. (2), the R valueis dominated by the Y and V values, and the B value isdominated by the Y and U values, and then the subsampled ( 𝐶𝑏, 𝐶𝑟 ) -pair of 𝐵 𝐶𝑏𝐶𝑟 equals ( 𝑈 , 𝑉 ) by considering theBayer CFA pattern. Although their method benefits the Rand B components of the reconstructed Bayer CFA image,it does not benefit the G components at all. To overcome thisdisadvantage, based on the COPY-based upsampling processto estimate the four chroma-pairs of 𝐵 𝐶𝑏𝐶𝑟 at the server side,Lin et al. [20] proposed a 2 × × et al. derived a closed formto determine the subsampled ( 𝐶𝑏, 𝐶𝑟 ) -pair of 𝐵 𝐶𝑏𝐶𝑟 . Theircombination is expressed as DM-COPY.
In the gradient descent-based (GD) method [7], the × Bayer CFA block-distortion function used in the DM method[20] is proved to be a convex function. Then, according to theshape similarity of the convex function in the real domain tothat in the integer domain, an iterative procedure, in whichthe closed form derived in DM [20] was used as the initiallysubsampled ( 𝐶𝑏, 𝐶𝑟 ) -pair of 𝐵 𝐶𝑏𝐶𝑟 , was proposed to betterimprove the DM method. In each iteration, the GD methodapplied the BILI method to estimate the four ( 𝐶𝑏, 𝐶𝑟 ) -pairsof each neighboring chroma block in the eight neighboringchroma blocks, and then the GD method selected the bestone with the minimal 2 × et al. ’s modified 4:2:0(A) chroma subsampling method [22]only considers the demosaiced RGB full-color block-distortionas the distortion minimization criterion. After performing themodified 4:2:0(A)-TN combination on the demosaiced RGBfull-color image, by Eq. (1), the reconstructed Bayer CFAimage can be extracted from the reconstructed RGB full-colorimage. Therefore, ‘modified 4:2:0(A)-TN’ is also applicablefor the input Bayer CFA image.In [8], based on the BILI-based chroma upsampling processat the server side, combining chroma subsampling, luma mod-ification, and the Bayer CFA pattern together, a 𝐶𝑆𝐿𝑀
𝐵𝑎𝑦𝑒𝑟 -BILI combination was proposed. For each 2 × 𝐵 𝑌 𝐶𝑏𝐶𝑟 , 𝐶𝑆𝐿𝑀
𝐵𝑎𝑦𝑒𝑟 -BILI determined the best solution ofthe subsampled ( 𝐶𝑏, 𝐶𝑟 ) -pair and the modified luma valuesfor 𝐵 𝑌 𝐶𝑏𝐶𝑟 simultaneously. In particular, after analyzing allthe sixteen (= ) luma-selection cases, only two luma pa-rameters, namely 𝑌 and 𝑌 , are modified such that the 2 × 𝐶𝑆𝐿𝑀
𝐵𝑎𝑦𝑒𝑟 -BILI combination[8] outperforms DM-COPY [20], GD-BILI [7], and modified4:2:0(A)-TN [22].
C. Motivation
From the introduction of the related chroma subsamplingworks for 𝐼 𝑅𝐺𝐵 and 𝐼 𝐵𝑎𝑦𝑒𝑟 , we find that under the current cod-ing system in Fig. 1, the traditional and state-of-the-art combi-nations tend to suffer from the coordinate-inconsistency (CI)problem and/or the chroma upsampling process-inconsistency(UPI) problem because at the client side, the decoder is blindto the chroma subsampling process used at the server side andthe future chroma upsampling process prefered by the chromasubsampling method. The two problems will be defined inSection III in detail.The CI and UPI problems lead to the quality degradation ofthe reconstructed images, which will be explained in SectionIII. The two problems prompted us to develop a systematicapproach to analyze them, and then to propose an effectiveapproach to solve them for enhancing the quality of thereconstructed images.
D. Contributions
In this paper, we first analyze the subsampled chromapositions of all considered chroma subsampling methods, and then we partition all these chroma subsampling meth-ods into four classes. Based on the four partitioned chromasubsampling classes and the allowable chroma upsamplingprocesses, we propose a flow diagram approach to analyzethe coordinate-inconsistency (CI) problem and the upsam-pling process-inconsistency (UPI) problem occurring in thetraditional and state-of-the-art combinations. We also explainwhy the two problems lead to the quality degradation of thereconstructed images.To solve the CI and UPI problems, we propose an effectivereduced codebook and re-interpolation (RCRI) approach forenhancing the quality of the reconstructed images. Based onthe testing RGB full-color images and Bayer CFA imagescollected from the IMAX [14] dataset, the Kodak [16] dataset,and the Video [32] dataset, the thorough experimental resultsjustified the significant quality enhancement effects of ourRCRI approach against the CI and UPI problems for thetraditional and state-of-the-art chroma subsampling methods.For 𝐼 𝑅𝐺𝐵 and 𝐼 𝐵𝑎𝑦𝑒𝑟 , the average CPSNR (color peak signal-to-noise ratio) gain and the average PSNR gain of our RCRIapproach are at least 1.4 dB and 2.4 dB, respectively.The rest of this paper is organized as follows. In SectionII, based on the subsampled chroma positions, all consideredchroma subsampling methods are partitioned into four classes.In Section III, the proposed flow diagram approach is pre-sented to analyze the CI problem and the UPI problem. In Sec-tion IV, the proposed RCRI approach is presented to solve thetwo problems. In Section V, the thorough experimental resultsare illustrated to justify the significant quality enhancementusing our RCRI approach. In Section VI, some concludingremarks are addressed.II. PARTITIONING ALL CONSIDERED CHROMASUBSAMPLING METHODS INTO FOUR CLASSESIn this section, based on the subsampled chroma positionsof all considered chroma subsampling methods, we partitionthese methods into four classes. It has been known that thesubsampled chroma positions of the five traditional chromasubsampling methods are depicted by the four red bullets inFigs. 2(a)-(d) corresponding to the top-left, left, middle, andright black bullets in Fig. 4, respectively. Similar to 4:2:0(D)and 4:2:0(MPEG-B), the subsampled chroma positions ofIDID [43] and JCDU [33] are all located at (0, 1), as depictedby the top-left black bullet of Fig. 4.Based on the subsampled chroma parameter-pair, ( 𝐶𝑏 𝑠 , 𝐶𝑟 𝑠 ), which is computationally located at ( , ), the DM[20], GD [7], and modified 4:2:0(A) [22] methods apply thesame COPY-based chroma upsampling process to estimatethe four chroma pairs of each 2 × 𝐵 𝐶𝑏𝐶𝑟 forbuilding up their own block-distortion functions. Accordingly,the determined values of the subsampled ( 𝐶𝑏 𝑠 , 𝐶𝑟 𝑠 )-pairsusing the above three methods are all located at ( , ),as depicted by the middle black bullet in Fig. 4. Based onthe subsampled chroma parameter-pair located at ( , ), theCSLM [4] and 𝐶𝑆𝐿𝑀
𝐵𝑎𝑦𝑒𝑟 [8] methods apply the BILI-basedchroma upsampling process to estimate the four chroma pairsof each 2 × 𝐵 𝐶𝑏𝐶𝑟 for building up their own block-distortion functions. Therefore, the subsampled chromaposition of the two methods is expressed as ( , ). Fig. 4. The four subsampled chroma positions for all considered chromasubsampling methods.
For all considered chroma subsampling methods, the foursubsampled chroma positions (SCPs) are denoted by
SCP = {(0, 1), (0, ), ( , ), (1, )} corresponding to the fourpartitioned chroma subsampling classes which are expressedas CS = { 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑚𝑖𝑑 , 𝐶 𝑟𝑖𝑔ℎ𝑡 }, as depicted in Fig.4. Accordingly, in our study, instead of considering the originalchroma subsampling method used in one combination, weonly consider its chroma subsampling class which the chromasubsampling method used belongs to.III. THE ANALYSIS OF THE UPI AND CIPROBLEMSWe first define the UPI problem and explain why it degradesthe quality of the reconstructed color images. Next, we proposea flow diagram approach to analyze the CI problem systemat-ically. According to the coordinate displacement analysis, weexplain why the CI problem also degrades the quality of thereconstructed color images. A. The UPI Problem
Without the loss of generality, we take the CSLM-BILIcombination [4] as the example to define the UPI problem.In CSLM-BILI, as introduced in Subsection I.A, at theserver side, a BILI-based chroma estimation of each 2 × 𝐵 𝐶𝑏𝐶𝑟 is deployed in the 2 × ( 𝐶𝑏, 𝐶𝑟 ) .At the client side, instead of ‘BILI’, if the decoder adoptsthe other upsampling process, e.g. ‘COPY’, to upsamplethe received subsampled chroma image, it causes an upsam-pling process-inconsistency (UPI) problem because the futurechroma upsampling process ‘BILI’ preferred by CSLM-BILIis misused as ‘COPY’ by the decoder. B. The CI Problem
Let the set symbol ‘ CU ’ denote the four chroma upsamplingprocesses, namely COPY, BILI, NEDI, and BICU, used atthe client side. Let CS x CU , where the symbol ‘x’ indicatesthe cross product operator, denote all combinations over theproduct of CS and CU . For each combination in CS x CU ,we propose a flow diagram approach to analyze whether atrue CI problem occurs in that combination and to report thecoordinate displacement of the true CI problem. Furthermore,we explain why the true CI problem leads to the qualitydegradation of the reconstructed images.
1) The proposed flow diagram to analyze the true CIproblem and to calculate the coordinate displacement:
Afterperforming one chroma subsampling 𝑐𝑠 ∈ CS on the chromaimage 𝐼 𝐶𝑏𝐶𝑟 , it yields a subsampled chroma image 𝐼 𝑠𝑢𝑏,𝐶𝑏𝐶𝑟𝑐𝑠 ,which can be expressed as 𝐼 𝑠𝑢𝑏,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 , 𝐼 𝑠𝑢𝑏,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐼 𝑠𝑢𝑏,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑚𝑖𝑑 ,or 𝐼 𝑠𝑢𝑏,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑟𝑖𝑔ℎ𝑡 . The four possible subsampled chroma imagesare depicted in Fig. 5(a). For compression, as depicted in Fig.5(b), each subsampled chroma image 𝐼 𝑠𝑢𝑏,𝐶𝑏𝐶𝑟𝑐𝑠 is further rear-ranged to a quarter-sized subsampled chroma image 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 which is stored in an array data structure under an integercoordinate system.However, at the server side, for compression, moving thesubsampled chroma pair of each 2 × ∈ SCP ) to the new position,namely (0, 1), often causes a coordinate displacement problem.Generally, as depicted by the four arrows between Fig. 5(a)and Fig. 5(b), the corresponding four coordinate displacementsequal (0, 0) (= (0, 1) - (0, 1)), (0, ) (= (0, 1) - (0, )), (- , ) (= (0, 1) - ( , )), and (-1, ) (= (0, 1) - (1, )) cor-responding to 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 , 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑚𝑖𝑑 , 𝑎𝑛𝑑𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑟𝑖𝑔ℎ𝑡 ,respectively. We conclude that for one chroma subsamplingmethod 𝑐𝑠 ∈ 𝐶 𝑙𝑒 𝑓 𝑡 ∪ 𝐶 𝑚𝑖𝑑 ∪ 𝐶 𝑟𝑖𝑔ℎ𝑡 , at the server side,preparing the quarter-sized subsampled chroma image 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 for compression causes a CI problem due to its nonzerocoordinate displacement (NCD) in the set NCD = (0, ), (- , ), (-1, ).After receiving the compressed quarter-sized subsampledchroma image 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 by the decoder at the client side,each subsampled chroma pixel 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 ( 𝑖, 𝑗 ) is moved to 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟𝑐𝑠 ( 𝑖, 𝑗 ) , where 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟𝑐𝑠 indicates the upsampledchroma image, for constructing the initially upsampled chromaimage 𝐼 𝑖𝑛𝑖,𝐶𝑏𝐶𝑟𝑐𝑠 , as depicted in Fig. 5(c). Because from Fig.5(a) to Fig. 5(b), for 𝑐𝑠 ∈ 𝐶 𝑙𝑒 𝑓 𝑡 ∪ 𝐶 𝑚𝑖𝑑 ∪ 𝐶 𝑟𝑖𝑔ℎ𝑡 , it causesa CI problem in the subsampled chroma image 𝐼 𝑞,𝐶𝑏𝐶𝑟𝑐𝑠 , theinitially upsampled chroma image 𝐼 𝑖𝑛𝑖,𝐶𝑏𝐶𝑟𝑐𝑠 in Fig. 5(c) thusinherits the CI problem in Fig. 5(b) and the associated nonzerocoordinate displacement in NCD .Further, all missing chroma pixels in 𝐼 𝑖𝑛𝑖,𝐶𝑏𝐶𝑟𝑐𝑠 of Fig.5(c) are reconstructed using the adopted chroma upsamplingprocess ‘ 𝑐𝑢 (cid:48) in CU . We first consider the chroma upsamplingprocess ‘COPY’. After performing the COPY-based upsam-pling process on each initially subsampled chroma image inFig. 5(c), the reconstructed chroma pixels are denoted by blackbullets of the upsampled chroma image in Fig. 5(d). For each2 × 𝐼 𝑖𝑛𝑖,𝐶𝑏𝐶𝑟𝑐𝑠 , as depicted in Fig. 5(d), thethree missing chroma pixels are reconstructed by copying the top-left subsampled chroma-pair of that block. According tothe analysis from Fig. 5(a) to Fig. 5(d), we conclude that forany combination ‘ 𝑐𝑠 -COPY’, where ‘cs’ is in CS , no true CIproblem occurs.Next, we consider the chroma upsampling process ‘ 𝑐𝑢 ’ ∈ {BILI, NEDI, BICU}. After performing the upsamplingprocess ‘ 𝑐𝑢 ’ on each initially subsampled chroma image inFig. 5(c), the reconstructed chroma pixels are denoted by blackcross-marked symbols of the reconstructed chroma image inFig. 5(d). In Fig. 5(d), each missing chroma pixel in 𝐼 𝑖𝑛𝑖,𝐶𝑏𝐶𝑟𝑐𝑠 is reconstructed by the upsampling process ‘ 𝑐𝑢 ’ referringto the neighboring chroma pairs of that missing chromapixel. Consequently, we conclude that for any combinationin { 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑚𝑖𝑑 , 𝐶 𝑟𝑖𝑔ℎ𝑡 }x{BILI, NEDI, BICU}, a true CIproblem occurs eventually, as depicted in Fig. 5(e). On theother hand, among the sixteen combinations in CS x CU , thetrue CI problems occur in only nine combinations in { 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑚𝑖𝑑 , 𝐶 𝑟𝑖𝑔ℎ𝑡 }x{BILI, NEDI, BICU}.From the above analysis of the true CI problems, we findthat there are only three distinct coordinate displacements,namely (0, ), (- , ), and (-1, ), corresponding to thethree combinations in 𝐶 𝑙𝑒 𝑓 𝑡 x{BILI, NEDI, BICU}, the threecombinations in 𝐶 𝑚𝑖𝑑 x{BILI, NEDI, BICU}, and the threecombinations in 𝐶 𝑟𝑖𝑔ℎ𝑡 x{BILI, NEDI, BICU}, respectively.Due to the nonzero coordinate displacement problem, for each2 ×
2) Why the true CI problem degrades the quality of thereconstructed image:
Because for any combination, at theserver side, we do not subsample the luma image 𝐼 𝑌 atall, the luma image has no CI problem from the chromasubsampling step to the chroma upsampling step at the clientside. For any combination in { 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑚𝑖𝑑 , 𝐶 𝑟𝑖𝑔ℎ𝑡 }x{BILI,NEDI, BICU}, we know a true CI problem occurs in thereconstructed chroma image 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟 . Consequently, at theclient side, the luma pixel 𝐼 𝑌 ( 𝑖, 𝑗 ) and the upsampled chromapixel 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟 ( 𝑖, 𝑗 ) lead to a coordinate displacement prob-lem each other. Finally, after converting the upsampled YCbCrimage to the reconstructed RGB full-color image (or thereconstructed Bayer CFA image) by Eq. (2), it degrades thequality of the reconstructed image.IV. T HE P ROPOSED
REDUCED CODEBOOK ANDRE-INTERPOLATION (RCRI) APPROACH TO SOLVETHE UPI AND CI P
ROBLEMS
To solve the UPI problem, in our RCRI (reduced codebookand re-interpolation) approach, we first build up a reducedcodebook in which each codeword occupies four bits. For eachcodeword, the first two bits represent the chroma subsamplingclass instead of the chroma subsampling method used at theserver side, and the last two bits represent the future chromaupsampling process preferred at the client side. This is why wecall it the reduced codebook. Later, a (7, 4)-Hamming code isproposed to increase the robustness against the communication interference. To solve the true CI problem, in our RCRIapproach, our coordinate displacement-based re-interpolationstrategy will be presented in Subsection IV.B.
A. The Reduced Codebook Design in Our RCRI Approach
From Fig. 4, we know there are four chroma subsamplingclasses, namely 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑚𝑖𝑑 , and 𝐶 𝑟𝑖𝑔ℎ𝑡 , in CS andthere are four chroma upsampling processes, namely COPY,BILI, NEDI, and BICU, in CU . To record the necessaryinformation of each combination in CS x CU , a reduced 4-bitcodebook is depicted in Table I. In Table I, for each codeword,the first two bits are used to represent the chroma subsamplingclass, which corresponds to the chroma subsampling methodused at the server side, and the last two bits are used torecord the future chroma upsampling process preferred atthe client side. For example, the combination ‘JCDU-BICU’corresponding to ‘ 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 -BICU’ is expressed as the 4-bitcodeword ‘0011’. TABLE ITHE REDUCED CODEBOOK USED FOR REPRESENTING THECONSIDERED COMBINATIONS.
COPY BILI NEDI BICU 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 𝐶 𝑙𝑒 𝑓 𝑡 𝐶 𝑚𝑖𝑑 𝐶 𝑟𝑖𝑔ℎ𝑡 To increase the robustness to tolerate one bit error againstthe communication interference, we deploy three redundantbits in the 4-bit codeword in Table I to form a (7, 4)-Hammingcode [11]. Let each 4-bit codeword in Table I be denoted by‘ 𝑚 𝑚 𝑚 𝑚 ’. Using the error correcting code technique, thecorresponding (7, 4)-Hamming code is represented as a 7-bit codeword “ 𝑟 𝑟 𝑚 𝑟 𝑚 𝑚 𝑚 ” in which the redundant 3-bit ‘ 𝑟 𝑟 𝑟 ’ is used for correcting the one bit error. For easyexposition, let 𝑝 𝑝 𝑝 𝑝 𝑝 𝑝 𝑝 = 𝑟 𝑟 𝑚 𝑟 𝑚 𝑚 𝑚 . Basedon the three equations with even parity: 𝑝 + 𝑝 + 𝑝 + 𝑝 = ( mod 2 ) , 𝑝 + 𝑝 + 𝑝 + 𝑝 = ( mod 2 ) , and 𝑝 + 𝑝 + 𝑝 + 𝑝 = ( mod 2 ) , the corrupted one bit can be detected and corrected.For example, suppose the decoder received the 7-bit errorcorrection code, namely 𝑝 𝑝 𝑝 𝑝 𝑝 𝑝 𝑝 = 1100000. Fromthe equality: 𝑝 𝑝 𝑝 𝑝 𝑝 𝑝 𝑝 = 𝑟 𝑟 𝑚 𝑟 𝑚 𝑚 𝑚 , it yields‘ 𝑚 𝑚 𝑚 𝑚 = 1000’. Based on three even parity equations, ityields 𝑝 + 𝑝 + 𝑝 + 𝑝 = ( mod 2 ) , 𝑝 + 𝑝 + 𝑝 + 𝑝 = ( mod 2 ) , and 𝑝 + 𝑝 + 𝑝 + 𝑝 = ( mod 2 ) . From theresultant 3-bit ‘011’ calculated from the above three evenparity equations, we know that the value of 𝑝 is corrupted.Therefore, the value of 𝑚 is corrected from 0 to 1. Equiv-alently, the corrected 7-bit code should be 1110000, and thecorrected 4-bit codeword equals 0001. Taking the first twobits of the corrected 4-bit codeword, namely 00, as a keyto query Table I at the client side, the chroma subsamplingclass used at the server side is reported as 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 . Takingthe last two bits of the corrected 4-bit codeword, namely 01,by Table I, the future chroma upsampling process preferred Fig. 5. The proposed flow diagram to analyze the CI problem for all combinations in CS x CU . (a) Subsampled chroma image. (b) Input subsampled chromaimage for encoding. (c) Initially upsampled chroma image. (d) Upsampled chroma image. (e) Status of true CI problems. at the client side is reported as BILI. Table II illustrates theerror correcting (7, 4)-Hamming codebook used in our RCRIapproach. Consequently, for any combination in CS x CU ,Table II can be used to solve the UPI problem in a robustway. Table II is kept by the server side and the client sidesimultaneously. TABLE IITHE (7, 4)-HAMMING CODEBOOK USED IN OUR RCRIAPPROACH.
COPY BILI NEDI BICU 𝐶 𝑡𝑜𝑝 − 𝑙𝑒 𝑓 𝑡 𝐶 𝑙𝑒 𝑓 𝑡 𝐶 𝑚𝑖𝑑 𝐶 𝑟𝑖𝑔ℎ𝑡 B. The Re-Interpolation Strategy for Correcting the Devia-tionly Upsampled Chroma Image
We first consider the three combinations in 𝐶 𝑙𝑒 𝑓 𝑡 x{BILI,NEDI, BICU}. As depicted in Fig. 5, we know their commonnonzero coordinate displacement vector is (0, ). To solve thetrue CI problem which occurred in the three combinations,based on the coordinate displacement vector (0, ), for each2 × 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑙𝑒 𝑓 𝑡 of Fig. 5(d).Next, we consider the three combinations in 𝐶 𝑚𝑖𝑑 x{BILI,NEDI, BICU}. As depicted in Fig. 5, we know their commonnonzero coordinate displacement vector is (- , ). To solvethe true CI problem which occurred in the three combinations,based on the coordinate displacement vector (- , ), for each2 × 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑚𝑖𝑑 of Fig. 5(d), the four temporarily up-sampled chroma pairs are replaced by re-interpolating the fourchroma pairs marked by the four red triangles in 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑚𝑖𝑑 .By the same argument, for each 2 × 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑟𝑖𝑔ℎ𝑡 of Fig. 5(d), the four temporarily up-sampled chroma pairs can be recovered by re-interpolatingthe four chroma pairs marked by the four red triangles in 𝐼 𝑟𝑒𝑐,𝐶𝑏𝐶𝑟𝑐𝑠 ∈ 𝐶 𝑟𝑖𝑔ℎ𝑡 . Consequently, the deviationly upsampled chromaimage occurred in the nine combinations (= { 𝐶 𝑙𝑒 𝑓 𝑡 , 𝐶 𝑚𝑖𝑑 , 𝐶 𝑟𝑖𝑔ℎ𝑡 }x{BILI, NEDI, BICU}) can be recovered using ourcoordinate displacement-based re-interpolation strategy.V. E XPERIMENTAL R ESULTS
Based on the Kodak, IMAX, and Video datasets, and underthe newly released versatile video coding (VVC) platformVTM-9.0 [31] for QP = 0, the thorough experimental resultsare illustrated to justify the quality enhancement effects ofour RCRI approach against the CI and UPI problems for thetraditional and state-of-the-art chroma subsampling methods.
TABLE IIIQUALITY ENHANCEMENT EFFECTS OF OUR RCRI APPROACH AGAINST THE UPI AND TRUE CI PROBLEMS FOR 𝐼 𝑅𝐺𝐵 . 𝐼 𝑅𝐺𝐵
IDID [43] JCDU [33] 4:2:0(L) 4:2:0(R) CSLM [4]COPY BILI NEDI [43] BICU COPY BILI NEDI BICU [33] COPY BILI NEDI BICU COPY BILI NEDI BICU COPY BILI [4] NEDI BICUCPSNR 40.0832 45.2248 45.2343 44.5094 40.7675 45.3701 45.0005 45.6042 41.7039 43.6801 43.4977 43.6427 41.6744 40.6781 40.6171 40.5475 41.2692 42.2481 42.1925 41.3915(45.2343) (45.2343) (45.2343) (45.2343) (45.6042) (45.6042) (45.6042) (45.6042) (45.1837) (45.1837) (45.1837) (45.1837) (44.4567) (44.4567) (44.4567) (44.4567) (46.2224) (46.2224) (46.2224) (46.2224)Average 1.4715 1.4186 2.0526 3.5774 4.4471CPSNR gainSSIMc 0.9710 0.9863 0.9865 0.9855 0.9742 0.9864 0.9858 0.9873 0.9779 0.9838 0.9834 0.9840 0.9778 0.9742 0.9741 0.9737 0.9750 0.9792 0.9792 0.9762(0.9865) (0.9865) (0.9865) (0.9865) (0.9873) (0.9873) (0.9873) (0.9873) (0.9870) (0.9870) (0.9870) (0.9870) (0.9860) (0.9860) (0.9860) (0.9860) (0.9896) (0.9896) (0.9896) (0.9896)Average 0.0042 0.0039 0.0047 0.0111 0.0122SSIMc gainFSIMc 0.9992 0.9996 0.9997 0.9996 0.9993 0.9997 0.9996 0.9997 0.9995 0.9996 0.9995 0.9996 0.9995 0.9992 0.9992 0.9992 0.9993 0.9995 0.9994 0.9993(0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997) (0.9997)Average 0.0002 0.0001 0.0002 0.0004 0.0003FSIMc gain
All the concerned combinations are implemented on acomputer with an Intel Core i7-7700 CPU 3.6 GHz and 24 GBRAM. The operating system is the Microsoft Windows 10 64-bit operating system. The program development environmentis Visual C++ 2017.We adopt the quality metrics, namely CPSNR (color peaksignal-to-noise ratio), PSNR, SSIM (structure similarity index)[35], and FSIM (feature similarity index) [40], to illustrate thequality enhancement effects of our RCRI approach against theUPI and CI problems existing in the traditional and state-of-the-art combinations. The related quality metrics are definedbelow.CPSNR is used to evaluate the average quality of thereconstructed RGB full-color images for one dataset with Nimages, and it is defined byCPSNR = 𝑁 𝑁 ∑︁ 𝑛 =
10 log 𝐶 𝑀𝑆𝐸 (3)with
𝐶 𝑀𝑆𝐸 = 𝑊 𝐻 (cid:205) 𝑝 ∈ 𝑃 (cid:205) 𝑐 ∈{ 𝑅,𝐺,𝐵 } [ 𝐼 𝑅𝐺𝐵𝑛,𝑐 ( 𝑝 ) − 𝐼 𝑅𝐺𝐵𝑛,𝑐 ( 𝑝 )] in which P = {( 𝑥, 𝑦 )| ≤ 𝑥 ≤ 𝐻, ≤ 𝑦 ≤ 𝑊 } denotesthe set of pixel coordinates in one 𝑊 × 𝐻 image. Here, N= 24, N = 18, and N = 200 for the Kodak, IMAX, and Videodatasets, respectively. 𝐼 𝑅𝐺𝐵𝑛,𝑐 ( 𝑝 ) and 𝐼 𝑅𝐺𝐵𝑛,𝑐 ( 𝑝 ) denote the c-color value of the pixel at position p in the 𝑛 th original RGBfull-color image and the reconstructed one, respectively. In ourexperience, for fairness, each image in the Kodak dataset isdownsampled to a quarter-sized one such that the average sizeof the downsampled images is close to the average size of theimages in the IMAX dataset. The average CPSNR value equalsthe mean of the three CPSNR values for the three datasets.Similarly, the average PSNR value is used to evaluate thequality of the reconstructed Bayer CFA images for the threedatasets.For 𝐼 𝐵𝑎𝑦𝑒𝑟 , SSIM [35] is used to measure the joint preser-vation effects of luminance, contrast, and structure similaritybetween the original Bayer CFA image and the reconstructedone. For 𝐼 𝑅𝐺𝐵 , the SSIMc value is measured by the mean ofthe three SSIM values for the R, G, and B color planes.For 𝐼 𝐵𝑎𝑦𝑒𝑟 , FSIM [40] is an image quality metric with highconsistency with the subjective evaluation. FSIM first utilizesthe primary feature “phase congruency (PC)” and the minorfeature “gradient magnitude” to obtain the local quality map,and then FSIM utilizes PC as a weighting function to obtaina quality score. For 𝐼 𝑅𝐺𝐵 , the FSIMc value is measured bythe mean of the three FSIM values for the R, G, and B colorplanes.
A. Quality Enhancement Merit of Our RCRI Approach for 𝐼 𝑅𝐺𝐵
For 𝐼 𝑅𝐺𝐵 , this subsection presents the quality enhancementeffects of our RCRI approach against the UPI and true CIproblems for the traditional and state-of-the-art combinations[43], [33], and [4]. In the five traditional chroma subsamplingmethods, 4:2:0(L) and 4:2:0(R) are selected to balance ourdiscussion of the experiments. In addition, the quality en-hancement effect of our RCRI approach to the state-of-the-artcombination [22] is also investigated.Table III illustrates the CPSNR gains of the reconstructedRGB full-color images using our RCRI approach againstthe UPI and true CI problems existing in the concernedcombinations. For clarifying the quality enhancement effectof our RCRI approach, the CPSNR value of the reconstructedRGB full-color images using our RCRI approach for eachcombination is tabulated in the parenthesis ‘()’.After deploying our RCRI approach in IDIDx CU , exceptfor IDID-NEDI [43] without UPI and true CI problems,the CPSNR gains are 5.1511 (= 45.2343 - 40.0832 ) dB,0.01 (= 45.2343 - 45.2248) dB, and 0.7249 dB w.r.t. IDID-COPY, IDID-BILI, and IDID-BICU, respectively. Suppose theprobability of selecting each chroma upsampling process in CU at the client side is the same and equals . For IDID-NEDI, the average CPSNR gain using our RCRI approachequals 1.4715 (= (5.1511 + 0.01 + 0.7249)) dB, achieving aclear quality enhancement effect.Similarly, after deploying our RCRI approach in JCDUx CU ,except for JCDU-BICU [33] without UPI and true CI prob-lems, the CPSNR gains using our RCRI approach are 4.8367dB, 0.2341 dB, and 0.6037 dB w.r.t. JCDU-COPY, JCDU-BILI, and JCDU-NEDI, respectively; the average CPSNR gainequals 1.4186 (= (4.8367 + 0.2341 + 0.6037)) dB, alsoachieving a clear quality enhancement effect.After deploying our RCRI approach in 4:2:0(L)x CU ,4:2:0(R)x CU , and CSLMx CU , the average CPSNR gainsequal 2.0526 (= (3.4798 + 1.5036 + 1.6860 + 1.5410))dB, 3.5774 (= (2.7823 + 3.7786 + 3.8396 + 3.9092)) dB,and 4.4471 (= (4.9532 + 3.9743 + 4.0299 + 4.8309)) dB,respectively, also achieving significant quality enhancementeffects.In addition, our experimental results indicate that for 𝐼 𝑅𝐺𝐵 , after deploying our RCRI approach in ‘modified4:2:0(A)’x( CU ∪ {TN}), except for ‘modified 4:2:0(A)’-TN[22], the CPSNR gains for the five combinations are 2.1622dB, 2.3352 dB, 2.5043 dB, and 2.2331 dB, respectively.Accordingly, the average CPSNR gain of our RCRI approach equals 1.84696 (= (2.1622 + 2.3352 + 2.5043 + 2.2331)) dB,also achieving a clear quality enhancement effect.Besides the CPSNR improvement, Table III also demon-strates the 𝑆𝑆𝐼 𝑀 𝑐 and 𝐹𝑆𝐼 𝑀 𝑐 improvements of our RCRIapproach for 𝐼 𝑅𝐺𝐵 . In fact, more chroma upsampling pro-cesses [42], [37], [36], [10], [30], [34] can be included in ourstudy to justify the quality enhancement effect of the proposedRCRI approach.
B. Quality Enhancement Merit of Our RCRI Approach for 𝐼 𝐵𝑎𝑦𝑒𝑟
For 𝐼 𝐵𝑎𝑦𝑒𝑟 , this subsection presents the quality enhance-ment effects of our RCRI approach against the UPI problemsand true CI problems for 4:2:0(L), 4:2:0(R), and the state-of-the-art combinations [20], [7], [8], and [22] under the currentcoding environment.From Table IV, we observe that after deploying our RCRIapproach in DMx CU , GDx CU , 4:2:0(L)x CU , 4:2:0(R)x CU ,and 𝐶𝑆𝐿𝑀
𝐵𝑎𝑦𝑒𝑟 x CU , the average PSNR gains equal 4.8134(= (5.2940 + 5.6031 + 8.3564)) dB, 5.7962 (= (0.1047 +7.5368 + 7.2515 + 8.2917)) dB, 2.4466 (= (2.7829 + 2.3125+ 2.7198 + 1.9712)) dB, 4.1248 (= (3.2360 + 4.3958 +4.4658 + 4.4017)) dB, and 10.1278 (= (9.8250 + 10.0740+ 9.7277 + 10.8843)) dB, respectively, achieving significantquality enhancement effects.In addition, our experimental results indicate that for 𝐼 𝐵𝑎𝑦𝑒𝑟 , after deploying our RCRI approach in ‘modified4:2:0(A)’x( CU ∪ {TN}), except for ‘modified 4:2:0(A)’-TN[22], the CPSNR gains are 2.0860 dB, 3.1487 dB, 3.5034dB, and 2.8881 dB, respectively. Accordingly, the averagePSNR gain of our RCRI approach equals 2.3252 (= (2.0860 +3.1487 + 3.5034 + 2.8881)) dB, also achieving a clear qualityenhancement effect.Besides the PSNR improvement, Table IV also demonstratesthe SSIM and FSIM improvements of our RCRI approach for 𝐼 𝐵𝑎𝑦𝑒𝑟 . VI. C
ONCLUSION
For 𝐼 𝑅𝐺𝐵 and 𝐼 𝐵𝑎𝑦𝑒𝑟 , we have presented the proposedRCRI (reduced codebook and re-interpolation) approach tosolve the UPI and true CI problems existing in the traditionaland state-of-the-art combinations under the current codingenvironment. Based on the Kodak, IMAX, and Video datasets,the comprehensive experimental results have justified theclear quality enhancement effects after deploying our RCRIapproach in the traditional and state-of-the-art combinations[43], [33], [4], [20], [7], [22], [8].Our future work is to integrate our RCRI approach, the dis-crete cosine transform (DCT) based subsampling method [44],and the DCT based quantization error-minimization method[45] to achieve better quality of the reconstructed images inJPEG [12]. Moreover, our additional future work is to applyour RCRI approach to enhance the accuracy of the cross-component linear model for chroma component coding [17],[39] and to solve the accuracy degradation problem existing inthe luma-guided winner-first voting strategy [6] at the clientside for identifying the chroma subsampling method used atthe server side for high QP values. A
CKNOWLEDGMENT
The authors appreciate the proofreading help of Ms. C.Harrington to improve the manuscript.R
EFERENCES[1] J. Allebach and P. W. Wong, “Edge-directed interpolation,”
IEEE Inter-national Conference on Image Processing, pp. 707-710, Sep. 1996.[2] B. E. Bayer, “Color imaging array,”
U.S. Patent
IEEE Transactions onCircuits and Systems for Video Technology, vol. 19, no. 12, pp. 1891-1896, Dec. 2009.[4] K. L. Chung, J. S. Cheng, and H. B. Yang, “Effective chroma subsamplingand luma modification for RGB full-color images using the multiplelinear regression technique,”
IEEE Access, vol. 8, pp. 118315-118323,Jun. 2020.[5] K. L. Chung, T. C. Hsu, and C. C. Huang, “Joint chroma subsampling anddistortion-minimization-based luma modification for RGB color imageswith application,”
IEEE Transactions on Image Processing, vol. 26, no.10, pp. 4626-4638, Oct. 2017.[6] K.L. Chung, C.C. Huang, and T.C. Hsu, ”Adaptive Chroma Subsampling-binding and Luma-guided Chroma Reconstruction Method for ScreenContent Images,"
IEEE Trans. Image Processing, vol. 26, no. 12, pp.6034-6045, Dec. 2017.[7] K. L. Chung, Y. L. Lee, and W. C. Chien, “Effective gradient descent-based chroma subsampling method for Bayer CFA images in HEVC,”
IEEE Transactions on Circuits and Systems for Video Technology, vol.29, no. 11, pp. 3281-3290, Nov. 2019.[8] K. L. Chung, T. Y. Liu, and J. S. Cheng, “Novel and optimal lumamodification-based chroma downsampling for Bayer color filter arrayimages,”
IEEE Open Journal of Circuits and Systems , vol.1, pp. 48-59,May 2020.[9] L. Condat, “A generic variational approach for demosaicking from anarbitrary color filter array,"
IEEE 272 International Conference on ImageProcessing, pp. 1625–1628, Nov. 7–10, 2009.[10] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolutionconvolutional neural network,” in
Proc. Eur. Conf. Comput. Vis.,
Aug.2016, pp. 1–16.[11] R. W. Hamming,
Coding and Information Theory,
IEEE Multimedia, vol. 24, no. 2, pp. 96–103, Apr./Jun. 2017.[13] “ITU-R Recommendation BT-601-5: Studio encoding parameters ofdigital television for standard 4:3 and wide-screen 16:9 aspect ratios.”
International Telecommunications Union,
Proc. IEEE Int. Conf. Image Process.(ICIP), ∼ lucier/PHOTO_CD/BMP_IMAGES/[17] J. Li, M. Wang, L. Zhang, K. Zhang, S. Wang, S. Wang, S. Ma, and W.Gao, “Sub-sampled cross-component prediction for chroma componentcoding,” in Data Compression Conference (DCC), pp. 203-212, Mar.2020.[18] X. Li, B. Gunturk, and L. Zhang, ”Image demosaicing: A systematicsurvey,"
Proc. SPIE, vol. 6822, pp. 68221J-1-68221J-15, Jan. 2008.[19] X. Li and M. T. Orchard, “New edge-directed interpolation,”
IEEETransactions on Image Processing, vol. 10, no. 10, pp. 1521-1527, Oct.2001.[20] C. H. Lin, K. L. Chung, and C. W. Yu, “Novel chroma subsamplingstrategy based on mathematical optimization for compressing mosaicvideos with arbitrary RGB color filter arrays in H.264/AVC and HEVC,”
IEEE Transactions on Circuits and Systems for Video Technology, vol.26, no. 9, pp. 1722-1733, Sep. 2016.[21] T. L. Lin, B. H. Liu, and K. H. Jiang, “An efficient algorithm forluminance optimization in chroma downsampling,”
IEEE Transactions onCircuits and Systems for Video Technology, acceptance for publication,2020.[22] T. L. Lin, Y. C. Yu, K. H. Jiang, C. F. Liang, and P. S. Liaw,“Novel chroma sampling methods for CFA video compression in AVC,HEVC and VVC,”
IEEE Transactions on Circuits and Systems for VideoTechnology, vol. 30, no. 9, pp. 3167-3180, Sep. 2020.
TABLE IVQUALITY ENHANCEMENT EFFECTS OF OUR RCRI APPROACH AGAINST THE UPI AND TRUE CI PROBLEMS FOR 𝐼 𝐵𝐴𝑌 𝐸𝑅 . 𝐼 𝐵𝑎𝑦𝑒𝑟
DM [20] GD [7] 4:2:0(L) 4:2:0(R)
𝐶𝑆𝐿𝑀
𝐵𝑎𝑦𝑒𝑟 [8]COPY [20] BILI NEDI BICU COPY BILI [7] NEDI BICU COPY BILI NEDI BICU COPY BILI NEDI BICU COPY BILI [8] NEDI BICUPSNR 46.8980 41.6040 41.2949 41.5416 47.4129 40.9808 41.2661 40.2259 42.4150 42.8854 42.4781 43.2267 41.2777 40.1179 40.0479 40.1120 40.9136 40.6646 41.0109 39.8543[46.8980] [46.8980] [46.8980] [46.8980] [48.5176] [48.5176] [48.5176] [48.5176] [45.1979] [45.1979] [45.1979] [45.1979] [44.5137] [44.5137] [44.5137] [44.5137] [50.7386] [50.7386] [50.7386] [50.7386]Average 4.8134 5.7962 2.4466 4.1248 10.1278PSNR gainSSIM 0.9981 0.9946 0.9943 0.9945 0.9983 0.9936 0.9942 0.9924 0.9955 0.9959 0.9955 0.9961 0.9943 0.9925 0.9924 0.9924 0.9937 0.9931 0.9938 0.9917[0.9981] [0.9981] [0.9981] [0.9981] [0.9987] [0.9987] [0.9987] [0.9987] [0.9973] [0.9973] [0.9973] [0.9973] [0.9969] [0.9969] [0.9969] [0.9969] [0.9991] [0.9991] [0.9991] [0.9991]Average 0.0027 0.0041 0.0016 0.0040 0.0059SSIM gainFSIM 0.9984 0.9975 0.9973 0.9975 0.9986 0.9969 0.9972 0.9965 0.9977 0.9981 0.9979 0.9982 0.9964 0.9967 0.9968 0.9968 0.9971 0.9968 0.9972 0.9964[0.9984] [0.9984] [0.9984] [0.9984] [0.9992] [0.9992] [0.9992] [0.9992] [0.9987] [0.9987] [0.9987] [0.9987] [0.9983] [0.9983] [0.9983] [0.9983] [0.9996] [0.9996] [0.9996] [0.9996]Average 0.0007 0.0019 0.0007 0.0016 0.0027FSIM gain [23] Y. Lu, S. Li, and H. Shen, “Virtualized screen: A third element forcloud-mobile convergence,”
IEEE Multimedia Magazine, vol. 18, no. 2,pp. 4–11, Feb. 2011.[24] R. Lukac and K. N. Plataniotis, “Color filter arrays: Design andperformance analysis,”
IEEE Trans. Consum. Electron., vol. 51, no. 4,pp. 1260–1267, Nov. 2005.[25] Y. Monno, D. Kiku, M. Tanaka, and M. Okutomi, “Adaptive residualinterpolation for color and multispectral image demosaicking,”
Sensors, vol. 17, no. 12, pp. 2787, Dec. 2017.[26] Z. Ni, K. K. Ma, H. Zeng, and B. Zhong, “Color image demosaicingusing progressive collaborative representation,”
IEEE Transactions onImage Processing, vol. 29, pp. 4952-4964, Mar. 2020[27] W. Pu, M. Karczewicz, R. Joshi, V. Seregin, F. Zou, J. Sole, Y. C. Sun,T. D. Chuang, P. Lai, S. Liu, S. T. Hsiang, J. Ye, and Y. W. Huang,“Palette mode coding in HEVC screen content coding extension,”
IEEEJ. Emerging and Selected Topics in Circuits and Systems, vol. 6, no. 4,pp. 420–432, Dec. 2016.[28] "Spatial Scalability Filters", document ISO/IEC JTC1/SC29/WG11 ITU-T SG 16 Q.6,
Jul. 2005.[29] D. S. Tan, W. Y. Chen, K. L. Hua, “DeepDemosaicking: adaptive imagedemosaicking via multiple deep fully convolutional networks,”
IEEETrans. Image Processing , vol. 27, no. 5, pp. 2408-2419, May. 2018.[30] T. Vermeir et al. , “Guided chroma reconstruction for screen contentcoding,”
IEEE Trans. Circuits Syst. Video Technol., vol. 26, no. 10, pp.1884–1892, Oct. 2016.[31]
Versatile Video Coding (VVC).
Available: https://vcgit.hhi.fraunhofer.de/jvet\/VVCSoftware_VTM[32] The Video Dataset. ftp://140.118.175.164/CFASS/[33] S. Wang, K. Gu, S. Ma, and W. Gao, “Joint chroma downsampling andupsampling for screen content image,”
IEEE Transactions on Circuits andSystems for Video Technology, vol. 26, no. 9, pp. 1595-1609, Sep. 2016.[34] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. C. Loy,“ESRGAN: Enhanced super-resolution generative adversarial networks,”in
Proc. Eur. Conf. Comput. Vis. Workshops (ECCVW),
Sep. 2018, pp.1–16.[35] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Imagequality assessment: from error measurement to structural similarity,”
IEEETransactions on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004.[36] Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep networksfor image super-resolution with sparse prior,” in
Proc. IEEE Int. Conf.Comput. Vis.,
Dec. 2015, pp. 370–378.[37] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution viasparse representation,”
IEEE Trans. Image Process., vol. 19, no. 11, pp.2861–2873, Nov. 2010.[38] W. Ye and K. K. Ma, “Color image demosaicing using iterative residualinterpolation,”
IEEE Transactions on Image Processing, vol. 24, no. 12,pp. 5879–5891, Dec. 2015.[39] K. Zhang, J. Chen, L. Zhang, X. Li, and M. Karczewicz, “Enhancedcross-component linear model for chroma intra-prediction in videocoding,”
IEEE Transactions on Image Processing, vol. 27, no. 8, pp.3983–3997, Aug. 2018.[40] L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity indexfor image quality assessment,”
IEEE Transactions on Image Processing, vol. 20, no. 8, pp. 2378-2386, Aug. 2011.[41] L. Zhang, X. Wu, A. Buades, X. Li, ”Color demosaicking by localdirectional interpolation and nonlocal adaptive thresholding,"
Journal ofElectronic imaging, vol. 20, no. 2, pp. 023016, Jun. 2011.[42] X. Zhang and X. Wu, “Image interpolation by adaptive 2-D autoregres-sive modeling and soft-decision estimation,”
IEEE Trans. Image Process., vol. 17, no. 6, pp. 887–896, Jun. 2008.[43] Y. Zhang, D. Zhao, J. Zhang, R. Xiong, and W. Gao, “Interpolation-dependent image downsampling,”
IEEE Transactions on Image Process-ing, vol. 20, no. 11, pp. 3291-3296, Nov. 2011. [44] S. Zhu, C. Cui, R. Xiong, Y. Guo, and B. Zeng, “Efficient chromasubsampling and luma modification for color image compression,”
IEEETransactions on Circuits and Systems for Video Technology, vol. 29, no.5, pp. 1559-1563, May 2019.[45] S. Zhu, M. Li, C. Chen, S. Liu, and B. Zeng, “Cross-space distortiondirected color image compression,”
IEEE Transactions on Multimedia, vol. 20, no. 3, pp. 525-538, 2018, Mar. 2018.
Kuo-Liang Chung (SM01)received his B.S., M.S.,and Ph.D. degrees from National Taiwan University,Taipei, Taiwan in 1982, 1984, and 1990, respectively.He has been one Chair Professor of the Depart-ment of Computer Science and Information Engi-neering at National Taiwan University of Scienceand Technology, Taipei, Taiwan since 2009. He wasthe recipient of the Distinguished Research Award(2004-2007; 2019-2022) and Distinguished ResearchProject Award (2009-2012) from the Ministry ofScience and Technology of Taiwan. In 2020, hereceived the K. T. Li Fellow Award from the Institute of InformationComputing Machinery, Taiwan. He has been an Editor and Associate Editor ofSignals and the Journal of Visual Communication and Image Representationsince 2020 and 2011, respectively. His research interests include machinelearning, image processing, and video compression.