ART-UP: A Novel Method for Generating Scanning-robust Aesthetic QR codes
Mingliang Xu, Qingfeng Li, Jianwei Niu, Xiting Liu, Weiwei Xu, Pei Lv, Bing Zhou
JJOURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1
ART-UP: A Novel Method for GeneratingScanning-robust Aesthetic QR codes
Mingliang Xu,
Member, IEEE,
Qingfeng Li,
Student Member, IEEE,
Jianwei Niu,
Member, IEEE,
Xiting Liu, Weiwei Xu,
Member, IEEE,
Pei Lv,
Member, IEEE, and Bing Zhou,
Member IEEE
Abstract —QR codes are usually scanned in different environments, so they must be robust to variations in illumination, scale, coverage,and camera angles. Aesthetic QR codes improve the visual quality, but subtle changes in their appearance may cause scanning failure.In this paper, a new method to generate scanning-robust aesthetic QR codes is proposed, which is based on a module-based scanningprobability estimation model that can effectively balance the tradeoff between visual quality and scanning robustness. Our methodlocally adjusts the luminance of each module by estimating the probability of successful sampling. The approach adopts the hierarchical,coarse-to-fine strategy to enhance the visual quality of aesthetic QR codes, which sequentially generate the following three codes: abinary aesthetic QR code, a grayscale aesthetic QR code, and the final color aesthetic QR code. Our approach also can be used tocreate QR codes with different visual styles by adjusting some initialization parameters. User surveys and decoding experiments wereadopted for evaluating our method compared with state-of-the-art algorithms, which indicates that the proposed approach has excellentperformance in terms of both visual quality and scanning robustness.
Index Terms —Aesthetic QR codes, error analysis, visualization optimization, scanning robustness, scanning probability calculation. (cid:70)
NTRODUCTION A QUICK response (QR) code is a matrix symbology con-sisting of an array of nominally square modules. Thepopularization of smartphones has brought about wide useof QR codes in connection with offline and online life. Thesecodes offer the advantages of large information capacity, lowcost, and easy manufacture, etc [1]. At the same time, theirappearance, consisting of black and white square modules,is difficult to meet the individualized demands of customersbecause of lack of aesthetic elements.The process of embellishing QR codes always aims toimprove the visual quality of their appearance, which makesthem more interesting and more appealing [2], [3], [4], [5],[6], [7]. They can incorporate high-level semantic featuressuch as faces, letters or logos into products of plain designand contribute to brand promotion. However, changing theappearance of QR codes manually is costly and difficult toachieve, as designers have to repeatedly confirm that theresult can be recognized by general QR scanners. Conse-quently, changing in an automatic way is desirable, for thepurpose of efficiency and low additional design cost.The main challenge in embellishing QR codes is toensure that the original information is not influenced andcan be scanned by general scanners correctly even when thecode’s appearance has been changed. Scanning robustnesscan be defined as the property describing whether theembellished QR code is easy to be scanned correctly byuniversal scanners. When QR codes are scanned in real-life applications, good scanning robustness is especiallyimportant, as it reduces the impact of various environmentalfactors, such as illumination, noises, coverage, and cameraangles, etc, during the scanning process.Traditional QR code images express information viamodules using two highly contrasting colors. While theReed-Solomon algorithm [8] is used to enhance fault tol-
Fig. 1. Various types of aesthetic QR codes erance and maintain the scanning robustness [1]. On theother hand, embellished aesthetic QR codes enhance visualquality and contain visual information by introducing othercolors, changing module shapes, embedding icons, or ad-justing codewords. For example, some representative resultsof aesthetic QR codes are shown in Fig. 1.In this work, we propose ART-UP codes - sc A nning- R obust aes T hetic QR (q U ick res P onse) codes. We establisha new module-based scanning probability estimation modelto measure the scanning robustness of aesthetic QR codes,resulting in QR codes with the same error tolerance butimproved embellishment. More specifically, we first ana-lyze the steps and fundamental reasons that cause errorsduring QR code scanning in accordance with QR decodingprinciples. Scanning errors are classified as either thresh-olding errors or sampling errors. Then, error models areaccordingly established, with corresponding estimations ofthe probability of correctly sampling each module.We then use an iterative luminance adjustment solutionto locally adjust the luminance of each module by combin-ing these probability estimation models, so as to increase theprobability of correctly sampling and to improve scanning a r X i v : . [ c s . MM ] M a r OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 2 robustness. In addition, in order to obtain better visualquality and make the modification more effective, we com-bine the image saliency with the corresponding probabilityconstraint, which generates QR codes that are scanning-robust and well embellished.The major contributions of this work are as follows: • The scanning and decoding process of QR codes isanalyzed, threshold and sampling errors that affectthe robustness of embellished QR codes are quanti-fied, and a module-based scanning probability esti-mation model is built in this paper, which is used toestimate the scanning robustness of the QR code. • An optimization strategy based on local luminanceadjustment algorithm is proposed, which can balancethe tradeoff between visual quality and scanning ro-bustness. And a new method of generating aestheticQR codes is established, which can achieve state-of-the-art visual quality and preserve error correctionabilities. • An interesting threshold estimation algorithm is pro-posed, which is flexible enough to generate aestheticQR code images with different styles by adjustinginitialization parameters. • An easy to implement and effective luminance ad-justment algorithm based on linear interpolation isproposed, which is fast, accurate. This algorithm canbe applied to convert a grayscale aesthetic QR codeinto a color one.
ELATED W ORK
QR codes were first invented for tracking vehicles and partsduring the manufacturing process, recently with the rapiddevelopment of mobile Internet, these have been widelyapplied to many different fields. This has resulted in anincreased research interest in the technology necessary forembellishing QR codes [2], [3], [4], [5], [9], [10], [11], [12],[13], [14], [15], [16].Currently, the existing foundation algorithms of generat-ing embellished QR codes can be classified into four groups:embedding icons [17], replacing colors [18], changing themodule shape [3], [14], and adjusting codewords [2]. Amongthe four ways, the one that embedding icons is the easiestto implement, but it relies on the inherent error correctioncapacity of the QR code, so its controllable area is relativelysmall and monotone, and reduces the error correction abilityof the algorithm [11], [17]. Replacing colors and changingthe module shape for attaching semantic information oftenneed manual intervention and difficult to handle. [14]. Gen-erating QR codes by adjusting codewords usually achievesscanning robustness, but the resulting visual quality is poorand the result is usually inconsistent [2].In recent studies, researchers usually try to mix QR codemodules with arbitrary input image in order to obtain a gen-eral method. For example, Cox [2] looked into the principleof QR code encoding and, by combining the characteris-tics of Reed-Solomon code with Gauss-Jordan eliminationalgorithm, proposed a complex method which adjusts thecodewords of QR code in the encoding procedure, makingthe resulting image similar to the binary image. However,this method is fit for encoding URL data only. Through constant optimization, this algorithm has become the mostefficient algorithm for adjusting codewords.Apart from this, in 2013 Chu et al. [3] proposed HalftoneQR Codes, combining the halftone algorithm with embel-lished QR codes. By separating each module into 3-by-3 sub-modules and binding the module color to the central sub-module color, a replacement algorithm of was proposed,which offered reliability and regularization in generatinghalftone QR codes. However, the algorithm adopts a nonlin-ear optimization procedure which suffers from inefficiencyand low-quality visual result issues.Subsequently, Garateguy et al. [9] proposed a new al-gorithm using halftone, which was based on the selectionof a set of modified pixels using a halftoning mask and aprobabilistic model predicting the distortion generated bythe embedded image. Although it can enhance the visualquality by sacrificing some scanning robustness, the gener-ated images still contain much image noise.To further improve the visual quality of aesthetic QRcodes, Yu-Hsun et al. [4] and Shih-Syun et al. [7] proposedtwo new algorithms. Yu-Hsun et al. took the saliency the em-bedded image into consideration so that pixels are modifiedmore effectively, which offers a remarkable improvementin the visual quality of high version QR codes, but theeffect of improving low version codes was limited. On theother hand, Shih-Syun et al. proposed an easy and efficientmethod, where the QR code module sequence was firstadjusted to maintain its global similarity with the inputimage based on the Gauss-Jordan elimination procedure,and then a rendering mechanism was designed to blend theinput image into the QR code according to a weight map fora better visual quality. Although the visual effect was quitegood, it severely degraded the scanning robustness.Apart from constantly improving the visual effect ofQR codes, maintaining scanning robustness is also a chal-lenge that researchers need to take into consideration. Alva et al. [19] proposed a commercial algorithm (Visualead) forgenerating scanning-robust embellished QR codes, but itsimplementation is closed and its appearance always givespeople a feeling of clutter. On the other hand, Zhang et al. [5] came up with an aesthetic QR code generation algorithmbased on two-stage image blending, as well as module-based and pixel-based blending. This resulted in a relativelyrobust embellished QR code generating algorithm, however,this algorithm resorts to empirical rather than theoreticalanalysis, which results in some images being hard to scancorrectly for unexplained reasons, making the algorithmunsuitable for general use.In this paper, we analyze theoretically and guaranteethe scanning robustness of QR codes by establishing anerror analysis model to estimate the probability of a singlemodule being scanned properly. Meanwhile, by designingan adaptive parametric method, visually appealing andhighly robust QR codes are obtained.
In order to generate an embellished QR code which isscanning robustness, we must analyze the details of thewhole scanning and decoding process, and determine theerror-generating factors that may affect scanning. This willbe discussed in this section.
OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 3
A QR code is a matrix that consists of white and blackmodules ordered by a specified encoding rule, where eachmodule’s white or black color represents a bit, i.e. a 0 or1, and 8 bits make up a codeword. Further, the codewordsequence is always subdivided into one or more blocks,and each block is divided into two parts: data codewordsand error correction codewords. More information about thegeneration of QR codes can be found in [1].Typically, the scanning process involves the use of ter-minal devices to obtain the QR code image from screens orprints via a camera and then using image processing tech-nology to locate and sample the black-and-white modules.According to the sampled information, the QR code imageis transformed into a matrix, which is subsequently decodedusing a decoding algorithm [20], [21], [22], [23].In accordance with the most well-known QR code gen-erating and scanning approach, based on the open-sourcelibrary ZXing [20], we divide the scanning process into threesteps: preprocessing, detection and recognition. Preprocess-ing includes transforming the RGB image into a grayscaleone, and then converting the 256-gray intensity level imageinto a binary one. Detection involves finding and confirmingthe exact location of QR code, while recognition consists ofdown-sampling and decoding, as shown in Fig. 2.
Images captured from the camera and transformed into abinary one during the preprocessing step, which mainlycontains two steps. Firstly, converting the RGB image into asingle-channel one, using the equation below: Y x = αC rx + βC gx + γC bx (1)where α = 0 . , β = 0 . , γ = 0 . , Y x is the outputvalue of the single-channel image at position of x , and C rx , C gx , C bx respectively correspond the input values of the red,green, and blue channels of the color image at x .Then, converting an image into a binary one is also basedon the commonly used approach of thresholding: H x = ζ ( Y x , t x ) (2)where H x and t x are respectively the binary result andthreshold at position x , ζ ( Y x , t x ) is a function which assignsthe value 1 to H x when Y x ≥ t x , and 0 otherwise.Choosing a proper threshold is key to the binarizationeffect that influences the performance of QR code detection,and ZXing uses a hybrid local block averaging method forit. That is, the obtained image is first divided into imageblocks, the size of each is 8-by-8, without overlapping, andthe average of each block is calculated. Then, a 5-by-5 setof blocks is formed around each block and the averages ofthese blocks are calculated as the final threshold.To facilitate the discussion, in the following, the i -thblock is denoted as S i , while the set of its neighboring blocksis denoted as Γ( S i ) . When pixel x is located inside block S i ,this is written as x ∈ S i . In the same block, all the pixelswill be compared against the same threshold T i . So, for anypixel x ∈ S i , we have t x = T i = 1 | Γ( S i ) | (cid:88) S j ∈ Γ( S i ) ( 1 | S j | (cid:88) k ∈ S j Y k ) (3) Graying Locate the
Finder Pattern
Original Message
DownsamplingRestore and Decode Thresholding
Fig. 2. QR code scanning process
For general QR codes, this threshold method helps re-duce the scanning error, especially for different illuminationenvironments. However, for aesthetic QR codes, the imagecolor will seriously affect the threshold calculation, and it iseasy to obtain an unexpected binary result because a toohigh or too low value of the threshold is calculated. Asa result, the reduction of this effect is the key factor formaintaining scanning robustness.
In the detection phase, the main task is to locate the positionof the QR code from the binary image generated after pre-processing. Pattern matching is used as the primary methodfor locating the finder pattern.The finder pattern is located on three corners of the QRcode image and each component comprises three concen-tric squares. The three squares are formed by black 7-by-7modules, white 5-by-5 modules, and black 3-by-3 modules,respectively. During detection, a black-white-black-white-black pattern with corresponding ratios of 1:1:3:1:1, corre-sponding to the cross-section of the finder pattern, will bematched in order to quickly identify the existence of a QRcode. Finally, according to the relative positions of the 3finder patterns, the exact position and direction of the imageare confirmed.
The process of recognizing QR codes mainly includes sam-pling and decoding. Sampling refers to first estimating thesize of each module, and then obtaining the number of themodules. After the center pixel of each module is sampledto obtain the information of the whole module, finally, amatrix is formed once all the modules have been sampled.On the other hand, the decoding process involves ex-amining the matrix that is obtained during sampling andparsing the contained information. This involves inverselyresolving according to the rules of QR encoding, includingdata masks, codeword rearrangements, error correction, anddecoding. The Reed-Solomon algorithm is used in the pro-cess of encoding, and is thus necessary for decoding errorcorrection. When the errors in a block are overwhelming, theoriginal data will not be retrievable, resulting in the failureof resolving the QR code.
When the QR code is located correctly and the data matrixobtained finally is decoded properly, we regard this code
OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 4 a.b.d.c. (1)Input Images (2)Binary Images (3)Sampling Results (4)Errors
Fig. 3. Error analysis of the intermediate results of scanning different QRcodes using the ZXing library. The last column shows the error images,where errors are depicted in red and correct measurements in green.a. QR code image generated using QART; b. QART QR code imagecaptured using a mobile device under ordinary illumination; c. QR codeimage generated using CA; d. CA QR code image captured using amobile device under ordinary illumination. to be an accepted one. However, during actual scanningprocesses, all kinds of noise and errors may occur. To furtheranalyze and show the various errors produced during thescanning process, we conducted two typical aesthetic QRcode implementations for experimentation, QART [2] andCA [5]. As shown in Fig. 3, we used the ZXing library tocompare and maintain the intermediate experimental resultsfor further identification of errors.In Fig. 3, column 1 shows the input images. Apart fromdisplaying the scanning process of the original QR codeimages (Rows (a) and (c)), we used mobile devices to obtaina group of images (Rows (b) and (d)) to compare resultswith those obtained in conditions of ordinary illumination,i.e. simulating the daily scanning. Column 2 shows the bi-narization results obtained after processing using the ZXinglibrary. Column 3 displays the downsampling results of QRcodes after detection, that is, the matrix of sampling results.In the end, we compare the real sampling result matrix withthe expected one, generating Column 4, with red for errorsand the green for correctly identified values, which directlyshows where errors were generated.As evident from the results, images a-(4) and b-(4) aretotally green, which indirectly shows the good scanningrobustness of QR codes generated by QART. Regardless ofwhether they were scanned directly or using mobile devices,the results turned out to be in line with expected ones.However, images c-(4) and d-(4) contain errors, and theerrors of d-(4) are much more severe than that of c-(4).According to scanning principles, if there are errors inthe matrix of sampling results, they need to be corrected byperforming data correction such as using the Reed-Solomoncode. When the percentage of the generated errors exceedsa certain threshold, error correction may fail, which resultsin a scanning failure. This paper proposes a method to estimate the probabilityof produced errors within a module during the generationprocess and taking actions to prevent the errors occurring.It obtains the final correct sampling result matrix and im-proves the scanning robustness, where the details will bedescribed in Section 4.2.1.
ESIGN AND I MPLEMENTATION
To present the generation of ART-UP codes more clearly,we divide the generation process into three stages: thebinary aesthetic stage, the grayscale aesthetic stage, and thecolor aesthetic stage. In the binary aesthetic stage, codewordadjustment is achieved using visual saliency and Gauss-Jordan elimination, and the order of modules is adjustedto match the binarization result of the input image, whichavoids visual conflicts globally. In the grayscale aestheticstage, thresholding error and sampling error models areestablished by analyzing the scanning process of generalscanners to simulate the process of thresholding and down-sampling. In this manner, the probability of correctly sam-pling each module can be estimated and used as feedbackfor improving the generation process and ensuring the scan-ning robustness. In the color aesthetic stage, the pixels ofeach channel are calculated accurately to create the coloredaesthetic QR codes by establishing a linear-based solution,which adjusts the luminance of original images to match thegrayscale QR codes.Fig. 4 shows the whole process of generating ART-UPcodes, where Q b is the binary aesthetic QR code, Q g is thegrayscale aesthetic one, and Q c is the colored aesthetic one.In the following three Sections, the methods adopted foreach stage will be discussed in details. The generation of binary aesthetic QR code is the main pre-step for the generation of grayscale code. The result willdirectly determine the structural visual effect [24] of thefinal colored code. In this step, we first generate a grayscaleimage I g in accordance with the original one I and then gen-erate a binary image I b using a module-based thresholdingalgorithm. After that, the codewords of the original QR code Q s will be adjusted according to the priority weight W inorder to generate the target binary aesthetic QR code Q b . To make the generated binary aesthetic QR code’s similarto the original one, we need to generate a binary imageby processing the original QR code. The side length ( n ) ofthe grayscale image is usually much larger than the lengthof individual modules ( l = 4 × V + 17 , where V is theversion of the QR code), so if we only scale the image I g to obtain its binary image, the visual effect is not ideal.Here, the approach used in [5] is adopted to avoid suchphenomena. First, the image I g is broken down into severalsquare modules. Let the side length of each such module be a = nl , that is, equal in size to the QR code module. Finally,we mark the k -th module as M k , so the binarization of themodule is given by I bk = ζ ( (cid:88) x ∈ M k I gx G ( x ) , (4) OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 5
Input Image( I ) update threshold Visual Saliency Weights( W ) Grayscale Aesthetic QR Code( Q g ) Expected Probability of Modules( η )Grayscale Image( I g ) Binary Image( I b ) Binary AestheticQR Code( Q b ) Standard QR Code( Q s ) Color AestheticQR Code( Q c )Local ThresholdDistribution( L ) Fig. 4. Overview of scanning-robust aesthetic QR code generation process. Lines of different colors represent different stages. 1 (cid:13) the binaryaesthetic stage; 2 (cid:13) the grayscale aesthetic stage; 3 (cid:13) the color aesthetic stage where I bk stands for the k -th element of image I b , G ( x ) isthe weight of the pixel x , and (cid:80) x ∈ M k G ( x )=1 . In this paper, G ( x ) is a Gaussian Distribution: G ( x ) = G M k ( i, j ) = 12 πσ e − ( i − a/ j − a/ σ (5) x belongs to M k , and G M k ( i, j ) refers to the weight of the k -th module at the position of ( i, j ) . It is necessary to notethat in Equation 5, the position of the k -th module ( i, j ) corresponds to the global position of x . In addition, in ourexperiment, we set σ to be a − . From the analysis of QR code scanning principles, it isevident that the scanned QR code will be accepted bythe scanners as long as the order of the modules passesthe error correction stage. QR codes employ Reed-Solomonencoding, which has the following properties: 1) It is asystematic code, where the input data is embedded in thefinal encoded output. That is, the first half is the originalinput code, with the latter half being the error correctioncode; 2) The result of an exclusive-or between two differentReed-Solomon encoded blocks, is also a valid Reed-Solomonencoded block.During encoding, each block is an independent Reed-Solomon code, and a block can be separated into 3 areas.One is for input data bits, one for padding bits, and the restfor correction bits, with the corresponding lengths being m , p , and c , respectively. According to the QR encoding rules,when the version and error correction level of a QR code areconfirmed, m + p and c are fixed constants, while m and p may change according to the size of the input data.To describe the process of codeword adjustment clearly,consider Fig. 5a as an example. The first row contains the original information. In order to generate the aestheticbinary QR code Q b in Fig. 4 similar to the binary image I b , let us assume that the k -th ( k > m and k ≤ m + p )digit needs to be adjusted from 1 to 0 in the block. Then, aspecial operator is constructed whose k -th digit is 1 whilethe other digits in the input data and padding bits are 0. Thecorrection bits are generated by the checking rules of Reed-Solomon coding. According to Property 2, a legal Reed-Solomon code block will be obtained through the exclusive-or operation of the special operator and the original code.The generated code will maintain the other digits of theinput data area and padding area unchanged, while the k -th digit will go through an inversion operation.As described above, a group of operators A can be con-structed, as shown in Fig. 5a. There are p operators markedas a , a , · · · , a p , respectively, where digits m +1 , m +2 , · · · , m + p of the corresponding operators are 1, while the restof the digits of the input data area and padding area are 0.That is, there is only one row with a 1 for each controllablemodule in the padding data area, which is marked with red.With this group of operators, we can apply an exclusive-oroperation on the current data with a k − m to manipulate the k -th digit to the desired value without affecting any of theother controllable modules. The intermediate result adjustedby Set A is shown in Fig. 5c.In order to maintain the input data bits unchanged andthe error correction bits updated, the controllable modulesmust be limited within the padding data area ( k > m and k ≤ m + p ). However, the Gauss-Jordan elimination methodcan be adopted to overcome these constraints and allowfurther flexibility. By combining the operators in set A , wecan create a new operator set B and apply an exclusive-or between the operators to create new basis operator thattrades data bits for error correction bits. In this way, the OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 6 uncontrollable modules are dispersed, and the noise pixelsare distributed over the canvas, as shown in Fig. 5d.It should be noted that the input data bits (including theending indicator) have not been changed during this pro-cess, so according to Property 1, the information containedin the QR code is not affected by the codewords’ adjustment,which is necessary to ensure a correct decoding outcome.
As known above, during codeword adjustment, the control-lable modules can be only moved within a certain rangedue to the QR codes’ encoding rules’ limitation, no newmodules can be added. In each block, the color of the p con-trollable modules can be freely controlled, while the otherareas will be affected by the input data and the checkingalgorithm. Therefore, a priority algorithm is proposed onhow to choose controllable modules.In this paper, a linear combination of saliency, edgedetection, and heuristic constraints is used to determine thispriority for each module of QR code: W = λ Edge + λ Sal + λ Heu (6)
Edge and
Sal are the results of edge detection and visualsaliency respectively for image I . To make the W and Q s equal in size, a mean-pooling operation is carried out. Heu defines some heuristic rules, so that in cases where twomodules have similar
Edge and
Sal , it allows the one whichis closer to the image center to obtain a larger weight.
Heu x,y = 1 − ( x − l ) + ( y − l ) l = xl + yl − x − y l (7)In Equation 7, Heu x,y refers to the heuristic variable’s resultat the module position ( x, y ) . l is the number of the moduleson each side of the code, that is, l = 4 V + 17 . In thispaper, the Canny [25] and Region-based Contrast (RC) [26]methods are adopted for the edge detection and saliencyarea extraction, respectively. After normalizing Edge , Sal and
Heu , the values of λ , λ , and λ are determinedto be 0.67, 0.23 and 0.10 respectively in our experiments.Although Heu only has a small weight, for most of imagesprovided by users, such as logos and human faces, thechoice of controllable modules always plays a key role inthe final processing.
To further enrich the details of QR code, the visual effectneeds to be optimized by combining it with an image togenerate a grayscale one. As shown in Fig. 4, first the initiallocal threshold distribution L according to the grayscaleimage I g and binary aesthetic QR code Q b is pre-estimated.Then, a module-based luminance modification algorithmis used to adjust the pixels of the modules in the corre-sponding grayscale image I g , ensure scanning robustnessin accordance with the expected probability distribution η ,and combine it with Q b to generate a grayscale QR code Q g . After that, the local threshold distribution L of thereal image is recalculated and updated through Q g , and OriginOperator
Target
Input Data Bits( m ) Padding Bits( p ) Correction Bits( c ) k = A Group of
Operators (A ) A Group of
Operators( B ) 00000000 (a)(b) (c) (d)Fig. 5. Codeword adjustment using the Gauss-Jordan eliminationmethod. a. Generation of the operator set; b. Original QR code; c. Binaryresult of codeword adjustment by operator set A; d. Binary result ofcodeword adjustment by operator set B; luminance modification is used to recreate Q g . With con-tinuous iterations that make the local threshold distributiongradually steady, the final Q g is obtained.During generation of grayscale aesthetic QR codes, amodule-based scanning probability estimation model is pro-posed in this paper, which is used to estimate the scanningrobustness of the QR code. After that, an iterative imagecolor updating algorithm is proposed to adjust the grayscaleaesthetic QR code, making it robust and visually appealing.These will be introduced in the following sections. According to the scanning process analyzed in Section 3.1and the scanning error analysis of Section 3.2, we classifyerrors resulting from scanning failure into two types: thresh-olding errors and sampling errors. Thresholding errorsmainly refer to those that result from the difference betweenthe real and expected threshold due to the environment,camera devices, and image content local thresholding. Sam-pling errors occur during sampling because of location andscaling errors, where the real sampling position is far fromthe module center. In the following, we will analyze these indetail and propose the module-based scanning probabilityestimation model.
A. Thresholding Error Estimation
As can be inferred from Section 3.1.1, generic QR codescanners use the thresholding method based on hybrid localblock averages. In theory, the threshold applied to a certainpixel only depends on image blocks and their neighboringpixels during the thresholding process. However, threshold-ing problems are complicated in reality, the following factorsmust be taken into account: 1) In most realistic scenarios, thecolors of the obtained pictures may differ from the original
OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 7 ones as they are affected by environmental conditions; 2) Inthe thresholding method based on the hybrid local block av-erages, the captured image is divided into non-overlappingimage blocks at first, the adapted thresholds are the samewithin the same block, but actually the position of thepixels cannot be confirmed during separation because of theunknown position and angle of the obtained QR code in theimage; 3) In theory, the size of the image blocks is definedaccurately, but in reality due to the scanning distance andimage scaling, the relative size of the image block and theQR code module is not fixed.Although an accurate determination of actual thresholdvalues of pixels may be difficult to achieve, good estimationsmay be possible under certain assumptions. Firstly, as forthe problem of color deviation, usually the captured color issimilar to the real color. Without considering the limitationsof color thresholding, we can suppose the probabilities thatthe captured color is more or less intense than the real oneare equal, and with the increase of the changing amplitude,these probabilities are gradually reduced. Secondly, due tothe randomness of the position and angle of the code withinan image, resulting in random pixel locations for the blocks,we may as well assume that the pixel is always located in thecenter of the block. In this way, in accordance with Equation3, the thresholding method based on hybrid local blockaverages can be replaced by the thresholding method basedon the local average, that is, the average of the neighboringpixels can be used to determine the threshold. Luckily, evenif the pixel is not located in the block’s center, its realthreshold can be estimated from the surrounding area as-suming that the local image area exhibits color continuity. Inaddition, the influence of the scanning distance and imagescaling have on the real threshold is random, and dependson the scanning performed by the users, resulting in anunknown percentage of the QR code lying within the image,usually clustered within a certain area, which means that thethreshold to change within the corresponding ranges.According to the above analysis and observations, it isevident that for the QR code, the same pixel under differentenvironments may be compared to different thresholds,resulting in different binarization results. General methodsof generating aesthetic QR codes are usually applied withthe assumption that the thresholds are constant. This as-sumption leads to a reduction in scanning robustness. Inthis paper, in order to simulate the real thresholding process,we assume that the expected threshold of a certain pixel x in an aesthetic QR code is obtained from the average ofits surrounding pixels, denoted as t xo . The actual thresholdvalue, which is affected by the environment, lies close tothis value, and follows a Gaussian distribution p τ ( t x ) , with t x = t xo being its maximum value. p τ ( t x ) = 1 √ πσ e − ( tx − txo )22 σ (8)Meanwhile, according to the above, we have t xo t xo = 1 | R( x ) | (cid:88) i ∈ R( x ) Y i (9), Where R( x ) represents the set of pixels neighboring thecenter x . In this paper, R( x ) is a square area, with the sidelength of a , where a is the side length of the QR module. Pixel Value (Y x ) P r ob a b ilit y ( p x t f o r Q kb = ) xo = 0t xo = 128t xo = 255 Fig. 6. Variation curve of p tx with Y x corresponding to different valuesof t xo when the color of corresponding module is black for a binaryaesthetic QR code ( Q bk = 1 ). When we obtain the distribution of thresholds in posi-tion x , we compare the original pixel value with the realthreshold of this pixel, according to Equation 2. In themeantime, due to the limitations of color thresholding, thereal threshold is actually distributed in the range of [0 , .Therefore, the probability that the thresholding result equals0 or 1 is, respectively P ( H x = 1) = P ( Y x ≥ t x ) = (cid:90) Y x p τ ( t x )d t x (10) P ( H x = 0) = P ( Y x < t x ) = (cid:90) Y x p τ ( t x )d t x (11)Similarly, because of the limitations of color threshold-ing, the sum of the two equations above will not be equalto 1. In this paper, it is normalized for adjustment, so whenthe pixel x is located in the module M k , that is, x ∈ M k , theprobability of correctly thresholding the pixel is p tx = p t ( Y x ) = P ( H x = 1) P ( H x = 0) + P ( H x = 1) , for Q bk = 1 P ( H x = 0) P ( H x = 0) + P ( H x = 1) , for Q bk = 0 (12)In Fig. 6, the Y x - p tx curves under different values of t xo are shown for σ = , Q bk = 1 . From this, it can bederived that when the expected threshold of the pixel isfixed, the probability of correctly thresholding the pixel ismonotonically related to the real color luminance, that is,we can adjust the grayscale value of a pixel to raise theprobability and thus increase robustness. B. Sampling Error Estimation
According to Section 3.1.3, in ideal conditions, the down-sampling results only relate to the center pixel of the modulein the scanning process. However, in real environments,images captured by cameras are likely to be different fromthe original ones, due to rotations, scaling, and even trans-figuring, which result in the sampling error being affectedby the surroundings.Furthermore, this sampling error can hardly be simu-lated realistically, as this would require different applicationsurroundings and large amounts of experiments. In thispaper, a reasonable assumption is made that the probabilityof a pixel being sampled in a module follows a Gaussian
OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 8 a Fig. 7. The probability distribution of pixels being sampled within amodule. We assume that the module is sampled one thousand times,and each sample corresponds to a blue point. distribution. As shown in Fig. 7, in a module, the closer thepoint is to the center, the more likely it will be sampled. p sx = p sM k ( i, j ) = 12 πσ e − ( i − a/ j − a/ σ (13)In addition, in the whole module, the sum of the probabili-ties for sampled pixels is 1, that is, (cid:80) x ∈ M k p sx = 1 . C. Probability of Correctly Scanning a Module
As can be inferred from the scanning process analysis,thresholding and downsampling are performed during dif-ferent steps of the process, and the result of thresholding apixel is independent of sampling. In this way, a pixel will beselected as the sampling result according to the samplingprocess, so the probability of sampling this pixel as thecorrect result is: p mx = p sx p tx (14)Therefore, the probability of correctly scanning a module is P M k = (cid:88) x ∈ M k p mx (15) To balance the scanning robustness and the visual quality ofaesthetic QR code, in this section, a luminance modificationalgorithm within the module is proposed to reconcile thebinary aesthetic QR code Q b and the grayscale image I g . Inaddition, the expected threshold of the pixels is establishedby means of an iterative algorithm. In this manner, thegrayscale aesthetic QR code Q g is finally obtained. A. Luminance Modification Algorithm
The whole QR code is made up of many modules. Byestimating the probability of correctly scanning a singlemodule, we can indirectly work out and adjust the errorprobability in the matrix of the sampling result. To guaran-tee the code’s scanning robustness, this probability must bemodified accordingly so as to be able to recover from errorsthat may occur through error correction.In this section, we assume that the expected threshold t xo of each pixel during the thresholding process is a knownvariable. So, according to Equation 12, the corresponding p tx can be calculated. Also, in accordance with Section 4.2.1,the probability P M k of correctly scanning each module inthe grayscale aesthetic QR code can be calculated, underthe constraint of correctly scanning the modules in order tomaintain the scanning robustness of the whole QR code, P M k ≥ η k (16) Algorithm 1
Luminance Modification Algorithm
Require: Y x , t xo , p sx , (cid:36) px for x ∈ M k , and η k Ensure: P M k ≥ η k in module k of Y
1: Calculate p tx for x ∈ M k , see equations 8,10,11,122: maxpt ← while true do P M k ← (cid:80) x ∈ M k p sx p tx if P M k ≥ η k then break end if ℘ ← (cid:80) x ∈ M k p sx (cid:36) px if ℘ = 0 then break end if for each x ∈ M k do p tx ← p tx + (cid:104) max( η k ,P Mk ) − P Mk (cid:105) (cid:36) px ℘ if p tx ≥ maxpt then p tx ← maxpt (cid:36) px ← end if end for end while for each x ∈ M k do
21: find Y ∗ x = argmin Y x (cid:12)(cid:12) p t ( Y x ) − p tx (cid:12)(cid:12) Y x ← Y ∗ x end for where η k is the minimum probability constraint for theexpected correct scanning of the k -th module. Here, aninitial fixed value can be given, such as . The choiceof this value will be discussed later in Section 4.4.As can be inferred from Equation 13, after confirmingthe size and version of the QR code image, the probability p sx of each pixel in the module is actually a fixed value.So, according to Equation 14 and 15, when P M k < η k ,where the probability of correctly scanning the module islower than the expected one, we have to adjust P M k to meetthe constraint by increasing the probability p tx of correctlythresholding the pixels in the module. In order to achievethis, the following equation is formed for adjusting thethresholding probability within a module: p tx ← p tx + [max( η k , P M k ) − P M k ] (cid:36) px (cid:80) x ∈ M k p sx (cid:36) px (17)where (cid:36) px is the weight for adjusting the thresholdingprobability at position x . It is not difficult to see thatwhen the equation is multiplied by p sx , and all the ele-ments within the module M k are summed, P M k ← P M k +[max( η k , P M k ) − P M k ] can be maintained. In this manner,the probability of scanning the module can be updated tobecome larger than or equal to η k .However, Equation 17 ignores the limitation of p tx , thatis, ≤ p tx ≤ . During the actual updating process, thismay result in a phenomenon where the updated proba-bility of correctly scanning the module may be smallerthan expected. Therefore, a simplified iterative updatingalgorithm is proposed (Algorithm 1). During the iterations, (cid:36) px is adjusted dynamically according to p tx , thus solvingthe above problems.Through this algorithm, the correct thresholding proba-bility of each pixel in an image can be assigned to acquire p tx and make the probability of correctly scanning each OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 9
Algorithm 2
Threshold Estimation Algorithm
Require: (cid:36) px for each x and η k for each k Ensure:
Grayscale aesthetic QR code Q g
1: Calculate p sx ← p sM k ( i, j ) = πσ e − ( i − a/ j − a/ σ for each x
2: Initialize Q g ← . × I g + . × Q b
3: Initialize L old , L , n ← while true do Y ← Q g for each x do t xo ← | R( x ) | (cid:80) i ∈ R( x ) Y i L x ← t xo end for if n (cid:54) = 0 then if L old = L then break end if end if n ← n + 1 Y ← I g for each M k do
18: prepare Y x , L x as t xo , p sx , (cid:36) px for x ∈ M k , and η k
19: update each Y x using algorithm 1 for x ∈ M k end for Q g ← Y L old ← L end while module meet the expected value. As evident from Fig. 6,the grayscale value Y x that each pixel adopts reflects theprobability of correctly thresholding pixel p tx when t xo isfixed. So, by looking up in the table, the grayscale value cor-responding to p tx can be easily obtained and the luminancemodification of the pixels in an image can be accomplished. B. Threshold Estimation Algorithm
In Algorithm 1, it was assumed that the adopted expectedthreshold t xo of each pixel is known, and the luminanceof the pixels was adjusted based this threshold. However,according to Equation 9, t xo is related to luminance Y x . So,when adjusting the luminance of the pixels, the expectedthreshold will be changed.To find a proper threshold which makes the adjustedluminance of the pixels in line with the assumed expectedthreshold, an iterative threshold estimation algorithm (Al-gorithm 2) is developed, which estimates the expectedthreshold and gradually converges to the desired value.According to observations, the algorithm iterates effec-tively, and the convergence of L is very fast even withan increased number of iterations. For example, in ourexperiments, dealing with a 512-by-512 image only needsabout 10 iterations to reach a steady state. The generationof a specific QR code is shown in Fig. 8. As the number ofiterations grows, the percentage of pixels that need updatingdecreases quickly, and the appearance of the QR code isaccordingly stabilized. After the generation of the grayscale QR code has been com-pleted, it needs to be transformed into a color aesthetic oneunder the constraint of not changing the luminance. As canbe inferred from Equation 1, the mapping between RGB andgrayscale is many-to-one, so there are many ways to convertthree-color-channel images to single-channel grayscale ones. P e r ce n t a g e o f M od i f i e d A r ea Iterations
Fig. 8. The percentage of the modified area decreases quickly as thenumber of iterations increases.
During color aesthetic QR code generation, not onlymust the luminance be maintained, but the generated colorimage Q c needs to resemble the original image I to guaran-tee the visual effect. The general method is to first convert I from RGB space to another color space (such as HSL, LAB),then to keep other channels from changing while only adjustthe luminance channel until the corresponding grayscaleimage equals Q g , and finally convert the image back toRGB space. However, when converting the color space, therange of color expression is always limited, so simply ad-justing luminance may not meet the expected requirementsin some cases. In addition, luminance adjustment is usuallytransformed into an optimization problem, which increasesthe cost of calculation [9].Inspired by the bilinear interpolation algorithm, in thispaper a linear-based luminance adjustment algorithm isproposed, which alleviates the above problems. In thismethod the process of luminance adjustment as is treatedas a linear interpolation process. For convenience, we firstdefine C m ( x ) , which represents the minimum or maximumluminance value of position x . That is, when x ∈ M k , C m ( x ) = (cid:40) [255 , , T , for Q bk = 1[0 , , T , for Q bk = 0 (18)After that, according to the principles above, let Q cx = I x + θ ( C m ( x ) − I x ) (19)where θ is a variable, which is mainly used for luminanceadjustment, whose range is from 0 to 1. While Q cx is theoutput color value of the color image at x . Here the symbol ω will be used to denote [ α, β, γ ] T . As can be inferredfrom Equation 1, in the process of scanning Q c , if thecorresponding grayscale image is to match Q g , then Q g ( x ) = ω T Q cx (20)Therefore, by multiplying both sides of Eq. 19 by ω T , wehave: ω T Q cx = ω T I x + θω T ( C m ( x ) − I x ) (21) OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 10 (a) Case I(b) Case IIFig. 9. Example of grayscale-to-colored QR code conversion which, when combined with the previous equation, givesus: θ = Q gx − ω T I x ω T C m ( x ) − ω T I x (22)Combining Equation 22 and Equation 19, we finally have Q cx = I x + Q gx − ω T I x ω T C m ( x ) − ω T I x ( C m ( x ) − I x ) (23)As can be seen from the above, all the variables on the rightside are known, therefore it is easy to combine the grayscaleaesthetic QR code Q g with the original image I , to generatea color aesthetic QR code Q c . Two examples of grayscale-to-colored QR codes are shown in Fig. 9. Through the above analysis, a complete aesthetic QR codegeneration algorithm is proposed focusing on scanningrobustness. The matrix η is used to ensure the scanningrobustness of the generated code and (cid:36) px to initialize theweight of each module. However, the question remainsregarding the choice of these values for actual applications,and their corresponding influence on the generation ofaesthetic QR codes.Firstly, to study the effect of η on the generated QRcode, we conduct a set of experiments. We first investigateeach element of η be equal, that is, η k = ˆ η . Then, a datasetcontaining 300 images was created, and aesthetic QR codeswere generated for each value of ˆ η in the range 1.00, 0.99, ...,0.00, respectively to create a total of 30300 QR code images.By scanning the resulting images, a curve which describesthe relationship between the rate of scanning success and ˆ η was obtained. As shown in Fig. 4.4, with the decrease of ˆ η , the rate of scanning success also declines, and graduallyreaches 0.In the generation algorithm of the aesthetic QR code,scanning robustness is mainly affected by ensuring that P M k , i.e. the probability of correctly scanning a single mod-ule, is not less than η k . When it comes to thresholding, ifeach module is considered as an independent experiment ofsuccess or failure, the probability of successfully scanning Fig. 10. Influence of η on the aesthetic QR code generation the whole QR code is equal the percentage of instanceswhere the failure rate is less than the necessary threshold forrecovering the codeword. Therefore, improving the η k of asingle module results in an increase of the overall scanningrobustness of the QR codes.As we observed in Fig. 10a-f, the larger ˆ η is, the moresimilar the result image and the binary aesthetic QR codewill be. When ˆ η = 1 . , that is, η k = 1 . for all k , thegenerated color aesthetic QR code is exactly the same asthe binary aesthetic QR code. On the contrary, the smaller ˆ η is, the more similar it will be to the original image.When ˆ η = 0 . , except for some functional pattern area ofthe generated color aesthetic QR code, the remaining areasresemble the original image exactly.As can be inferred from the above discussion, for a singlemodule, the larger η k is, the larger the probability of it beingscanned correctly will be. In real applications, setting η k = ˆ η for all k -s is not required, so η k is usually adjusted locallyin order to obtain a better aesthetic effect. For instance, onecan use η = 0 .
75 + 0 . − W ) (24)to keep η k in the range of [0 . , . .Finally, the influence of (cid:36) px on the generated coloraesthetic QR code is discussed. In Algorithm 1, it is usedto initialize the range-adjusting weight of the module’sprobability. In the example mentioned above, it is treated asequal to p sx by default. That is, as a 2-dimensional Gaussiandistribution, in order to have a prior to the color adjustmentof the module’s center. However, in reality, more styles ofcolor aesthetic QR code results can be obtained by usingdifferent values. Fig. 11 shows some results of the sameimage for 6 different initializations of (cid:36) px . The scanningrobustness can still be maintained for different appearances,which indicates that the constraint of η is effective.The use of the variables ( η and (cid:36) px ) provides flexibilityto the generation of color aesthetic QR, which enables usersto generate aesthetic QR codes of different robustness andvisual styles according to their requirements. OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 11
Fig. 11. Aesthetic QR codes of different styles using different initial (cid:36) px values. a. (cid:36) px using a 2-D Gaussian distribution; b. Constant value forall (cid:36) px ; c. initialization with a random in the range of [0,1]; d. Initializationusing pixel values from another image, for example, to generate a patchof QR in the bottom-right of the image; e. Greater central than peripheralweights to formulating an image similar to Visualead’s approach; f.Initialization using superposition of Gaussian matrix and image edgedetection result. XPERIMENTS
In this section, the visual effects and the robustness of QRcodes are evaluated through extensive experiments. Firstof all, a user survey was conducted to compare the visualeffects of different methods. After that, the robustness ofQR codes was quantified and compared through decodingexperiments and user evaluations.
To evaluate the visual effect of color aesthetic QR code,a user survey was conducted. Different types of imageswere first chosen, including cartoons, sceneries, buildings,animals, human faces, brands, etc. After that, 4 differentmethods of generating aesthetic QR codes were applied togenerate the corresponding code for each image. Amongthem, Halftone [3] is a popular generation method of aes-thetic QR codes. Visualead [19] is a widely used commercialalgorithm, and Efficient [7] is the latest QR code visual beau-tification algorithm which has only recently been presented.We use these methods to obtain 20 groups of images in total,each group containing an original image and 4 QR codeimages generated using different methods. In this way, anevaluation questionnaire for the visual effect was created.The sample images are shown in Table 1.40 volunteers, 25 males and 15 females, were invited toparticipate in the survey. The images of each group weredisplayed, they were asked to give subjective scores rangingfrom 1 to 5 according to the observed degree of visual appealand similarity to the original images.As shown in Fig. 12, the visual effect of the QR codeimages generated by the four algorithms and the averagescores of the similarities between the generated images andthe original ones were evaluated.It can be seen that the scores of the two measures are sim-ilar, and the visual effect generated by ART-UP which this
Algorithm
Halftone Visualead Efficient ART-UP A e s t h e ti c M ea s u r e (a) Aesthetic Measure Algorithm
Halftone Visualead Efficient ART-UP S i m il a r it y M ea s u r e (b) Similarity MeasureFig. 12. User study about aesthetic and similarity x yz Fig. 13. The test method for scanning robustness of angle variation. TheQR code was placed in the center of the coordinate system, with thecamera a fixed distance from the origin and aimed toward the center. paper proposes resembles the state-of-art Efficient method[7]. Besides, it behaves much better than the Visualead [19]and Halftone [3] algorithms.
Scanning robustness of the QR codes was evaluated fromvarious aspects, such as scanning angle variation, bright-ness variation, scale variation, coverage, and scanning inreal environments. 30 images were randomly selected fromthe dataset to establish a sub-dataset from the 300-imagedataset, and the robustness tests were carried out on thissubset.
Scanning Angle Variation.
QR code images were placed inthe origin of a 3-D axes system, and the camera was rotatedabout the x -, y - and z -axes respectively, as shown in Fig.13. For the x - and y -axes, the images were sampled in ◦ steps ranging from − ◦ to ◦ . For the z -axes, the imageswere sampled in ◦ steps ranging from ◦ to ◦ . In thisway, 972 measurements were obtained. The correspondingresults are shown in Fig. ?? , ?? , ?? , respectively. Meanwhile,the angle between the plane of the QR code and the oneof the camera was also calculated and the correspondingresults are shown in Fig. ?? . As can be inferred from the fourfigures, ART-UP is slightly lower than that of Visualead’sstate-of-the-art commercial algorithm in terms of scanningangle robustness, but better than the other two methods. Brightness Variation.
QR code scanning is easily affectedby illumination and capturing devices, resulting in differ-ences between the captured and the real luminance. Tostudy this effect further, it was simulated with a linearbrightness adjustment, adjusting the brightness of each im-
OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 12
TABLE 1Examples Generated using Different Algorithms
OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 13 age varying − to +255 . That is, each QR code imageresults in 511 images of different luminance levels. Asshown in Fig. ?? , with regard to small-range brightnesschange, ART-UP is obviously more robust and immune toillumination changes among all methods. Scale Variation.
For aesthetic QR codes, scale variation isalways a key factor that affects scanning. Whether printedor shown on a screen, QR codes face scaling issues. Foreach image in the dataset, a × QR code image wasgenerated, and was sampled at a . step per frame fora scaling ratio ranging from . to . . That is, each QRcode image yielded 61 images at different scaling ratios. Theimage scaling was performed through bi-cubic interpola-tion. As shown in Fig. ?? , ART-UP demonstrates the bestperformance compared to the other three schemes whenfaced with the scale variation and it is consistent. Efficientis consistent but has a low scanning success rate, whileVisualead shows serious oscillation and inconsistency. Coverage.
QR codes allow a considerable percentage oferror correction, which will be affected when the code is cov-ered or has a missing segment. Each QR code of the imagewas replaced with a fixed number of a × a square black-or-white blocks in random positions. After 30 rounds ofrepeated testing, the average performance was determined.As shown in Fig. ?? , ART-UP, Visualead, and Halftone showsimilar performance, while the Efficient algorithm suffersmore in these cases. Scanning in realistic conditions.
Besides simulating scan-ning experiments, one subjective experiment was carriedout, where 30 volunteers were invited to vote for the QRcodes’ performance under real scanning circumstances. Inthe experiment, no constraint on the scanning tools, scan-ning environments or scanning methods was set. Volunteerswere shown the QR codes generated by the different algo-rithms, and were required to scan each one. The subjectivescore ranged from 1 to 5 regarding the response time, wherea score of 1 indicates a scarce ability to successfully scanthe code and the score of 5 represents a prompt response.As shown in Fig. ?? , the result of ART-UP is almost equal to the Visualead’s, and both of them are ranked better thanHalftone and Efficient. This experiment demonstrates thatthe proposed algorithm is robust enough to adapt to mostcommon scenes. Especially when the codes are printed onpaper or other materials, the speed of response is still quickand stable. ONCLUSION
In this paper, a flexible generation algorithm of aesthetic QRcodes is proposed, where a QR code is obtained based ona probability model of successfully scanning the QR codes,controlling and adjusting the tradeoff between visual effectand scanning robustness. By adjusting the thresholdingand sampling errors, our proposed algorithm can gener-ate aesthetic QR codes with different appearances usinga hierarchical, coarse-to-fine strategy. Experimental resultsshow that ART-UP is scanning-robust and can avoid theerror occurrence in most realistic scenarios. This algorithmnot only provides aesthetic results, but also ensures codeusability, and is thus suitable for commercial needs. R EFERENCES [1] E. ISO/IEC, “Information technology - automatic identificationand data capture techniques - bar code symbology - qr code,”2000.[2] R. Cox, “Qart codes,” http://research.swtch.com/qart, 2012, ac-cessed October 16, 2016.[3] H.-K. Chu, C.-S. Chang, R.-R. Lee, and N. J. Mitra, “Halftone qrcodes,”
ACM Transactions on Graphics (TOG) , vol. 32, no. 6, p. 217,2013.[4] Y.-H. Lin, Y.-P. Chang, and J.-L. Wu, “Appearance-based qr codebeautifier,”
IEEE Transactions on Multimedia , vol. 15, no. 8, pp.2198–2207, 2013.[5] Y. Zhang, S. Deng, Z. Liu, and Y. Wang, “Aesthetic qr codes basedon two-stage image blending,” in
MultiMedia Modeling . Springer,2015, Conference Proceedings, pp. 183–194.[6] S. Ramya and C. S. Joice, “An optimized image and data embed-ding in color qr code,”
Structure , vol. 17, p. 4V, 2015.[7] S.-S. Lin, M.-C. Hu, C.-H. Lee, and T.-Y. Lee, “Efficient qr codebeautification with high quality visual content,”
IEEE Transactionson Multimedia , vol. 17, no. 9, pp. 1515–1524, 2015.
OURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 14 [8] F. J. MacWilliams and N. J. A. Sloane,
The theory of error correctingcodes . Elsevier, 1977.[9] G. J. Garateguy, G. R. Arce, D. L. Lau, and O. P. Villarreal, “Qr im-ages: optimized image embedding in qr codes,”
IEEE transactionson image processing , vol. 23, no. 7, pp. 2842–2853, 2014.[10] C. Fang, C. Zhang, and E.-C. Chang, “An optimization modelfor aesthetic two-dimensional barcodes,” in
MultiMedia Modeling .Springer, 2014, pp. 278–290.[11] L. Li, J. Qiu, J. Lu, and C.-C. Chang, “An aesthetic qr codesolution based on error correction mechanism,”
Journal of Systemsand Software , 2015.[12] B. Jiang and X. Liu, “Qr code embellishment with backgroundfacial image embedding,” in
SIGGRAPH Asia 2014 Posters . ACM,2014, p. 31.[13] Z. Baharav and R. Kakarala, “Visually significant qr codes: Imageblending and statistical analysis,” in
Multimedia and Expo (ICME),2013 IEEE International Conference on . IEEE, 2013, pp. 1–6.[14] Y.-S. Lin, S.-J. Luo, and B.-Y. Chen, “Artistic qr code embellish-ment,” in
Computer Graphics Forum , vol. 32, no. 7. Wiley OnlineLibrary, 2013, pp. 137–146.[15] Z. Gao, G. Zhai, and C. Hu, “The invisible qr code,” in
Proceedingsof the 23rd Annual ACM Conference on Multimedia Conference . ACM,2015, pp. 1047–1050.[16] Z. Yang, Y. Bao, C. Luo, X. Zhao, S. Zhu, C. Peng, Y. Liu, andX. Wang, “Artcode: preserve art and code in any image,” in
Pro-ceedings of the 2016 ACM International Joint Conference on Pervasiveand Ubiquitous Computing . ACM, 2016, pp. 904–915.[17] D. Samretwit and T. Wakahara, “Measurement of reading charac-teristics of multiplexed image in qr code,” in
Intelligent Networkingand Collaborative Systems (INCoS), 2011 Third International Confer-ence on
Zebra Crossing , 2013.[21] E. Ohbuchi, H. Hanaizumi, and L. A. Hock, “Barcode readersusing the camera device in mobile phones,” in
Cyberworlds, 2004International Conference on . IEEE, 2004, pp. 260–265.[22] Y. Liu and M. Liu, “Automatic recognition algorithm of quickresponse code based on embedded system,” in
Sixth InternationalConference on Intelligent Systems Design and Applications , vol. 2.IEEE, 2006, pp. 783–788.[23] Y. Liu, J. Yang, and M. Liu, “Recognition of qr code with mobilephones,” in . IEEE,2008, pp. 203–206.[24] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Imagequality assessment: from error visibility to structural similarity,”
IEEE transactions on image processing , vol. 13, no. 4, pp. 600–612,2004.[25] J. Canny, “A computational approach to edge detection,”
IEEETransactions on Pattern Analysis and Machine Intelligence(IEEETPAMI) , no. 6, pp. 679–698, 1986.[26] M.-M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, and S.-M. Hu,“Global contrast based salient region detection,”
IEEE Transactionson Pattern Analysis and Machine Intelligence(IEEE TPAMI) , vol. 37,no. 3, pp. 569–582, 2015.
Mingliang Xu is an associate professor in theSchool of Information Engineering of ZhengzhouUniversity, China, and currently is the director ofCIISR ( Center for Interdisciplinary InformationScience Research), and the general secretaryof ACM SIGAI China. His research interests in-clude virtual reality and artificial intelligence. Xugot his Ph.D. degree in computer science andtechnology from the State Key Lab of CAD&CGat Zhejiang University.
Qingfeng Li is a master student in Centerfor Interdisciplinary Information Science Re-search, Zhengzhou University, China. He re-ceived the B.E. degree in Computer Softwarefrom Zhengzhou University, in 2014. He is cur-rently working with Prof. Jianwei Niu in BeihangUniversity, his research interests include com-puter version, image processing, and computergraphics.
Jianwei Niu received the M.S. and Ph.D. de-grees in computer science from Beihang Uni-versity, Beijing, China, in 1998 and 2002, re-spectively. He was a visiting scholar at Schoolof Computer Science, Carnegie Mellon Univer-sity, USA from Jan. 2010 to Feb. 2011. He isa professor in the School of Computer Scienceand Engineering, BUAA, and an IEEE seniormember. His current research interests includemobile and pervasive computing, mobile videoanalysis.
Xiting Liu is currently a senior student in Bei-hang University, Beijing, China. His major isComputer science and technology. He oncestudied and worked with Prof. Jianwei Niu, in2016. His research interests include computerversion and image processing.
Weiwei Xu now is a researcher in State KeyLab of CAD&CG, College of Computer Scienceat Zhejiang University, recipient of the NSFCExcellent Young Scholars Program in 2013. Hismain research interests are digital geometry pro-cessing, physical simulation and virtual reality.
Pei Lv is an assistant professor in Center forInterdisciplinary Information Science Research,Zhengzhou University, China.His research inter-ests include video analysis and crowd simula-tion. He received his Ph.D in 2013 from theState Key Lab of CAD&CG, Zhejiang University,China.