On the Efficient Estimation of Min-Entropy
aa r X i v : . [ c s . CR ] S e p On the Efficient Estimation of Min-Entropy
Yongjune Kim, Cyril Guyot, and Young-Sik Kim
Abstract —The min-entropy is an important metric to quantifyrandomness of generated random numbers in cryptographicapplications; it measures the difficulty of guessing the most-likely output. One of the important min-entropy estimator isthe compression estimator of NIST Special Publication (SP) 800-90B, which relies on Maurer’s universal test. In this paper, wepropose two kinds of min-entropy estimators to improve compu-tational complexity and estimation accuracy by leveraging twovariations of Maurer’s test: Coron’s test (for Shannon entropy)and Kim’s test (for R´enyi entropy). First, we propose a min-entropy estimator based on Coron’s test which is computationallyefficient than the compression estimator while maintaining theestimation accuracy. The secondly proposed estimator relies onKim’s test that computes the R´enyi entropy. This proposedestimator improves estimation accuracy as well as computationalcomplexity. We analytically characterize an interesting trade-offrelation between theoretical gap of accuracy and variance ofmin-entropy estimates, which depends on the order of R´enyientropy. By taking into account this trade-off relation, we observethat the order of two is a proper assignment since the proposedestimator based on the collision entropy (i.e., the R´enyi entropyof order two) provides the most accurate estimates. Moreover,the proposed estimator based on the collision entropy has aclosed-form solution whereas both the compression estimatorand the proposed estimator based on Coron’s test do not haveclosed-from solutions. Numerical evaluations demonstrate thatthe first proposed estimator achieves the same accuracy as thecompression estimator with much less computations. Moreover,the second estimator can even improve the accuracy as well asreduce the computational complexity.
I. I
NTRODUCTION
Random numbers are essential for generating cryptographicinformation such as secret keys, nonces, salt values, etc . Thesecurity of cryptographic systems crucially depends on the ran-domness of generated random numbers [1]–[5]. Randomnessof generated numbers should be quantified and entropies arewidely used metrics as in AIS 31 [6], NIST SP 800-22 [7],and NIST SP 800-90B [1].There are several kinds of entropies such as Shannonentropy, R´enyi entropy, and min-entropy. The Shannon entropyquantifies the difficulty of guessing a typical output of randomsources [2], which was used in AIS 31 [6] and NIST SP800-22 [7]. The min-entropy corresponds to the difficulty ofguessing the most likely output of random sources [1], [3].The NIST SP 800-90B [1] supports the use of min-entropy toquantify the randomness.However, it is difficult to estimate the min-entropy ofsources that are not independent and identically distributed,
Y. Kim is with the Department of Information and Communication En-gineering, DGIST, Daegu 42988, South Korea (e-mail: [email protected]). C.Guyot is with Western Digital Research, Milpitas, CA 95035 USA (e-mail:[email protected]). Y.-S. Kim is with the Department of Informationand Communication Engineering, Chosun University, Gwangju 61452, SouthKorea (e-mail: [email protected]). i.e., non-IID sources [6]. Hence, the NIST SP 800-90Badopted ten different algorithms to estimate the min-entropyof non-IID sources [1, Ch 6.3]. Each estimator independentlyperforms its own estimation and then the NIST SP 800-90Bconservatively selects the minimum among these ten estimatesas a final estimate of min-entropy.Although this conservative approach is preferred in securityapplications, it incurs a problem of detrimental underestimate .Even if only one estimator provides a significant underesti-mate, it determines the final estimation no matter how accuratethe other estimators are [3], [8], [9]. Hence, it is important toavoid significant underestimates to obtain more accurate min-entropy estimates.In this paper, we focus on improving the compressionestimator among ten min-entropy estimators of the NIST SP800-90B (see Table I) since it often underestimates the min-entropy. The compression estimator theoretically relies onMaurer’s universal test [10]. Maurer’s test quantifies random-ness by taking into account the minimum distance betweenmatching patterns. Maurer proposed a computationally effi-cient algorithm whose output is closely related to the Shannonentropy [10]. Maurer’s test is a widely used randomness test;it was included in the NIST SP 800-22 [7]. Hagerty andDraper [2] proposed an algorithm to estimate the lower boundon the min-entropy by leveraging Maurer’s test. Afterwards, itbecame the compression estimator of the NIST SP 800-90B.Although the compression estimator is theoretically well-justified by Maurer’s test, it is prone to underestimate themin-entropy as discussed in [2], [3], [8]. The reasons ofunderestimates are twofold: • Variance of Maurer’s test statistic : To ensure the confi-dence level of 99 %, the lower bound of the confidenceinterval for the Maurer’s test value is used to estimate themin-entropy. Hence, the larger variance of Maurer’s testsleads to the lower underestimate of min-entropy. • Theoretical gap : The compression estimator estimates thelower bound on the min-entropy instead of the actual min-entropy [2]. Depending on the distributions of sources,the gap between the lower bound and the actual min-entropy can be large [2], [3], [8].The impact of variance of Maurer’s test can be reducedby including more samples. However, the computational com-plexity of the compression estimator is O ( K ) (where K denotes the number of samples), which limits the improvementof estimation by including more samples.We propose two types of computationally efficient min-entropy estimators. First, we propose a min-entropy estimatorby using Coron’s test [11] instead of Maurer’s test. Its com-putational complexity is O ( K ) instead of O ( K ) , so we caneffectively include more data samples to reduce the impactof variance. The proposed estimator based on Coron’s test TABLE IC
LASSIFICATION OF
NIST SP 800-90B E
STIMATORS [1], [9]Statistic-based estimator [2] Prediction-based estimator [3]Most common value estimator MultiMCW prediction estimatorCollision estimator Lag prediction estimatorMarkov estimator MultiMMC prediction estimatorCompression estimator LZ78Y prediction estimator t -Tuple estimatorLRS estimator is motivated by observing that the compression estimator’sapproach [2] is essentially similar to the approach in [12]–[14],which relate the lower bound on the probability of error andthe Shannon entropy. The proposed estimator based on Coron’stest is more efficient than the compression estimator whileachieving the same accuracy as the compression estimator.Hence, the proposed estimator based on Coron’s test wouldbe an appealing alternative to the compression estimator ofthe NIST SP 800-90B.In spite of the computational advantage of the min-entropyestimator based on Coron’s test, it does not reduce the theo-retical gap. The reason is that the test values by Maurer’s testand Coron’s test are inherently similar [10], [11]. In order toreduce the theoretical gap, we propose a min-entropy estimatorbased on the R´enyi entropy.Recently, Kim [15] proposed a variation of Maurer’s testto estimate the R´enyi entropy. By leveraging Kim’s test, wepropose a min-entropy estimator which effectively reduces thetheoretical gap compared to the compression estimator. Weshow that the theoretical gap can be decreased by adopting ahigher order of the R´enyi entropy. However, the higher orderincreases the variance of min-entropy estimates. Hence, wefocus on the min-entropy estimator based on the collisionentropy (i.e., R´enyi entropy of order two) by taking intoaccount the trade-off relation between the the theoretical gapand the variance of estimates.The proposed min-entropy estimator based on the collisionentropy is computationally efficient than other estimators.Furthermore, it has a closed-form solution for the min-entropyestimate whereas other estimators rely on bisection method orbinary search to calculate an approximated estimate. Due toits computational efficiency, the min-entropy estimator basedon the collision entropy can effectively suppress the varianceby including more samples.In addition, we propose an online estimator that updatesthe min-entropy estimate as a new sample is received. Notethat the compression estimator is inherently an offline (orbatch) algorithm that requires the whole samples to outputthe estimate. Unlike the compression estimator, the proposedonline estimator can provide a min-entropy estimate withlimited samples, then the accuracy of estimates is improvedas obtaining more samples. Moreover, the proposed on-lineestimator does not need to store the entire samples, hence, itis proper for applications with stringent resource constraints.The rest of this paper is organized as follows. Section IIprovides an overview of entropies, statistical tests, and thecompression estimator. Section III presents a proposed min-entropy estimator based on Coron’s test. Section IV proposes an min-entropy estimator based on Kim’s test and Section Vfocuses on the estimator based on the collision entropy. Sec-tion VI provides numerical results and Section VII concludes.II. P RELIMINARY : E
NTROPIES , S
TATISTICAL T ESTS , AND C OMPRESSION E STIMATOR
A. Entropies
Suppose that N -bit sample s = ( s , . . . , s N ) is generatedfrom a given source. The sample sequence s is partitioned intonon-overlapping L -bit blocks as follows: b ( s ) = (cid:0) b , . . . , b ⌊ N/L ⌋ (cid:1) (1)where b n = ( s L ( n − , . . . , s Ln ) denotes the n th block of s ,i.e., b n ∈ { , . . . , B − } and B = 2 L .The Shannon entropy of L -bit blocks b ( s ) is defined as H ( B ) = − B − X b =0 p b log p b (2)where B denotes a random variable over the alphabet { , . . . , B − } and p b = P ( b ) . The corresponding per-bitentropy is given by H ( S ) = H ( B ) L (3)where S denotes the random variable of binary sample s ∈{ , } .The R´enyi entropy of order α is defined as H ( α ) ( B ) = 11 − α log B − X b =0 p αb (4)where α > and α = 1 . For α = 2 , the R´enyi entropycorresponds to the collision entropy, which is defined as H (2) ( B ) = − log B − X b =0 p b . (5)The corresponding per-bit R´enyi entropy is given by H ( α ) ( S ) = H ( α ) ( B ) L .The min-entropy of b ( s ) is defined as H ( ∞ ) ( B ) = − log (cid:18) max b ∈{ ,...,B − } p b (cid:19) = − log θ (6)where θ = max b ∈{ ,...,B − } p b . The corresponding per-bitmin-entropy is given by H ( ∞ ) ( S ) = H ( ∞ ) ( B ) L . Remark 1:
The following relations are well known: H ( B ) = lim α → H ( α ) ( B ) , (7) H ( ∞ ) ( B ) = lim α →∞ H ( α ) ( B ) . (8) Remark 2:
The R´enyi entropy is non-increasing in α [16].Hence, H ( ∞ ) ( B ) ≤ H ( α ) ( B ) . Hence, the min-entropy is thesmallest of the R´enyi family of entropies. B. Maurer’s Test
Maurer’s test is a common randomness test, capable ofdetecting a wide range of statistical defects [10]. Maurer’stest detects whether or not the sequence can be significantlycompressed without loss of information [7], [10]. The formu-lation of Maurer’s test was motivated by the universal sourcecoding algorithms of Elias [17] and Willems [18]. Maurer’stest is also universal since Maurer’s test is designed withoutknowing the distribution of the source. Maurer’s universal testwas adopted by the NIST SP 800-22 for a randomness testand the compression estimator of NIST SP 800-90B relies onMaurer’s test.Maurer’s test takes as input three integers { L, Q, K } and an N -bit sample s = ( s , . . . , s N ) where N = ( Q + K ) × L . Thesample sequence s is partitioned into non-overlapping L -bitblocks b ( s ) = ( b , . . . , b Q + K ) . The first Q blocks are used toinitialize the test. The remaining K blocks are used to computethe following test function: f M ( s ) = 1 K Q + K X n = Q +1 log D n ( s ) (9)where D n ( s ) is given by D n ( s )= ( n, if b n − i = b n , ∀ i < n ;min { i : i ≥ , b n = b n − i } , otherwise. (10)Note that D n ( s ) is the minimum distance between the current n th block and any preceding block with the same pattern.Maurer’s test can be implemented in an efficient manner andthe detailed algorithm is described in [10].The size of initial blocks Q should be chosen to be at least × L so as to have a high likelihood that each of the L blocks occurs at least once in the first Q blocks. A larger K for test blocks is preferred; usually, it is recommended to use K ≥ × L [7], [10].Maurer’s test is closely related to the source’s Shannonentropy. In [10] and [19], it was shown that there is a gapbetween Maurer’s test and the Shannon entropy as follows: lim L →∞ [ E ( f M ( s )) − H ( B )] , Z ∞ e − ξ log ξdξ ≃ − . . (11) C. Coron’s Test for Shannon Entropy
Coron’s test was proposed to estimate the Shannon entropyby modifying Maurer’s test [11]. Coron’s test f C ( s ) is givenby f C ( s ) = 1 K Q + K X n = Q +1 g C ( D n ( s )) (12)where the function g C ( · ) should be chosen to satisfy thecondition E ( f C ( s )) = H ( B ) . Coron showed that the followingfunction g C ( · ) achieves this equality condition: g C ( i ) = ( , if i = 1; P i − k =1 1 k , if i ≥ . (13) The computational complexity of Coron’s test is comparableto Maurer’s test [11]. To improve the computational efficiency, g C ( i ) can be approximated for large i (e.g., i ≥ ) as follows: i − X k =1 k ≃ log ( i −
1) + γ + 12( i − − i − (14)where γ is Euler’s constant, i.e., γ = − R ∞ e − x log xdx ≃ . . The detailed algorithm of Coron’s test is describedin [11]. D. Kim’s Test for R´enyi Entropy
As Coron modified Maurer’s test to obtain the Shannonentropy, Kim [15] proposed a variant of Maurer’s test toestimate the R´enyi entropy of order α . Kim’s test f K ( s , α ) is given by f K ( s , α ) = 1 K Q + K X n = Q +1 g K ( D n ( s ) , α ) , (15)where g K ( i, α ) is defined as g K ( i, α ) = ( , if i = 1;( − i − · (cid:0) α − i (cid:1) , if i ≥ . (16)Here, (cid:0) α − i (cid:1) denotes the generalized binomial coefficient, i.e., (cid:0) α − i (cid:1) = ( α − i i ! where ( · ) i is the Poccharmmer symbol.In [15], it was shown that the function g K ( i, α ) satisfies thefollowing condition: E ( f K ( s , α )) = B − X b =0 p αb . (17)Then, the R´enyi entropy of order α can be estimated by H ( α ) ( B ) = 11 − α log f K ( s , α ) . (18)For the collision entropy (i.e., R´enyi entropy of α = 2 ), g K ( i, α = 2) is simplified to g K ( i, α = 2) = ( , if i = 1;0 , if i ≥ . (19) Remark 3 (Collision Entropy by Kim’s Test):
The colli-sion entropy can be estimated by counting only the case of D n ( s ) = 1 (i.e., the current block b n is the same as theprevious block b n − which can be interpreted as a collision of consecutive samples). Hence, Q = 1 is sufficient for theinitialization of Kim’s test whereas Q ≥ × L are requiredfor the initialization stages of Maurer’s test and Coron’s test.The computational complexity of Kim’s test for the collisionentropy is less than those of Maurer’s test and Coron’s test [15,Table 2]. Algorithm 1
The compression estimator of NIST 800-90B [1]
Input: L -bit blocks b ( s ) = ( b , . . . , b Q + K ) Output: H ( ∞ ) ( S ) Compute f M ( s ) := K P Q + Kn = Q +1 log D n ( s ) X := f M ( s ) and b σ := c p Var (log D n ( s )) X ′ := X − . · b σ √ K By the bisection method, solve the following equation forthe parameter θ ∈ [ B , : X ′ = G ( θ ) + ( B − G ( ϕ ) (20)where G ( · ) is given by (24) and ϕ = − θB − . The estimated per-bit min-entropy is given by H ( ∞ ) ( S ) := ( − log θL , if Step 4 yields a solution ;1 , otherwise . E. Compression Estimator of NIST 800-90B
The compression estimator of NIST 800-90B first computesMaurer’s test and then estimate the lower bound on the min-entropy from the statistics of Maurer’s test [2]. The compres-sion estimator is described in Algorithm 1. NIST SP 800-90Bsets L = 6 and c = 0 . . The corrective factor c dependson L and K , which reduces the standard deviation to accountfor dependencies between D n ( s ) [10]. The corrective factor c = 0 . is obtained by setting L = 6 and K → ∞ [19].Without loss of generality, we can assume that p ≥ p ≥ . . . ≥ p B − (21)where B = 2 L . For a given Maurer’s test f M ( s ) , the following near-uniform distribution can estimate the maximum value of θ , which corresponds to the lower bound on the min-entropy[2]: P θ ( b ) = ( θ, if b = 0; − θB − , otherwise . (22)Then, the maximum value of θ can be obtained from thefollowing equation [2]: f M ( s ) = G ( θ ) + ( B − G ( ϕ ) (23)where ϕ = − θB − and G ( z ) = 1 K Q + K X n = Q +1 n X i =1 F ( z, n, i ) · log i (24) F ( z, n, i ) = ( z (1 − z ) i − , if i < n ; z (1 − z ) n − , if i = n. (25)The key equation in Step 4 of Algorithm 1 is formulated from(23) by considering the confidence interval.Algorithm 1 should solve the non-closed-form equationby the bisection method. The computational complexity is O ( M K ) where O ( K ) is required to compute G ( z ) and M corresponds to the number of iterations of the bisectionmethod. Note that M determines numerical accuracy of thebisection method. The number of samples K should be limited due to thecomputational complexity of O ( K ) . It affects the estimationaccuracy because a larger K reduces the variance b σ . Wepropose a computationally efficient min-entropy estimator soas to include more samples readily and reduce the variance.III. P ROPOSED E STIMATOR B ASED ON C ORON ’ S T EST
In this section, we propose a min-entropy estimator whosecomputational complexity is less than that of the compressionestimator while maintaining the estimation accuracy of thecompression estimator.
A. Proposed Estimator Based on Coron’s Test
In [12]–[14], the relation between the Shannon entropy andthe probability of error was investigated. In the absence ofany other knowledge regarding the random variable B over thealphabet { , . . . , B − } , the estimator of B that minimizes theerror probability is the value with the highest probability, i.e., θ = max b ∈{ ,...,B − } p b . Then, the minimal error probability π in guessing the value of B is π = 1 − θ. (26)The lower bound on the minimal error probability is derivedby a special case of Fano’s inequality [14]: h ( π ) + π log ( B − ≥ H ( B ) (27)where h ( π ) = − π log π − (1 − π ) log (1 − π ) . Also, thebound is achieved with equality by the following distribution: ( p , p , . . . , p B − ) = (cid:18) − π, πB − , . . . , πB − (cid:19) , (28)which is equivalent to the near-uniform distribution of (22).Fano’s inequality is sharp since the equality is actuallyachieved [14], [20]. Because of π = 1 − θ , (27) can be modifiedto h ( θ ) + (1 − θ ) log ( B − ≥ H ( B ) (29)where h ( π ) = h ( θ ) .By using Coron’s test and assuming the near-uniform dis-tribution, we can estimate the maximum value of θ from thefollowing equation: f C ( s ) = h ( θ ) + (1 − θ ) log ( B − . (30)By solving (30), we can estimate a lower bound on the min-entropy. Theorem 4:
For θ ∈ [ B , , there exists only one solutionof (30). The solution θ ∗ minimizes the min-entropy, i.e., H ( ∞ ) ( B ) ≥ − log θ ∗ . Proof:
Suppose that ζ ( θ ) = h ( θ ) + (1 − θ ) log ( B − (31)For θ ∈ ( B , , ζ ( θ ) is a strictly decreasing function, i.e., ζ ( θ ) ′ = log (cid:16) − θθ · B − (cid:17) < . Also, ζ ( B ) = log B and ζ (1) = 0 . Since ≤ H ( B ) ≤ log B , there exists only onesolution θ ∗ , which is the maximum value that achieves (29)with equality. Hence, H ( ∞ ) ( B ) = − log θ ≥ − log θ ∗ . Algorithm 2
Proposed estimator based on Coron’s test
Input: L -bit blocks b ( s ) = ( b , . . . , b Q + K ) Output: H ( ∞ ) ( S ) Compute f C ( s ) := K P Q + Kn = Q +1 g C ( D n ( s )) X := f C ( s ) and b σ := c ′ p Var ( g C ( D ( s ))) X ′ := X − . · b σ √ K By the bisection method, solve the following equation forthe parameter θ ∈ [ B , : X ′ = h ( θ ) + (1 − θ ) log ( B − (32) The estimated per-bit min-entropy is given by H ( ∞ ) ( S ) := ( − log θL , if Step 4 yields a solution ;1 , otherwiseWe propose Algorithm 2 by using Coron’s test instead ofMaurer’s test. The key equation of Step 4 of Algorithm 2 isformulated by (30). The corrective factor of Coron’s test is c ′ = 0 . [11], which is obtained by setting L = 6 and K → ∞ as in NIST SP 800-90B. Note that c ′ is close to c = 0 . of the compression estimator. Remark 5:
Unlike the compression estimator, the RHS of(30) (i.e., ζ ( θ ) of (31)) does not depend on K . For a givennumber of iteration of the bisection method M , the complexityof solving (32) is O ( M ) (see Table II). If we store a table for ( θ, ζ ( θ )) , then we can readily estimate θ ∗ = arg min | ζ ( θ ) − f C ( s ) | for a given B .Hence, we can estimate the min-entropy efficiently in spiteof a large number of samples. Hence, it is very effective toreduce the variance by including more samples.The estimated values of the compression estimator and theproposed estimator are almost identical for the same K (seeSection VI). It is mainly because Maurer’s test and Coron’stest are closely related and both estimates are obtained byassuming the near-uniform distribution. Since the proposedestimator achieves the identical estimation accuracy with muchless computations, the proposed estimator of Algorithm 2 isan appealing alternative to the compression estimator. B. Theoretical Gaps of Compression Estimator and ProposedEstimator
Although we propose a computationally efficient estimatorbased on Coron’s test, it would suffer from a large theoreticalgap of the compression estimator. As shown in Fig. 1, both thecompression estimator and the proposed estimator have similartheoretical gaps between the lower bound and the upper bound.For a given Maurer’s test value f M ( s ) , the compressionestimator outputs the lower bounded value, which is achievedby the near-uniform distribution. However, the actual min-entropy will be between the lower bound and the upper bound(see Fig. 1(a)). In the worst case, the actual min-entropy cancorrespond to the upper bounded value, which is achieved by Maurer's Test M i n - en t r op y Inverted near-uniform (UB)Near-uniform (LB) (a)
Coron's Test (Shannon Entropy) M i n - en t r op y Inverted near-uniform (UB)Near-uniform (LB) (b)Fig. 1. The theoretical gap for L = 6 : (a) Compression estimator based onMaurer’s test and (b) proposed estimator based on Coron’s test. The maximumvalue of Maurer’s test is 5.2177 [10] and the maximum value of Coron’s testis L (i.e., the maximum value of Shannon entropy). the following inverted near-uniform distribution [2]: P ψ ( b ) = ψ, if b ∈ n , . . . , j ψ k − o ;1 − j ψ k ψ, if b = j ψ k ;0 , otherwise. (33)The proposed estimator based on Coron’s test also estimatesthe lower bound on the min-entropy for a given Coron’s testvalue f C ( s ) . As in the compression estimator, the actual min-entropy will be between the lower bound and the upper bound(see Fig. 1(b)).Importantly, these theoretical gaps cannot be tightened forthe compression estimator and the proposed estimator basedon Coron’s test. It is because these bounds are sharp (i.e.,the near-uniform distribution and the inverted near-uniformdistribution achieve the lower and upper bounds with equality,respectively). Remark 6:
The theoretical gap will be zero for only twoextreme points, i.e., H ( ∞ ) ( B ) = 0 and H ( ∞ ) ( B ) = L (i.e., H ( ∞ ) ( S ) = 1 ).Since most sample sequences would not correspond to thesetwo extreme points, both the compression estimator and theproposed estimator might output significant underestimates. TABLE IIC
OMPARISON OF C OMPRESSION E STIMATOR AND P ROPOSED E STIMATORS
Compression Estimator Estimator (Coron’s Test) Estimator (Kim’s Test)Complexity of Test O ( K ) O ( K ) O ( K ) Complexity of Key Equation O ( MK ) O ( M ) O (1) IV. P
ROPOSED E STIMATOR B ASED ON K IM ’ S T EST
In this section, we attempt to address the theoretical gapby using the R´enyi entropy. The proposed estimator caneffectively reduce the theoretical gap and be computationallyefficient.
A. Proposed Estimator Based on Kim’s Test
In order to reduce the theoretical gap, we take into accountthe relation between the min-entropy and the R´enyi entropyof Remark 1. Fig. 2(a) shows that the theoretical gap betweenlower bound and upper bound can be suppressed by increasingthe order of R´enyi entropy.Since the sharp upper bound by the inverted near-uniformdistribution is changed by α , we consider a common upperbound, which is not affected by α . This upper bound canbe explained by Fig. 2(b) showing the relation between θ (the maximum probability of near-uniform distribution) andentropies. For a given near-uniform distribution with θ , it isclear that the min-entropy (i.e., − log θ ) will be the minimumamong all entropies, which corresponds to the upper bound inFig. 2(a). Note that the sharp upper bounds are close to thiscommon upper bound for a large L as discussed in [14, Fig. 2].We estimate the lower bound on the min-entropy by as-suming the near-uniform distribution as in the compressionestimator and the proposed estimator based on Coron’s test. Lemma 7:
Suppose that θ = max b ∈{ ,...,B − } p b . Then, thefollowing inequality holds: − α log (cid:18) θ α + (1 − θ ) α ( B − α − (cid:19) ≥ H ( α ) ( B ) (34)for α > . The near-uniform distribution of (22) achieves thisbound with equality. Proof:
Without loss of generality, suppose that θ = p .For α > , maximization of H ( α ) ( B ) is equivalent to thefollowing optimization problem:minimize ( p ,...,p B − ) B − X b =1 p αb subject to B − X b =1 p b = 1 − θ, p b ≥ , ∀ b (35)which is a convex optimization problem because of α > and p b ≥ . From the Karush-Kuhn-Tucker (KKT) conditions, weobtain the optimal solution p ∗ = · · · = p ∗ B − = − θ − B , i.e.,the near-uniform distribution. The R´enyi entropy becomes theLHS of (34) for the near-uniform distribution.We estimate the lower bound on the min-entropy by lever-aging Kim’s test. For α > , (34) is equivalent to θ α + (1 − θ ) α ( B − α − ≤ (1 − α ) H ( α ) ( B ) . (36) Test Values for Entropies M i n - en t r op y Shannon ( = 1): LBCollision ( = 2): LBRenyi ( = 5): LBRenyi ( = 10): LBMin-entropy ( = ): UB (a) E n t r op i e s Shannon ( = 1)Collision ( = 2)Renyi ( = 5)Renyi ( = 10)Min-entropy ( = ) (b)Fig. 2. The theoretical gap between upper bound and lower bounds. (a)Relation between the test values for entropies and the estimated min-entropy;(b) Relation between the test values for entropies and θ = max p b where H ( ∞ ) ( B ) = − log θ . By assuming the near-uniform distribution as in (30), we canestimate the maximum value of θ from the following equation: f K ( s , α ) = θ α + (1 − θ ) α ( B − α − (37)where (1 − α ) H ( α ) ( B ) = f K ( s , α ) because of (18). The follow-ing theorem shows that the lower bound on the min-entropycan be estimated by Lemma 7 and Kim’s test. Theorem 8:
For θ ∈ [ B , and α > , there exists only onesolution of (37). The solution θ ∗ minimizes the min-entropy,i.e., H ( ∞ ) ( B ) ≥ − log θ ∗ . Proof:
Suppose that ζ ( θ ) = θ α + (1 − θ ) α ( B − α − . For θ ∈ ( B , , ζ ( θ ) is a strictly increasing function, i.e., ζ ( θ ) ′ > .Also, ζ ( B ) = B − α and ζ (1) = 1 . Since ≤ H ( α ) ( B ) ≤ log B , we observe that B − α ≤ f K ( s , α ) ≤ . Hence, thereexists only one solution θ ∗ , which is the maximum value thatsatisfies (34). Hence, H ( α ) ( B ) = − log θ ≥ − log θ ∗ . Algorithm 3
Proposed estimator based on Kim’s test
Input: L -bit blocks b ( s ) = ( b , . . . , b Q + K ) Output: H ( ∞ ) ( S ) Compute f K ( s , α ) := K P Q + Kn = Q +1 g K ( D n ( s ) , α ) X := f K ( s , α ) and b σ := c ′′ p Var ( g K ( D ( s ) , α )) X ′ := X − . · b σ √ K By the bisection method, solve the following equation forthe parameter θ ∈ [ B , : X ′ = θ α + (1 − θ ) α ( B − α − (38) The estimated per-bit min-entropy is given by H ( ∞ ) ( S ) := ( − log θL , if Step 4 yields a solution ;1 , otherwiseBased on Theorem 8, we propose Algorithm 3 to estimatethe min-entropy. As in Algorithm 1 and Algorithm 2, the keyequation of Step 4 is formulated from (37) by taking intoaccount confidence interval. The corrective factor of Kim’stest depends on α as well as L and K . B. Theoretical Gap and Variance of Estimates
Here, we show that the order α is a parameter of trade-off relation between the theoretical gap and the variance ofmin-entropy estimates.It is clear that the maximum theoretical gap decreases for ahigher order α as shown in Fig. 2(a). The following Theoremshows how a higher order α can improve the min-entropyestimates. Theorem 9:
Suppose that θ ( α ) and θ ( α +1) are estimatedvalues by f K ( s , α ) and f K ( s , α + 1) , respectively. If θ ( α ) ≫ B − α − α , then H ( ∞ ) ≥ − log θ ( α +1) L ≥ − log θ ( α ) L , (39)for α > . Hence, the estimated lower bounds on the min-entropy improve with the order α > for a large B . Proof:
The proof is given in Appendix A.If we consider only the theoretical gap, then a higher order α would be preferred in Algorithm 3. However, we shouldtake into account the variance of estimates which dependson α . Fig. 3 shows the relation between Kim’s test f K ( s , α ) and the estimated θ . The derivative dθdf K ( s ,α ) increases with α especially for the higher entropy regime (i.e., θ is close to B ).The derivative dθdf K ( s ,α ) is given by dθdf K ( s , α ) = z ( θ, α ) = 1 α (cid:26) θ α − − (cid:16) − θB − (cid:17) α − (cid:27) . (40)Then, Var ( θ ( α ) ) = z ( θ, α ) · Var ( f K ( s , α )) (41) = z ( θ, α ) K · Var ( g K ( D, α )) (42) f K ( s , ) = 2 = 3 = 4 = 5 Fig. 3. The relation between Kim’s test f K ( s , α ) and the estimated θ of (37)for L = 6 . where D denotes the random variable of D n ( s ) in (15) and θ ( α ) denotes the estimated θ via f K ( s , α ) .It is clear that a larger K (more samples) reduces Var ( θ ( α ) ) .We observe that Var ( θ ( α ) ) → ∞ as θ → B . It is because z ( θ, α ) → ∞ as θ → B by (40). Moreover, Fig. 3 shows that z ( θ, α ) < z ( θ, α + 1) for higher min-entropy (i.e., for lower θ ), which is characterized in the following theorem. Theorem 10:
For θ = B + δ where δ ≪ B , z ( θ, α ) isapproximated to z ( θ, α ) ≃ B α − α ( α − · B − δ (43)and ξ = z ( θ, α + 1) z ( θ, α ) ≃ α − α + 1 · B. (44)For B = 64 (i.e., the given parameter of NIST SP 800-90B), ξ > (i.e., z ( θ, α ) < z ( θ, α + 1) ) if α > . Proof:
The proof is given in Appendix B.The following theorem shows
Var ( θ (2) ) < Var ( θ (3) ) formost sample sources. Note that Var ( θ (2) ) < Var ( θ (3) ) isequivalent to σ (2) < σ (3) where σ ( α ) denotes the standarddeviation of H ( ∞ ) ( B ) estimated by f K ( s , α ) . Theorem 11:
For a sample s with θ , σ (2) < σ (3) if θ < − B − . (45) Proof:
The proof is given in Appendix C.
Remark 12:
For B = 64 (i.e., the given parameter of NISTSP 800-90B), this condition of θ corresponds to θ < ≃ . and H ( ∞ ) ( S ) > . . Hence, we claim that σ (2) <σ (3) for the most of random sources. Remark 13:
A higher order α can reduce the theoreticalgap, which improves the accuracy (Theorem 9). On the otherhand, a higher order α increases the variance of min-entropyestimate. The order α is a parameter of the trade-off relationbetween the gap and the variance.By considering the trade-off between the theoretical gapand the variance, we observe that α = 2 is a proper valuefrom numerical evaluations in Section VI. Note that the R´enyientropy with α = 2 corresponds to the collision entropy. In thefollowing section, we propose estimation algorithms based onKim’s test for the collision entropy. Fortunately, the proposedalgorithms are very efficient for α = 2 . V. P
ROPOSED E STIMATOR B ASED ON C OLLISION E NTROPY
A. Proposed Estimator Based on Collision Entropy
For the collision entropy, we show that the proposed esti-mator of Algorithm 3 has the following advantages:1) The computations are simplified because a closed-formsolution of Step 4 in Algorithm 3 can be derived (seeCorollary 14);2) Samples for initialization are not required. Note that boththe compression estimator and the proposed estimatorbased on Coron’s test require Q ( > × L ) samplesfor initialization (see Remark 3). Corollary 14:
For a estimated collision entropy H (2) ( B ) = f K ( s , α = 2) , the min-entropy is lower bounded as follows: H ( ∞ ) ( B ) ≥ − log θ (2) (46)where θ (2) = ( B , if ≤ f K ( s , ≤ B ; √ ( B − B · f K ( s , − B , if B < f K ( s , ≤ (47)where f K ( s ,
2) = f K ( s , α = 2) . Proof:
First, we note that ≤ f K ( s , ≤ by (15)and (19). If f K ( s , ≤ B , then we set f K ( s ,
2) = B because H (2) ( B ) = − log f K ( s , ≤ L by entropy defi-nition. Hence, θ (2) = B . If B < f K ( s , ≤ , then wederive θ (2) = ± √ ( B − B · f K ( s , − B from (37). We choose θ (2) = √ ( B − B · f K ( s , − B because of the given conditionof B ≤ θ (2) ≤ .It is worth mentioning that the proposed estimator basedon the collision entropy has advantages over the compressionestimator of the NIST 800-90B in terms of accuracy (reducedtheoretical gap), computational complexity (closed-form solu-tion), and data efficiency (skipped initialization stage). B. Online Estimator Based on Collision Entropy
We propose online estimator by leveraging the advantagesof the collision entropy (see Algorithm 4). Since the proposedon-line estimator processes the data samples in a serial manner,it can estimate the min-entropy with limited samples and thenimprove its estimation accuracy as getting more samples. Theproposed on-line estimator does not need to store the entiresamples, hence, the proposed online estimator is lightweightand proper for applications with stringent resource constraints.The proposed online estimator has two parts: 1) Estimationof the collision probability; 2) estimation of the min-entropyfrom the collision probability. The first part (Steps 3–10) isan online algorithm to estimate the Collision entropy. For thecollision entropy, g K ( i, is given by (19). Hence, it countsonly an event that a new block is the same as its previous one(Step 6) (i.e., collision counting in consecutive blocks). Then,the collision probability η converges to f K ( s , as gettingmore blocks.The second part (Steps 11–16) estimates the min-entropyfrom the collision probability η in an online manner. This part Algorithm 4
Proposed online min-entropy estimator
Input: L -bit blocks b ( s ) = ( b , . . . , b K ) Output: H ( ∞ ) k ( S ) for k ∈ { , . . . , K } and the collisionindex set C k := 1 , c := 0 , v := b , C = ∅ ⊲ Initialization while k < K do k := k + 1 u := b k if u = v then c := c + 1 ⊲ Count collision C := C ∪ k end if v := u η := ck ⊲ Compute collision probability if η > B then θ := √ ( B − B · η − B else if η ≤ B then θ := B end if H ( ∞ ) k ( S ) := − log θL end while relies on the closed-form solution of θ in Corollary 14. Theproposed online algorithm is computationally simple and canoutput a new estimate of min-entropy H ( ∞ ) k ( S ) as getting anew block b k .The proposed online estimator is helpful to detect low en-tropy sources with limited samples. It is because the estimationvariance of low entropy sources is not large, so its estimate canbe obtained reliably with limited samples (see Fig. 8). Hence,the proposed online estimator can filter out the low entropysources very effectively.VI. N UMERICAL R ESULTS
We evaluate our proposed estimators for simulated andreal world data samples. We also compare our results tothe compression estimator in NIST SP 800-90B. Amongestimators of NIST SP 800-90B, we focus on the compressionestimator because the compression estimator and the proposedestimators attempt to estimate the min-entropy based on theminimum distance between the matching blocks D n ( s ) of(10). We note that our proposed estimators can be appealingalternatives to the compression estimator.Datasets of simulated samples are produced using the fol-lowing distribution families as in [3]: • Binary memoryless source (BMS):
Samples are generatedby Bernoulli distribution with P ( S = 1) = p and P ( S =0) = 1 − p (IID); • Near-uniform distribution:
Samples are generated bynear-uniform distribution of (22) (IID); • Inverted near-uniform distribution:
Samples are gener-ated by inverted near-uniform distribution of (33) (IID); • Normal distribution rounded to integers:
Samples aredrawn from a normal distribution and rounded to integervalues (IID); p E s t i m a t ed m i n - en t r op y CorrectCompression estimator (Algo. 1)Proposed estimator (Algo. 2)Proposed estimator (Algo. 3, = 2)Proposed estimator (Algo. 3, = 3)
Fig. 4. Comparison of min-entropy estimators for binary memoryless sourceswith p . • Markov model:
Samples are generated using the firstorder Markov model (non-IID).One hundred simulated sources were created in each ofthe above datasets. A sequence of 6,000,000 bits (1,000,000blocks) was generated from each simulated sources. Note thatthe compression estimator of NIST SP 800-90B sets L = 6 .For each source, the correct min-entropy is derived from thegiven probability distribution as in [3].Fig. 4 compares the min-entropy estimators for BMSwith p . The correct min-entropy is given by H ( ∞ ) ( S ) = − log max { p, − p } . We observe that the compression es-timator (Algorithm 1) of NIST SP 800-90B and the proposedestimator based on Coron’s test (Algorithm 2) are almostidentical. By comparing the computational complexities ofestimators (see Table II), the proposed estimator based onCoron’s test is an appealing alternative to the compressionestimator.The proposed estimators based on Kim’s test (Algorithm 3)with α = 2 (i.e., collision entropy) provides better estimatesthan the compression estimator and the proposed estimatorbased on Coron’s test since the theoretical gap can be reduced.However, for a BMS with p = 0 . , Algorithm 3 with α = 2 isslightly worse. It is because the theoretical gap is zero for BMSwith p = 0 . (see Remark 6) and the variance of estimatesincreases for the higher α . Fig. 4 shows that Algorithm 3 with α = 3 suffers from large variances for high entropy sources.Hence, we focus on α = 2 for Algorithm 3 since the varianceis manageable and its computations are very efficient.Table III shows the mean squared error (MSE) and the meanpercentage error (MPE) of all the estimators for the BMS.Suppose that the correct (actual) min-entropy is h and theestimates are b h n for n ∈ { , . . . , N } . Then, the MSE andMPE are defined as: MSE = 1 N N X n =1 ( h − b h n ) , (48) MPE = 100 % N N X n =1 h − b h n h . (49)The MPEs are used to capture the sign of the error, whichis not captured by MSE [3]. We observe that the proposed Min-entropy E s t i m a t ed m i n - en t r op y CorrectCompression estimator (Algo. 1)Proposed estimator (Algo. 2)Proposed estimator (Algo. 3, = 2) (a)
Min-entropy E s t i m a t ed m i n - en t r op y CorrectCompression estimator (Algo. 1)Proposed estimator (Algo. 2)Proposed estimator (Algo. 3, = 2) (b)Fig. 5. Comparison of min-entropy estimators for (a) near-uniform distributedsources and (b) inverted near-uniform distributed sources. estimator based on the collision entropy (Algorithm 3) canimprove the estimation accuracy compared to other estimators.Fig. 5 compares the min-entropy estimators for near-uniform distributed sources and inverted near-uniform dis-tributed sources. As shown in Fig. 5(a), all the estimatorsare accurate for near-uniform distributed sources since all theestimators perform their estimation tasks by assuming near-uniform distribution, i.e., the lower bound on the min-entropyis the same as the actual min-entropy.On the other hand, the min-entropy estimates for invertednear-uniform distributed sources are quite underestimated asshown in Fig. 5(b). It is because the inverted near-uniformdistribution corresponds to the upper bounds in Fig. 1, whichleads to the maximal theoretical gap. The proposed estimatorbased on the collision entropy effectively reduces the theoret-ical gap, so it provides much more accurate estimates than theother estimators.Fig. 6 compares the min-entropy estimators for the normaldistributed sources (rounded to integers). For this distribution,it is known that the compression estimator is prone to sig-nificant underestimates [3]. The proposed estimator based onCoron’s test is slightly better than the compression estimator.More importantly, the proposed estimator based on the colli-sion entropy provides much more accurate estimates.Fig. 7 compares the min-entropy estimators for the firstorder Markov sources with p = p (1 |
0) = p (0 | . The TABLE IIIE
RROR M EASURES FOR
BMS
WITH pp = 0 . p = 0 . p = 0 . p = 0 . p = 0 . MSE of Algo. 1 0.0043 0.0179 0.0365 0.0434 0.0105MSE of Algo. 2 0.0036 0.0157 0.0336 0.0416 0.0107MSE of Algo. 3 ( α = 2 ) 0.0001 0.0012 0.0068 0.0184 0.0217MPE of Algo. 1 43.09 41.56 37.14 28.26 10.10MPE of Algo. 2 39.23 38.96 35.65 27.67 10.21MPE of Algo. 3 ( α = 2 ) 5.29 10.86 16.05 18.42 14.62 Min-entropy E s t i m a t ed m i n - en t r op y CorrectCompression estimator (Algo. 1)Proposed estimator (Algo. 2)Proposed estimator (Algo. 3, = 2)
Fig. 6. Comparison of min-entropy estimators for normal distributed sources. p E s t i m a t ed m i n - en t r op y CorrectCompression estimator (Algo. 1)Proposed estimator (Algo. 2)Proposed estimator (Algo. 3, = 2)
Fig. 7. Comparison of min-entropy estimators for the first order Markovsources with p = p (1 |
0) = p (0 | .TABLE IVP ER - BIT M IN - ENTROPY E STIMATE FOR R EAL W ORLD S OURCES
Algo. 1 Algo. 2 Algo. 3RANDOM.ORG 0.9110 0.9006 0.8690Ubld.it 0.8811 0.8811 0.8175LKRNG 0.9219 0.9006 0.8690 compression estimator and the proposed estimator based onCoron’s test are almost identical. The proposed estimatorbased on the collision entropy is much better than the otherestimators except at H ( ∞ ) ( S ) = 1 .We also evaluate min-entropy estimates using random num-ber generators deployed in the real world as in [3]. The trueentropies for these sources are unknown, so the MSE and MPEcannot be calculated. The estimates of the real world sources are presented in Table IV.We evaluate RANDOM.ORG, Ubld.it, and Linux kernelrandom number generator (LKRNG). RANDOM.ORG [21] isa service that provides random numbers based on atmosphericnoise and Ubld.it generates random numbers by a TrueRNGdevice by [22]. As we expected, the compression estimator(Algorithm 1) and the proposed min-entropy estimator basedon Coron’s test (Algorithm 2) are almost identical. The pro-posed estimator based on the collision entropy (Algorithm 3)is slightly lower than the others. It is because these real worldsources are high entropy sources, which make the varianceof estimates by Algorithm 3 larger than the variance of otherestimators as observed in Figs. 4 and 7.Fig. 8 shows the min-entropy estimates by online algorithm(Algorithm 4) for BMS with p . Due to the computationalefficiency, Algorithm 4 can output an estimate as getting a newblock b k . As collecting more samples, the estimate is improvedand its variance is reduced. We observe that higher entropysources result in larger variances as discussed in Section IV-B.We note that Algorithm 4 is very effective to detect lowentropy sources. It is because the proposed online estimatorprovides estimates as getting new blocks and low entropysources can be detected reliably with limited samples. Forexample, Fig. 8 shows that a low entropy source whose min-entropy is less than 0.5 can be detected by testing only severalhundred blocks, which is comparable to the required samples( > × L ) for initialization of the compression estimator.VII. C ONCLUSION
We proposed computationally efficient min-entropy estima-tors by leveraging the variations of Maurer’s test. The proposedestimator based on Coron’s test achieves the identical accuracywith much less computations compared to the compressionestimator. Moreover, we propose the min-entropy estimatorbased on the collision entropy. It has advantages over thecompression estimator in terms of estimation accuracy, com-putational complexity, and data efficiency. We also proposea lightweight estimator which processes data samples in anonline manner without having the entire samples.A
PPENDIX AP ROOF OF T HEOREM θ ( α ) ≥ θ ( α +1) for θ ( α ) ≫ B − α − α ,which is equivalent to (39). For convenience, suppose that x = θ ( α ) and y = θ ( α +1) . E s t i m a t ed m i n - en t r op y (a) E s t i m a t ed m i n - en t r op y (b) E s t i m a t ed m i n - en t r op y (c)Fig. 8. Online min-entropy estimates by Algorithm 4 for BMS with p (20sample sources): (a) p = 0 . , (b) p = 0 . , and (c) p = 0 . . In [23], it was shown that α − α H ( α ) ≤ β − β H ( β ) for β > α and αβ > . If β = α + 1 and α > , H ( α ) ( B ) ≤ α α − H ( α +1) ( B ) (50)Then, we obtain the following inequality for the near-uniformdistribution: − α log (cid:18) x α + (1 − x ) α ( B − α − (cid:19) ≤ α − α log (cid:18) y α +1 + (1 − y ) α +1 ( B − α (cid:19) , (51) which is equivalent to (cid:18) x α + (1 − x ) α ( B − α − (cid:19) α ≥ (cid:18) y α +1 + (1 − y ) α +1 ( B − α (cid:19) α +1 . (52)If x α ≫ (1 − x ) α ( B − α − and y α +1 ≫ (1 − y ) α +1 ( B − α , then (52) becomes x ≥ y . Hence, θ ( α ) ≥ θ ( α +1) for θ ( α ) ≫ B − α − α .A PPENDIX BP ROOF OF T HEOREM θ = B + δ where δ ≪ B . Then, (40) becomes z ( θ, α ) = 1 α · (cid:0) B + δ (cid:1) α − − (cid:16) − B − δB − (cid:17) α − (53) = 1 α · B α − (1 + Bδ ) α − − (cid:16) − BB − δ (cid:17) α − (54) ≃ α · B α − { α − Bδ } − { − ( α − BδB − } (55) = B α − α ( α − · B − δ . (56)where (55) follows from (1 + Bδ ) α − ≃ α − Bδ and (cid:16) − BB − δ (cid:17) α − ≃ − ( α − BB − δ for δ ≪ B . Itis straightforward to derive (44) from (56).A PPENDIX CP ROOF OF T HEOREM
Var ( f K ( s , ≤ Var ( f K ( s , . Afterwards, we show that z ( θ, < z ( θ, if θ < − B − . Then, Var ( θ (2) ) < Var ( θ (3) ) , i.e., σ (2) < σ (3) if θ < − B − .i) By (15), Var ( f K ( s , α )) = K Var ( g K ( D, α )) where D denotes D n ( s ) . For α = 2 , g K ( D, is given by (19). Hence, E ( g K ( D, P ( D = 1) . Also, E (cid:0) g K ( D, α = 2) (cid:1) = K X k =1 P ( D = k ) g K ( D = k, α = 2) = P ( D = 1) . (57)Then, Var ( g K ( s , α = 2)) = P ( D = 1) − P ( D = 1) . (58)From (16), we obtain g K ( i,
3) = , if i = 1; − , if i = 2;0 , otherwise . (59) Then, we can derive E ( g K ( D, P ( D = 1) − P ( D = 2) and E (cid:0) g K ( D, (cid:1) = P ( D = 1) + P ( D = 2) . Hence, Var ( g K ( s , α = 3))= P ( D = 1) + P ( D = 2) − { P ( D = 1) − P ( D = 2) } = Var ( g K ( s , α = 2)) + { P ( D = 2) − P ( D = 2) } + 2 P ( D = 1) P ( D = 2) (60) ≥ Var ( g ( D, α = 2)) (61)where (60) follows from (58) and (61) follows from P ( D =2) ≥ P ( D = 2) and P ( D ) ≥ .ii) From (40), the inequality z ( θ, α ) < z ( θ, α + 1) isequivalent to ( α + 1) (cid:26) θ α − (cid:18) − θB − (cid:19) α (cid:27) < α ( θ α − − (cid:18) − θB − (cid:19) α − ) . (62)For α = 2 , (62) becomes (cid:18) θ − B (cid:19) (3( B − θ − B + 5) < , (63)which is equivalent to B < θ < B − B − = − B − .Note that B < − B − for B > , which holds L ≥ .Since θ > B by definition, we obtain z ( θ, α ) < z ( θ, α + 1) if θ < − B − . R EFERENCES[1] M. S. Turan, E. Barker, J. Kelsey, K. A. McKay, M. L. Baish, andM. Boyle,
Recommendation for the entropy sources used for random bitgeneration , NIST Special Publication 800-90B Std., Jan. 2018.[2] P. Hagerty and T. Draper, “Entropy bounds and statistical tests,” in
Proc.NIST Random Bit Generation Workshop , Dec. 2012, pp. 1–28.[3] J. Kelsey, K. A. McKay, and M. S. Turan, “Predictive models formin-entropy estimation,” in
Proc. Int. Workshop Cryptograph. Hardw.Embedded Syst. (CHES) , Berlin, Heidelberg, Sep. 2015, pp. 373–392.[4] T. Amaki, M. Hashimoto, Y. Mitsuyama, and T. Onoye, “A worst-case-aware design methodology for noise-tolerant oscillator-based truerandom number generator with stochastic behavior modeling,”
IEEETrans. Inf. Forensics Security , vol. 8, no. 8, pp. 1331–1342, Aug. 2013.[5] Y. Ma, T. Chen, J. Lin, J. Yang, and J. Jing, “Entropy estimation forADC sampling-based true random number generators,”
IEEE Trans. Inf.Forensics Security , vol. 14, no. 11, pp. 2887–2900, Nov. 2019.[6] W. Killmann and W. Schindler,
A proposal for: Functionality classesfor random number generators , German Federal Office for InformationSecurity (BSI) Std., Rev. 2, Sep. 2011.[7] A. Rukhin, J. Soto, J. Nechvatal, M. Smid, E. Barker, S. Leigh,M. Levenson, M. Vangel, D. Banks, A. Heckert, J. Dray, and S. Vo,
Astatistical test suite for random and pseudorandom number generatorsfor cryptographic applications , NIST Special Publication 800-22 Std.,Rev. 1a, Apr. 2010.[8] S. Zhu, Y. Ma, T. Chen, J. Lin, and J. Jing, “Analysis and improvementof entropy estimators in NIST SP 800-90B for non-IID entropy sources,”
IACR Trans. Symmetric Cryptol. , vol. 2017, no. 3, pp. 151–168, Sep.2017.[9] S. Zhu, Y. Ma, X. Li, J. Yang, J. Lin, and J. Jing, “On the analysis andimprovement of min-entropy estimation on time-varying data,”
IEEETrans. Inf. Forensics Security , vol. 15, pp. 1696–1708, Oct. 2020.[10] U. M. Maurer, “A universal statistical test for random bit generators,”
J. Cryptol. , vol. 5, no. 2, pp. 89–105, Jan. 1992.[11] J.-S. Coron, “On the security of random sources,” in
Proc. Int. WorkshopPublic Key Cryptography , Mar. 1999, pp. 29–42.[12] D. Tebbe and S. Dwyer, “Uncertainty and the probability of error,”
IEEETrans. Inf. Theory , vol. 14, no. 3, pp. 516–518, May 1968. [13] J. Golic, “On the relationship between the information measures and theBayes probability of error,”
IEEE Trans. Inf. Theory , vol. 33, no. 5, pp.681–693, Sep. 1987.[14] M. Feder and N. Merhav, “Relations between entropy and error proba-bility,”
IEEE Trans. Inf. Theory , vol. 40, no. 1, pp. 259–266, Jan. 1994.[15] Y.-S. Kim, “Low complexity estimation method of R´enyi entropy forergodic sources,”
Entropy , vol. 20, no. 9, pp. 1–14, Aug. 2018.[16] C. Beck and F. Sch¨ogl,
Thermodynamics of Chaotic Systems: AnIntroduction , ser. Cambridge Nonlinear Science Series. CambridgeUniversity Press, 1993.[17] P. Elias, “Interval and recency rank source coding: Two on-line adaptivevariable-length schemes,”
IEEE Trans. Inf. Theory , vol. 33, no. 1, pp.3–10, Jan. 1987.[18] F. M. J. Willems, “Universal data compression and repetition times,”
IEEE Trans. Inf. Theory , vol. 35, no. 1, pp. 54–58, Jan. 1989.[19] J.-S. Coron and D. Naccache, “An accurate evaluation of Maurer’suniversal test,” in
Proc. Int. Workshop Sel. Areas Cryptography , Aug.1999, pp. 57–71.[20] T. M. Cover and J. A. Thomas,
Elements of Information Theory