aa r X i v : . [ c s . I T ] S e p Starting CLuP with polytope relaxation
Mihailo Stojnic ∗ Abstract
The Controlled Loosening-up (CLuP) mechanism that we recently introduced in [23] is a generic conceptthat can be utilized to solve a large class of problems in polynomial time. Since it relies in its core on aniterative procedure, the key to its excellent performance lies in a typically very small number of iterationsneeded to execute the entire algorithm. In a separate paper [22], we presented a detailed complexity analysisthat indeed confirms the relatively small number of iterations. Since both papers, [23] and [22] are theintroductory papers on the topic we made sure to limit the initial discussion just to the core of the algorithmand consequently focused only on the algorithm’s most basic version. On numerous occasions though, weemphasized that various improvements and further upgrades are possible. In this paper we present a firststep in this direction and discuss a very simple upgrade that can be introduced on top of the basic CLuPmechanism. It relates to the starting of the CLuP and suggests the well-known so-called polytope-relaxationheuristic (see, e.g. [24, 25]) as the starting point. We refer to this variant of CLuP as the CLuP-plt andproceed with the presentation of its complexity analysis. As in [22], a particular complexity analysis periteration level type of complexity analysis is chosen and presented through the algorithm’s application onthe well-known MIMO ML detection problem. As expected, the analysis confirms that CLuP-plt performseven better than the original CLuP. In some of the most interesting regimes it often achieves within the firstthree iterations an excellent performance. We also complement the theoretical findings with a solid set ofnumerical experiments. Those also happen to be in an excellent agreement with the analytical predictions.
Index Terms: Controlled Loosening-up (CLuP); Polytope relaxation; ML - detection; MIMOsystems; Algorthms; Random duality theory . To handle famous MIMO ML detection problem, we in [23] presented the so-called Controlled Loosening-up(CLuP) algorithm. Since the CLuP algorithm will be the main topic of this paper as well, and since we willstudy its behavior when applied for solving the MIMO ML detection problems, we first briefly recall on thebasics of the MIMO ML.As usual, one start with the most basic linear system: y = A x sol + σ v , (1)where y ∈ R m is the output vector, A ∈ R m × n is the system matrix , x sol ∈ R n is the input vector, v ∈ R m is the noise vector at the output, and σ is a scaling factor that determines the ratio of the useful signaland the noise (the so-called SNR (signal-to-noise ratio)). It goes without saying that this type of systemmodeling is among the most useful/popular in various scientific/engineering fields (a particularly popularapplication of this model in the fields of information theory and signal processing is its utilization in modelingof multi-antenna systems).Also, we will here continue the trend that we have started in [23] and [22], and consider a statistical setupwhere both v and A are comprised of i.i.d. standard normal random variables. A similar continuing thetrend from [23] and [22] regarding the so-called linear regime will be in place as well. That means that inthis paper we will also view n and m as large but with a constant proportionality between them, i.e. we will ∗ e-mail: [email protected] m = αn where α ∈ R + is a number that doesn’t change as both n and m grow. The followingoptimization problem is the simplest yet most fundamental version of the MIMO ML-detection problemˆ x = min x ∈X k y − A x k , (2)where X is the set of all available input vectors x . Now, many interesting scenarios/variants of the MIMOML problem appear depending on the structure of X (for example, LASSO/SOCP variants of (2) often seenin statistics, machine learning, and compressed sensing are just a tiny subset of many very popular scenariosof interest; more on these considerations can be found in e.g. [1–3, 10, 13, 26, 27]). Here, we follow into thefootsteps of [22, 23] and consider the standard information theory/wireless communications binary scenariowhich assumes X = {− √ n , √ n } n . It goes trivially, basically almost without saying, that x sol ∈ X is naturallyassumed as well. We will also without a loss of generality assume even further that x sol = { √ n , √ n , . . . , √ n } .The above problem (2) can be solved either exactly or approximately (for more on various relaxingheuristics see, e.g. [5, 6, 9, 28]). What makes it particularly interesting is that in the above mentioned binaryscenario, (2) is typically viewed as a very hard combinatorial optimization type of problem. As such it wasobviously the topic of interest in various research communities over last at least half a century. Many excellentalgorithms and algorithmic heuristics have been introduced over this period of time. As a detailed discussionabout such developments is more suited for review papers we here just in passing mention that some ofthe very best results regarding to the perspective of the problem that is of interest here can be found ine.g. [4,7,8,24,25]. In addition, we also emphasize two probably the most important points: 1) the problem in(2) is hard if one wants/needs to solve it exactly and 2) polynomial heuristics typically offer an approximatesolution that does trail the exact one by a solid margin in almost all interesting scenarios. In [23] and [22] weintroduced the above mentioned CLuP mechanism as a way of attacking the MIMO ML on the exact level.Compared to [24, 25], which also attacked the MIMO ML on the exact level, CLuP did so by running only afew (fairly often not more than 10) simplest possible quadratic programming type of iterations. In [22], thisrather remarkable property was analytically characterized. Here we provide a similar characterization for aslightly different upgraded variant of CLuP. Before we proceed with the characterization of this new CLuPvariant we below first recall on the CLuP’s basics. As is by now well-known from [22, 23], CLuP is effectively a very simple iterative procedure that in its coreform assumes choosing a starting x (0) ∈ X = {− √ n , √ n } n , radius r , and running the following x ( i +1) = x ( i +1 ,s ) k x ( i +1 ,s ) k with x ( i +1 ,s ) = arg min x − ( x ( i ) ) T x subject to k y − A x k ≤ r x ∈ (cid:20) − √ n , √ n (cid:21) n . (3)As one can guess, the choice for x (0) and r has a very strong effect on the way how the algorithm progresses.For the simplest possible choice of x (0) (each component of x (0) is generated with equal likelihood as − √ n or √ n ) we in Figure 1 show both, the theoretical and the simulated CLuP’s performance. Without going intotoo much details, we just briefly mention that as r increases from 1 . r plt to 1 . r plt (more on the definitionand importance of r plt can be found in [23]) CLuP’s performance gets closer to the ML. Further detailedexplanations related to the figure can be found in [23]. Those among other things include a discussionregarding the appearance of a vertical line (the so-called line of corrections). We of course skip repeatingsuch discussion and just mention that in this paper (similarly to [22]) we will be interested in the regimesabove the line, i.e. in the regimes where the SNR, 1 /σ , is to the right of the line.What is of a bit more interest to the present paper though (and what can’t exactly be seen from Figure1) is the complexity of the above CLuP algorithm. The analysis of the CLuP’s complexity was of course themain topic of [22]. The remarkable CLuP’s property that it fairly often runs not only in a polynomial butrather fixed number of iterations was through such an analysis fully characterized. What may have escaped2 / σ in [db] p e rr -4 -3 -2 -1 L i n e o f m il d o r no c o rr ec t i on s CLuP – approaching the exact ML p ( plt ) err – theoryˆ p ( CLuP ) err – theory r sc = 1ˆ p ( CLuP ) err – theory r sc = { . , . , . } ˆ p ( CLuP ) err – theory ultimate CLuPˆ p ( ml ) err – theory (1FL ML (1RSB))ˆ p ( CLuP ) err – simulated r sc = 1 . p ( CLuP ) err – simulated r sc = 1 . p ( CLuP ) err – simulated r sc = 1 . . [db] r sc = . sc = . sc = . Figure 1: p err as a function of 1 /σ ; α = 0 . x (0) is basically completely random and as such in a way completely disconnected from theproblem at hand. On the other hand, it seems rather natural that a bit more clever choice could help CLuPachieve even better performance. There are a tone of possible choices for x (0) and the next natural questionwould be which of such choices would be the best or at least better than the random one. Such a discussionrequires a careful analysis and we will present it in a separate companion paper. To insure that the initialdiscussion in this direction is as simple as possible, we here focus on a particular choice of the starting x (0) that we view as pretty much the simplest, most natural one after the fully random one considered in [22,23]. The new CLuP’s variant that we consider in this paper (and to which we refer as CLuP-plt), assumes simplygenerating x (0) as the solution to the standard polytope-relaxation heuristic (see, e.g. [24,25]) of the originalML problem (2) x (0 ,plt ) = arg min x k y − A x k subject to x ∈ (cid:20) − √ n , √ n (cid:21) n . (4)Then one can define CLuP-plt as x ( i +1) = x ( i +1 ,s ) k x ( i +1 ,s ) k with x ( i +1 ,s ) = arg min x − ( x ( i ) ) T x subject to k y − A x k ≤ r x ∈ (cid:20) − √ n , √ n (cid:21) n , (5)3here i starts from zero and x (0) = x (0 ,plt ) . Alternatively, one can increment the indices and start countingthe iterations by first setting x (1 ,s ) = x (0 ,plt ) and x (1) = x (1 ,s ) k x (1 ,s ) k = x (0 ,plt ) k x (0 ,plt ) k , (6)and then continuing with (5) for i ≥
1. To be in an alignment with what we have done in [22] and toaccurately account for the (4) as the first iteration of the algorithm (as we should) we will rely on (6) and(5) with i ≥ x (0 ,plt ) is expected to be closer tothe targeted optimal solution and as such might help getting to the optimum faster. Below we will providean analysis that will confirm these expectations. We will formally focus on the algorithm’s complexity,which due to its iterative nature amounts to handling the number of iterations. However, we will presenta particular type of analysis that we typically refer to as the complexity analysis per iteration level ,where we basically fully characterize all system parameters and how they change through each of the runningiterations. Such an analysis is of course way more demanding than just mere computation of the total numberof needed iterations.Through the presentation below we will see that the analysis of CLuP-plt can be designed so that it to alarge degree parallels what we have done when we analyzed the complexity of the original CLuP in [22]. Wewill therefore try to avoid repeating many explanations that are to a large degree similar or even identicalto the corresponding ones in [22] and instead focus on the key differences. Also, we will emphasize it onmultiple occasions but do mention it here as well that we chose a very simple upgrade to showcase potentialof the CLuP’s core mechanism. Since we will be utilizing the main concepts of the analysis from [22] in someof our companion papers as well, we also found this particular upgrade as a very convenient choice to quicklyget fully familiar with all the key steps of [22]. In a way, we will essentially through a reconsideration in thispaper bring those steps (that at first may appear complicated) to a level of a routine. This will turn out tobe particularly useful when we switch to discussion of a bit more advanced structures.Parallelling what was done in [22] the presentation will be split into several parts. The characterizationof the algorithms’s first iteration will be briefly discussed at the beginning and then in the second part wewill move to the second and higher iterations. We will also present a large set of simulations results andobserve that they are in a rather nice agreement with the theoretical findings. As mentioned above, to facilitate the exposition and following we will try to parallel as much as possible theflow of the presentation from [22]. That means that the core of the complexity analysis will again be theso-called complexity analysis on per iteration level .We start things off by noting that a combination of (1) and (4) gives the following version of the CLuP-plt’s first iterations x (0 ,plt ) = arg min x k σ v + A ( x sol − x ) k subject to x ∈ (cid:20) − √ n , √ n (cid:21) n , (7)which with a cosmetic change easily becomes x (0 ,plt ) = arg min x k σ v + A z k subject to z ∈ (cid:20) , √ n (cid:21) n . (8)Following considerations from [22, 23] and ultimately those from [11–21] and utilizing the concentration4trategy we set k z k = c ,z and instead of (8) consider ξ p, ( α, σ, c ,z ) = lim n →∞ √ n E min z k σ v + A z k subject to k z k = c ,z z ∈ (cid:2) , / √ n (cid:3) n . (9)It is now not that hard to note that the problem in (9) is conceptually identical to the one in equation (7)in [22]. In fact, it can be thought of a special case of the one from [22] with s and the components of x (0) in equation (7) in [22] being equal to zero. This basically means that one can completely repeat the restof the analysis from the second section of [22]. The only substantial difference will be that the ν variablefrom [22]’s second section will now be zero. In particular, instead of [22]’s equation (16) one now has for theoptimizing z i z i = 1 √ n min (cid:18) max (cid:18) , − (cid:18) h γ (cid:19)(cid:19) , (cid:19) . (10)Moreover, analogously to [22]’s equations (18) and (19) one now has I , ( γ ) = − ( exp ( − . γ ) )( − γ ) + p π/ √ γ + 1 / √ / (4 √ πγ ) I , ( γ ) = 2 γ erfc((4 γ ) / √ − exp ( − / γ ) ) / √ π, (11)and ξ (1) RD ( α, σ ; c ,z , γ ) = √ α q c ,z + σ + I , ( γ ) + I , ( γ ) − γc ,z . (12)The following theorem summarizes what we presented above. Theorem 1. (CLuP-plt – RDT estimate – first iteration) Let ξ p, ( α, σ, c ,z ) and ξ (1) RD ( α, σ ; c ,z , γ ) be as in(9) and (12), respectively. Then ξ p, ( α, σ, c ,z ) = max γ ξ (1) RD ( α, σ ; c ,z , γ ) . (13) Consequently, min c ,z ξ p, ( α, σ, c ,z ) = min c ,z max γ ξ (1) RD ( α, σ ; c ,z , γ ) . (14) Proof.
Follows automatically from [22] and ultimately the RDT mechanisms from [12–16, 18, 19] (as in [22],the strong random duality is trivially in place here as well).We do mention in passing also that one can trivially first solve the optimization over c ,z and effectivelytransform/simplify the above optimization problem to an optimization over only γ . However, to maintainparallelism with [22] and ultimately with what we will present below, we avoided doing so. Since the above theorem is very similar to the corresponding one in [22], we below continue to follow intothe footsteps of [22] and in a summarized way formalize how it can be utilized to finally obtain all of the keyalgorithm’s parameters in the first iteration.
Summary of the CLuP-plt’s first iteration
We first solve { ˆ γ (1) , ˆ c (1)1 ,z } = arg min ≤ c ,z ≤ max γ ξ (1) RD ( α, σ ; c ,z , γ ) (15)and then as in [22]’s equations (23) define s x, ( γ ) = 1 / /γ/ √ π (1 − exp ( − (4 γ ) / xsq, ( γ ) = − I , ( γ ) /γs x, ( γ ) = 2( . γ ) / √ s xsq, ( γ ) = 2 s x, . (16)Moreover, analogously to [22]’s (24),(25), and (26) we now have √ n E z i = s x, (ˆ γ (1) ) + s x, (ˆ γ (1) ) n E z i = s xsq, (ˆ γ (1) ) + s xsq, (ˆ γ (1) ) , (17)and with x i = x sol − z i also √ n E x i = 1 − ( s x, (ˆ γ (1) ) + s x, (ˆ γ (1) )) n E x i = s xsq, (ˆ γ (1) ) + s xsq, (ˆ γ (1) ) + √ n E x i − , (18)and finally E (( x sol ) T x ) = 1 − ( s x, (ˆ γ (1) ) + s x, (ˆ γ (1) )) E k x k = s xsq, (ˆ γ (1) ) + s xsq, (ˆ γ (1) ) + 2 E (( x sol ) T x ) − . (19)As in [22], the strong random duality ensures that the above are not only the expected values but alsothe concentrating points of the corresponding quantities (concentration of course is exponential in n ). Asin [22]’s (27) one can also obtain for the probability of error p (1) err = 1 − P (cid:18) z i ≤ √ n (cid:19) = 1 −
12 erfc (cid:18) − γ (1) √ (cid:19) . (20)The theoretical values for all key system parameters that can be obtained utilizing the above Theorem 1are shown in Table 1 for SNR, 1 /σ = 13[db]. To maintain the parallelism with [22] and with what we willTable 1: Theoretical values for key system parameters obtained based on Theorem 1 /σ [db] ˆ ν (1) ˆ γ (1) ˆ c (1)1 ,z ˆ s (1)1 ξ (1) RD p (1) err k x (1 ,s ) k ( x sol ) T x (1 ,s ) . . − . . . . present below, we artificially keep two additional parameters ˆ ν (1) and ˆ s (1)1 and assign the value 0 to them. The move from the first to the second iteration is of course of critical importance for understanding all latermoves from k -th to ( k + 1)-th iteration for k ≥
2. The CLuP’s second iteration assumes computing x (2) as x (2) = x (2 ,s ) k x (2 ,s ) k with x (2 ,s ) = arg min x − ( x (1) ) T x subject to k y − A x k ≤ r x ∈ (cid:20) − √ n , √ n (cid:21) n , (21)where we recall from (6), x (1) = x (1 ,s ) k x (1 ,s ) k = x (0 ,plt ) k x (0 ,plt ) k . One can then also rewrite (21) in the following waymin z ( x (1) ) T z subject to k σ v + A z k ≤ r ∈ (cid:2) , / √ n (cid:3) n . (22)Utilizing once again the concentration strategy we set k z k = c ,z and ( x (1 ,s ) ) T z = s and consider ξ p, ( α, σ, c ,z , s ) = lim n →∞ √ n E min z k σ v + A z k subject to k z k = c ,z ( x (1 ,s ) ) T z = s z ∈ (cid:2) , / √ n (cid:3) n . (23)The above problem is structurally literally identical to [22]’s (31). One can then repeat all the steps between[22]’s (31) and (56) to arrive at the following set of equations that determine the optimizing z i and x (2 ,s ) i z (2) i = 1 √ n min max , − h (1 ,p ) i + ν x (1 ,s ) i + ν γ !! , ! x (2 ,s ) i = 1 √ n − z (2) i = 1 √ n − min max , − h (1 ,p ) i + ν x (1 ,s ) i + ν γ !! , !! , (24)where one also recalls from (6) x (1 ,s ) i = x (0 ,plt ) = 1 − z (1) i = 1 √ n (cid:18) − min (cid:18) max (cid:18) , − (cid:18) h i γ (1) (cid:19)(cid:19) , (cid:19)(cid:19) . (25)As in [22], h (1 ,p ) = p (1) h + p − ( p (1) ) h (1) and the components of both h and h (1) are i.i.d. standardnormals. Setting I (2)1 ( γ, ν, ν ) = Z Z (( h (1 ,p ) i + ν x (1 ,s ) + ν ) z (2) i + γ (cid:16) z (2) i (cid:17) ) exp − (cid:16) h (1) i (cid:17) + h i d h (1) i d h i π , (26)where if negative, the term under the integral is zero for γ <
0. Analogously to [22]’s (60) one can define ξ (2) RD ( α, σ ; p (1) , q (1) , c ,z , s , s , γ, ν, ν ) = √ α q c ,z + σ (cid:18) q (1) p (1) + q − ( q (1) ) q − ( p (1) ) (cid:19) + I (2)1 ( γ, ν, ν ) − νs − ν s − γc ,z . (27)Finally, from [22]’s (76)-(78) one has φ (2) b = arg min s,d (2)1 ,d (2)2 s subject to max p (1) min ≤ c ,z ≤ max γ,ν,ν ξ (2) RD ( α, σ ; p (1) , q (1) , c ,z , s , s , γ, ν, ν ) = rs = d (1)1 + s q d (1)2 s = 1 − d (2)1 c ,z = d (2)2 − d (2)1 + 1 q (1) = s − s + σ p c ,z + σ p ˆ c ,z + σ , (28)7nd p (2) err = 1 − Z Z ((sign( x (2 ,s ) ) + 1) / exp − (cid:16) h (1) i (cid:17) + h i d h (1) i d h i π . (29)To obtain the remaining key parameters one can utilizeˆ d (2)2 = Z Z (( x (2 ,s ) i ) exp − (cid:16) h (1) i (cid:17) + h i d h (1) i d h i π ˆ d (2)1 = Z Z (( x (2 ,s ) i ) exp − (cid:16) h (1) i (cid:17) + h i d h (1) i d h i π ˆ s (2)2 = Z Z (( x (1 ,s ) i ) z (2) i exp − (cid:16) h (1) i (cid:17) + h i d h (1) i d h i π . (30)To make things easier to follow one can define a set of the key output parameters at the end of the seconditeration (of course, it goes without emphasizing that x (2 ,s ) is the main output of the second iteration). Thisset consists of critical plus auxiliary parameters φ (2) = { p (2) err , ˆ s (2) , ˆ d (2)2 , ˆ d (2)1 , ˆ ν (2) , ˆ ν (2)2 , ˆ γ (2) , ˆ p (1) , ˆ q (1) , ˆ c (1)2 ,z , ˆ s (2)2 , ˆ s (2)3 } , (31)where p (2) err − probability of error after the second iterationˆ s (2) = E (( x (1) ) T x (2 ,s ) ) − objective value after the second iterationˆ d (2)2 = E k x (2 ,s ) k − squared norm after the second iterationˆ d (2)1 = E x Tsol x (2 ,s ) − inner product with x sol after the second iteration . (32)with the last three quantities being not only the expected but also the concentrating values as well. Beforeproceeding with the numerical results for the second iteration we recall the output of the first iteration φ (1) = { p (1) err , ˆ s (1) , ˆ d (1)2 , ˆ d (1)1 , ˆ ν (1) , ˆ γ (1) , ˆ c (1)1 ,z } = { . , − , . , . , , . , . } . (33)The theoretical values for the output parameters after the second iteration (i.e. for the parameters from(31) that are obtained through the discussion presented above for SNR, 1 /σ = 13[db], α = 0 .
8, and r sc = 1 .
3) are included in Table 2. One can also characterize the remaining auxiliary parameters fromTable 2:
Theoretical values for various parameters at the output of the second iteration /σ [db] ˆ ν (2) ˆ ν (2)2 ˆ γ (2) ˆ p (1) − ˆ s (2) ξ (2) RD p (2) err k x (2 ,s ) k ( x sol ) T x (2 ,s ) . − . . .
769 0 . . . . . φ (2) , i.e. { ˆ q (1) , ˆ c (2)2 ,z , ˆ s (2)2 , ˆ s (2)3 } relying on the equality constraints in (28). Table 3 shows the results for theseparameters that can be obtained through both, the equality constraints in (28) and (30).8able 3: Theoretical values for { ˆ q (1) , ˆ c (2)2 ,z , ˆ s (2)2 , ˆ s (2)3 } obtained utilizing (28) ( bold ) as well as (30) ( purple ) /σ [db] ˆ s (2)2 = ˆ d (1)1 + ˆ s (2) q ˆ d (1)2 ˆ s (2)3 = 1 − ˆ d (2)1 ˆ c (2)2 ,z = ˆ d (2)2 − d (2)1 + 1 ˆ q (1) = ˆ s − ˆ s + σ √ ˆ c ,z + σ √ ˆ c ,z + σ . / . . / . . / . . / . ( k + 1) -th iteration analysis The heart of the analysis mechanism is the move from the first to the second iteration. Such a move isconceptually then identical to the move from any k -th to ( k + 1)-th iteration. However, there are still a fewtechnical differences that require a special attention. These differences are of course the main reason why weseparately discuss a generic move from k -th to ( k + 1)-th iteration for any k >
1. On the other hand, we havealready faced a similar situation in [22] and all the results obtained there in this regard can be reutilized.We start by recalling that CLuP’s ( k + 1)-th iteration is basically the following optimization problem x ( k +1) = x ( k +1 ,s ) k x ( k +1 ,s ) k with x ( k +1 ,s ) = arg min x − ( x ( k ) ) T x subject to k y − A x k ≤ r x ∈ (cid:20) − √ n , √ n (cid:21) n . (34)This is of course structurally identical to (85) in [22]. One can then again utilize the Random Duality Theoryand repeat all the steps between (85) and (108) in [22] to arrive at the following for the optimizing z i and x ( k +1 ,s ) i z ( k +1) i = 1 √ n min max , − h ( k,p ) i + P kj =1 ˜ ν j x ( j,s ) i + ν γ !! , ! x ( k +1 ,s ) i = 1 √ n − z (2) i = 1 √ n − min max , − h ( k,p ) i + P kj =1 ˜ ν j x ( j,s ) i + ν γ !! , !! , (35)where x ( j,s ) i , ≤ j ≤ k are obtained after the k -th iteration as the optimizing variables after each of the first k iterations. One can also set as in [22]’s (110) I ( k +1)1 ( γ, ν, ν , ˆ ν (1) ) = E (( h ( k,p ) i + k X j =1 ˜ ν j x ( j,s ) i + ν ) z ( k +1) i + γ (cid:16) z ( k +1) i (cid:17) ) , (36)where for γ < ξ ( k +1) RD ( α, σ ; P ( k +1) , Q ( k +1) , c ,z , s ,j , s , γ, ˜ ν j , ν ) = √ α q c ,z + σ f ( k +1) sph + I ( k +1)1 ( γ, ν, ν , ˆ ν (1) ) − k X j =1 ˜ ν j s ,j − ν s − γc ,z , (37)where P ( k +1) and Q ( k +1) are as in [22]’s (90) and f ( k +1) sph is as in [22]’s (101). We also note from [22]’s(112)-(114) that the key output parameters after the k -th iteration are x ( j,s ) i , z ( j ) i , λ ( j − , ≤ j ≤ k, (38)9nd φ ( k ) = { p ( k ) err , ˆ s ( k ) , ˆ d ( k )2 , ˆ d ( k )1 , ˆ ν ( k ) , ˆ ν ( k )2 , ˆ γ ( k ) , ˆ P ( k ) , ˆ Q ( k ) , ˆ c ( k )2 ,z , ˆ s ( k )2 , ˆ s ( k )3 } , (39)where ˆ ν ( k ) and ˆ s ( k )2 are the ( k − ν and s vectors at the k -th iteration (i.e. for k > ν ( k ) = [˜ ν , ˜ ν , . . . , ˜ ν k − ] for optimal ˜ ν j and ˆ s ( k )2 = [ s , , s , , . . . , s ,k − ] for optimal s ,j ). We also recallfrom [22] that four probably most important output parameters after the k -th iteration are p ( k ) err − probability of error after the k -th iterationˆ s ( k ) = E (( x ( k − ) T x ( k,s ) ) − objective value after the k -th iterationˆ d ( k )2 = E k x ( k,s ) k − squared norm after the k -th iterationˆ d ( k )1 = E x Tsol x ( k,s ) − inner product with x sol after the k -th iteration . (40)The above is what one essentially has avaialable before approaching handling the ( k + 1)-th iteration. Key part – Handling the ( k + 1) -th iteration Analogously to [22]’s (117) we have φ ( k +1) b = arg min s,d ( k +1)1 ,d ( k +1)2 ,s ,j s subject to max P ( k +1) min ≤ c ,z ≤ max γ,ν,ν ξ ( k +1) RD ( α, σ ; P ( k +1) , Q ( k +1) , c ,z , s ,j , s , γ, ˜ ν j , ν ) = rs ,k = ˆ d ( k )1 + s q ˆ d ( k )2 s = 1 − d ( k +1)1 c ,z = d ( k +1)2 − d ( k +1)1 + 1 Q ( k +1) k +1 ,j = s − s ,j + σ p c ,z + σ q ˆ c ( j )2 ,z + σ , (41)with φ ( k +1) b = { ˆ P ( k +1) , ˆ Q ( k +1) , ˆ ν ( k +1) , ˆ ν ( k +1)2 , ˆ γ ( k +1) , ˆ s ( k +1) , ˆ d ( k +1)2 , ˆ d ( k +1)1 , ˆ s ( k +1)2 } . (42)Moreover, analogously to [22]’s (118)-(119) we also have p ( k +1) err = 1 − E ((sign( x ( k +1 ,s ) ) + 1) / . (43)Finally, in addition to the solution, x ( k +1 ,s ) , we also have the following as the full set of all critical plusauxiliary parameters that appear at the output of the ( k + 1)-th iteration: φ ( k +1) = { p ( k +1) err , ˆ s ( k +1) , ˆ d ( k +1)2 , ˆ d ( k +1)1 , ˆ ν ( k +1) , ˆ ν ( k +1)2 , ˆ γ ( k +1) , ˆ P ( k +1) , ˆ Q ( k +1) , ˆ c ( k +1)2 ,z , ˆ s ( k +1)2 , ˆ s ( k +1)3 } , (44)where analogously to (40) p ( k +1) err − probability of error after the ( k + 1)-th iterationˆ s ( k +1) = E (( x ( k ) ) T x ( k +1 ,s ) ) − objective value after the ( k + 1)-th iterationˆ d ( k +1)2 = E k x ( k +1 ,s ) k − squared norm after the ( k + 1)-th iterationˆ d ( k +1)1 = E x Tsol x ( k +1 ,s ) − inner product with x sol after the ( k + 1)-th iteration . (45) Numerical results – third iteration ( k = 2 ) As we have mentioned in [22], the above discussion is in principle enough to compute all the criticalparameters and basically fully characterize the CLuP-plt’s performance. We will in a separate paper presenta systematic way to compute all these parameters. In [22] we gave a quick estimate whose an analogue wecould obtain here as well. However, as it will turn out later on, in the case of interest that we chose to10ighlight in this paper ( α = 0 .
8, 1 /σ = 13[db], r sc = 1 . k = 2, i.e. in the third iteration.First, we recall from (33) that for the set of key output parameters after the first iteration we obtainedthe following φ (1) = { p (1) err , ˆ s (1) , ˆ d (1)2 , ˆ d (1)1 , ˆ ν (1) , ˆ γ (1) , ˆ c (1)1 ,z } = { . , − , . , . , , . , . } . (46)This set of parameters is then utilized to obtain in Tables 2 and 3 a similar set of parameters after the seconditeration (basically the one from (31)) φ (2) = { p (2) err , ˆ s (2) , ˆ d (2)2 , ˆ d (2)1 , ˆ ν (2) , ˆ ν (2)2 , ˆ γ (2) , ˆ p (1) , ˆ q (1) , ˆ c (1)2 ,z , ˆ s (2)2 , ˆ s (2)3 } = { . , − . , . , . , . , − . , . , . , . , . , . , . } . (47)The φ (2) set can then be utilized according to the above mechanism to obtain φ (3) set from (44). In Tables 4and 5 we show some of these results. In particular, we show the results for { p (3) err , ˆ s (3) , ˆ d (3)2 , ˆ d (3)1 , ˆ ν (3) , ˆ ν (3)2 , ˆ γ (3) } Table 4:
Theoretical values for various parameters at the output of the third iteration /σ [db] ˆ ν (3) ˆ ν (3)2 ˆ γ (3) − ˆ s (3) ξ (3) RD p (3) err k x (3 ,s ) k ( x sol ) T x (3 ,s )
13 [ − . , . ] − . . . . . . . in Table 4. In Table 5, we show the results for { ˆ c (3)2 ,z , ˆ s (3)2 , ˆ s (3)3 } . The second component of ˆ s (3)2 based on (41)is ˆ d (2)1 + ˆ s (3) q ˆ d (2)2 . The results for optimized matrices P and Q aer shown separatelyTable 5: Theoretical values for { ˆ c (3)2 ,z , ˆ s (3)2 , ˆ s (3)3 } /σ [db] ˆ s (3)2 ˆ s (3)3 = 1 − ˆ d (3)1 ˆ c (3)2 ,z = ˆ d (3)2 − d (3)1 + 113 [ . , . ] . . ˆ P (3) = .
769 0 . .
769 1 0 . .
771 0 .
984 1 (48)ˆ Q (3) = .
839 0 . .
839 1 0 . .
792 0 .
984 1 . (49)We also recall from (41) ˆ Q (3)3 , = ˆ s (3)3 − ˆ s (3)2 , + σ q ˆ c (3)2 ,z + σ q ˆ c (1)2 ,z + σ ˆ Q (3)3 , = ˆ s (3)3 − ˆ s (3)2 , + σ q ˆ c (3)2 ,z + σ q ˆ c (2)2 ,z + σ , (50)where ˆ s (3)2 , is the second component of vector ˆ s (3)2 and for cosmetic reasons ˆ c (1)2 ,z = ˆ c (1)‘ ,z . From (28) and [22]’s1190) we also recall that ˆ Q (3)2 , = q (1) = ˆ s (2)3 − ˆ s (2)2 , + σ q ˆ c (2)2 ,z + σ q ˆ c (1)2 ,z + σ , (51)where ˆ s (2)2 , is the first (and only) component of vector ˆ s (2)2 .In Table 6, we show the estimated values for p ( k ) err , ˆ s ( k ) , ˆ d ( k )2 , and ˆ d ( k )1 and how they progress throughthe iterations We also add that the discussion from [22] regarding f (3) sph applies here as well with the valuesTable 6: Change in p ( k ) err , ˆ s ( k ) , k x ( k,s ) k , and ( x sol ) T x ( k,s ) as k grows; α = 0 . r sc = 1 .
3; CLuP-plt i p ( i ) err − ˆ s ( k ) ˆ d ( k )2 = k x ( i ) k ˆ d ( k )1 = ( x sol ) T x ( i ) . . . . . . . . . . . . limit . . . . even closer to one. Namely, from the above considerations and what we presented in [22], we now have f (3) sph ≥ . third iteration CLuP-plt approaches the limiting CLuP performance (such a performance does not have an a priori restrictionon the number of iterations). In fact, to be completely fair towards CLuP, the overall CluP-plt variantdiscussed above needs three iterations but the CLuP mechanism itself needs only two iterations (the firstiteration technically speaking is not really a part of the CLuP’s inherent structure; instead it belongs to thepolytope-relaxation heuristic).
We in this section complement the above theoretical findings with a set of results obtained through numericalsimulations. As mentioned above, we considered a standard scenario, α = 0 .
8, 1 /σ = 13[db], r sc = 1 . n = 800. The results are shown in Table 7. We observe a very good agreementTable 7: CLuP-plt – change in p ( k ) err , ˆ s ( k ) , k x ( k,s ) k , and ( x sol ) T x ( k,s ) as k grows; α = 0 . r sc = 1 . Simulated ( n = 800)/ Theory–computed ( n → ∞ ) k p ( k ) err − ˆ s ( k ) ˆ d ( k )2 = k x ( k,s ) k ˆ d ( k )1 = ( x sol ) T x ( k,s ) . / . . / . . / . . / . . / . . / . . / . . / . . / . . / . . / . . / . limit . . . . between the theoretical predictions and the results obtained through numerical simulations. We also obtainedthe following simulated results for matrices P and QP (3) = . . . . . . ˆ P (3) = .
769 0 . .
769 1 0 . .
771 0 .
984 1 (52)and Q (3) = . . . . . . ˆ Q (3) = .
839 0 . .
839 1 0 . .
792 0 .
984 1 . (53)12e again observe a very solid agreement between the theoretical and simulated results.For a comparison, we in Table 8 show how the performance of the CLuP algorithm itself progressesthrough the iterations for the same parameters as above. We observe that it takes one iteration less forTable 8: CLuP – change in p ( k ) err , ˆ s ( k ) , k x ( k,s ) k , and ( x sol ) T x ( k,s ) as k grows; α = 0 . r sc = 1 . n = 800; Simulated k p ( k ) err ˆ s ( k ) ˆ d ( k )2 = k x ( k,s ) k ˆ d ( k )1 = ( x sol ) T x ( k,s ) . . . . . . . . . . . . . . . . . . . . limit . . . . CLuP-plt to get to a better performance level than the original CLuP. One should here also keep in mindwhat we mentioned earlier. Namely, the first iteration in CLuP-plt is not really a part of the CLuP structureitself, whereas the first iteration in [22] is. That basically means that starting CLuP with a better initial x (0) in this case saves two out of four iterations. We also simulated two different SNR scenarios as well. We first decreased the SNR to 1 /σ = 12[db] and thento 1 /σ = 11[db] while havng all other parameters take the same values as in the above 1 /σ = 13[db] case.The results are shown in Tables 9 and 10. In particular, in Table 9 we show the results for 1 /σ = 12[db].Table 9: CLuP-plt – change in p ( k ) err , ˆ s ( k ) , k x ( k,s ) k , and ( x sol ) T x ( k,s ) as k grows; 1 /σ = 12[db]; α = 0 . r sc = 1 . n = 800; Simulated k p ( k ) err ˆ s ( k ) ˆ d ( k )2 = k x ( k,s ) k ˆ d ( k )1 = ( x sol ) T x ( k,s ) . . . . . . . . . . . . . . . . . . . . limit . . . . Again after a fairly small number of iterations one approaches the limiting performance. Moreover, althoughone is now a bit closer to the line of corrections (and supposedly a tougher to handle SNR regime), theincrease in the number of iterations is rather minimal. Instead of three iterations that were needed for1 /σ = 13[db] here five iterations suffice. The same discussion regarding counting the starting iterationsthat we emphasized above applies again. That means that in terms of CLuP’s own iterations, instead of twofor 1 /σ = 13[db] one now needs four for 1 /σ = 12[db].In Table 9 we show the results for 1 /σ = 11[db]. One is now really close to the line of corrections anda significant increase in the number of iterations might be expected. However, as results in the table showwithin 8 iterations one is reaching performance level literally identical to the optimal one. However, alreadyafter the fifth iteration one is very close to the optimum with the margin of error being on the fourth decimal(of course this iteration numbering accounts for the first iteration; if one is more fair towards CLuP thenthese 8 and 5 iterations should be replaced by 7 and 4 of CLuP’s own iterations). In [23] we introduced the so-called CLuP (Controlled Loosening-up) method as a way to solve the MIMO MLproblem exactly . As observed already in [23], one of the CLuP’s very best features is its very low running13able 10: CLuP-plt – change in p ( k ) err , ˆ s ( k ) , k x ( k,s ) k , and ( x sol ) T x ( k,s ) as k grows; 1 /σ = 11[db]; α = 0 . r sc = 1 . n = 800; Simulated k p ( k ) err ˆ s ( k ) ˆ d ( k )2 = k x ( k,s ) k ˆ d ( k )1 = ( x sol ) T x ( k,s ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . limit . . . . complexity. Basically, as an iterative algorithm, its complexity is mainly driven by the number of iterationsthat it needs to achieves required convergence precision. Along those lines, what was essentially observedin [23] was the fact that the typical number of iterations needed to achieve an excellent performance is notonly polynomial but actually rather a fixed number that does not depend on the problem dimension. Onewould typically assume that if a polynomial number of iterations suffices then that would have already beenan unprecedent feature in algorithms that attack MIMO ML. The discovery that a fixed number of iterationsworks as well was beyond remarkable.We then in the followup paper [22] looked at this property more carefully and designed a Random DualityTheory type of mechanism to precisely quantify not only the required number of iterations but rather thebehavior of all important systems parameters as they move/change through the CLuP’s iterations. Such anapproach is of course the most complete type of performance analysis that one can hope for. As expected,the analysis confirmed all the observations from [23]. As we wanted to maintain the introductory paperson this topic to be related to the simplest possible underlying structures, we in [22, 23] considered only themost basic CLuP version. However, on numerous occasions we did emphasize that various more advancedstructures can now easily be built and analyzed. In this paper, we provide a first step in those directions.Namely, the standard basic CLuP from [22, 23] is here modified to its a variant where for the starting stepof the algorithm instead of a random initialization one utilizes the well-known so-called polytope-relaxationheuristic. We provided again a detailed per iteration level analysis similar to the one that we providedin [22] for the standard CLuP. It turns out that the new version of CLuP, to which we refer as CLuP-plt, isindeed faster and requires a smaller number of iterations.We should here also emphasize as in [22,23], that we again didn’t utilize the most advanced concepts butrather a very simple upgrade. As earlier, we wanted to showcase the conceptual opportunity for upgraderather than its a best realization (we will address those in separate papers that will deal a bit more withfurther engineering of the main concepts rather than with their fundamental structuring). Nonetheless, thefact that in the regimes of interest CLuP-plt needed between 3-5 iterations in total (2-4 if one excludes theinitialization) continues to sound almost unbelievable. After the theoretical considerations, we proceededfurther and presented a solid set of numerical simulations results. They turned out to be in an excellentagreement with the theoretical predictions.As mentioned above, a large class of way more sophisticated CLuP versions we will discuss in separatepapers. Moreover, we will also in parallel present how they behave when applied for solving problems fromdifferent scientific fields as well. References [1] F. Bunea, A. B. Tsybakov, and M. H. Wegkamp. Sparsity oracle inequalities for the lasso.
ElectronicJournal of Statistics , 1:169–194, 2007. 142] S.S. Chen and D. Donoho. Examples of basis pursuit.
Proceeding of wavelet applications in signal andimage processing III , 1995.[3] D. Donoho, A. Maleki, and A. Montanari. The noise-sensitiviy thase transition in compressed sensing.available online at http://arxiv.org/abs/1004.1218 .[4] U. Fincke and M. Pohst. Improved methods for calculating vectors of short length in a lattice, includinga complexity analysis.
Mathematics of Computation , 44:463–471, April 1985.[5] M. Goemans and D. Williamnson. Improved approximation algorithms for maximum cut and satisfia-bility problems using semidefinite programming.
Journal of ACM , 42(6):1115–1145, 1995.[6] G. Golub and C. Van Loan.
Matrix Computations . John Hopkins University Press, 3rd edition, 1996.[7] B. Hassibi and H. Vikalo. On the sphere decoding algorithm. Part I: The expected complexity.
IEEETrans. on Signal Processing , 53(8):2806–2818, August 2005.[8] J. Jalden and B. Ottersten. On the complexity of the sphere decoding in digital communications.
IEEETrans. on Signal Processing , 53(4):1474–1484, August 2005.[9] L. Lovasz M. Grotschel and A. Schriver.
Geometric algorithms and combinatorial optimization . NewYork: Springer-Verlag, 2nd edition, 1993.[10] N. Meinshausen and B. Yu. Lasso-type recovery of sparse representations for high-dimensional data.
Ann. Statist. , 37(1):246270, 2009.[11] M. Stojnic. Block-length dependent thresholds in block-sparse compressed sensing. available online at http://arxiv.org/abs/0907.3679 .[12] M. Stojnic. Discrete perceptrons. available online at http://arxiv.org/abs/1306.4375 .[13] M. Stojnic. A framework for perfromance characterization of
LASSO algortihms. available online at http://arxiv.org/abs/1303.7291 .[14] M. Stojnic. A performance analysis framework for
SOCP algorithms in noisy compressed sensing.available online at http://arxiv.org/abs/1304.0002 .[15] M. Stojnic. A problem dependent analysis of
SOCP algorithms in noisy compressed sensing. availableonline at http://arxiv.org/abs/1304.0480 .[16] M. Stojnic. Regularly random duality. available online at http://arxiv.org/abs/1303.7295 .[17] M. Stojnic. Upper-bounding ℓ -optimization weak thresholds. available online at http://arxiv.org/abs/1303.7289 .[18] M. Stojnic. Various thresholds for ℓ -optimization in compressed sensing. available online at http://arxiv.org/abs/0907.3666 .[19] M. Stojnic. Recovery thresholds for ℓ optimization in binary compressed sensing. ISIT, IEEE Inter-national Symposium on Information Theory , pages 1593 – 1597, 13-18 June 2010. Austin, TX.[20] M. Stojnic. Box constrained ℓ optimization in random linear systems – asymptotics. 2016. availableonline at http://arxiv.org/abs/1612.06835 .[21] M. Stojnic. Box constrained ℓ optimization in random linear systems – finite dimensions. 2016. availableonline at http://arxiv.org/abs/1612.06839 .[22] M. Stojnic. Complexity analysis of the controlled loosening-up (CLuP) algorithm. 2019. available onlineat arxiv.[23] M. Stojnic. Controlled loosening-up (CLuP) – achieving exact MIMO ML in polynomial time. 2019.available online at arxiv. 1524] M. Stojnic, Haris Vikalo, and Babak Hassibi. A branch and bound approach to speed up the spheredecoder.
ICASSP, IEEE International Conference on Acoustics, Signal and Speech Processing , 3:429–432, March 2005.[25] M. Stojnic, Haris Vikalo, and Babak Hassibi. Speeding up the sphere decoder with H ∞ and SDP inspired lower bounds.
IEEE Transactions on Signal Processing , 56(2):712–726, February 2008.[26] R. Tibshirani. Regression shrinkage and selection with the lasso.
J. Royal Statistic. Society , B 58:267–288, 1996.[27] S. van de Geer. High-dimensional generalized linear models and the lasso.
Ann. Statist. , 36(2):614–645,2008.[28] H. van Maaren and J.P. Warners. Bound and fast approximation algorithms for binary quadraticoptimization problems with application on MAX 2SAT.