MIP and Set Covering approaches for Sparse Approximation
aa r X i v : . [ c s . D M ] S e p MIP and Set Covering approaches for Sparse Approximation ∗ Diego Delle Donne , Matthieu Kowalski and Leo Liberti . LIX CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91128 Palaiseau, France. Laboratoire des Signaux et Systmes, UMR 8506 Univ Paris-Sud – CNRS – Centralesupelec, 91192 Gif-sur-Yvette Cedex, France
Abstract—
The sparse approximation problem asks to find asolution x such that || y − Hx || < α , for a given norm || · || , mini-mizing the size of the support || x || := { j | x j = 0 } . We presentvalid inequalities for Mixed Integer Programming (MIP) formu-lations for this problem and we show that these families are suffi-cient to describe the set of feasible supports. This leads to a refor-mulation of the problem as an Integer Programming (IP) modelwhich in turn represent a minimum set covering formulation, thusyielding many families of valid inequalities which may be used tostrengthen the models up. We propose algorithms to solve sparseapproximation problems including a branch & cut for the MIP,a two-stages algorithm to tackle the set covering IP and a heuris-tic approach based on Local Branching type constraints. Thesemethods are compared in a computational experimentation withthe goal of testing their practical potential. The sparse representation of a vector y ∈ R n in a dictionary H ∈ R n × m aims to find a solution x ∈ R m to the system Hx = y , having the minimum number of non-zero compo-nents, i.e., minimizing the so-called ℓ pseudo-norm of x , de-fined by || x || := |{ j | x j = 0 }| . The sparse approximation problem takes also into account noise and model errors. It re-laxes the equality constraint aiming to minimize the misfit datameasure || y − Hx || , for a given norm || · || . In this context,several optimization problems may be stated as such:1. minimize || x || subject to a given threshold for the datamisfit || y − Hx || ≤ α ,2. minimize the data misfit || y − Hx || subject to a givenbound || x || ≤ k ,3. minimize a weighted sum λ || y − Hx || + λ || x || for some λ , λ ∈ R .In this work, we study mixed integer programming (MIP) for-mulations for the problem stated in Item 1 when the norm usedfor the data misfit measure is the ℓ and ℓ ∞ norms (we addressthe reader to [4] for the rest of the cases). Following the nota-tion from [4], we define these problems as P /p ( α ) : min x || x || s. t. || y − Hx || p ≤ α, In the remaining, we may write just P /p whenever α p isclear from the context and/or irrelevant. Also, for any naturalnumber t , we may use [ t ] as a shortcut for the set { , . . . , t } .Some natural mixed-integer programming (MIP) formula-tions for P / and P / ∞ are introduced in [4]. These models ∗ This research was partially supported by Labex DigiCosme (project ANR-11-LABEX-0045-DIGICOSME) operated by ANR as part of the program “In-vestissement d’Avenir” Idex Paris-Saclay (ANR-11-IDEX-0003-02). use decision variables x j ∈ R , for each j ∈ [ m ] to determinethe solution and binary support variables b j to state whether x j has a non-zero value or not. They require to (artificially)bound | x j | with a value M in order to properly state themodels. Then, P j ∈ [ m ] b j is minimized subject to appropriateconstraints. We call these formulations MIP / and MIP / ∞ ,respectively, and we state them here for completeness. [ MIP / ] min X j ∈ [ m ] b j (1) − M b j ≤ x j ≤ M b j ∀ j ∈ [ m ] (2) − w i ≤ y i − X j ∈ [ m ] h ij x j ≤ w i ∀ i ∈ [ n ] (3) X i ∈ [ n ] w i ≤ α (4) w i , x j ∈ R , b j ∈ { , } ∀ i ∈ [ n ] , j ∈ [ m ] (5) [ MIP / ∞ ] min X j ∈ [ m ] b j (6) − M b j ≤ x j ≤ M b j ∀ j ∈ [ m ] (7) − w ≤ y i − X j ∈ [ m ] h ij x j ≤ w ∀ i ∈ [ n ] (8) w ≤ α ∞ (9) x j ∈ R , b j ∈ { , } ∀ j ∈ [ m ] (10)In [4], these formulations are solved directly by CPLEX. Asfar as we know, no polyhedral studies have been done for theseformulations, with the goal of developing more powerful reso-lution (e.g., cutting planes based) algorithms.In this context, we study the polytopes arising from these for-mulations and derive valid inequalities for them which we thenuse within an initial branch and cut algorithm for MIP /p . Ad-ditionally, we prove that the obtained inequalities are sufficientto describe the projection of these polytopes into the space ofthe binary variables b j and that this projections are in fact setcovering polytopes. Based on this fact, we introduce a novelIP approach for P /p which consists in solving a pure com-binatorial set covering formulation (with exponentially manycovering constraints) and we propose a two-stages algorithm totackle this IP. Furthermore, we propose a heuristic approach re-sorting to the Variable Neighborhood Search metaheuristic [7]and Local Branching [6] type constraints. We say that a set of columns J ⊆ [ m ] is a forbidden support for P /p if there exist no solutions with J as support, i.e., if in x {|| y − H J x J || p } > α p , where H J (resp. x J ) is thesubmatrix of H (resp. subvector of x ) involving only thosecolumns indexed by J . Proposition 2.1. If J ⊆ [ m ] is a forbidden support for P /p ,then the forbidden support inequality X j ∈ [ m ] \ J b j ≥ (11) is valid for M IP /p . From Proposition 2.1, we derive some subfamilies of valid in-equalities for which we developed separation procedures (bothexact and heuristics) and implemented a branch & cut algo-rithm using them as cutting planes. We omit here these ele-ments due to space limitations. We state next an interestingtheoretical result about the forbidden support inequalities (11).
Proposition 2.2.
The projection on the variables b j of all fea-sible solutions of formulation M IP /p can be described by theforbidden support inequalities (11) as P fs = { b ∈ { , } m | b satisfies (11) for each forb. supp. J ⊆ [ m ] } . Proposition 2.2 lets us obtain a minimum support ˆ b of a solu-tion to P /p by solving the following integer programming (IP)formulation: [ IP cov /p ] min X j ∈ [ m ] b j (12) X j ∈ [ m ] \ J b j ≥ ∀ forb. supp. J ⊆ [ m ] (13) b j ∈ { , } ∀ j ∈ [ m ] (14)We remark that by solving IP cov /p we do not obtain a solutionfor P /p but just an optimal support S ⊆ [ m ] . However, asolution x for this support can be efficiently obtained after-wards. Precisely, the non-zero values of x can be obtainedby minimizing || y − H S x S || p (which for p ∈ { , ∞} canbe achieved by solving a linear program). Moreover, as thesupport is already fixed for this last step, there is no need touse the (usually artificial) big-M bounds for x , which gives animportant advantage against formulation MIP /p .An initial drawback towards the computational resolutionof IP cov /p , is that the formulation may have exponentially manyconstraints (13). However, we note that we can efficiently testwhether a vector b ∈ { , } m satisfies all these constraints ornot, without the need of enumerating them. Due to Proposition2.2, a vector b ∈ { , } m satisfies (13) if and only if thereexists a feasible solution to MIP /p with support b , i.e., min x ∈ R m {|| y − H S x S || p } ≤ α p , where S is the supportdescribed by b . As mentioned before, for p ∈ { , ∞} , thiscan be tested by solving a linear program. Based on thischaracteristic, we propose a two stages algorithm which startsfrom a combinatorial relaxation of IP cov /p with a few (or none)constraints and dynamically adds constraints (13) whenever anoptimal integer but not feasible solution is found.The equivalence given by Proposition 2.2 has also useful im-plications. In particular, we note that IP cov /p represents a mini-mum set covering problem and this kind of problems has been widely studied in the literature both in the polyhedral and in thecombinatorial aspects [1, 2, 3, 5, 8, 9, 10, 11]. A direct impli-cation of this is the fact that any valid inequality for these setcovering polytopes can provide a valid inequality for MIP /p .We give next an example in which we depict a (non intuitive)family of valid inequalities for MIP /p obtained from a knownfamily of set covering facets [1]. Proposition 2.3.
Let
J ⊆ [ m ] be a family of forbidden sup-ports for P /p , and define J none := [ m ] \ S J ∈J J , and J some := [ m ] \ ( J none ∪ T J ∈J J ) . Then the forbidden supportfamily inequality X j ∈ J none b j + X j ∈ J some b j ≥ (15) is valid for MIP /p . Given an integer solution (ˆ x, ˆ w, ˆ b ) for MIP /p , the idea of localbranching is to impose a constraint forcing the solution to be“similar” to (ˆ x, ˆ w, ˆ b ) . In our setting, this constraint ensures thatthe difference in the support sizes should not exceed a prespec-ified value δ , i.e., X j ∈ J b j + X j ∈ J (1 − b j ) ≤ δ. (16)with J i := { j ∈ [ m ] : ˆ b j = i } , for i ∈ { , } . The additionof Constraint (16) to MIP /p reduces the feasible region to asort of δ -neighborhood of the given point (ˆ x, ˆ w, ˆ b ) , aiming toobtain a faster (although heuristic) resolution, which should bethe case for small values of δ . We propose a heuristic algo-rithm (based on the Variable Neighborhood Search [7] meta-heuristic) which starts from an initial solution and explores its δ -neighborhood for increasing values of δ , beginning from apredefined value δ . Every time a better solution is found, δ isreseted to δ and the process is repeated from this new solution. We proposed some MIP based approaches for the sparse ap-proximation problem, including both exact and heuristic meth-ods. All these methods can be improved and/or combined to-wards the development of a general algorithm. As a first stepon that direction, in this work we aimed to test the potential ofthe proposed methods by conducting a computational experi-mentation over a set of hard instances (arising from a patholog-ical example from the literature). We omit to show numericalresults here due to space limitations. According to our experi-mentation, the heuristic algorithm (from Section 3) provides agood starting point towards the development of an efficient ap-proximation algorithm for the problem addressed in this work.Within this work, an interesting relation between P /p andthe minimum set covering problem was established. As a fu-ture line of research, we believe that this relation shall be ex-ploited. In particular, it would be interesting to derive morefamilies of valid inequalities from set covering polytopes, in or-der to strengthen the initial branch and cut implemented for thiswork. Additionally, the combinatorial aspects of the minimumset covering problem may uncover interesting tools towards theefficient resolution of MIP /p . We leave these aspects for afuture work. eferences [1] E. Balas and S.M. Ng. On the set covering polytope: I.All the facets with coefficients in { , , } . MathematicalProgramming , 43:57–69, 1989.[2] E. Balas and S.M. Ng. On the set covering polytope: II.All the facets with coefficients in { , , } . MathematicalProgramming , 45:1–20, 1989.[3] Ralf Bornd¨orfer.
Aspects of Set Packing, Partitioning, andCovering . PhD thesis, 1998.[4] S. Bourguignon, J. Ninin, H. Carfantan, and M. Mon-geau. Exact Sparse Approximation Problems via Mixed-Integer Programming: Formulations and ComputationalPerformance.
IEEE Transactions on Signal Processing ,64(6):1405–1419, March 2016.[5] G. Cornu´ejols and A. Sassano. On the 0, 1 facets of the setcovering polytope.
Mathematical Programming , 43:45–55, 1989.[6] M. Fischetti and A. Lodi. Local branching.
MathematicalProgramming , 98:23–47, 2003.[7] P. Hansen, N. Mladenovic, and J.A.M. Perez. Variableneighbourhood search: methods and applications.
Annalsof Operations Research , 175:367–407, 2010.[8] Laurent. A Generalization of Antiwebs to IndependenceSystems and Their Canonical Facets.
Mathematical Pro-gramming , 45:97–108, 1989.[9] P. Nobili and A. Sassano. Facets and lifting procedures forthe set covering polytope.
Mathematical Programming ,45:111–137, 1989.[10] M. S´anchez-Garc´ıa, M.I. Sobr´on, and B. Vitoriano. Onthe set covering polytope: Facets with coefficients in { , , , } . Annals of Operations Research , 81:343–356,1998.[11] A. Sassano. On the facial structure of the set cover-ing polytope.