Vulnerability Coverage for Secure Configuration
11 Vulnerability Coverage for Secure Configuration
Shuvalaxmi Dass and Akbar Siami Namin
Computer Science DepartmentTexas Tech University shuva93.dass, [email protected]
Abstract
We present a novel idea on adequacy testing called “vulnerability coverage.” The introduced coverage measure examines theunderlying software for the presence of certain classes of vulnerabilities often found in the National Vulnerability Database (NVD)website. The thoroughness of the test input generation procedure is performed through the adaptation of evolutionary algorithmsnamely Genetic Algorithms (GA) and Particle Swarm Optimization (PSO). The methodology utilizes the Common VulnerabilityScoring System (CVSS), a free and open industry standard for assessing the severity of computer system security vulnerabilities,as a fitness measure for test inputs generation. The outcomes of these evolutionary algorithms are then evaluated in order toidentify the vulnerabilities that match a class of vulnerability patterns for testing purposes.
Index Terms
Software Vulnerability Testing, Vulnerability Coverage, Genetic Algorithms, Particle Swarm Optimization
I. I
NTRODUCTION
The National Vulnerability Database (NVD) [3] lists over , instances of vulnerabilities identified by their uniqueCVE (Common Vulnerabilities and Exposures) numbers. Some of these vulnerabilities exist partly due to improper settingsof configuration parameters that govern the functionality of the given software. Testing software applications against thesevulnerabilities can become tedious, and as a result, infeasible if the test has to target all the possible settings of a configurationin order to check the vulnerability of the software against some known attacks. This calls for a systematic configurationtesting framework and mechanism in order to mitigate the efforts put into inspecting the given software by identifying anarrowed down set of configuration test inputs. However, from vulnerability perspective, it is not easy to check whether thetests generated for testing examines the software system against certain classes of vulnerabilities. Hence, it makes it impracticalfor the administrator to exercise the given software under test against any vulnerability reported in NVD.To illustrate the overall mechanism of the proposed adequacy coverage, consider an example for vulnerabilities in MySQL.The “high” severity level of the MySQL CVE-2019-12463 vulnerability is . out of 10. There are several other vulnerabilitiesreported for MySQL with similar patterns as vulnerability vector and with similar severity scores. Then the major testingquestion is whether it is essential to examine the software under test for all the reported vulnerabilities with certain CVEnumbers. It is possible to view the problem as an instance of general software testing problem and thus develop a specificadequacy criterion for covering vulnerabilities and examine the software under test (SUT) for the vulnerabilities.This paper extends our initial idea [6] on vulnerability coverage and thus presents the novel concept of “ vulnerabilitycoverage ,” in an analogous way to conventional adequacy criterion in software testing. We apply evolutionary algorithms togenerate test inputs with certain patterns and use Common Vulnerability Scoring System (CVSS) as a primary criterion toassess the vulnerability coverage (i.r., fitness) of the SUT. The paper makes the following key contributions:1) We present the idea of vulnerability coverage as a test adequacy criterion for inspecting the given software against certaintypes of vulnerabilities (Section III).2) We perform evolutionary algorithms such as genetic algorithms (GA) and particle swarm optimisations (PSO) to generatevulnerability vector patterns (Section IV).3) We compare the performance of both GA and PSO in generating such vulnerability vector patterns (Section V). Accordingto our results, PSO managed to generate a more stable trend of secure vulnerability vector patterns than that of GA ina single generation.We provide relevant background on the Common Vulnerability Scoring System (CVSS) is presented in Section II. SectionIII presents the idea of vulnerability coverage as an adequacy test criterion. Section IV presents the fitness function for theevolutionary algorithms. In Section V, we present experimental setup and results. Section VI reviews the related work. SectionVII concludes the paper. Throughout the paper, we will be using the words “ vulnerability pattern ” and “ CVSS vector pattern ”interchangeably. II. V
ULNERABILITY S CORING S YSTEM
The Common Vulnerability Scoring System (CVSS) is an open-standard industry framework, which helps cyber-securityprofessionals to seek out information regarding ranking mechanism for the severity of vulnerabilities. CVSS captures the a r X i v : . [ c s . CR ] J un principle characteristics of the vulnerability by assigning a Base score rating ranging from 0 to 10 which is representativeof the ease of exploitation and the damaging effect of the concerned vulnerability where 10.0 is the most easily exploitablevulnerability. The numerical scores have a qualitative assessment (low, medium, high, and critical) to provide organizations withbetter understanding and assessment of vulnerabilities. Some vulnerability are also given temporal and environmental scoresthat may modify the base score. As a proof of concept in generating CVSS patterns, the GA and PSO optimization algorithmsare applied to the Base metrics. The Base metric consists of three sub-main metrics where each metric group comprises of aset of vector fields and the associated values it takes: • Exploitability Sub-Metric:
It addresses how the attack is captured. Table I lists down the vector fields. • Impact Sub-Metric:
It reflects the “ characteristics ” of the impacted components as shown in Table I. • Scope Sub-Metric:
It is a vector field acting as a separate metric which describes the change in the scope of the attackby determining whether other components are affected along with the original vulnerability. It accepts only two values:
Unchanged (U) and
Changed (C) . Sub-Metric Fields Description Values Score
Exploitability Network(N) 0.85Adjacent(A) 0.62Local (L) 0.55Attack Vector (AV) Reflects the proximity of the attacker to attackthe vulnerable component. Physical (P) 0.2Low(L) 0.77Attack Complexity (AC) Reflects the resources and conditions required toconduct the exploit on the vulnerable component. High(H) 0.44Low(L) 0.62(or 0.68 if Scope is Changed)High(H) 0.27(or 0.5 if Scope is Changed)Privileges Required (PR) Represents the level of privileges required by anattacker to successfully launch an exploit. None(N) 0.85None(N) 0.85User Interaction (UI) Reflects whether the participation of the user isrequired for launching a successful attack. Required (R) 0.62Impact Availability Impact (A) Measures the severity of the attack on the availabilityof the impacted component. Low(L) 0.00Integrity Impact (I) Measures the severity of the attack on the integrityof the impacted component. High(H) 0.22Confidentiality Impact (C) Measures the severity of the attack on the confiden-tiality of the impacted component. None(N) 0.56
TABLE I: Sub-Metrics.Base score formula is calculated as follows [1]:
Impact Sub − Score ( ISS ) = 1 − [(1 − C ) ∗ (1 − I ) ∗ (1 − A )] (1) Impact ( IM ) = . ∗ ISS if Scope is Unchanged . ∗ ( ISS − . − . ∗ ( ISS − . if Scope is Changed (2)
Exploitability ( EX ) = 8 . ∗ AV ∗ AC ∗ P R ∗ UI (3) Base Score = if Impact < = 0
Round ( Min [( IM + EX ) , if Scope is Unchanged
Round ( Min [1 . ∗ ( IM + EX ) , if Scope is changed (4)
Every known vulnerability’s severity can be represented as a vulnerability/CVSS vector pattern, which comprises all theaforementioned vector fields. For instance, Figure 1 shows the CVSS score and the vulnerability pattern vector for CVE-2019-14389 [3]. As shown in the figure, the score for this vulnerability is high and is quantified as . out of 10. The generatedrepresentation of the vulnerability/CVSS vector is [AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H] . Table II lists thevector pattern description for this vulnerability. Parameter Description
AV: L Denotes the vulnerability is exploited by the attacker throughaccessing the target system locally (L).AC: L Represents that the vulnerability has a Low (L) complexity ofbeing attacked.PR: L Shows that a Low (L) number of Privileges are required forsuccessfully exploiting this vulnerability.UI: N Denotes that no (N) User Interaction and involvement is requiredto launch a successful attack.S: U Shows the Scope (S) of the attack is Unchanged (U).C: H Total loss (High) of confidentiality.I: H Total loss (high) of integrity, or a complete loss of protection.A: H Total loss (High) of availability, full access denial to resources inthe impacted component.
TABLE II: CVSS vector pattern description.
Fig. 1: The description of CVE-2019-14389 for MySQL.III. V
ULNERABILITY C OVERAGE A S A DEQUACY T ESTING
In an analogous way to the conventional definition given for “code coverage” in software testing, the vulnerability coverageis a measurement of how many known and reported vulnerabilities of the system under test (SUT) are inspected against.Similarly, the vulnerability coverage (VC) for a software system (S) can be measured as follows:
V C S = ( vulnerabilities inspected ) S ( T otal of vulnerabilities reported ) S ∗ (5) Where ( vulnerabilities inspected ) S is the number of vulnerabilities inspected for the system S , and ( T otal of vulnerabilities reported ) S is the total number of vulnerabilities reported for system S . Without loss of generality,this paper uses vulnerability patterns provided by CVSS score to measure the adequacy testing of vulnerabilities give for asoftware system.It is important to note that vulnerabilities of the same pattern might be found in CVE database directly. However, giventhe evolutionary search algorithms presented in this paper, it is possible to identify vulnerabilities of different patterns butsimilar CVSS score. Hence, the use of CVSS score, as a fitness function, enables us to identify various forms and patternsof vulnerabilities within the specific level of CVSS score. Therefore, given the desired level of CVSS score, the problem ofadequacy testing for vulnerability testing will be exercising the vulnerabilities with different patterns but equal CVSS scores.In following sections, we adopt two evolutionary algorithms that enable us search the input space (i.e., vulnerability pattern)that achieve a certain level of CVSS score (i.e., the fitness value).IV. F ITNESS F UNCTION : CVSS S
CORE
This section explains the genetic and optimization algorithms developed in which CVSS scores are used as fitness functions.For the ease of naming convention and understanding, we considered each CVSS pattern as a separate configuration in thesealgorithms.
A. Genetic Algorithm (GA)
Genetic Algorithms are based on the biological process of evolution. The idea is that over time, a pool of chromosomeswill evolve to be even better (i.e., better fitness value) than the previous generation. A new generation (equal to the poolsize) of chromosomes (i.e., configurations) is created with any iteration of the algorithm. This is achieved by the processes ofselection, crossover, and mutation [9]. A fitness score metric is adopted as a measure to select the two fittest chromosomesfrom the pool that are called parent chromosomes. Then crossover takes place between the parents to produce a new childchromosome, which will have the best traits from both the parents followed by mutating of some of the characteristics of thechild to introduce new traits. This process is repeated until an entirely new generation gets created.
B. GA implementation for secure configuration pool
We implemented the algorithm in Python. We first created a CVSS vector pool with the fitness score of . (i.e., the best andmore secure fitness score). We refer to vector as a “string” in our implementation. We set the number of iteration/generationas 50. The entire algorithm (Algorithm 1) is divided into five parts: 1) configuration generation , 2) fitness score, 3) breeder’sSelection, 4) crossover, and 5) mutation. These parts are explained below in-depth.
1) Initial Configuration Generation:
As shown on lines 1 – 9 of Algorithm 1, we created a pool of 100 possible CVSSvector strings by randomly choosing corresponding permissible values from the ‘ val ’ list to produce the initial pool of vectorstrings.
2) Fitness Score:
As shown on lines 11 – 20 of Algorithm 1, the fitness score of the initial population of the vector stringsis evaluated. We imported the cvss [2] python library and thus utilized the base metric score method
CVSS3 . The CVSSscores were considered as the fitness scores. The scores were valid if they were in the range of [2 . , . . Anything outsideof that range was assigned the score as . We chose score . to be the upper limit since it is roughly the average score aconfiguration can take to be deemed reasonably secure. Algorithm 1
Genetic Alg. for generation of configurations. (cid:46) Generating initial pool of configuration vectors. procedure C ONFIGURATION3: val = [ (cid:48) H (cid:48) , (cid:48) L (cid:48) , (cid:48) N (cid:48) , (cid:48) A (cid:48) , (cid:48) P (cid:48) , (cid:48) U (cid:48) , (cid:48) C (cid:48) , (cid:48) N (cid:48) , (cid:48) R (cid:48) ] vector f ield = [ (cid:48) AV (cid:48) , (cid:48) AC (cid:48) , (cid:48) P R (cid:48) , (cid:48) U I (cid:48) , (cid:48) S (cid:48) , (cid:48) C (cid:48) , (cid:48) I (cid:48) , (cid:48) A (cid:48) ] for each vf in vector f ield do vf = random.choice [ val ] (cid:46) ’val’ takes permissible set of values based on vf chosen. end for return vector end procedure (cid:46) Assigning fitness score based on Best score. procedure F ITNESS ( BestScore, vector ) score = CV SS vector ) .score () if ( score < = BestScore & score < = 5 . then f it = score else f it = 100 end if Return f it end procedure (cid:46)
Breeder’s Selection: Select best vector samples. procedure S ELECTION ( population, best sample, lucky f ew ) nextGen = [ ] sortedP op = Sorted ( population ) (cid:46) descending order of fitness values = low cvss score to high for i in range(best sample) do nextGen.append ( sortedP op [ i ]) (cid:46) first ’i’ end for for i in range(lucky few) do nextGen.append ( random.choice ( sortedP op )) end for Return nextGen end procedure (cid:46)
Creating new vector from 2 parent vectors. procedure C REATE C HILD ( vector , vector ) child vector = ”” for i in range(len(vector1)) do if random.random < . then child vector = child vector + vector i ] else child vector = child vector + vector i ] end if end for Return child vector end procedure (cid:46)
Mutating: randomly changing a value of the vector. procedure M UTATION ( vector ) vf = random.choice ( vector f ield ) modif y = random.choice ( val ) index = get position ( vf ) vector = vector [ :index ] + modif y + vector [ index+1: ] (cid:46) inserting ’modify’ in the vector string Return vector end procedure
3) Breeder’s Selection:
As shown on lines 21 – 32 of Algorithm 1, we then used Breeder’s selection method. This methodselects a combination of the best solutions generated by the algorithm (i.e., vectors with the low score). Furthermore, in orderto avoid the problem of falling into local minima, the algorithm also picks some lucky few vectors with random vector scores.
4) Crossover:
For crossover, As shown on lines 33 – 44 of Algorithm 1, we randomly swapped the values of metrics amongthe two parent vectors. We used a random value generator to select which parent vector to use for crossover. If value < . ,parent 1 is chosen, otherwise parent 2 would be the choice.
5) Mutation:
As shown on lines 45 – 53 of Algorithm 1, the algorithm performs mutation on the CVSS vector strings byrandom selection of vector field whose value is also randomly selected from its permissible set of values.We ran the GA script 100 times. Each run of the algorithm produced different pool of CVSS vector strings with differentnumber and combinations of vector of fitness score . . C. Particle Swarm Optimization (PSO) Algorithm
PSO is a widely used swarm-based optimization technique. It draws its inspiration from bee swarm, and bird flocking socialbehavior of particles. PSO and GA, both being different forms of evolutionary computation techniques, share some similarities.Both techniques start off with a random set of initial population/solutions and keep updating generations until it reaches anoptimum solution space with respect to the fitness function. In case of PSO, it is a swarm consisting of various particles,where each particle represents a solution. Unlike GA, PSO does not make use of crossover and mutation operators to updatethe particles. Instead, these techniques are directed towards the global optimum by their personal best position along with theswarm’s best position in the search space. PSO is also easier to implement than GA and has comparatively fewer parametersto adjust [7].
D. PSO implementation for secure pool configuration
We compared the performance of GA in generating a set of best configurations with that of PSO. We implemented the PSOalgorithm in Python 3.6. To make the comparison meaningful and fair, the number of iterations and population size (swarmsize) were kept similar to GA, which are 50 and 100, respectively.The PSO algorithm is described in Algorithm 2. The algorithm takes two list as parameters: 1) pbest_fitness and 2) particle_vel . These lists maintain the initial pbest fitness and velocity values associated to every particle in a swarm.We defined swarm as a collection (list) of 100 initial particles whose implementation (line 2) is similar to the procedure configuration in GA. The algorithm returns a pool of particles with varied scores in each iteration and also stores thecount of particles with score = 2.0 in every iteration in order to check how many most secure particles (configurations) aregenerated by PSO. We also focus only on the scores, which belong in the range [2.0, 5.0] in every iteration.In a nutshell, the algorithm searches for the best fitness and velocity values for each particle until a threshold is reached(lines 7 – 34). In every iteration (lines 9 – 13), the algorithm picks the pbest_fitness (i.e., particle best) values as theirCVSS scores cvss_fit with the assumption that the fitness would be better (i.e., lesser is better) than its current pbest fitness value. After the
For loop ends, it then picks the global best ( gbest ) value of the swarm by the the best pbest value (lines 14 – 16), in this case, the least value.The next step in the algorithm is to calculate the velocity (lines 17 - 31) where particle vel ( particle ) fetches the velocityof the given particle. The lines 18 - 28 describe how the velocity is evaluated for every particle. The velocity metric measuresthe distance between the fitness score (pbest) and the best score. The particle is updated whenever its current velocity valueis greater than its previous one.The particles are updated using update_particle in a similar manner to the configuration mutation in GA. Morespecifically, any one out of the eight vector fields (i.e., AV, AC, etc.) is constructed whose value is chosen randomly from itscorresponding set of permissible values. For example, if ‘AV’ is selected, then any value in the list of { H, L, N, A } can berandomly selected.The target global best value was set to . , particle velocity in the range [0 , where 0 and 8 are the minimum andmaximum number of differences between two particles, respectively. Since each particle (CVSS vector) constitutes of only 8vector fields (AV, AC, etc). The fitness range is set between the range [2 , where . is deemed as the best fitness scoreand 10.0 is the maximum CVSS score any particle can get which means highly unfit.V. E XPERIMENTATION AND R ESULTS
We ran our Python scripts, developed for implementing the GA and PSO algorithms, 100 times on the CVSS population inorder to evaluate the performance of the evolutionary algorithms in generating the most secure patterns. The performance wasmeasured on the basis of three evaluation metrics:1) Number of instances of vulnerability patterns with the target score (e.g., score = (2 . , . ) in each run.2) Mean hamming distance (diversity) of the CVSS vectors.3) Standard deviation of the scores calculated for the set of population produced. Algorithm 2
PSO for generation of configuration. procedure PSO(pbest fitness, particle vel) swarm = [particle() for i in range(swarm size)] (cid:46) Initialize 100 particles iteration = 0 Threshold = 50 total count = [ ] (cid:46) To store count of particles with score = 2.0 in every iteration best score = 2.0 while iteration < Threshold do count = 0 for each particle do (cid:46) Calc Fitness if cvss f it ( particle ) < pbest f itness ( particle ) then pbest fitness(particle) = cvss fit(particle) end if end for if pbest f itness ( particle ) < gbest ( swarm ) then gbest(swarm) = pbest(particle) end if for each particle do (cid:46) Calc Velocity if pbest f itness ( particle ) 1) Diversity of Vulnerability Patterns: It is important to produce a diverse set of instances of vulnerability vector patternsto ensure the thoroughness of test inputs (i.e., vulnerability pattern) generation and thus avoid generating redundant test inputswhere test input refers to an instance of vulnerability vector pattern produced by the algorithms. We collected the data for thethree evaluation metrics for various range of target scores including S ∈ S ∈ (2.0, 3.0], S ∈ (2.0, 4.0] and S ∈ (2.0,5.0]. As a representative example, Figure 2 shows the plots for all the aforementioned three metrics for CVSS vector stringsfalling into S ∈ (2.0, 3.0] for both GA and PSO.As expected, the number of instances of the generated vulnerability patterns for each run is smaller for the target score of . (i.e., most secure) and it is higher when the target fitness score is in range (2 . , . (i.e., least secure). It implies that whena lower level of vulnerability is targeted (i.e., more secure with CVSS score = 2 . ), there are “not” too many alternativesfor patterns. On the other hand, if the vulnerability levels and security is relaxed (i.e., CVSS score < = 5 . then over 60alternatives could be produced for pattern matching. A combination of such varying level of CVSS scores might be beneficialto increase the search space when implementing a moving target defense platform.The bar plots depicted in Figure 2.(a) and 2.(d) demonstrate the number of occurrences of CVSS patterns (i.e., y-axis)against the number of runs (i.e., x-axis) when the target target CVSS scores is (2 . , . for GA and PSO, respectively. Aglance at the charts indicates that GA is able to generate more instances of the CVSS patterns targeting the desired level ofsecurity (i.e., (2 . , . ).The scatter plots given in Figure 2.(b)-(e) and Figure 2.(c)-(f) denote distribution of the mean hamming distance and standarddeviation against the runs, respectively. To ease comprehending the trend of the mean values, a regression line is fitted into the (a) Histogram Plot (GA) (b) Mean Hamming Distance (GA) (c) Standard Deviation (GA)(d) Histogram Plot (PSO) (e) Mean Hamming Distance (PSO) (f) Standard Deviation (PSO) Fig. 2: Histograms, Mean Hamming Distances, Standard Deviations of CVSS vectors for S ∈ (2 . , . .scatter plots to capture the overall trend. The Hamming distance addresses the “ diversity ” of the vulnerability vector patternsgenerated by the algorithms based on the count of corresponding unequal values of each vector fields among strings. Thesmoothing lines for mean values of Hamming distance demonstrate similar trends for each target value for the fitness score.The mean values of the hamming distance (y-axis) in all the cases remain unchanged over the runs and are mostly scatteredbetween . and . for GA (i.e., a diversity of the vulnerability vector pattern generated) and between . and . for PSO.As demonstrated in scatter plots shown in Figure 2.(b) and 2.(c), the instances generated by the GA algorithm is less diversecompare to the instances generated by PSO. The mean of the hamming distances between the instances generated by GA andPSO are . and . , respectively. This indicates that even though the PSO algorithm generates far fewer instances of CVSSpatterns for the given fitness, it produces more diverse instances of patterns.To illustrate the variations of such vulnerability patterns generated, plots 2.(c) and 2.(f) illustrate the trend of the values ofthe standard deviations for Hamming distance over the runs. There is a light reduction in standard deviations while runningGA for all cases. The observed standard deviations for all cases is somewhere between . and . . When combined together,the mean and standard deviations of the hamming distance can serve as an indication of the diversity of the vulnerability vectorpatterns produced by the algorithms and thus helps in measuring the thoroughness of test case generation and thus vulnerabilityselections in which the generation of redundant patterns (i.e., test inputs) is avoided. 2) The Contributions of Each Permissible Value in each Vector Field: It is also important to investigate whether certainsettings of each vector field contributes to security configuration differently than its counterpart. Table III shows the frequency(i.e., in terms of percentage) of each value permissible for each vector field, as listed in the base metrics. As reported in TableIII:– AV : The most contributing value is P (i.e., Physical) for GA (ranging from . to . ). The PSO algorithm highlightstwo values of P and L as the most contributing to the security level of the patterns. This observation indicates that if theseverity of the vulnerability needs to be reduced, no other values or means of attacks (i.e., Network (N), Adjacent (A),and somewhat Local (L) is allowed for exploiting the vulnerability.– AC : There is a mixed situation for attack complexity and there is no clear winner between Low (L) and High (H)complexity level to launch the exploitation.– S : The dominant setting for this variable is C, except the case for GA when the target score is . .– UI : There is a mixed situation for the level of user involvement for exposing the vulnerability.– C : There is also a mixed situation for confidentiality settings among GA and PSO algorithms.– I : A similar mixed situation for this case. However, it is also observed that in most cases a None (N) risk to integrity isneeded to reduce the impact of exploiting the vulnerability.– A : Furthermore, there is a a mixed situation for availability where there is no clear dominant setting value.– PR : The two dominant setting values for the level privileges are L and H. Vector Values [2.0] (2.0, 3.0] (2.0, 4.0] (2.0, 5.0]Field GA PSO GA PSO GA PSO GA PSO P A 13.76 26.26 18.67 26.15 18.41 25.25 19.11 25.33 AV N 12.98 22.22 10.60 23.34 11.83 24.56 13.13 25.45L 32.98 44.44 48.63 H S C 39.74 N 33.24 43.43 48.51 R C H 0.0 N I H 0.0 26.26 0.0 31.06 0.18 34.90 1.07 N A H 0.0 26.26 0.0 34.76 0.05 N 40.25 26.26 29.75 28.05 30.81 27.95 30.58 27.17L 0.0 PR H TABLE III: % of contribution of each permissible value in all the score ranges across 100 runs of GA and PSO. (a) GA (b) PSO Fig. 3: th iteration along with highly unstable trend; whereas, the PSO algorithm demonstrateda more stable trend with four pattern generated with score 2.0.VI. R ELATED W ORK Crouse and Fulp [4] used genetic algorithms to deploy a Moving Target Defense (MTD) platform and make computersystems more secure through temporal/spatial diversity in configuration parameters that govern how a system operates. Lateron, they developed an MTD by simulating 256 virtual machines of similarly purposed computers where each computer wasinitially configured with an extremely vulnerable configuration making them prone to all sorts of attacks.Post and Sinz [8] bridged the gap between configuration information and verification process by introducing a new techniquenamed Configuration Lifting . The technique converts all the variants over which a software is verified into a meta-programthereby making the application of configuration-aware verification techniques like static analysis, and model checking moreefficient.Dai et al. [5] introduced the concept of configuration fuzzing in order to check the vulnerabilities that appear onlyat certain conditions by randomly modifying the configuration of the running application at specific execution points. Duringthe deployment phase, this technique ceaselessly fuzzes the configuration and looks for a vulnerability that rises due to theviolation of of security invariants. VII. C ONCLUSION We introduced the novel idea of “vulnerability coverage,” a methodology to examine software under test against certainclasses of vulnerabilities as reported by National Vulnerability Database (NVD) adequately. The introduced idea makes useof an open industry standard tool called Common Vulnerability Scoring System (CVSS) as a metric to measure fitness inorder to generate a pool of vulnerability vector patterns that attains a secure level of CVSS score. For adequacy testing of the underlying software, the software under test is then inspected against all those filtered representative sets of vulnerabilities withsimilar vulnerability vector pattern that were selected from the generated pool. The paper compared two evolutionary-basedalgorithms namely Genetic and Participle Swarm Optimization algorithms on the basis of their performance in generating apool of vulnerability patterns and the results indicated a similar performance achieved by both algorithms.The concept of adequacy criterion is a new approach and hence has a larger scope of improvement. An adequacy criterionbased on vulnerability coverage is a novel technique in the best of our knowledge. This approach can be further improvedby taking into consideration several other metrics including temporal and environmental ones present in CVSS and NationalVulnerability Database (NVD). We also built our experiments based on the range of . and . . Additional experimentationwould be needed to further study the effect of such range. Moreover, the concept needs tool support and further empiricalstudies which can aid in thorough and systematic searching for vulnerabilities reported in the NVD database based on thematching property for the goal of security testing and then investigate the effectiveness of such adequacy criterion.A CKNOWLEDGMENT This work is supported in part by funding from National Science Foundation under grants no: 1516636 and 1821560.R Symposium on Configuration Analyticand Automation (SAFECONFIG) , 2011.[5] H. Dai, C. Murphy, and G. Kaiser. Configuration fuzzing for software vulnerability detection. In , pages 525–530, 2010.[6] Shuvalaxmi Dass and Akbar Siami Namin. Vulnerability coverage for adequacy security testing. In SAC ’20: The 35th ACM/SIGAPP Symposium onApplied Computing , pages 540–543, 2020.[7] Mei-Ping Song and Guo-Chang Gu. Research on particle swarm optimization: a review. In International Conference on Machine Learning and Cybernetics ,2004.[8] H. Post and C. Sinz. Configuration lifting: Verification meets software configuration. In IEEE/ACM International Conference on Automated SoftwareEngineering , pages 347–350, 2008.[9] Kumara Sastry, David Goldberg, and Graham Kendall.