Can we distinguish modes of selective interactions using linkage disequilibrium?

Abstract

Selected mutations interfere and interact with evolutionary processes at nearby loci, distorting allele frequency trajectories and correlations between pairs of mutations. A number of recent studies have used patterns of linkage disequilibrium (LD) between selected variants to test for selective interference and epistatic interactions, with some disagreement over interpreting observations from data. Interpretation is hindered by the relative lack of analytic or even numerical expectations for patterns of variation between pairs of loci under the combined effects of selection, dominance, epistasis, and demography. Here, I develop a numerical approach to compute the expected two-locus sampling distribution under diploid selection with arbitrary epistasis and dominance, recombination, and variable population size. I use this to explore how epistasis and dominance affect expected signed LD, including for non-steady-state demography relevant to human populations. Finally, I use whole-genome sequencing data from humans to assess how well we can differentiate modes of selective interactions in practice. I find that positive LD between missense mutations within genes is driven by strong positive allele-frequency correlations between pairs of mutations that fall within the same conserved domain, pointing to compensatory mutations or antagonistic epistasis as the prevailing mode of interaction within but not outside of conserved genic elements. The heterogeneous landscape of both mutational fitness effects and selective interactions within protein-coding genes calls for more refined inferences of the joint distribution of fitness and interactive effects, and the methods presented here should prove useful in that pursuit.