Shaoxia Chen

Prevention of overfitting in cryo-EM structure determination

In the field of single-particle analysis of electron cryo-microscopy (cryo-EM) data, a growing concern that some resolution claims might not be substantiated by the data has been one of the instigators of community-wide efforts to develop new validation tools1. A known issue with commonly used cryo-EM structure determination procedures is their liability to overfit the data. Most procedures counter overfitting by low-pass filtering, but the effective frequencies for these filters are often based on suboptimal Fourier Shell Correlation2 (FSC) procedures. In the suboptimal procedure, FSC curves are calculated between reconstructions from two halves of the data, while a single model is used to determine the relative orientations of all particles. It is well known that bias towards noise in the single model may inflate the resulting resolution estimates. To illustrate this, we applied the suboptimal procedure to a simulated cryo-EM data set of 20,212 GroEL particles. Whereas the reported resolution was 4.6 A, the true resolution of the map was only 7.8 A. Also the presence of expected density features in the map does not necessarily provide sufficient evidence for a resolution claim: we could make convincingly looking figures of apparent side-chain density that in reality corresponded to overfitted noise (Supplementary Figure 1). Consequently, overfitting may remain undetected and interpretation of cryo-EM maps may be subject to errors. n nThe dangers of overfitting have been recognized, and refinement procedures with resolution-dependent weighting schemes to reduce overfitting have been proposed3,4. However, two known solutions to prevent it are not in common use. By refining two models independently (one for each half of the data), so-called gold-standard1 FSC curves may be calculated that are free from spurious correlations. Alternatively, the data used for the orientation determination may be limited to a user-specified frequency, so that model bias beyond that frequency may be avoided. However, the argument that withholding part of the data from the refinement would substantially deteriorate the orientations and thereby the quality of the structure has prevented the wide-spread use of either of these solutions. In what follows, we prove this thesis to be false. n nAnalysis of simulated data with realistic signal-to-noise ratios (SNRs) indicates that the accuracy of the orientation determination is not affected by the exclusion of high-frequency terms, nor by the use of a model that is reconstructed from only half of the particles (Supplementary Figure 2). These simulations illustrate that only the low-medium frequency terms in the individual particles contain sufficiently high SNRs to contribute significantly to the orientation determination, which is in good agreement with experimental evidence that cryo-EM particles may be aligned accurately using only low-frequency data5. Because in most cryo-EM studies the low-medium frequencies of reconstructions from half of the particles are not expected to be significantly worse than those of reconstructions from all particles, we hypothesize that overfitting may be prevented without a notable loss of resolution using either frequency-limited refinement or refinement based on gold-standard FSCs. Since the former involves a decision by the user, i.e. choosing the frequency at which to limit the refinement, we favour gold-standard FSCs and implemented a procedure to independently refine two models as a script on top of the conventional projection matching protocol in the XMIPP package6 (Supplementary Figure 3 & Supplementary Software). n nWe tested our hypothesis using three cryo-EM data sets: 5,053 GroEL particles that are distributed by the National Center for Macromolecular Imaging; an in-house collected data set of 50,330 β-galactosidase particles (Supplementary Methods); and 5,403 hepatitis B capsid particles from a previously published study7. High-resolution crystal structures are available for all three data sets, and these were used to assess the “true” resolution obtained using refinements based on either gold-standard or conventional FSC procedures (Figure 1). For all three cases, the conventional procedure reported apparently better FSC curves than the gold-standard procedure, but in no case did the gold-standard procedure actually result in a lower resolution map compared to the crystal structure. On the contrary, for the β-galactosidase data the gold-standard procedure yielded a structure that correlated up to higher frequencies with the crystal structure than the conventional procedure, which suffered from severe overfitting and gave rise to strong artefacts in the map. We also note that, in the absence of overfitting, the frequency at which the gold-standard FSC drops below 0.143 is a good indicator of the true resolution of the map (Supplementary Table 1), which is as expected from theory8. Finally, in the limit of very small data sets, division of the data into two halves might affect resolution. However, calculations with subsets of the GroEL particles suggest that this only becomes an issue for data sets that are much smaller than those typically used in cryo-EM reconstructions (Supplementary Figure 4). n n n nFigure 1 n nThe prevention of overfitting n n n nThe principal conclusion is therefore that overfitting of noise using suboptimal FSCs causes worse orientations and leads to a worse structure. In contrast, the use of gold-standard FSCs provides a realistic estimate of the true signal, which ultimately leads to a better map. The procedures proposed here are straightforward to implement in existing programs, and their application will eradicate the hazards of overfitting from cryo-EM structure determination procedures.

High-resolution noise substitution to measure overfitting and validate resolution in 3D structure determination by single particle electron cryomicroscopy

Three-dimensional (3D) structure determination by single particle electron cryomicroscopy (cryoEM) involves the calculation of an initial 3D model, followed by extensive iterative improvement of the orientation determination of the individual particle images and the resulting 3D map. Because there is much more noise than signal at high resolution in the images, this creates the possibility of noise reinforcement in the 3D map, which can give a false impression of the resolution attained. The balance between signal and noise in the final map at its limiting resolution depends on the image processing procedure and is not easily predicted. There is a growing awareness in the cryoEM community of how to avoid such over-fitting and over-estimation of resolution. Equally, there has been a reluctance to use the two principal methods of avoidance because they give lower resolution estimates, which some people believe are too pessimistic. Here we describe a simple test that is compatible with any image processing protocol. The test allows measurement of the amount of signal and the amount of noise from overfitting that is present in the final 3D map. We have applied the method to two different sets of cryoEM images of the enzyme beta-galactosidase using several image processing packages. Our procedure involves substituting the Fourier components of the initial particle image stack beyond a chosen resolution by either the Fourier components from an adjacent area of background, or by simple randomisation of the phases of the particle structure factors. This substituted noise thus has the same spectral power distribution as the original data. Comparison of the Fourier Shell Correlation (FSC) plots from the 3D map obtained using the experimental data with that from the same data with high-resolution noise (HR-noise) substituted allows an unambiguous measurement of the amount of overfitting and an accompanying resolution assessment. A simple formula can be used to calculate an unbiased FSC from the two curves, even when a substantial amount of overfitting is present. The approach is software independent. The user is therefore completely free to use any established method or novel combination of methods, provided the HR-noise test is carried out in parallel. Applying this procedure to cryoEM images of beta-galactosidase shows how overfitting varies greatly depending on the procedure, but in the best case shows no overfitting and a resolution of ~6 Å. (382 words)

Detective quantum efficiency of electron area detectors in electron microscopy.

Recent progress in detector design has created the need for a careful side-by-side comparison of the modulation transfer function (MTF) and resolution-dependent detective quantum efficiency (DQE) of existing electron detectors with those of detectors based on new technology. We present MTF and DQE measurements for four types of detector: Kodak SO-163 film, TVIPS 224 charge coupled device (CCD) detector, the Medipix2 hybrid pixel detector, and an experimental direct electron monolithic active pixel sensor (MAPS) detector. Film and CCD performance was measured at 120 and 300 keV, while results are presented for the Medipix2 at 120 keV and for the MAPS detector at 300 keV. In the case of film, the effects of electron backscattering from both the holder and the plastic support have been investigated. We also show that part of the response of the emulsion in film comes from light generated in the plastic support. Computer simulations of film and the MAPS detector have been carried out and show good agreement with experiment. The agreement enables us to conclude that the DQE of a backthinned direct electron MAPS detector is likely to be equal to, or better than, that of film at 300 keV.

Tilt-pair analysis of images from a range of different specimens in single-particle electron cryomicroscopy.

The comparison of a pair of electron microscope images recorded at different specimen tilt angles provides a powerful approach for evaluating the quality of images, image-processing procedures, or three-dimensional structures. Here, we analyze tilt-pair images recorded from a range of specimens with different symmetries and molecular masses and show how the analysis can produce valuable information not easily obtained otherwise. We show that the accuracy of orientation determination of individual single particles depends on molecular mass, as expected theoretically since the information in each particle image increases with molecular mass. The angular uncertainty is less than 1° for particles of high molecular mass (∼ 50 MDa), several degrees for particles in the range 1–5 MDa, and tens of degrees for particles below 1 MDa. Orientational uncertainty may be the major contributor to the effective temperature factor (B-factor) describing contrast loss and therefore the maximum resolution of a structure determination. We also made two unexpected observations. Single particles that are known to be flexible showed a wider spread in orientation accuracy, and the orientations of the largest particles examined changed by several degrees during typical low-dose exposures. Smaller particles presumably also reorient during the exposure; hence, specimen movement is a second major factor that limits resolution. Tilt pairs thus enable assessment of orientation accuracy, map quality, specimen motion, and conformational heterogeneity. A convincing tilt-pair parameter plot, where 60% of the particles show a single cluster around the expected tilt axis and tilt angle, provides confidence in a structure determined using electron cryomicroscopy.

The BRC repeats of human BRCA2 differentially regulate RAD51 binding on single- versus double-stranded DNA to stimulate strand exchange

The breast and ovarian cancer suppressor BRCA2 controls the enzyme RAD51 during homologous DNA recombination (HDR) to preserve genome stability. BRCA2 binds to RAD51 through 8 conserved BRC repeat motifs dispersed in an 1127-residue region (BRCA2[BRC1–8]). Here, we show that BRCA2[BRC1–8] exerts opposing effects on the binding of RAD51 to single-stranded (ss) versus double-stranded (ds) DNA substrates, enhancing strand exchange. BRCA2[BRC1–8] alters the electrophoretic mobility of RAD51 bound to an ssDNA substrate, accompanied by an increase in ssDNA-bound protein assemblies, revealed by electron microscopy. Single-molecule fluorescence spectroscopy shows that BRCA2[BRC1–8] promotes RAD51 loading onto ssDNA. In contrast, BRCA2[BRC1–8] has a different effect on RAD51 assembly on dsDNA; it suppresses and slows this process. When homologous ssDNA and dsDNA are both present, BRCA2[BRC1–8] stimulates strand exchange, with delayed RAD51 loading onto dsDNA accompanying the appearance of joint molecules representing recombination products. Collectively, our findings suggest that BRCA2[BRC1–8] targets RAD51 to ssDNA while inhibiting dsDNA binding and that these contrasting activities together bolster one another to stimulate HDR. Our work provides fresh insight into the mechanism of HDR in humans, and its regulation by the BRCA2 tumor suppressor.

Structure of Hepatitis B Surface Antigen from Subviral Tubes Determined by Electron Cryomicroscopy

Hepatitis B virus consists of an icosahedral core containing the double-stranded DNA genome, enveloped by a membrane with embedded surface proteins. The crystal structure of the core protein has been solved but little information about the structure of the surface proteins has so far been available. There are three sizes of surface protein, small (S), medium (M) and large (L), which form disulfide-bonded homo- and heterodimers. The three proteins, expressed from different start sites in the coding sequence, share the common C-terminal S region; the M protein contains an additional preS2 sequence N-terminal to S, and the L protein a further preS1 sequence N-terminal to M. In infected individuals, the surface proteins are produced in huge excess over the amount needed for viral envelopment and are secreted as a heterogeneous mixture of isometric and tubular subviral particles. We have used electron cryomicroscopy to study tubular particles extracted from human serum. Helical Fourier-Bessel analysis was used to calculate a low-resolution map, although it showed that the tubes were quite disordered. From the symmetry derived from this analysis, we used single-particle methods to improve the resolution. We found that the tubes had a diameter of approximately 250 A, with spike-like features projecting from the membrane. In the plane of the membrane the proteins appear to be close packed. We propose a model for the packing arrangement of surface protein dimers in the tubes.

High-resolution structure of the presynaptic RAD51 filament on single-stranded DNA by electron cryo-microscopy

Homologous DNA recombination (HR) by the RAD51 recombinase enables error-free DNA break repair. To execute HR, RAD51 first forms a presynaptic filament on single-stranded (ss) DNA, which catalyses pairing with homologous double-stranded (ds) DNA. Here, we report a structure for the presynaptic human RAD51 filament at 3.5–5.0Å resolution using electron cryo-microscopy. RAD51 encases ssDNA in a helical filament of 103Å pitch, comprising 6.4 protomers per turn, with a rise of 16.1Å and a twist of 56.2°. Inter-protomer distance correlates with rotation of an α-helical region in the core catalytic domain that is juxtaposed to ssDNA, suggesting how the RAD51–DNA interaction modulates protomer spacing and filament pitch. We map Fanconi anaemia-like disease-associated RAD51 mutations, clarifying potential phenotypes. We predict binding sites on the presynaptic filament for two modules present in each BRC repeat of the BRCA2 tumour suppressor, a critical HR mediator. Structural modelling suggests that changes in filament pitch mask or expose one binding site with filament-inhibitory potential, rationalizing the paradoxical ability of the BRC repeats to either stabilize or inhibit filament formation at different steps during HR. Collectively, our findings provide fresh insight into the structural mechanism of HR and its dysregulation in human disease.

CTF Challenge: Result summary.

Image formation in bright field electron microscopy can be described with the help of the contrast transfer function (CTF). In this work the authors describe the CTF Estimation Challenge, called by the Madrid Instruct Image Processing Center (I2PC) in collaboration with the National Center for Macromolecular Imaging (NCMI) at Houston. Correcting for the effects of the CTF requires accurate knowledge of the CTF parameters, but these have often been difficult to determine. In this challenge, researchers have had the opportunity to test their ability in estimating some of the key parameters of the electron microscope CTF on a large micrograph data set produced by well-known laboratories on a wide set of experimental conditions. This work presents the first analysis of the results of the CTF Estimation Challenge, including an assessment of the performance of the different software packages under different conditions, so as to identify those areas of research where further developments would be desirable in order to achieve high-resolution structural information.

Three-dimensional reconstruction of Heterocapsa circularisquama RNA virus by electron cryo-microscopy

Heterocapsa circularisquama RNA virus is a non-enveloped icosahedral ssRNA virus infectious to the harmful bloom-forming dinoflagellate, H. circularisquama, and which is assumed to be the major natural agent controlling the host population. The viral capsid is constructed from a single gene product. Electron cryo-microscopy revealed that the virus has a diameter of 34 nm and Tu200a=u200a3 symmetry. The 180 quasi-equivalent monomers have an unusual arrangement in that each monomer contributes to a bump on the surface of the protein. Though the capsid protein probably has the classic jelly roll β-sandwich fold, this is a new packing arrangement and is distantly related to the other positive-sense ssRNA virus capsid proteins. The handedness of the structure has been determined by a novel method involving high resolution scanning electron microscopy of the negatively stained viruses and secondary electron detection.

The Detective Quantum Efficiency of Electron Area Detectors

Recent progress in detector design has created the need for a careful side-by-side comparison of the modulation transfer function (MTF) and resolution-dependent detective quantum efficiency (DQE) of existing electron detectors, including film, with detectors based on new technology. I will present the results of measurements of the MTF and DQE of several detectors at 120 and 300keV. We have used the method of Meyer & Kirkland [1] modified to use analytically fitted curves, as shown in Figure 1. Figure 2 shows some of the MTF and DQE measurements for 300 keV electrons. Computer simulations have also been carried out, showing good agreement with the experimental results. We will conclude that the DQE to be expected from direct detection by back-thinned CMOS designs is likely to be equal to or better than film at 300 keV.