Using gradient boosting regression to improve ambient solar wind model predictions
R. L. Bailey, M. A. Reiss, C. N. Arge, C. Möstl, M. J. Owens, U. V. Amerstorfer, C. J. Henney, T. Amerstorfer, A. J. Weiss, J. Hinterreiter
DDraft version August 19, 2020
Typeset using L A TEX default style in AASTeX63
Improving ambient solar wind model predictions with machine learning
Rachel L. Bailey ,
1, 2
Martin A. Reiss ,
1, 3, 4
Charles N. Arge, Christian M¨ostl ,
1, 4
Mathew J. Owens , Carl J. Henney , Ute V. Amerstorfer, Tanja Amerstorfer , Andreas J. Weiss,
1, 7 andJ¨urgen Hinterreiter
1, 7 Space Research Institute, Austrian Academy of Sciences, 8042 Graz, Austria Conrad Observatory, Zentralanstalt f¨ur Meteorologie und Geodynamik, 1190 Vienna, Austria NASA Goddard Space Flight Center, Greenbelt, MD, USA Institute of Geodesy, Graz University of Technology, Graz, Austria Space and Atmospheric Electricity Group, Department of Meteorology, University of Reading, Reading, UK Air Force Research Laboratory, Space Vehicles Directorate, Kirtland AFB, NM, USA Institute of Physics, University of Graz, Universit¨atsplatz 5, 8010 Graz, Austria (Received August 19, 2020; Revised -; Accepted -)
Submitted to The Astrophysical JournalABSTRACTThe study of ambient solar wind, a continuous pressure-driven plasma flow emanating from our Sun,is an important component of space weather research. The ambient solar wind flows in interplanetaryspace determine how solar storms evolve through the heliosphere before reaching Earth, and especiallyduring solar minimum are themselves a driver of activity in the Earth’s magnetic field. Accuratelyforecasting the ambient solar wind flow is therefore imperative to space weather awareness. Here wepresent a machine learning approach in which solutions from magnetic models of the solar corona areused to output the solar wind conditions near the Earth. The results are compared to observations andexisting models in a comprehensive validation analysis, and the new model outperforms existing modelsin almost all measures. The final model discussed here represents an extremely fast, well-validated andopen-source approach to the forecasting of ambient solar wind at Earth.
Keywords: slow solar wind INTRODUCTIONIt was only in the 1970s, through the view of X-ray telescopes on the NASA Skylab satellite, that the dark patchesin the Sun’s polar regions were identified as coronal holes. Over the past decades, it has become clear that these areareas of open magnetic field lines, along which the solar plasma leaving the Sun in a continuous flow is accelerated tosupersonic speeds into the vast reaches of our solar system. This fast component of the ambient solar wind flow and themagnetic field embedded within it corotate with the Sun in an ever expanding spiral, the Parker spiral. Understandingof the ambient solar wind is important in space weather forecasting to determine the evolution and possible impact ofsolar storms, particularly as they influence the evolution of transient events such as coronal mass ejections (CMEs),the catalysts for the strongest geomagnetic storms, on their path from Sun to Earth (Temmer et al. 2011; Owens &Forsyth 2013). Ambient solar wind flows are themselves also an important driver of recurrent geomagnetic activity atEarth (Zhang et al. 2007; Verbanac et al. 2011; Nakagawa et al. 2019), determining critical properties in interplanetaryspace such as the solar wind speed, magnetic field strength and orientation (Luhmann et al. 2002). A clear picture andunderstanding and accurate modeling of the ambient solar wind are therefore essential in all aspects of space weatherresearch.
Corresponding author: R. L. [email protected] a r X i v : . [ phy s i c s . s p ace - ph ] A ug Bailey et al.
In today’s ambient solar wind modeling approaches, the solar surface, corona and inner heliosphere are treated as aconnected system to simulate the dynamics of the ambient solar wind flow from the Sun to the vicinity of Earth. Thesecoupled modeling approaches commonly span the range from 1 solar radius ( R (cid:12) ) to Earth at 1 AU, with the coronalmodel from 1 R (cid:12) to 5 R (cid:12) (or 30 R (cid:12) ) and the heliospheric model spanning the range from 5-30 R (cid:12) to 1 AU. Despitethe discovery of an empirical relationship between the configuration of open magnetic field lines on the Sun and thesolar wind properties measured at the Sun-Earth Lagrange Point 1 (L1) near the Earth (Wang & Sheeley 1990), it stillproves challenging to develop and optimize empirical techniques for specifying the solar wind conditions at the innerboundary of the heliospheric domain (Arge & Pizzo 2000; Riley et al. 2001; Arge et al. 2004). Well-known empiricalrelationships in this context are the Wang-Sheeley (WS) model (Wang & Sheeley 1990), Distance from the CoronalHole Boundary (DCHB) model (Riley et al. 2001) and the Wang-Sheeley-Arge (WSA) model (Arge et al. 2003). Moresophisticated three-dimensional magnetohydrodynamic (MHD) codes such as CORHEL (Linker et al. 2016), LFM-Helio (Merkin et al. 2016), SIP-CESE (Feng et al. 2015) and COIN-TVD MHD (Shen et al. 2018) are also used, withfurther examples being the Magnetohydrodynamics Algorithm outside a Sphere (Linker et al. 1999), Enlil (Odstrcil2003), the Space Weather Modeling Framework (T´oth et al. 2005), and the recently developed European HeliosphericForecasting Information Asset (Pomoell & Poedts 2018). Besides these MHD models, other modeling approaches basedon empirical relationships and statistics (e.g. Owens et al. 2020) have also been developed. In the world of machinelearning, early work of Wintoft & Lundstedt (1997) and Wintoft & Lundstedt (1999) developed neural networks toforecast the solar wind speed near Earth using values from coronal magnetic models, and similar ideas were developedfurther in Liu et al. (2011), Yang et al. (2018) and Chandorkar et al. (2020).From the available models, the WSA model approach enjoys widespread use in operational settings in the Met Officeand NOAA’s Space Weather Prediction Center, primarily in the prediction of high-speed solar wind streams (MacNeice2009a; Owens et al. 2005; Reiss et al. 2016; Reiss et al. 2019). It also has applications for many different scientificproblems such as the evolution of CMEs in interplanetary space and their subsequent arrival at Earth (Mays et al.2015; Riley et al. 2018; Scolini et al. 2019; Taktakishvili et al. 2009; Verbeke et al. 2019; Wold et al. 2018). Predictionof solar energetic particle events (Luhmann et al. 2017; MacNeice et al. 2011) and studies evaluating the interactionbetween the solar wind and planetary magnetospheres (Dewey et al. 2015) have also been carried out using the WSAmodel, highlighting its importance in current research.Despite all the recent developments, ambient solar wind forecasting models still only outperform 27-day solar windpersistence by a small margin, often with uncertainties in the timing of high-speed streams of roughly one day (Allenet al. 2020; Devos et al. 2014; Hinterreiter et al. 2019; Jian et al. 2011; Kohutova et al. 2016; Owens et al. 2008;Owens et al. 2013; Reiss et al. 2016). One large source of uncertainty is in the modeling of the topology of the coronalmagnetic field, which the WSA model takes as input, as we typically only have data for one side of the Sun. This istackled in one way using the Air Force Data Assimilative Photospheric Flux Transport or ADAPT model (Arge et al.2010; Henney et al. 2012; Hickmann et al. 2015), which provides multiple realizations of the possible magnetic fieldon the far side of the Sun. As can be seen by the many and varied approaches used, forecasting ambient solar windconditions remains a challenge.Continued efforts are needed to improve our capabilities for predicting conditions in the ambient solar wind. Herewe improve ambient solar wind forecasts by coupling machine learning techniques to modeling results from the outerboundary condition of coronal magnetic models. Machine learning techniques are undergoing a renaissance and aregrowing in popularity among both the public and the space weather community as a new tool for current challenges,with promising results. An overview of recent developments in machine learning methods applied to the topics of spaceweather and the heliosphere can be found for example in Camporeale (2019), with studies tackling problems such asforecasting geomagnetic indices and automated solar image classification. So far, however, beside the application ofstatistical techniques (Bussy-Virat & Ridley 2014; Riley et al. 2017), there have been few studies on ambient solarwind forecasting using machine learning. Here we will couple machine learning with established solar wind methodsin combination with the ADAPT realizations.The machine learning methods we apply in this study are based on Gradient Boosting Regressors (GBRs), which area powerful technique that builds a single estimator from a collection of weak learners called decision trees (Friedman2001). We use GBRs to predict the ambient solar wind bulk speed at Earth using the magnetic topology solutions atthe outer boundary of the coronal magnetic model. Figure 1 shows a depiction of the WSA coronal model solutionsfor one Carrington rotation and the variables extracted for model training. The top part represents maps of the fluxtube expansion factor f p and distance to the coronal hole boundary d computed at a reference height of 5 R · . The mbient solar wind from machine learning model
50 100 150 200 250 300 350
Carrington Longitude [deg] La t i t ude [ deg ] a Coronal model solution map of f p for CR f p
50 100 150 200 250 300 350
Carrington Longitude [deg] La t i t ude [ deg ] b Coronal model solution map of d for CR d [ deg ]
50 100 150 200 250 300 350
Carrington Longitude [deg] l o g ( f p ) c
50 100 150 200 250 300 350
Carrington Longitude [deg] d [ deg ] d
01 23 45 67 89 1011 mean
Figure 1.
The flux tube expansion factor f p (left) and distance to the coronal hole boundary d (right) are shown for oneCarrington rotation (CR R (cid:12) ) with the sub-Earth track drawn in with a dashed white line. (c-d) The values extracted along the sub-Earth track.Output from all 12 ADAPT realisations are shown in colour, while the black line depicts the mean. The log of f p is shownso that the behaviour is more easily interpretable in the plot, but the value did not need to be scaled for the model training.Note that, from a time perspective, time is running in the opposite direction with decreasing Carrington Longitude as the Sunrotates from left to right according to the map. lower part shows each variable extracted from the map along the sub-Earth track, which is the path along which aprojection of the Earth crosses during the Carrington rotation. All 12 ADAPT realizations can be included in themachine learning approach to account for uncertainties in the magnetic modeling.We present the application of machine learning techniques to complement and inform established solar wind modelingapproaches. The machine learning model is trained on 14 years of past data (1992 till 2006) and tested on one fullsolar cycle length of 11 years (2006 till 2017) from the recent Solar Cycle 24. We present a comprehensive validationanalysis of the machine learning model (GBR) solutions and compare the results to commonly applied ambient solarwind models and reference baseline models, including a 27-day persistence model, which uses the approximation thatthe conditions in the ambient solar wind flow repeat after each Carrington rotation. This is followed by a discussion ofthe importance and then outline possible future investigations. This is the first study in which all ADAPT realizationsare coupled with a machine learning approach. METHODS2.1.
Magnetic models of the Sun
In this section, we present the numerical framework for reconstructing the global magnetic field topology in the solarcorona. Our framework relies on magnetic maps of the photospheric magnetic field from the Global Oscillation NetworkGroup from the National Solar Observatory. Specifically, we make use of the ADAPT (Air Force Data AssimilativePhotospheric Flux Transport) (Arge et al. 2010; Henney et al. 2012; Hickmann et al. 2015) model. ADAPT is aflux transport model (e.g., includes differential rotation and meridional flows) to provide an ensemble of estimates ofthe global spatial distribution of the solar photospheric magnetic field. For this study, the ADAPT model producedensembles of twelve realizations representing the uncertainty driven by supergranulation (Worden & Harvey 2000).
Bailey et al.
Based on the twelve different ADAPT realisations, we reconstruct the global coronal magnetic field topology usingthe Potential Field Source Surface (PFSS) (Altschuler & Newkirk 1969; Schatten et al. 1969) and Schatten CurrentSheet (SCS) (Schatten 1971) model combination. The well-established PFSS model computes the potential magneticfield solution in the solar corona with an outer boundary condition that the magnetic field is radial at the source surfaceat 2 . R (cid:12) . Similarly, the SCS model in the modeling domain between 2 . . R s accounts for the latitudinalinvariance of the radial field as observed by the Ulysses mission (Wang & Sheeley 1995).We compute magnetic properties such as the areal expansion factor and the distance to the coronal hole boundaryfrom the modeled global magnetic field configuration. The areal expansion factor f p is the rate at which the flux tubeexpands between the photosphere and a reference height in the corona (Wang & Sheeley 1990): f p = (cid:18) R (cid:12) . R (cid:12) (cid:19) | B ( R (cid:12) , θ , φ ) || B (2 . R (cid:12) , θ , φ ) | . (1) θ is the longitude and φ the latitude, where the indices 0 and 1 represent the longitude/latitude at the solar surfaceand 2.5 solar radii, respectively.The distance to the coronal hole boundary d refers to the great circle distance that an open-field footpoint in thephotosphere lies from the nearest coronal hole boundary. The underlying idea of d is that the solar wind is slow nearcoronal hole boundaries and fast inside regions of open field topology (Riley et al. 2001). From the ensemble solutionsof ADAPT, we obtain a set of twelve different results for all the magnetic properties computed. Besides the expansionfactor and the distance to the nearest coronal hole boundary at the sub-Earth point, we also study the usefulness ofthe magnetic field strength at the photospheric footpoint, B phot , and the outer boundary of the coronal model, B cor .As a reference baseline model, we use the traditional WSA approach (Arge et al. 2003) for specifying the conditionsin the solar wind near the Sun. The WSA model used here is a combination of the expansion factor and the distancefrom the nearest coronal hole boundary (Arge et al. 2003). Specifically, the WSA relation used in this study is givenby v wsa ( f p , d ) = c + c (1 + f p ) c (cid:26) c − c exp (cid:20) − (cid:18) dc (cid:19) c (cid:21)(cid:27) c , (2)where c i are the model coefficients, for which we use the following settings: c = 250 km s − , c = 650 km s − , c = 0 . c = 1, c = 0 . c = 3 ◦ , c = 1 .
75 and c = 3 (Arge et al. 2003).To map the solar wind solutions near the Sun to the Earth, we employ the Heliospheric Upwind eXtrapolation model(HUX) (Riley & Lionello 2011), which simplifies the fluid momentum equation as much as possible. Furthermore, theHUX model solutions match the dynamical evolution explored by global heliospheric MHD codes reasonably wellwhile having low computing requirements (Owens et al. 2020). In this way, we can efficiently study the results andimplications of the ensemble ADAPT realisations. For more details on the HUX model, we would like to refer thereader to Riley & Lionello (2011) and Owens et al. (2017).2.2. Application of Machine Learning
We use a machine learning approach to predict the solar wind speed at L1 from the output of the coronal magneticmodel. The steps worked through in order to train and finalise the machine learning model can be summarised asfollows. To begin with, the data at a height of 5 R (cid:12) is extracted from the coronal magnetic model along the sub-Earthtrack to produce a time series. Next, this data set is split into training and testing data, with 14.5 years being used totrain the model, and one full solar cycle length (11 years) is left to test the model on. The model that will be trainedis a Gradient Boosting Regressor (GBR), and an initial iterative training through sets of machine learning-specificparameters is carried out on the data set to determine those best-suited for this specific problem. With the modelparameters determined, the model is first trained on the full data set before feature selection is carried out to reducethe number of input features. This produces a set of trained models, which we compare using validation metrics. Theoptimal model is saved for later use.The results extracted from the coronal magnetic model are the properties f p , d , B cor and B phot calculated alongthe sub-Earth track, i.e. the path the Earth traces through the rotation projected along the solar surface. Therewere 12 coronal magnetic model solutions per day based on 12 different ADAPT solutions. The sub-Earth track wasextracted from each of these, and all ADAPT solutions were included. The data was extracted in a way equivalentto an operational setting, in which the time series is updated once per day with the newest coronal magnetic model mbient solar wind from machine learning model features are the magnetic model properties described above, and the target is thesolar wind speed v sw at the location of the Sun-Earth L1 point. The model is trained to be able to produce the targetfrom any given set of features. The solar wind speed is taken from the OMNI hourly data set. In models developed toforecast the ambient solar wind speed from magnetic model results, there is usually a heliospheric model to account forexpansion times from the outer boundary of the coronal model to L1. In this approach, the use of a singular definedtime shift is avoided and instead the variables over the past 2-6 days are taken as input, meaning that at a resolutionof 3.64 hours, there are 29 time-shifted values of four properties taken from 12 possible ADAPT realisations, resultingin a total of 1,392 magnetic model features. In addition to the magnetic properties, a solar wind bulk speed persistencevariable, v pers , was included, under the assumption that the variation in the solar wind speed repeats itself every 27.27days. Three persistence features were included: the solar wind speed 26, 27 and 28 days before the target forecast,totaling 1,395 features. Contrary to other machine learning approaches, the features do not need to be scaled for aGBR, so no further feature engineering is carried out.In order to test the accuracy of the final model, the input data is split into training and testing data for the sakeof fair testing and model validation. The test data set constitutes an entire average solar cycle length, none of whichhas been seen by the model in the training data set. Keeping an entire set of varied solar cycle conditions unseen andusing this for validation ensures we evaluate the model’s ability to extrapolate to new behaviour. The training dataset covers 14.5 years, most of solar cycle 23, and runs from May 1992 till October 2006, while the test set covers the11 years from October 2006 till October 2017, which is the end of solar cycle 23 and a large part of 24.The machine learning method chosen for this study relies on Gradient Boosting Regressors (GBRs), and the specificimplementation applied here is the Python version of XGBoost (Chen & Guestrin 2016). GBRs are based on anensemble of decision trees, which on their own are weak learners, but in a forest can form a powerful prediction tool.The term gradient boosting refers to the gradient descent technique implemented for optimised fitting. For moredetails on the algorithm, see Friedman (2001). A summary of the general usage of GBRs can be found in Natekin &Knoll (2013). There are many benefits to GBRs, which include the ease of use and interpretation coupled with fastcomputation times.In the first step of model training, a grid search is carried out to evaluate the best combination of five GBR-specificinput parameters (number of estimators, learning rate, maximum depth of nodes and L1 regularisation terms), whichare optimised for this specific set of features. This is done by exploring every point in the 4-D parameter space withincertain parameter ranges and training a model on each. The combination with the best predictive ability accordingto the selected scoring measure is used for further training and feature selection. The training data set is shuffled andprovided to the model with an 80/20 split between train and validation data. Through stratified 5-fold cross-validation,the validation data set is kept aside and used during training to choose the best option among five models.2.3. Feature selection
In machine learning, feature selection describes the act of reducing or adjusting the input features to optimise thefinal model, as well as reduce time needed for training. Training the model on the full set of 1,395 features with around60,000 data points each is computationally heavy, and so the first step in feature selection is in finding a reduced setof variables that produces a model with an equal level of accuracy. A straightforward approach, in which the inputvariables over time are reduced so that only values at every -2, -3, -4, -5 and -6 days are taken, performs just as wellor better than the feature list with all possible time steps, and the number of features drops to 288. More detailedfeature selection is carried out through different methods, firstly using an evaluation of the feature importances, whichthe trained model provides directly in terms of percentages for each individual feature, and then through training ofthe model on individual variables and different combinations thereof. More details on feature importances in GBRscan be found in (Hastie et al. 2009).Figure 2 shows an evaluation of all feature importances with the features divided into groups. This shows theimportance of each of the four variables to the model, followed by the relative importance of each ADAPT modelrealization, and lastly the importance each time shift (grouped into days) has for the model training. This plot largely
Bailey et al. d f p B cor B phot Variable F ea t u r e I m po r t an c e i n % a ADAPT Model b Relative importances of different feature selections
Time shift (days) c Figure 2.
Importance of features when training the machine learning model. The feature importances of all features accordingto either (a) the variable used, (b) the ADAPT model realization the variable was taken from, or (c) the time shift applied tothe variable. follows expected patterns: d and f p are clearly the most important properties as expected from the WSA approach,and among the ADAPT ensemble members some are notably better at predicting than others. The other two variablesrelated to the coronal and photospheric magnetic fields clearly do not contain enough information to carry an ambientsolar wind model on their own merits and can be excluded. Among the time-shifted variables, the most importantare around a shift of -4 days, which is the average time it takes for solar wind leaving the corona to reach the Earth,although data at other time shifts also play into it. These feature importances were evaluated without the influenceof the v pers variable, which trained alongside the others makes up almost 20% of the model variance.The best model, deduced from the feature importances, is one that uses both f p and d with v pers , with input fromthe three best ADAPT models (realization Validation analysis
Here we assess the correspondence between the predicted and observed time series of solar wind bulk speed in threedifferent ways. Firstly, we present the validation analysis in terms of a continuous variable validation based on simplepoint-to-point comparison metrics. Secondly, we study the skill in terms of binary metrics, where we label each timestep in the predicted and observed timeline as an event/non-event based on a selected threshold value. Thirdly, wecomplement the discussed validation techniques with an event-based approach where periods of enhanced solar windspeed in observations and forecasts are automatically detected and compared against each other. This validationanalysis is based on previous research as discussed in Owens (2018) and Reiss et al. (2020).2.4.1.
Point-to-point error functions
The predictive abilities of ambient solar wind models are assessed most easily by comparing the observed solar windtime series to ambient solar wind model solutions. A straightforward way is to compare the underlying statisticaldistributions in terms of standard measures such as mean, median, and standard deviation. These basic statisticalmeasures already contain essential information on the tendency of the model to over- or underestimate the observedsolar wind conditions. We complement these basic measures with established error functions. We compute differenterror functions for continuous variables such as the mean error,ME = 1 n n (cid:88) k =1 ( f k − o k ) = ¯ f − ¯ o, (3) mbient solar wind from machine learning model f k , o k ) is the k -th element of n total forecast and observation pairs. Here the ME is simply the differencebetween the average prediction and the average observation. Furthermore, we compute the mean absolute error,MAE = 1 n n (cid:88) k =1 | f k − o k | . (4)The MAE is the arithmetic mean of the absolute differences between the prediction and the observation pairs. Itrepresents the typical magnitude for the prediction error. Similar to the MAE, the root-mean-square error,RMSE = (cid:118)(cid:117)(cid:117)(cid:116) n n (cid:88) k =1 ( f k − o k ) , (5)is the mean squared difference between prediction and observation value pairs. The RMSE is an estimate of themagnitude of the forecast error being more sensitive to outliers than the MAE. The last error measure is the skillscore: SS = 1 − MSE pred
MSE ref , (6)which compares the mean-squared errors (MSE) between the prediction and reference. These measures are equal tozero in the case that the forecast errors are equal to zero (that is f k = o k ) and increase with increasing disagreementbetween predictions and observations. Although strictly not an error function per definition, we complement the errorfunctions with the Pearson correlation coefficient.2.4.2. Binary metrics
Aside from approaches assessing the magnitude of the prediction error at every time step in comparison to theobserved value, we use another approach where each time step in the solar wind time series is labelled in binary valuesas an event or non-event. A threshold value is selected, and all values above the threshold are defined as events whilethose below are non-events. The advantage of this method is that slow and fast solar wind, of which fast solar wind isof more interest to end-users, are not given equal importance (Owens 2018). It is also generally insensitive to outliers,which are irrelevant for a forecast but can greatly affect point-to-point error measures.The definition of events and non-events in the solar wind time series uses a selected threshold value (Reiss et al. 2020)of 450 km/s. We cross-check events and non-events in the predicted and observed solar wind time series and countthe number of hits (true positives; TPs), false alarms (false positives; FPs), misses (false negatives; FNs) and correctrejections (true negatives; TNs), which are listed in a contingency table. A hit is a correctly predicted event, while amiss is an event that is observed but not predicted. In contrast, a false alarm is a predicted event that was not observedand a correct rejection is a correctly predicted non-event. The total counts are summarised in the so-called contingencytable, and from these we compute a set of skill measures such as the true positive rate
TPR = TP / (TP + FN), falsepositive rate FPR = FP / (FP + TN), threat score TS = TP / (TP + FP + FN), bias BS = (TP + FP) / (TP + FN), and true skill statistics TSS = TPR − FPR. The threat score is a measure of the overall performance of a model definedbetween 0 and 1 (best performance), while the bias depicts how well the model tends towards overpredicting (BS >
0) or underpredicting (BS <
0) events. A meaningful skill measure is the true skill statistics (TSS), which is definedin the range [ − , Event-based validation
The interpretation of simple point-to-point comparisons as described above can be misleading due to a lack ofknowledge on the extent of uncertainties related to timing errors (Owens et al. 2005; MacNeice 2009b,a). In particular,this is the case when large variations in the solar wind time series are generally well predicted, but the arrival times
Bailey et al. l o g f p a Ambient solar wind ML model input ( f p and d at time t days from 12 ADAPT models) and output during Solar Cycle 24
012 345 678 910 11mean
Time d [ r ad i an s ] b Day S o l a r w i nd s peed [ k m / s ] c GBR predictionObservationWSA/HUX
Figure 3.
Input (a-b) and output (c) from ambient solar wind model. Variation in flux tube expansion factor f p (a) anddistance to the coronal hole boundary d (b) over five Carrington Rotations (CR 2182-2187) at a time shift of t − differ between prediction and observations. The use of an event-based validation technique is commonly applied toaccount for the uncertainties in the arrival times. More specifically, validation analysis on the example of solar windbulk speed is carried out in three steps. First, events of enhanced solar wind speed, also called high-speed enhancements(HSEs) in forecast and observation data are defined as periods exceeding a certain threshold. Next, HSEs detectedin the solar wind measurements are paired with HSEs in the predictions and each event pair is labelled as a hit, falsealarm, or miss. The predictive abilities of the model are determined using the validation summary variables. A moredetailed description of the steps discussed above applied to solar wind speed measurements is given in Reiss et al.(2016) in Section 3.2. RESULTSA validation of results for one solar cycle from the machine learning model with common metrics such as point-to-point measures shows that the GBR outperforms WSA/HUX model combination and 27-day persistence in almostall measures. The results of the model are plotted alongside example input in Fig. 3 for five Carrington rotations inSolar Cycle 24, which the model was tested on. Observed solar wind evolution is plotted in black in the lower panel(with solar wind speed determined by the WSA/HUX model in orange for comparison), and we see good agreement.The model applied to the entire test data set is plotted in Fig. 4. From a different perspective, figure 5 shows theoutput of the model over the full temporal range of the test data as the development over all 146 Carrington rotations.The good visual agreement between the observations (left) and predictions (right) shows that the model approximatesthe ambient solar wind flows well. Shorter, transient solar wind flows that are seen primarily around solar maximum(middle of images) do not appear in the model predictions, but this is as expected as the model only predicts theevolving ambient solar wind.Tables 1, 2 and 3 summarize the results obtained from model validation in terms of point-to-point measures, skillmeasures and event-based metrics, respectively. Multiple machine learning models with different input variables areconsidered, and the input variables are listed in the brackets. Besides the flux tube expansion factor f p and distanceto the coronal hole boundary d , the solar wind speed from 27 days ago ( v pers ) was also included. We find that the mbient solar wind from machine learning model Predictions from MLObserved solar wind speed S o l a r w i nd s peed [ k m / s ] Jan Mar May Jul Sep Nov Jan
Month in year
Figure 4.
From top to bottom: consecutive years of solar wind observations at Earth (black line) plotted alongside predictionsfrom the machine learning model (teal). All y-axes are scaled to the range 200–800 km/s. Bailey et al.
Day in Carrington Rotation C a rr i ng t on R o t a t i on N u m be r a Obs. solar wind speed per Carrington Rotation
Day in Carrington Rotation C a rr i ng t on R o t a t i on N u m be r b Pred. solar wind speed per Carrington Rotation S o l a r w i nd s peed [ k m / s ] Figure 5.
The development of solar wind speed in observations and predictions over time. (a) and (b) depict the change insolar wind speed per Carrington rotation (27 days) for the entirety of the testing dataset, 11 years or 146 Carrington rotations.Time is increasing from top to bottom, and each line of the image is one Carrington rotation. (a) shows the solar wind speedin the observations at L1, while (b) shows the predicted solar wind speed from the machine learning model. As can be seen,recurrent patterns match well between observations and predictions, although short-term signals as seen mostly around solarmaximum (middle of images) are not reproduced by the model.
Table 1.
Classic point-to-point error measures for model comparison. Table of ambient solar wind prediction error metricsin terms of arithmetic mean (AM), standard deviation (SD), mean error (ME), mean absolute error (MAE), root mean squareerror (RMSE), the skill score (SS) relative to the climatological mean, and the Pearson correlation coefficient (PCC).Model AM SD ME MAE RMSE SS PCC[km s − ] [km s − ] [km s − ] [km s − ] [km s − ]WSA/HUX 386.1 80.5 34.9 72.3 98.9 0.02 0.49GBR( d ) 430.7 57.4 -9.7 70.6 89.4 0.20 0.47GBR( f p ) 429.1 49.6 -8.1 72.6 91.3 0.16 0.42GBR( f p , d ) 427.7 61.6 -6.7 66.8 85.0 0.28 0.54GBR( f p , d, v pers ). 422.3 71.6 -1.3 59.9 78.3 0.39 0.63Persistence (27-days) 420.8 99.8 0.1 71.9 98.2 0.03 0.52Observation 421.0 99.9 - - - - - best results in terms of established error functions are obtained with the GBR model based on f p , d and v pers with anRMSE of 78 . − and a PCC of 0 .
63. In comparison, the RMSE for the traditional WSA/HUX model combinationis 98 . − and the PCC is 0 .
49. The results for the model of 27-day persistence with an RMSE of 98 . − anda PCC of 0 .
52 indicate that it provides a strong benchmark. The persistence model greatly benefits from having thesame statistics as the observations as indicated by the similar AM and SD, and the repeating nature of the ambientsolar wind flow. We also find that all the machine learning approaches investigated in this study improve the resultsin comparison to the climatological mean, as indicated by an SS of 0 .
02, and an SS greater than 0 .
15 for all the GBRs.In terms of the contingency table in Table 2, we find that the application of the best machine learning approachimproves all the skill measures, and particularly that the GBRs based on different feature combinations improve thenumber of hits and decrease the number of misses. At the same time, the machine learning techniques increase thenumber of false alarms and decrease the number of correct rejections. The TSS is greater than the value of 0 .
28 for mbient solar wind from machine learning model Table 2.
Contingency table with skill measures of solar wind speed events. The event threshold is set at v >
450 km s − . Thetable shows the number of hits (true positives; TPs), false alarms (false positives; FPs), misses (false negatives, FNs), correctrejections (true negatives, TNs), followed by metrics derived from these values: the true positive rate (TPR) and false positiverate (FPR). The last three entries in each row show the threat score (TS), true skill statistics (TSS), and bias (BS).Model TP FP FN TN TPR FPR TS TSS BSWSA/HUX 3106 1711 5306 16410 0.37 0.09 0.31 0.28 0.57GBR( d ) 4440 3722 3972 14399 0.53 0.21 0.37 0.32 0.97GBR( f p ) 3800 3118 4612 15003 0.45 0.17 0.33 0.28 0.82GBR( f p , d ) 4377 2969 4035 15152 0.52 0.16 0.38 0.36 0.87GBR( f p , d, v pers ) 4714 2368 3698 15753 0.56 0.13 0.44 0.43 0.84Persistence (27-days) 4873 3464 3472 14546 0.58 0.19 0.41 0.39 1.00 False Positive Rate T r ue P o s i t i v e R a t e WSA/HUXGBR (d)GBR (fp)GBR (fp+d)GBR (fp+d+pers)Persistance (27-days)Clim. Mean
Figure 6.
Receiver operator characteristic curves showing the true positive rate (TPR) versus the false positive rate (FPR) forthe investigated machine learning techniques and the reference models. the WSA/HUX model for all the machine learning approaches. This positive increase in the TSS indicates that theoverall performance of the machine learning techniques shows a positive trend towards correctly predicted enhancedsolar wind predictions. While the traditional WSA/HUX model combination tends to under-predict the number ofperiods of enhanced solar wind speeds (BS=0 . f p , d and v pers outperforms the WSA/HUX model combination for the full spectrum of selected eventthresholds. However, we also find that a model of persistence outperforms all the other models for nearly all the eventthresholds.The last evaluation was an event-based analysis, shown in Table 3. We find that the machine learning model slightlyimproves the results of the WSA/HUX model combination. We also find that the number of correctly predictedevents increases for all the machine learning techniques and the number of misses decreases. However, the numberof false alarms increases for the machine learning techniques. While the TS for the WSA/HUX model is 0 .
33, theTS for the GBR based on f p , d and v pers is 0 .
40. This indicates that most machine learning approaches provide a2
Bailey et al.
Table 3.
Statistics on event-based error metrics. An evaluation on the detections of high-speed enhancements was carried outin terms of event-based metrics including the number of observed (P) and forecast (P F ) events, the bias (BS), the number ofhits (TPs), false alarms (FPs), and misses (FNs) together with the probability of detection (POD), false negative rate (FNR),positive predictive value (PPV), false alarm ratio (FAR), and threat score (TS).Model P P F BS TP FP FN POD FNR PPV FAR TSWSA/HUX 451 223 0.49 167 56 284 0.37 0.63 0.75 0.25 0.33GBR( d ) 451 305 0.68 201 104 250 0.45 0.55 0.66 0.34 0.36GBR( f p ) 451 279 0.62 179 100 272 0.40 0.60 0.64 0.36 0.33GBR( f p , d ) 451 307 0.68 211 96 240 0.47 0.53 0.69 0.31 0.39GBR( f p , d, v pers ) 451 292 0.65 213 79 235 0.48 0.52 0.73 0.27 0.40Persistence (27-days) 451 449 1.00 287 162 161 0.64 0.36 0.64 0.36 0.47 reasonable improvement in comparison to the traditional approach. However, the model of persistence again providesa challenging comparison in terms of an event-based validation analysis.A wide range of other considerations were tested but then excluded from the analysis. An example is the inclusionof features related to the solar activity in model training, with the F10.7 of MgII indices being prime examples of sucha feature. We would have expected the predictions to be improved when provided information on the activity level onthe Sun. Surprisingly, models trained with either or both were found to perform no better on new data, i.e. sufficientinformation with regards to solar activity had already been provided via the variables used. In the cases of F10.7, themodels were more easily overtrained. If the training were to be carried out on more than one solar cycle where thedifferences in the F10.7 index per solar cycle can be identified by the model, this may prove useful, but due to theovertraining effect it has not been included in this work. DISCUSSIONIn this study we have presented an approach for predicting the ambient solar wind conditions in the vicinity of Earth.Specifically, we propose replacing the static WSA relation at the inner boundary of the heliospheric model domainby a machine learning technique, directly associating what is happening close to the Sun to what is happening nearthe Earth. We find that the features that are commonly employed in empirical techniques for specifying solar windconditions near the Sun, namely the flux tube expansion factor f p and the distance to the coronal hole boundary d ,are the most decisive features. While it is possible to use all ADAPT realizations to predict the ambient solar windflow, we find that three specific models in combination provide the best results. This is the first time a model hascoupled numerical solutions with machine learning and incorporated all possible ADAPT solutions, thereby removinga large amount of uncertainty from one section of the modeling.The final model provides a forecast of the ambient solar wind with a lead time of 4.5 days in an ideal case, in whichonly data within 60 degrees west of the solar meridian is taken to eliminate possible inaccuracies resulting from lackof imaging and model projection at greater longitudes. This lead time could be extended to 9 days with a mission atthe Sun-Earth Lagrange point 5 ( − ◦ from the Earth) or 14 days if one entire half of the solar magnetic map from-180 ◦ to the meridian 0 ◦ is used.There are some aspects to take into account when evaluating this work. First, the current modeling approach doesnot include any effects from the complex dynamic evolution of the ambient solar wind flow, and does therefore notconsider any interactions between fast and slow ambient solar wind. Hence, it does not provide a picture of the solarwind conditions in the heliosphere and provides no self-consistent way to propagate the solutions from the Sun to theEarth. In order to account for the uncertainty in the time series, we make use of features based on time shifts at acadence of 3 .
64 hours ranging from 2 to 6 days. During the process of identifying the most important features, wefind that those between 3-4 days are most critical, as would be expected from the typical transit time of ambient solarwind flows.Second, we note that the results in the present study have been deduced from an operational perspective. Thismeans that magnetic features were computed once per day. Since most of the published literature uses the magneticfeatures computed only once per Carrington rotation, we also tested this setting for all the investigated machinelearning models. We find that the model quality in terms of the RMSE of the GBR using this new setting varies mbient solar wind from machine learning model . . The computationtime of the final machine learning product is very fast, requiring around 100 µs (on a Macbook Pro 13” A1708) toprovide a set of 4-day predictions from data extracted from solar magnetic maps. This speed makes ensemble runspossible, with 1 million runs requiring 100 s - in an optimised setting on a dedicated server with multiprocessing itwould only be a matter of seconds. One possible application could include uncertainties in the coronal field solutionsin the solar wind speed predictions by running ensembles of all magnetic model features within ± ◦ of the sub-Earthtrack. In the future, we shall work on several topics related to the improvement of the approach and on the extensionthereof to other physical properties such as magnetic polarity, and magnetic sector boundary crossings. Furthermore,we plan on computing the global solar wind solutions near the Sun based on similar methodology.With the presented results, we conclude that the machine learning technique, for which the code is provided open-source, enables a robust and fast approach to predict the solar wind speed near the Earth. The algorithm is easilytransferable to other solar wind frameworks and is therefore an important contribution to the space weather community,and can serve as a benchmark for future development of numerical ambient solar wind models. In a broader context,this study lays the foundation for future work on this subject, which will look into improving the modeling of solarwind conditions near the Sun (see Yang & Shen 2019, for example) and provide important input for MHD codes. DATA AVAILABILITY https://iswat-cospar.org/H1-01 Bailey et al.
The materials used in this work are as follows: • • WSA coronal magnetic model maps: anyone interested in acquisition of this data should contact C.N.A. Thedata set can not be provided openly due to the competitive nature of the model. Alternatively, coronal magneticmaps from other models providing the same output could be used. • OMNI data set for solar wind speed observations: https://spdf.gsfc.nasa.gov/pub/data/omni/low res omni/ • Jupyter Notebook used to train the model and produce the plots in this paper: https://github.com/bairaelyn/ambsowi-ml ACKNOWLEDGMENTSR.L.B., M.A.R., C.M., U.V.A., T.A., A.J.W. and J.H. thank the Austrian Science Fund (FWF): P31659-N27, J4160-N27, P31521-N27, and P31265-N27. M.A.R. would like to thank NASA’s CCMC for travel support. Part of this workwas carried out during a research stay at NASA Goddard in December 2019.REFERENCES