Software Uncertainty in Integrated Environmental Modelling: the role of Semantics and Open Science
SSoftware Uncertainty in Integrated EnvironmentalModelling: the role of Semantics and Open Science
Daniele de Rigo
European Commission, Joint Research Centre, Institute for Environment and Sustainability,Via E. Fermi 2749, I-21027 Ispra (VA), Italy Politecnico di Milano, Dipartimento di Elettronica, Informazione e Bioingegneria,Via Ponzio 34/5, I-20133 Milano, ItalyCopyright © 2013 Daniele de Rigo.This work is licensed under a Creative Commons Attribution 3.0 Unported License( http://creativecommons.org/licenses/by/3.0/ ).See:
This is the author’s version of the work. The definitive version has been published in the Vol.15 of Geophysical Research Abstracts (ISSN 1607-7962) and presented at the European Geo-sciences Union (EGU) General Assembly 2013, Vienna, Austria, 07–12 April 2013
Cite as:de Rigo, D., 2013.
Software Uncertainty in Integrated Environmental Modelling: the role ofSemantics and Open Science . Geophys Res Abstr 15 , 13292+Author’s version DOI: 10.6084/m9.figshare.155701 , arXiv: 1311.4762 C OMPUTATIONAL aspects increasinglyshape environmental sciences [1]. Ac-tually, transdisciplinary modelling of com-plex and uncertain environmental systemsis challenging computational science (CS)and also the science-policy interface [2–7].Large spatial-scale problems fallingwithin this category – i.e. wide-scale trans-disciplinary modelling for environment(WSTMe) [8–10] – often deal with factors (a) for which deep-uncertainty [2, 7, 11, 12] mayprevent usual statistical analysis of mod-elled quantities and need different waysfor providing policy-making with science-based support. Here, practical recommen-dations are proposed for tempering a pe-culiar – not infrequently underestimated – source of uncertainty. Software errors incomplex WSTMe may subtly affect the out-comes with possible consequences even oncollective environmental decision-making.Semantic transparency in CS [2, 8, 10, 13, 14]and free software [15, 16] are discussed aspossible mitigations (b) . Software uncertainty,black-boxes and free software
Integrated natural resources modellingand management (INRMM) [17] fre-quently exploits chains of nontrivial data-transformation models (D-TM), each ofthem affected by uncertainties and errors.1 a r X i v : . [ c s . S Y ] F e b e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science. Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013).
Those D-TM chains may be packaged asmonolithic specialized models, maybe onlyaccessible as black-box executables (if ac-cessible at all) [18]. For end-users, black-boxes merely transform inputs in the finaloutputs, relying on classical peer-reviewed publications for describing the internalmechanism.While software tautologically plays a vitalrole in CS, it is often neglected in favour ofmore theoretical aspects. (a)
Complexity = Transdisciplinary integration (e.g. systems ofsystems)Environmental system(s) heterogeneity (e.g.geospatial fragmentation)Data heterogeneity (formats, definitions,spatiotemporal density, ...)Software complexity (algorithms, dependencies,languages, interfaces, ...)Uncertainty = Incomplete scientific knowledge (e.g. climatescenarios [19–21], tipping points [22–24], ... )Modelling assumptions and simplifications [25–27]Uncertainty of measured/derived dataSoftware uncertaintyDynamicbehaviour = Uncertainty propagation via:Propagation in the network of interconnectedWSTMe components [2, 14, 17, 28–33]Iterations within nonlinear optimizationsteps [5, 34–40]Data fusion, harmonization,integration [9, 41–44]Steps for computing and aggregating criteriaand indices [6, 7, 11, 45–48]This paradox has been provocatively de-scribed as “the invisibility of software inpublished science. Almost all published pa- pers required some coding, but almost nonemention software, let alone include or link tosource code” [49].2 e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science.
Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013).
Recently, this primacy of theory over reality[50–52] has been challenged by new emerg-ing hybrid approaches [53] and by the grow-ing debate on open science and scientificknowledge freedom [2, 54–57].In particular, the role of free software hasbeen underlined within the paradigm of re- producible research [18, 56–58]. In the spec-trum of reproducibility, the free availabil-ity of the source code is emphasized [56]as the first step from non-reproducible re-search (only based on classic peer-reviewedpublications) toward reproducibility. Y = f ∗ ( X ) = f ( θ ∗ , X ) Theoretic D-TM whose algorithm is typicallydescribed in peer reviewed publications. TheD-TM may e.g. implement a given WSTMe asinstance of a suitable family of functions f bymeans of selected parameters θ ∗ . θ ∗ may bethe result of an optimization (regression, con-trol problem, ...). Y = f ζ = f ( θ ζ , X , ζ ) Real D-TM where the software uncertainty ζ may affect both the function family f and theoptimality of the selected parameters θ ζ .:: (cid:175)(cid:175)(cid:175) f ( θ , X , ζ ) (cid:175)(cid:175)(cid:175) :: sem Semantically enhanced D-TM (e.g. SemAP).The D-TM is subject to the semantic checks sem as pre-, post-conditions and invariantson inputs, outputs and the D-TM itself: Y = :: (cid:175)(cid:175) f ( θ , X , ζ ) (cid:175)(cid:175) :: sem ⇔ (cid:189) Y = f ( θ , X , ζ ) (cid:3) sem ( Y , f , θ , X , ζ ) (b) where X is the input array of data X = { X , X , · · · X i · · · X n } X i ∈ C N i ×···× N ini is a multi-dimensional array (e.g. a two-dimensionalraster layer) Y is analogously the output array of datathe modal/deontic logic operator (cid:3) p means: it ought to be that p .Applying this paradigm to WSTMe, an alter-native strategy to black-boxes would suggestexposing not only final outputs but also key intermediate layers of data and informationalong with the corresponding free softwareD-TM modules.3 e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science. Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013). “Software errors in complexWSTMe may subtly affectthe outcomes with possibleconsequences even oncollective environmentaldecision-making”“The chain of free-softwaremodules should betransparent”
A concise, semantically-enhanced modularization[13,14] may help not only tosee the code (as a very ba-sic prerequisite for seman-tic transparency) but alsoto understand – and cor-rect – it [59]. Semantically-enhanced, concise modu-larization is e.g. supportedby semantic array program-ming (SemAP) [13, 14] andits extension to geospatial problems [8, 10].Some WSTMe may surely be classifiedin the subset of software systems which“are growing well past the ability of a smallgroup of people to completely understandthe content”, while “data from these sys-tems are often used for critical decision making” [50]. In this con-text, the further uncertaintyarising from the unpre-dicted “(not to say unpre-dictable)” [51] behaviour ofsoftware errors propagationin WSTMe should be explic-itly considered as softwareuncertainty [60, 61] (see b ).The data and informationflow of a black-box D-TM isoften a (hidden) composi-tion of D-TM modules:This chain of free-software D-TM mod-ules (each of them semantically-enhanced)should be transparent: Semantics and design diversity
Silent faults [62] are a critical class of soft-ware errors altering computation outputwithout evident symptoms – such as compu-tation premature interruption (exceptions,error messages, ...), obviously unrealistic re- sults or computation patterns (e.g. no-ticeably shorter/longer or endless computa-tions). As it has been underlined, “many sci-entific results are corrupted, perhaps fatallyso, by undiscovered mistakes in the soft-ware used to calculate and present those re-sults” [63].4 e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science.
Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013). “Semantic modularizationmight help to catch at leasta subset of silent faults,when misusingintermediate data outsidethe expected semanticcontext”“Where the complexity andscale may lead to deepuncertainty, techniquessuch as ensemble modellingmay be recommendable”
Despite the ubiquity of soft-ware errors [60–68], thestructural role of scien-tific software uncertaintyseems dramatically under-estimated [2, 51]. Seman-tic D-TM modularizationmight help to catch at leasta subset of silent faults,when misusing interme-diate data outside the ex-pected semantic context ofa given D-TM module (b) .Where the complexity andscale of WSTMe may leadunavoidable software-uncertainty to induceor worsen deep-uncertainty [2], techniquessuch as ensemble modelling may be rec-ommendable [7, 11, 12]. Adapting those techniques for glancing atthe software-uncertaintyof a given WSTMe wouldimply availability of multi-ple instances (implemen-tations) of the same ab-stract WSTMe. Indepen-dently re-implementing thesame WSTMe (design diver-sity [69]) might of course beextremely expensive. How-ever, partly independent re-implementations of criti-cal D-TM modules may bemore affordable and exam-ples of comparison between supposedlyequivalent D-TM algorithms seem to cor-roborate the interest of this research op-tion [49, 57, 70].
References [1] Casagrandi, R., Guariso, G., 2009.
Impact of ICT in environmental sciences: A citation analysis1990-2007.
Environmental Modelling & Software 24 (7), 865-871. DOI:10.1016/j.envsoft.2008.11.013Google Scholar: 10214045160670186637 (page 1).[2] de Rigo, D., (exp.) 2013.
Behind the horizon of reproducible integrated environmental modelling atEuropean scale: ethics and practice of scientific knowledge freedom.
F1000 Research . Submitted (pages 1,2, 3, and 5).[3] Gomes, C. P., 2009.
Computational sustainability: Computational methods for a sustainableenvironment, economy, and society.
The Bridge 39 (4), 5-13. (page 1).[4] Easterbrook, S. M., Johns, T. C., 2009.
Engineering the software for understanding climate change.
Computing in Science & Engineering 11 (6), 65-74. DOI:10.1109/MCSE.2009.193Google Scholar: 5281658010101603741 (page 1).[5] Hamarat, C., Kwakkel, J. H., Pruyt, E., 2012.
Adaptive robust design under deep uncertainty.
TechnologicalForecasting and Social Change. DOI:10.1016/j.techfore.2012.10.004 Google Scholar: 1225653171705658912(pages 1 and 2).[6] Bankes, S. C., 2002.
Tools and techniques for developing policies for complex and uncertain systems.
Proceedings of the National Academy of Sciences of the United States of America 99 (Suppl 3), 7263-7266.DOI:10.1073/pnas.092081399 Google Scholar: 2377027768488561757 (pages 1 and 2).[7] Kandlikar, M., Risbey, J., Dessai, S., 2005.
Representing and communicating deep uncertainty inclimate-change assessments.
Comptes Rendus Geoscience 337 (4), 443-455. DOI:10.1016/j.crte.2004.10.010Google Scholar: 8596177795590020821 (pages 1, 2, and 5).[8] de Rigo, D., Corti, P., Caudullo, G., McInerney, D., Di Leo, M., San-Miguel-Ayanz, J., 2013.
Toward OpenScience at the European scale: Geospatial Semantic Array Programming for Integrated EnvironmentalModelling.
Geophysical Research Abstracts 15 , 13245+ DOI:10.6084/m9.figshare.155703Google Scholar: 17118262245556811911 (pages 1 and 4). e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science. Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013).[9] Rodriguez Aseretto, D., Di Leo, M., de Rigo, D., Corti, P., McInerney, D., Camia, A., San Miguel-Ayanz, J.,2013.
Free and Open Source Software underpinning the European Forest Data Centre.
GeophysicalResearch Abstracts 15 , 12101+ DOI:10.6084/m9.figshare.155700 Google Scholar: 14482024956822192435(pages 1 and 2).[10] de Rigo, D., Corti, P., Caudullo, G., McInerney, D., Di Leo, M., San-Miguel-Ayanz, J., (exp.) 2013.
Supporting Environmental Modelling and Science-Policy Interface at European Scale with GeospatialSemantic Array Programming.
In prep. (pages 1 and 4).[11] Lempert, R. J., 2002.
A new decision sciences for complex systems.
Proceedings of the National Academyof Sciences of the United States of America 99 (Suppl 3), 7309-7313. DOI:10.1073/pnas.082081699Google Scholar: 3105059191542042306 (pages 1, 2, and 5).[12] Gober, P., Kirkwood, C. W., 2010.
Vulnerability assessment of climate-induced water shortage inPhoenix.
Proceedings of the National Academy of Sciences 107 (50), 21295-21299.DOI:10.1073/pnas.0911113107 Google Scholar: 13404393835259418552 (pages 1 and 5).[13] de Rigo, D., 2012.
Semantic Array Programming for Environmental Modelling: Application of theMastrave library.
In: Seppelt, R., Voinov, A. A., Lange, S., Bankamp, D. (Eds.), International EnvironmentalModelling and Software Society (iEMSs) 2012 International Congress on Environmental Modelling andSoftware. Managing Resources of a Limited Planet: Pathways and Visions under Uncertainty, Sixth BiennialMeeting. pp. 1167-1176.
Google Scholar: 6628751141895151391 (pages 1 and 4).[14] de Rigo, D., 2012.
Semantic Array Programming with Mastrave - Introduction to SemanticComputational Modelling. http://mastrave.org/doc/MTV-1.012-1
Google Scholar: 6848554969929557252 (pages 1, 2, and 4).[15] Free Software Foundation, 2012.
What is free software? (revision 1.118 archived at ) Google Scholar: 7470103647360812109 (page 1).[16] Stallman, R. M., 2009.
Viewpoint: Why "open source" misses the point of free software.
Communications of the ACM 52 (6), 31-33. DOI:10.1145/1516046.1516058 (free access version: )Google Scholar: 17751536887456926788 (page 1).[17] de Rigo, D., 2012.
Integrated Natural Resources Modelling and Management: minimal redefinition of aknown challenge for environmental modelling.
Excerpt from the
Call for a shared research agenda towardscientific knowledge freedom , Maieutike Research Initiative. (pages 1 and 2).[18] Morin, A., Urban, J., Adams, P. D., Foster, I., Sali, A., Baker, D., Sliz, P., 2012.
Shining light into black boxes.
Science 336 (6078), 159-160. DOI:10.1126/science.1218263 Google Scholar: 12575758499484368256 (pages 2and 3).[19] Lempert, R., Schlesinger, M. E., 2001.
Climate-change strategy needs to be robust.
Nature 412 (6845),375. DOI:10.1038/35086617 Google Scholar: 3059012525521592083 (page 2).[20] Shell, K. M., 2012.
Constraining cloud feedbacks.
Science 338 (6108), 755-756.DOI:10.1126/science.1231083 Google Scholar: 17586687489186735372 (page 2).[21] van der Sluijs, J. P., 2012.
Uncertainty and dissent in climate risk assessment: A Post-Normalperspective.
Nature and Culture 7 (2), 174-195. DOI:10.3167/nc.2012.070204Google Scholar: 2453345655060253463 (page 2).[22] Lenton, T. M., Held, H., Kriegler, E., Hall, J. W., Lucht, W., Rahmstorf, S., Schellnhuber, H. J., 2008.
Tippingelements in the earth’s climate system.
Proceedings of the National Academy of Sciences 105 (6), 1786-1793.DOI:10.1073/pnas.0705414105 Google Scholar: 9014054360036592667 (page 2).[23] Hastings, A., Wysham, D. B., 2010.
Regime shifts in ecological systems can occur with no warning.
Ecology Letters 13 (4), 464-472. DOI:10.1111/j.1461-0248.2010.01439.xGoogle Scholar: 10306576888461248930 (page 2). e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science. Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013).[24] Barnosky, A. D., Hadly, E. A., Bascompte, J., Berlow, E. L., Brown, J. H., Fortelius, M., Getz, W. M., Harte, J.,Hastings, A., Marquet, P. A., Martinez, N. D., Mooers, A., Roopnarine, P., Vermeij, G., Williams, J. W., Gillespie,R., Kitzes, J., Marshall, C., Matzke, N., Mindell, D. P., Revilla, E., Smith, A. B., 2012.
Approaching a state shiftin earth’s biosphere.
Nature 486 (7401), 52-58. DOI:10.1038/nature11018Google Scholar: 17836601480741683208 (page 2).[25] Milly, P. C. D., Betancourt, J., Falkenmark, M., Hirsch, R. M., Kundzewicz, Z. W., Lettenmaier, D. P., Stouffer,R. J., 2008.
Stationarity is dead: Whither water management?
Science 319 (5863), 573-574.DOI:10.1126/science.1151915 Google Scholar: 16810436092791821916 (page 2).[26] Sloan, S., Pelletier, J., 2012.
How accurately may we project tropical forest-cover change? A validation ofa forward-looking baseline for REDD.
Global Environmental Change 22 (2), 440-453.DOI:10.1016/j.gloenvcha.2012.02.001 Google Scholar: 10187012353585922701 (page 2).[27] Nabuurs, G. J., van Putten, B., Knippers, T. S., Mohren, G. M. J., 2008.
Comparison of uncertainties incarbon sequestration estimates for a tropical and a temperate forest.
Forest Ecology and Management 256 (3), 237-245. DOI:10.1016/j.foreco.2008.04.010 Google Scholar: 6421873803139267676 (page 2).[28] Green, D. G., Sadedin, S., 2005.
Interactions matter – complexity in landscapes and ecosystems.
Ecological Complexity 2 (2), 117-130. DOI:10.1016/j.ecocom.2004.11.006Google Scholar: 558815488529198010 (page 2).[29] Baker, R., Koch, F., Kriticos, D., Rafoss, T., Venette, R., van der Werf, W. (Eds.), 2012.
Advancing riskassessment models for invasive alien species in the food chain: contending with climate change,economics and uncertainty.
Bioforsk FOKUS 7 . Bioforsk, Frederik A. Dahls vei 20, 1432 Ås, Norway. (page 2).[30] de Rigo, D., Caudullo, G., San-Miguel-Ayanz, J., Stancanelli, G., 2012.
Mapping European forest treespecies distribution to support pest risk assessment.
In: Baker, R., Koch, F., Kriticos, D., Rafoss, T., Venette,R., van der Werf, W. (Eds.), Advancing risk assessment models for invasive alien species in the food chain:contending with climate change, economics and uncertainty.
Bioforsk FOKUS 7 . Bioforsk, Frederik A. Dahlsvei 20, 1432 ås, Norway.
Google Scholar: 6508055261897528514 (page 2).[31] Thompson, I., Mackey, B., McNulty, S., Mosseler, A., 2009.
Forest resilience, biodiversity, and climatechange: a synthesis of the biodiversity/resilience/stability relationship in forest ecosystems.
Vol. 43 ofTechnical Series. Secretariat of the Convention on Biological Diversity. ISBN: 9292251376Google Scholar: 16541384134391848156 (page 2).[32] Center for International Forestry Research., FAO Regional Office for Asia and the Pacific, 2005.
Forestsand floods: drowning in fiction or thriving on facts?
Center for International Forestry Research; Food andAgriculture Organization of the United Nations, Regional Office for Asia and the Pacific. (page 2).[33] Bonan, G. B., 2008.
Forests and climate change: Forcings, feedbacks, and the climate benefits of forests.
Science 320 (5882), 1444-1449. DOI:10.1126/science.1155121 Google Scholar: 12659549051072980947(page 2).[34] Ferreira, L., Constantino, M. F., Borges, J. G., Garcia-Gonzalo, J., 2012.
A stochastic dynamicprogramming approach to optimize Short-Rotation coppice systems management scheduling: Anapplication to eucalypt plantations under wildfire risk in portugal.
Forest Science , 353-365.DOI:10.5849/forsci.10-084 Google Scholar: 15823418833962927194 (page 2).[35] de Rigo, D., Rizzoli, A. E., Soncini-Sessa, R., Weber, E., Zenesi, P., 2001.
Neuro-dynamic programming forthe efficient management of reservoir networks.
In: Proceedings of MODSIM 2001, International Congresson Modelling and Simulation. Vol. 4. Modelling and Simulation Society of Australia and New Zealand, pp.1949-1954. DOI:10.5281/zenodo.7481 Google Scholar: 16120008708786398621 (page 2).[36] Bond, C. A., Champ, P., Meldrum, J., Schoettle, A., 2011.
Investigating the optimality of proactivemanagement of an invasive forest pest.
In: Keane, R. E., Tomback, D. F., Murray, M. P., Smith, C. M. (Eds.),The future of high-elevation, five-needle white pines in Western North America: Proceedings of the HighFive Symposium. U.S. Department of Agriculture, Forest Service, Rocky Mountain Research Station, pp.295-302.
Google Scholar: 15999737409186644552(page 2). e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science. Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013).[37] de Rigo, D., Castelletti, A., Rizzoli, A. E., Soncini-Sessa, R., Weber, E., 2005.
A selective improvementtechnique for fastening neuro-dynamic programming in water resources network management.
In:Zítek, P. (Ed.), Proceedings of the 16th IFAC World Congress. Vol. 16. International Federation of AutomaticControl (IFAC), pp. 7-12. DOI:10.3182/20050703-6-CZ-1902.02172 Google Scholar: 3101827154043147997(page 2).[38] Phillis, Y. A., Kouikoglou, V. S., 2012.
System-of-Systems hierarchy of biodiversity conservationproblems.
Ecological Modelling 235-236 , 36-48. DOI:10.1016/j.ecolmodel.2012.03.032Google Scholar: 14693363263013674243 (page 2).[39] Cavallo, A., Nardo, A., 2008.
Optimal fuzzy management of reservoir based on genetic algorithm.
In:Lowen, R., Verschoren, A. (Eds.), Foundations of Generic Optimization. Vol. 24 of Mathematical Modelling:Theory and Applications. Springer Netherlands, pp. 139-159. DOI:10.1007/978-1-4020-6668-9_2Google Scholar: 596109054217603623 (page 2).[40] Castelletti, A., de Rigo, D., Tepsich, L., Soncini-Sessa, R., Weber, E., 2008.
On-Line design of waterreservoir policies based on inflow prediction.
In: Myung, C., Misra, P. (Eds.), Proceedings of the 17th IFACWorld Congress. Vol. 17. International Federation of Automatic Control (IFAC), pp. 14540-14545.DOI:10.3182/20080706-5-KR-1001.02463 Google Scholar: 17611910683549006538 (page 2).[41] Kempeneers, P., Sedano, F., Seebach, L. M., Strobl, P., San-Miguel-Ayanz, J., 2011.
Data fusion of differentspatial resolution remote sensing images applied to Forest-Type mapping.
IEEE Transactions onGeoscience and Remote Sensing 49 (12), 4977-4986. DOI:10.1109/TGRS.2011.2158548Google Scholar: 1622266323511963573 (page 2).[42] Sedano, F., Kempeneers, P., Strobl, P., McInerney, D., San-Miguel-Ayanz, J., 2012.
Increasing Spatial Detailof Burned Scar Maps Using IRS-AWiFS Data for Mediterranean Europe.
Remote Sensing 4 (3), 726-744.DOI:10.3390/rs4030726 Google Scholar: 16654372463982241297 (page 2).[43] de Rigo, D., Bosco, C., 2011.
Architecture of a Pan-European Framework for Integrated Soil WaterErosion Assessment.
IFIP Advances in Information and Communication Technology 359 , 310-318.DOI:10.1007/978-3-642-22285-6_34 Google Scholar: 3950024085016158193 (page 2).[44] Voinov, A., Shugart, H. H., 2013. ’Integronsters’, integral and integrated modeling.
EnvironmentalModelling & Software 39 , 149-158. DOI:10.1016/j.envsoft.2012.05.014Google Scholar: 17438791388360294950 (page 2).[45] Mendoza, G. A., Martins, H., 2006.
Multi-criteria decision analysis in natural resource management: Acritical review of methods and new modelling paradigms.
Forest Ecology and Management 230 (1-3), 1-22.DOI:10.1016/j.foreco.2006.03.023 Google Scholar: 17320958744182864860 (page 2).[46] O’Farrell, P. J., Anderson, P. M. L., 2010.
Sustainable multifunctional landscapes: a review toimplementation.
Current Opinion in Environmental Sustainability 2 (1-2), 59-65.DOI:10.1016/j.cosust.2010.02.005 Google Scholar: 11738195672718592653 (page 2).[47] Dale, V. H., Beyeler, S. C., 2001.
Challenges in the development and use of ecological indicators.
Ecological Indicators 1 (1), 3-10. DOI:10.1016/S1470-160X(01)00003-6Google Scholar: 6821471585672677713 (page 2).[48] Gilbert, N., 2010.
Balancing water supply and wildlife.
Nature . DOI:10.1038/news.2010.505Google Scholar: 4413680200627277742 (page 2).[49] Barnes, N., Jones, D., 2011.
Clear climate code: Rewriting legacy science software for clarity.
Software,IEEE 28 (6), 36-42. DOI:10.1109/MS.2011.113 Google Scholar: 12280553201622082598 (pages 2 and 5).[50] Sanders, R., Kelly, D., 2008.
Dealing with risk in scientific software development.
Software, IEEE 25 (4),21-28. DOI:10.1109/MS.2008.84 Google Scholar: 5979969088190020135 (pages 3 and 4).[51] Cerf, V. G., 2012.
Where is the science in computer science?
Commun. ACM 55 (10), 5.DOI:10.1145/2347736.2347737 Google Scholar: 15391984385653800704 (pages 3, 4, and 5).[52] Pincas, U., 2011.
Program verification and functioning of operative computing revisited: How aboutmathematics engineering?
Minds and Machines 21 (2), 337-359. DOI:10.1007/s11023-011-9237-zGoogle Scholar: 8660462933112825884 (page 3).[53] Sanders, P., 2009.
Algorithm engineering – an attempt at a definition.
In: Albers, S., Alt, H., Näher, S.(Eds.), Efficient Algorithms. Vol. 5760 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp.321-340. DOI:10.1007/978-3-642-03456-5_22 Google Scholar: 16310481551473795186 (page 3). e Rigo, D., 2013. Software Uncertainty in Integrated Environmental Modelling: the role of Semantics andOpen Science. Geophys Res Abstr 15 , 13292+. ISSN 1607-7962. (EGU General Assembly 2013).[54] Kleiner, K., 2011.
Data on demand.
Nature Climate Change 1 (1), 10-12. DOI:10.1038/nclimate1057Google Scholar: 13711304819619162709 (page 3).[55] Nature, 2011.
Devil in the details.
Nature 470 (7334), 305-306. DOI:10.1038/470305b (page 3).[56] Peng, R. D., 2011.
Reproducible research in computational science.
Science 334 (6060), 1226-1227.DOI:10.1126/science.1213847 Google Scholar: 905554772905069177 (page 3).[57] Cai, Y., Judd, K. L., Lontzek, T. S., 2012.
Open science is necessary.
Nature Climate Change 2 (5), 299.DOI:10.1038/nclimate1509 Google Scholar: 10156741028963436768 (pages 3 and 5).[58] Ghisla, A., Rocchini, D., Neteler, M., Förster, M., Kleinschmit, B., 2012.
Species distribution modellingand open source GIS: why are they still so loosely connected?
In: Seppelt, R., Voinov, A. A., Lange, S.,Bankamp, D. (Eds.), International Environmental Modelling and Software Society (iEMSs) 2012International Congress on Environmental Modelling and Software. Managing Resources of a Limited Planet:Pathways and Visions under Uncertainty, Sixth Biennial Meeting. pp. 1481-1488.
Google Scholar: 6265264564814017260 (page 3).[59] Iverson, K. E., 1980.
Notation as a tool of thought.
Communications of the ACM 23 (8), 444-465. http://awards.acm.org/images/awards/140/articles/9147499.pdf
Google Scholar: 15203139354397204728 (page 4).[60] Lehman, M. M., 1989.
Uncertainty in computer application and its control through the engineering ofsoftware.
J. Softw. Maint: Res. Pract. 1 (1), 3-27. DOI:10.1002/smr.4360010103Google Scholar: 11699944162229777276 (pages 4 and 5).[61] Lehman, M. M., Ramil, J. F., 2002.
Software uncertainty.
In: Bustard, D., Liu, W., Sterritt, R. (Eds.),Soft-Ware 2002: Computing in an Imperfect World. Vol. 2311 of Lecture Notes in Computer Science.Springer Berlin / Heidelberg, Ch. 14, pp. 477-514. DOI:10.1007/3-540-46019-5_14Google Scholar: 15328955307076618013 (pages 4 and 5).[62] Hook, D., Kelly, D., 2009.
Testing for trustworthiness in scientific software.
In: Software Engineering forComputational Science and Engineering, 2009. SECSE ’09. ICSE Workshop on. IEEE, Washington, DC, USA,pp. 59-64. DOI:10.1109/SECSE.2009.5069163 Google Scholar: 9959381250960735693 (pages 4 and 5).[63] Hatton, L., 2007.
The chimera of software quality.
Computer 40 (8), 104-103. DOI:10.1109/MC.2007.292Google Scholar: 691364900894680251 (pages 4 and 5).[64] Hatton, L., 1997.
The t experiments: errors in scientific software.
Computational Science & Engineering,IEEE
Defects, scientific computation and the scientific method uncertainty quantificationin scientific computing.
IFIP Advances in Information and Communication Technology 377 , 123-138.DOI:10.1007/978-3-642-32677-6_8 Google Scholar: 1399353442026811576 (page 5).[66] Lehman, M. M., 1996.
Laws of software evolution revisited software process technology.
In:Montangero, C. (Ed.), Software Process Technology. Vol. 1149 of Lecture Notes in Computer Science.Springer Berlin/Heidelberg, Ch. 12, pp. 108-124. DOI:10.1007/BFb0017737Google Scholar: 3754503458653974527 (page 5).[67] Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., Alvin, K. F., 2002.
Error and uncertaintyin modeling and simulation.
Reliability Engineering & System Safety 75 (3), 333-357.DOI:10.1016/S0951-8320(01)00120-X Google Scholar: 10613684146262342334 (page 5).[68] Wilson, G., 2006.
Where’s the real bottleneck in scientific computing?
American Scientist 94 (1), 5+.DOI:10.1511/2006.1.5 Google Scholar: 12828974620192082182 (page 5).[69] Rebaudengo, M., Reorda, M., Violante, M., 2011.
Software-Level Soft-Error mitigation techniques.
In:Nicolaidis, M. (Ed.), Soft Errors in Modern Electronic Systems. Vol. 41 of Frontiers in Electronic Testing.Springer US, pp. 253-285. DOI:10.1007/978-1-4419-6993-4_9 Google Scholar: 15742657402676184752(page 5).[70] Beaudette, D., 2008.
Simple comparison of two Least-Cost path approaches.
In: Open Source SoftwareTools for Soil Scientists. http://casoilresource.lawr.ucdavis.edu/drupal/node/544 (archived at: ) Google Scholar: 14623451910171059303 (page 5).) Google Scholar: 14623451910171059303 (page 5).