CONTEXT: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. OBJECTIVE: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. METHOD: We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software. RESULTS: We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them. CONCLUSIONS: Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.
CONTEXT: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. OBJECTIVE: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. METHOD: We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software. RESULTS: We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them. CONCLUSIONS: Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.
Entities:
Keywords:
Scientific software; Software quality; Software testing; Systematic literature review
Authors: Joe Pitt-Francis; Miguel O Bernabeu; Jonathan Cooper; Alan Garny; Lee Momtahan; James Osborne; Pras Pathmanathan; Blanca Rodriguez; Jonathan P Whiteley; David J Gavaghan Journal: Philos Trans A Math Phys Eng Sci Date: 2008-09-13 Impact factor: 4.226
Authors: David W Kane; Moses M Hohman; Ethan G Cerami; Michael W McCormick; Karl F Kuhlmman; Jeff A Byrd Journal: BMC Bioinformatics Date: 2006-05-30 Impact factor: 3.169
Authors: Kavishwar B Wagholikar; Michael Mendis; Pralav Dessai; Javier Sanz; Sindy Law; Micheal Gilson; Stephan Sanders; Mahesh Vangala; Douglas S Bell; Shawn N Murphy Journal: Biomed Inform Insights Date: 2018-06-04
Authors: Anna-Lena Lamprecht; Magnus Palmblad; Jon Ison; Veit Schwämmle; Mohammad Sadnan Al Manir; Ilkay Altintas; Christopher J O Baker; Ammar Ben Hadj Amor; Salvador Capella-Gutierrez; Paulos Charonyktakis; Michael R Crusoe; Yolanda Gil; Carole Goble; Timothy J Griffin; Paul Groth; Hans Ienasescu; Pratik Jagtap; Matúš Kalaš; Vedran Kasalica; Alireza Khanteymoori; Tobias Kuhn; Hailiang Mei; Hervé Ménager; Steffen Möller; Robin A Richardson; Vincent Robert; Stian Soiland-Reyes; Robert Stevens; Szoke Szaniszlo; Suzan Verberne; Aswin Verhoeven; Katherine Wolstencroft Journal: F1000Res Date: 2021-09-07
Authors: Paul Brack; Peter Crowther; Stian Soiland-Reyes; Stuart Owen; Douglas Lowe; Alan R Williams; Quentin Groom; Mathias Dillen; Frederik Coppens; Björn Grüning; Ignacio Eguinoa; Philip Ewels; Carole Goble Journal: PLoS Comput Biol Date: 2022-03-24 Impact factor: 4.475