A Detailed Characterization of the Expert Problem-Solving Process in Science and Engineering; Guidance for Teaching and Assessment
Argenta Price, Candice Kim, Eric Burkholder, Amy Fritz, Carl Wieman
11 Title
A Universal Structure of Science and Engineering Problem-Solving
Authors
Argenta M. Price , Candice J. Kim , Eric W. Burkholder , Amy V. Fritz , and Carl E. Wieman Affiliations Stanford University, Department of Physics Stanford University, Graduate School of Education Stanford University, School of Medicine Stanford University, Department of Electrical Engineering * Corresponding Author
Author Contributions:
C. W. and A. P. conceptualized and designed the study, all authors acquired and analyzed the data, and A. P. and C. W. primarily wrote the manuscript. All authors approved the submitted manuscript.
This PDF file includes:
Main Text Figures 1 to 2 Table 1 Supplementary materials
Abstract
A primary goal of science and engineering (S & E) education is to produce good problem solvers, but much of the problem-solving process remains poorly understood. To characterize this process in terms of measurable and teachable pieces, we analyzed how S & E experts solved authentic problems. We identified an unexpectedly universal set of 29 decisions-to-be-made that were consistent across fields and provide a flexible structure underlying the S & E problem-solving process. We also find that the process of making those decisions (selecting between alternative actions) relies heavily on predictive models that embody the relevant specialized disciplinary knowledge and standards. This detailed characterization of the problem-solving process will allow better assessment and teaching of S & E problem-solving.
Significance Statement
We studied the problem-solving process used by skilled practitioners, “experts”, across the science, engineering, and medical disciplines. Through interviews with many such experts, we discovered that their problem-solving process can be characterized by a set of 29 specific decisions that they essentially all make. This result was quite surprising. The experts select and use particular frameworks of specific disciplinary knowledge to make those decisions. This distinguishes the domain general and domain specific elements of problem solving. This set of decisions defines the set of mental skills a student must practice to become a skilled problem solver. This work will enable better assessment and teaching of problem-solving skills across science, engineering, and medical disciplines.
Main Text Introduction
The importance of problem-solving as an educational outcome has long been recognized, but too often post-secondary S & E graduates show serious deficiencies when confronted with authentic problems (sometimes labelled as “critical thinking”). Much of the reason is that “problem-solving” is a complex process, but the details of that process are not well-understood and hence not taught. Better characterization of the detailed problem-solving process would allow it to be better measured and taught much more effectively. Various aspects of problem-solving have been studied across multiple domains and using a variety of methods (1–3). These range from expert self-reflections (4), to studies on knowledge-lean-tasks to discover general problem-solving heuristics (5), to comparing expert and novice performance on simplified problems across a variety of disciplines (6–10). These studies revealed important novice-expert differences – notably, that experts are better at identifying important features and have knowledge structures that allow them to reduce demands on working memory. Studies that specifically gave the experts unfamiliar problems in their disciplines found that they did more extensive planning and managing of their own behavior than novices, and could use their knowledge base to better define the problem (11–13). There have also been many theoretical proposals as to expert practices, but with little evidence as to their completeness or accuracy (e.g. (4, 14)). We built on this past work to achieve a complete and empirically-based characterization of the process by which scientists and engineers solve problems, focusing on identifying the decisions they had to make in the process, and comparing across fields of expertise. There has been extensive work in S & E education to teach students’ problem-solving skills. A variety of instructional methods have been used to generally teach science and engineering practices, but there has been limited research on their efficacy at improving problem-solving (see (3) for review). Research explicitly on problem-solving has primarily focused on textbook-type exercises and utilized step-by-step strategies. These studies have shown limited success, with potential drawbacks (e.g. (15–19) in physics). Of particular relevance to the teaching of problem-solving is the process, well established in the cognitive psychology literature (20), of gaining expertise through “deliberate practice” of the individual relevant skills. However, the specific cognitive skills to be deliberately practiced in the context of S & E problem-solving have not been well defined. We believe that deliberate practice to achieve mastery at making individual problem-solving decisions will be an efficient and effective way of training S & E problem-solving expertise. We argue that the decisions we identify here define both the set of cognitive skills a student should practice to become a skilled S & E problem solver, and the goals of an S & E educational program. In this study, we identified the decisions S & E experts had to make in the process of solving a problem from their work, and asked, “To what extent is this set of decisions-to-be-made consistent both within and across disciplines?” To develop this list, we interviewed many experts across different S & E fields. We first conducted 22 relatively unstructured interviews, where we asked about problem-solving expertise in their field, as it related to their own experience. The problems they described involved far more complexity and decisions than the problems previously used in expert-novice comparison research, or that college students see in their course work. From these interviews, we developed an initial list of decisions-to-be-made in S & E problem-solving. To refine and validate the list, we then carried out a set of 31 semi-structured interviews, in which experts chose a problem from their work and described the solution process in detail. These interviews were coded for the decisions represented, either explicitly stated or implied by a choice of action. Through iterative analysis of the interviews, we identified a framework of decisions that universally characterize the problem-solving process across S & E disciplines.
Results
We identified a total set of 29 decisions-to-be-made (plus 5 other themes), all of which were identified in a large fraction of the interviews across all disciplines (Table 1, Fig. 1). There was a surprising degree of overlap across the different fields with all the experts mentioning similar decisions-to-be-made, most of which were on our initial list. All 29 emerged by the fifth semi-structured interview, and on average, each interview contained 85% of the decisions. This consistency is particularly striking, as the percentages only represent what was described in the interviews and may miss some used in the actual problem-solving process but not described. Many decisions occurred multiple times in an interview, with the number of times varying widely, depending on the length and complexity of the solving process discussed. While the decisions-to-be-made were the same across disciplines, how the experts made those decisions (deciding on a set of alternatives and choosing among them) varied greatly by discipline. The process of making the decisions relied on specialized disciplinary knowledge and experience. However, while that knowledge was distinct and specialized, it was consistently organized according to a common structure we call a “predictive framework,” as discussed below. The set of decisions represent a remarkably consistent structure underlying S & E problem-solving. For the purposes of presentation, we have categorized the decisions as shown in Fig. 2, roughly based on the purposes they achieve. However, the process is far less orderly and sequential than implied by this diagram, or in fact any characterization of an orderly “scientific method.” There are flexible connections between decisions and repeated iterations – jumping back to the same type of decision multiple times in the solution process, often prompted by reflection as new information and insights are developed. The experts also often described considering multiple decisions simultaneously. In summary, while the specific decisions themselves are fully grounded in expert practice, the categories and order shown here are artificial simplifications for presentation purposes. The decisions contained in the seven categories are summarized below. See Table S1 for specific examples of each decision across multiple disciplines.
A. Selection and goals of the problem
This category involves deciding on the importance of the problem, what criteria a solution must meet, and how well it matches the capabilities, resources, and priorities of the expert. As an example, an earth scientist described the goal of her project (decision
B. Frame problem
These decisions lead to a more concrete formulation of the solution process and potential solutions. This involves identifying the key features of the problem and deciding on predictive frameworks to use, as discussed below, as well as narrowing down the problem, often forming specific questions or hypotheses. Many of these decisions are guided by past problem solutions with which the expert is familiar and sees as relevant. The framing decisions of a physician can be seen in his discussion of a patient with liver failure who had previously been diagnosed with HIV, but had features (
C. Plan process for solving
These decisions establish the specifics needed to solve the problem and include: how to simplify the problem and decompose it into pieces, what specific information is needed, how to obtain that information, and what are the resources needed and priorities? Planning by an ecologist can be seen in her extensive discussion of her process of simplifying (
D. Interpret information and choose solution(s)
This category includes deciding how to analyze, organize, and draw conclusions from available information, reacting to unexpected information, and deciding upon a solution. A biologist studying aging in worms described how she analyzed results from her experiments, which included representing her results in survival curves and conducting statistical analyses (
E. Reflect
Reflection decisions, which are ongoing, include deciding whether assumptions are justified, whether additional knowledge or information is needed, how well the solution approach is working, and if potential and then final solutions are adequate. Our decisions match the categories of reflection identified by Salehi (21). A mechanical engineer described developing a model (to inform surgical decisions) of which muscles allow the thumb to function in the most useful manner (
F. Implications and communication
These are decisions about the broader implications of the work, and how to communicate results most effectively. For example, a theoretical physicist developing a method to calculate the magnetic moment of the muon decided on who would be interested in his work (
G. Ongoing skill and knowledge development
Although we focused on decisions in the problem-solving process, the experts volunteered general skills and knowledge they saw as important elements of expertise in their field. These included teamwork and interpersonal skills (strongly emphasized), acquiring experience and intuition, and keeping abreast of new developments in their field.
Predictive framework
This work focused on the decisions-to-be-made, not how those decisions were made. The latter were highly dependent on the discipline and the problem. However, there was one element in how all these decisions were made that was fundamental and common across all interviews: the early adoption of a “predictive framework” that the experts used throughout their problem-solving process. We define this framework as “a mental model of key features of the problem and the relationships between the features.” All the predictive frameworks involved some degree of simplification and approximation and an underlying level of mechanism that established the relationships between key features. The frameworks provided a structure of knowledge and facilitated the application of that knowledge to the problem at hand, allowing experts to “run mental simulations” to make predictions for dependencies and observables and to interpret new information. As an example, an ecologist described her predictive framework for migration, which incorporated important features such as environmental conditions and genetic differences between species and the mechanisms by which these interacted to impact the migration patterns for a species. She used this framework to guide her meta-analysis of changes in migration patterns, affecting everything from her choice of datasets to include to her interpretation of why migration patterns changed for different species. In many interviews, the frameworks used evolved as additional information was obtained, with additional features being added or underlying assumptions modified. For some problems, the relevant framework was well established and used with confidence, while for other problems there was considerable uncertainty as to a suitable framework, so developing and testing the framework was a substantial part of the solution process. A predictive framework contains the expert knowledge organization that has been observed in previous studies of expertise (6) but goes farther, as here it serves as an explicit tool that guides most decisions and actions during the solving of complex problems. While the use of predictive frameworks was universal, the frameworks themselves explicitly reflected the relevant specialized knowledge, structure, and standards of the discipline, and arguably largely define a discipline (22).
Field-specific variation
While the set of decisions-to-be-made was largely consistent across fields, there were some discipline-specific differences. In particular, the solution methods are very different. For example, planning in some experimental sciences may involve formulating a multiyear construction and data collection effort, while in medicine it may be deciding on a simple blood test. Some decisions, notably in categories A, D, and F, were less likely to be mentioned in particular disciplines because of the nature of the problems. Specifically, decisions
Discussion
Many of the decisions we identified are supported by previous work on expertise and scientific problem solving. This is particularly true for those listed in “planning” and “interpreting information (6).” The priority experts give to framing and planning decisions over execution compared to novices (23) has been noted repeatedly. Expert reflection has been discussed, but less extensively (10), and there has been little previous discussion of the decisions in “selection” or “implications and communication” categories. Thus, our framework of decisions is consistent with previous work on “scientific practices” and expertise, but it is more complete, specific, empirically based, and generalizable across S & E disciplines. There are several limitations to this study. First, we have small sample from each of the disciplines. The consistency within those samples does argue that the results are fairly general, but does not rule out the possibility of there being experts who follow different individualistic paths in problem-solving. There also may be field-specific differences in prevalence of particular decisions that we are not able to identify in this study because of the small sample in each individual discipline. A second limitation is that our interview methods could have limited our detection of any decisions that were so intuitive to an expert that they didn’t think to mention the decision or associated action. Therefore the proportions of interviews in which particular decisions were identified is likely an underrepresentation of the true prevalence of that decision. It is conceivable that there were some decisions in this category for every interview and so we have missed from our list, but this is unlikely because of the high frequency of use of every decision that we did identify. We based our coding of the structured interviews on a pre-defined list of decisions (developed from the initial round of informal interviews), so our identification of decisions may also have been affected by confirmation bias. However, we did explicitly look for any decisions or actions taken that were not reflected in our list. The fact that we consistently identified nearly all of the decisions in every interview across diverse S & E fields gives us confidence that the list is essentially complete. A third limitation is that decisions often overlap and co-occur in particular interviews, so the division between decision items is often somewhat ambiguous and could be defined somewhat differently. As noted, a number of these decisions can be interconnected, and in particular fields some are nearly always interconnected. If one was focusing on a single field, the cognitive processes we list in the descriptions would be unchanged overall, but for some cases they would be combined into a slightly smaller set of numbered items. In summary, we have established a framework for characterizing and discussing S & E problem-solving, based on the set of decisions experts make when solving authentic problems. The small and strikingly consistent set of decisions represent a universal underlying structure in expert S & E problem-solving, with flexible linkages between decisions that are guided by reflection in a continually evolving process. We have also identified the nature of the “predictive frameworks” that S & E experts consistently use in problem-solving. This work has important implications for the assessment and teaching of S & E problem-solving. These decisions define the set of cognitive skills a student needs to deliberately practice and master to learn to perform as an expert in S & E. They can be used to improve undergraduate, graduate, and professional training, by focusing on the need to practice making the specific relevant decisions. It should be of use to graduate mentors and graduate program directors. Assessments based on our decision list and the associated discipline-specific predictive frameworks will allow a detailed determination of an individual’s discipline-specific problem-solving strengths and weaknesses. Such assessments can be used as formative assessments to provide needed feedback to students on their progress. They can also be used to evaluate the quality of potential employees and educational programs aimed at educating future members of the S &E workforce. Our preliminary work with such decision-based assessments is showing great promise, revealing large and specific variations in how expert-like post-secondary students are at authentic problem-solving.
Materials and Methods Experimental Design
This work involved interviewing many experts across different fields about their solving of authentic problems in their disciplines, and then identifying the problem-solving decisions-to-be-made that were mentioned in those interviews. We defined experts as practicing scientists, engineers, or physicians with considerable experience working as faculty at highly rated universities or working in moderately high level technical positions at successful companies. We also included a few long time post-docs and research staff in biosciences to capture more details of experimental decisions from which faculty members in those fields often were more removed. This broad definition of expert allows us to make general claims about the practice of skilled professionals; we are not attempting to study what makes only the most exceptional experts unique. Experts were recruited through direct contact via the research team’s personal and professional networks, and referrals from experts in our networks. The medical experts were chosen from a select group of medical school faculty chosen to serve as clinical reasoning mentors for medical students at a prestigious university. We defined an “authentic problem” to be one that these experts solve in their actual jobs. Such problems are characterized by complexity, with many factors involved and no obvious solution process, and involving substantial time, effort, resources, and decisions. Such problems involve far more complexity and many more decisions than the typical problems used in previous problem-solving research or used with students in instructional settings. We first carried out a round of relatively unstructured interviews with 22 experts (Table S2) where we asked them about expertise in problem-solving in their field as it related to their own experience. This typically resulted in their discussing examples of one or more problems they had solved. Based on the first several interviews, plus reflections from the research team and review of the literature on expert problem-solving and teaching of scientific practices (7, 24, 25), we created a generic list of decisions that were made in S & E problem-solving. In the rest of the unstructured interviews (~15), we also provided experts with our list and asked them to comment on any additions or deletions they would suggest. Faculty who had close supervision of graduate students and industry experts who had extensively supervised inexperienced staff were particularly informative. Their observations of the way inexperienced people could fail made them sensitive as to the different elements of expertise and where incorrect decisions could be made. To refine and validate the list of decisions generated from the informal interviews, we then conducted semi-structured interviews with 31 experts from across science, engineering, and medicine fields (Table S3). This work was approved by the Stanford IRB and informed consent was obtained from all the participants after the nature and possible consequences of the studies were explained.
Semi-structured interviews
In the semi-structured interviews, experts were asked to choose a problem or two from their work that they could recall the details of solving, and then describe the process, including all the steps and decisions they made. Our interview protocol (see supplement) was inspired in part by the critical decision method of cognitive task analysis (26, 27). Both protocols seek to elicit an accurate and complete reporting of the steps taken and decisions made in the process of solving a problem. We made several adaptations, however, to accommodate our different focus and to limit the interview time to 1hr. The critical decision method focuses on identifying critical decisions made during an unusual or important event, and the analysis focuses on identifying the factors involved in making those critical decisions. In contrast, our focus was to identify the complete set of decisions experts needed to make in solving a specific, but not necessarily unusual, problem in their work, without focusing on their reasons for making those decisions or identifying which specific decisions were most critical. The specific order of problem-solving steps was also less important to us, in part because it was clear that there was no consistent order that was followed. Our different focus allowed us to eliminate some of the more time-consuming steps from the critical decision method interview protocol, leaving us with a general strategy of 1) Having the expert explain their problem and step by step talk through the decisions involved in solving it, with relatively few interruptions from the interviewer except to keep the discussion focused on the specific problem and occasionally to ask for clarifications, followed by 2) Asking follow-up questions to probe for more detail about particular steps and aspects of the problem-solving process, and 3) occasionally asking for general thoughts on how a novice’s process might have differed. While some have questioned the reliability of information from retrospective interviews (e.g. (28)), we believe we avoid these concerns, because we are only identifying a decision-to-be-made, which in this case, means identifying a well-defined action that was chosen from alternatives. This is less subjective and much more likely to be accurately recalled than is the rationale behind such a decision. See Ericsson and Simon (29).
Interview coding
We coded the semi-structured interviews in terms of decisions made, through iterative rounds of coding (30), following a “directed content analysis approach,” which involves coding according to pre-defined theoretical categories and updating the codes as needed based on the data (31). Our theoretical categories were the list of decisions we had developed during the informal interviews. The goals of each iterative round of coding are described below. To code for decisions in general, we matched decisions from the list to statements in each interview, linking most to decisions because they were an: 1) explicit statement of a decision or choice made or needing to be made, 2) description of the outcome of a decision, such as listing important features of the problem or conclusions made, or 3) statement of actions taken that indicated a decision about the appropriate action had been made, usually from a set of alternatives. Coding the interviews in terms of decisions made often required substantial expertise in the disciplines in question, but our team included a broad range of expertise. A single statement could be coded as multiple decisions if they were occurring simultaneously in the story being recalled, or if they were intimately interconnected in the context of that interview. Our goal for the first iterative round of coding was to check for any decisions that were missing, as indicated by either an action taken or a stated decision that was not clearly connected to a decision on our initial list. In this round, we also clarified wording and combined decisions that we were consistently unable to differentiate during the coding. A sample of 3 interviews (from biology, medicine, and electrical engineering) were first coded independently by 4 coders, then discussed. The decision list was modified to add decisions and update wording based on that discussion. Then the interviews were re-coded with the new list and re-discussed, leading to more refinements to the list. Two additional 0 interviews (from physics and chemical engineering) were then coded by 3 coders and further similar refinements were made. Throughout the subsequent rounds of coding, we continued to check for missing decisions, but after the additions and adjustments made based on these 5 interviews, we did not identify any more missing decisions. In our next round of coding, we focused on condensing overlapping decisions and refining wording to improve the clarity of descriptions as they applied across different disciplinary contexts and to ensure consistent interpretation by different coders. Two or three coders independently coded an additional 11 interviews, iteratively refining wording, condensing the list of decisions, and then using the updated list to code subsequent interviews. We condensed the list by combining decisions that represented the same cognitive process taking place at different times, that were discipline-specific variations on the same decision, or that were occasional sub-steps involved in making a larger decision. We noticed that some decisions were frequently co-coded with others, particularly in some disciplines. But if they were identified as distinct a reasonable fraction of the time in any discipline, we listed them as separate. This provided us with a list, condensed from 42 to 29 discrete decisions (plus 5 additional non-decision themes), that gave good consistency between coders. Finally, we used the resulting codes to tabulate which decisions occurred in each interview. Individual coders identified decisions mentioned in the remaining 15 interviews. Interviews that had been coded with the early versions of the list were also re-coded to ensure consistency. Coders flagged any decisions they were unsure about occurring in a particular interview, and 2-4 coders met to discuss those unsure codes, with most uncertainties being resolved by explanations from a team member who had more technical expertise in the field of the interview. Minor wording changes were made during this process to ensure that each description of a decision captured all instantiations of the decision across disciplines, but no significant changes to the list were made. See “complete decisions list” in supplement for final list and descriptions of decisions. Table S4 shows the final tabulation of decisions identified in each interview. In the tabulation, most decisions were marked as either “yes” or “no” for each interview, though 65 out of 1054 total were marked as “implied,” for the following reasons: a) 40 because, based on the coder’s knowledge of the field, it was clear that a step must have been taken to achieve an outcome or action, even though that decision wasn’t explicitly mentioned (for example, they describe collecting X raw data and coming to Y conclusion, so they must have decided how to analyze the data, even if they didn’t mention the analysis explicitly); b) 15 were similar to those in a) but more tied to the interview context, in that there was not a single statement in which the expert had mentioned a decision but wording cues or the overall structure of the interview implied that decision must have happened; c) 10 involved a decision that was discussed as an important step in problem-solving but not stated in direct relation to the problem at hand, or was stated only in response to a direct prompt from the interviewer. The proportion of decisions identified in each interview, broken down by either explicit or explicit + implied, is presented in tables S3 and S4. Table 1 and Fig 1 of the main text show explicit + implied decision numbers. To check inter-rater reliability using the final decisions list, two of the interviews (one physics, one medicine) that had initially been coded by only a single coder were re-coded by a second coder using the final list. The tabulation of decisions identified in each interview was compared for the two coders. For both interviews there was disagreement on whether only one of the 29 (= 97% agreement) decisions occurred. Codes of “implied” were counted as agreement if the other coder selected either “yes” or 1 “implied.” As a side note, there was also one disagreement per interview on the coding of the 5 other themes, but those were not a focus of this work nor the interviews.
References
1. A. Newell, H. A. Simon,
Human problem solving (Prentice-Hall, 1972). 2. K. Dunbar, How Scientists Think in the Real World: Implications for Science Education.
Journal of Applied Developmental Psychology , 49–58 (2000). 3. National Research Council, “5 Problem Solving, Spatial Thinking, and the Use of Representations in Science and Engineering” in Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering , (The National Academies Press), pp. 75–118. 4. G. Polya,
How to Solve It: A New Aspect of Mathematical Method (Princeton University Press, 1945). 5. D. E. Egan, J. G. Greeno, “Theory of rule induction: Knowledge acquired in concept learning, serial pattern learning, and problem solving” in
Knowledge and Cognition , (Lawrence Erlbaum, 1974). 6. K. A. Ericsson, N. Charness, P. J. Feltovich, R. R. Hoffman, Eds.,
The Cambridge handbook of expertise and expert performance (Cambridge University Press, 2006). 7. K. A. Ericsson, R. R. Hoffman, A. Kozbelt, A. M. Williams, Eds.,
The Cambridge handbook of expertise and expert performance, 2nd ed (Cambridge University Press, 2018). 8. Particularly relevant in (6, 7) is the work by Ericsson, Chi, Feltovich, Glaser, Simon, and Larkin. 9. W. G. Chase, H. A. Simon, Perception in chess.
Cognitive Psychology , 55–81 (1973). 10. M. T. H. Chi, P. J. Feltovich, R. Glaser, Categorization and representation of physics problems by experts and novices. Cognitive Science , 121–152 (1981). 11. A. H. Schoenfeld, Mathematical Problem Solving (Elsevier, 1985). 12. S. Wineburg, Reading Abraham Lincoln: An Expert/Expert Study in the Interpretation of Historical Texts.
Cognitive Science , 319–346 (1998). 13. C. Singh, When physical intuition fails. American Journal of Physics , 1103–1109 (2002). 14. OECD, PISA 2018 Science Framework. 97–117 (2019). 15. J. I. Heller, F. Reif, Prescribing Effective Human Problem-Solving Processes: Problem Description in Physics. Cognition and Instruction , 177–216 (1984). 16. D. Huffman, Effect of explicit problem solving instruction on high school students’ problem-solving performance and conceptual understanding of physics. Journal of Research in Science Teaching , 551–570 (1997). 2 17. P. Heller, R. Keith, S. Anderson, Teaching problem solving through cooperative grouping. Part 1: Group versus individual problem solving. American Journal of Physics , 627–636 (1992). 18. A. F. Heckler, Some Consequences of Prompting Novice Physics Students to Construct Force Diagrams. International Journal of Science Education , 1829–1851 (2010). 19. E. Kuo, N. R. Hallinen, L. D. Conlin, When procedures discourage insight: epistemological consequences of prompting novice physics students to construct force diagrams. International Journal of Science Education , 814–839 (2017). 20. K. A. Ericsson, “The differential influence of experience, practice, and deliberate practice on the development of superior individual performance of experts” in The Cambridge Handbook of Expertise and Expert Performance, 2nd Ed , (Cambridge University Press, 2018), pp. 745–769. 21. S. Salehi, “Improving problem-solving through reflection /,” Stanford University, Stanford, CA. (2018). 22. C. E. Wieman, Expertise in University Teaching & the Implications for Teaching Effectiveness, Evaluation & Training.
Daedalus , 47–78 (2019). 23. M. T. H. Chi, R. Glaser, M. J. Farr, Eds.,
The nature of expertise (Lawrence Erlbaum Associates, Inc, 1988). 24. C. Wieman, Comparative Cognitive Task Analyses of Experimental Science and Instructional Laboratory Courses.
The Physics Teacher , 349–351 (2015). 25. National Research Council, A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas (National Academies Press, 2012) (September 21, 2016). 26. R. R. Hoffman, G. Lintern, “Eliciting and Representing the Knowledge of Experts” in
The Cambridge Handbook of Expertise and Expert Performance , (Cambridge University Press, 2006), pp. 203–222. 27. B. Crandall, G. A. Klein, R. R. Hoffman,
Working Minds: A Practitioner’s Guide to Cognitive Task Analysis (MIT Press, 2006). 28. R. E. Nisbett, T. D. Wilson, Telling more than we can know: Verbal reports on mental processes.
Psychological Review , 231–259 (1977). 29. K. A. Ericsson, H. A. Simon, Verbal reports as data. Psychological Review , 215–251 (1980). 30. M. T. H. Chi, Quantifying Qualitative Analyses of Verbal Data: A Practical Guide. Journal of the Learning Sciences , 271–315 (1997). 31. H.-F. Hsieh, S. E. Shannon, Three Approaches to Qualitative Content Analysis. Qual Health Res , 1277–1288 (2005). Acknowledgments
We acknowledge the many experts who agreed to be interviewed for this work, M. Flynn for contributions on expertise in mechanical engineering, and Shima Salehi for useful discussions. This work was funded by the Howard Hughes Medical Institute through an HHMI Professor grant to C. Wieman. Figures and Tables
Figure 1.
Proportion of decisions coded in interviews by field.
This tabulation includes decisions
Figure 2.
Representation of problem-solving decisions by categories.
The black arrows represent a hypothetical order of operations, the blue arrows represent more realistic iteration paths. Numbers indicate the number of decisions in the category. Knowledge and skill development were commonly mentioned themes but are not decisions. Table 1.
Problem-solving decisions a and percentages of expert interviews in which they occur. A. Selection and goals
Occur in 100% b c What is important in field? 61% 2. Opportunity fits solver’s expertise? 77% 3. Goals, criteria, constraints? 100%
B. Frame problem
4. Important features and info? 100% 5. What predictive framework? d C. Plan process for solving
10. Approximations and simplifications 81% 11. Decompose into sub-problems
68% 12. Most difficult or uncertain areas? 90% 13. What info needed? 100% 14. Priorities 87% 15. Specific plan for getting information 100%
D. Interpret information and choose solutions
16. Calculations and data analysis 81% 17. Represent and organize info 68% 18. How believable is information? 77% 19. Compare info to predictions 100% 20. Any significant anomalies? 71% 21. Appropriate conclusions? 97% 22. What is best solution? 97%
E. Reflect e
23. Assumptions + simplifications appropriate?
24. Additional knowledge needed?
25. How well is solving approach working?
26. How good is solution?
F. Implications and communication of results
27. Broader implications? 65% 28. Audience for communication? 55% 29. Best way to present work? 68%
G. Knowledge and skill dev. f Stay up to date in field 84% Intuition and experience 77% Interpersonal, teamwork 100% Efficiency 32% Attitude 68%
Footnotes: a) See supplementary text and Table S1 for full description and examples of each decision. b) Percentage of interviews in which category or decision was mentioned. c) Numbering is for reference. In practice ordering is fluid – involves extensive iteration with other possible starting points. d) Predictive framework will inform all other decisions. e) Reflection occurs throughout process, and often leads to iteration. Reflection on solution occurs also specifically at end. f) Not decisions – other themes mentioned frequently as important to professional success.
7 Supplementary Information for A Universal Structure of Science and Engineering Problem-Solving Authors: Argenta M. Price, Candice J. Kim, Eric W. Burkholder, Amy V. Fritz, and Carl E. Wieman Corresponding Author: Argenta Price.
This PDF file includes:
Supplementary text: interview protocol, full decisions list, notes on trends in specific decisions Tables S2 to S3 Legends for Tables S1 and S4 Legend for Dataset S1
Other supplementary materials for this manuscript include the following:
Tables S1 and S4 Dataset S1 Supplementary Information Text Supplementary Methods Complete semi-structured interview protocol
Notes: Inspired by the “Critical Decision Method” protocol of cognitive task analysis (1)
Semi-structured interview with many questions optional, depending on course of interview. Questions in bold were prioritized, so were usually asked. Aside from initial prompts and prompts to keep them on target or to provide enough detail, interviewers will say very little during the story telling part of the interview, then will ask elaboration and deepening questions to follow up.
Introduction We are interviewing you as part of a project to identify how experts think as they solve problems during their research/work. Our goal is to identify what students ought to be learning to do in order to improve education. So today we’d like to learn how you solve problems by having you recall a problem you have solved or project you’ve completed, and walk us through all the detailed steps. Particularly focus on the detailed decisions you made when solving the problem. Eliciting the scenario/problem: So, think about a problem you’ve solved in your work (or project you’ve completed). Choose a problem in which you can remember all the detailed steps and decisions. Then please walk me/us through how you went about solving that problem: What were the goals you were trying to achieve? What did you do step by step? What decisions did you make? Optional guidance questions to ask during story a.
If they’re having trouble thinking of a problem: i.
What is the most recent paper you’ve published/project you’ve worked on? How did you go about tackling that project? ii.
What was a particularly challenging research problem (or design problem or work problem) you’ve dealt with? b.
If they need more guidance starting to tell the story: i.
What was the first decision you needed to make? ii.
What did you do first? What did you do next? (phrase appropriately to respond to something in their narrative if they’ve stalled) iii.
You mentioned X goal, what did you do to accomplish that? iv.
What were the most important things for you to think about? c.
If they give too short or high-level of an account 9 i. (Especially at beginning to set the tone) probe for decisions made that were unstated: How did you decide X, that you just mentioned? ii.
What led to your decision X? iii.
What did you mean by X? Check-in, clarifications, and elaboration (may be asked during story, as needed): 3.
Ask for clarification about parts of the story that were unclear a.
In particular, if they used specific words like “model,” ask them to elaborate on their meaning of the term. i.
Ask for examples of where and how they used term/models (if not already stated) – elaborating on meaning may be best done through examples. 4.
Ask for elaboration on parts of the story that were glossed over (can interrupt story to ask, but give them time to get there themselves) a.
How did you decide…? b.
What led to your decision X? c.
Why did (or didn’t) you…? d.
What did you do next? More specific elaboration questions
Note: Can ask after story telling if they weren’t naturally covered, or don’t need to ask if they came up on their own. Can also bring up during story to help move story along or away from excess detail or attempts to teach. Prioritize bolded questions (combine “elaboration” and “deepening”). How did you decide to tackle the problem in the first place? You said you did X first; how did you decide to tackle that aspect first? a.
Or more general (preferred): How did you decide which way to go first? 7.
You said you chose X method/route; how/why did you pick that method over other possible methods? a.
Or more general (preferred): Why did you choose the path or methods you chose? 8.
What information or data did you need to collect? a.
Where did you get this information? b.
What did you do with this information? 9.
How did you interpret the results you collected (at X point)? 10.
Were there other solutions you considered? a.
How did you differentiate the possible solutions? “Deepening” questions about the whole process
What did you think were critical decisions you made during the process? a. If they don’t mention a particular point of story that interviewer thinks involved critical decisions, can ask for elaboration about decisions made during that particular point 0 b.
What decisions were you given vs. that you made (or what were the parameters you had at the start)?
What challenges did you encounter in solving the problem? a. How did you deal with those challenges?
What (if any) new knowledge or skills did you need to acquire for solving your problem? a. How did you acquire those? 14.
What tests or experiments did you run? a.
How did you decide on them? b.
How did you interpret the results? (or ask alternative about a specific point – question 8) 15. (optional) How does this connect with prior work you’ve done? 16.
How did you decide you had an adequate result (to publish paper/submit design, etc.) a. How did you know you were done? b.
When did you decide your design/solution/conclusion was satisfactory? 17.
What are the implications (or next steps) for your project? 18.
How were your findings/product received by the community? What-if scenarios: 19.
What alternative decisions could you have made (in general or at specific point X), and what might have happened differently?
If your student/trainee had been solving this problem instead of you (or without your guidance), how do you think their approaches would have differed from yours?
Other questions about perspective on expertise in their field (if time)
What are particular difficulties in problem-solving you’ve noticed in people you have trained or mentored? (be specific) 22.
How do you use models in your work?
What do you see as differences between experts and novices? 24.
What do you think was your particular expertise that made you successful (in solving this problem, and more generally in your career)? 25.
What do you want your trainees to be able to do (While trainees? After they’re done training?)?
Supplementary Text Complete list of decisions-to-be made by experts when solving authentic problems A.
Selection and goals What is important in field?
What are important questions or problems? Where is the field heading? Are there advances in the field that open new possibilities? 2)
Opportunity fits solver's expertise?
1 If and where are there gaps/opportunities to solve in field? Given experts’ unique perspectives and capabilities, are there opportunities particularly accessible to them? (could involve challenging the status quo, questioning assumptions in the field) 3)
Goals, criteria, constraints?
What are the goals for this problem? Possible considerations include: a.
What are the goals, design criteria, or requirements of the problem or its solution? b.
What is the scope of the problem? c.
What constraints are there on the solution? d.
What will be the criteria on which the solution is evaluated? B. Frame Problem Important features and info?
What are the important underlying features or concepts that apply? Could include: a.
Which available information is relevant to solving and why? b. (when appropriate) Create/find a suitable abstract representation of core ideas and information Examples: physics – equation representing process involved, chemistry – bond diagrams/potential energy surfaces, biology – diagram of pathway steps. 5)
What predictive framework?
Which potential predictive frameworks to use? (decide among possible predictive frameworks or create framework) This includes deciding on the appropriate level of mechanism and structure that the framework needs to embody to be most useful for the problem at hand. 6)
Narrow down problem.
How to narrow down the problem? Often involves formulating specific questions and hypotheses. 7)
Related problems?
What are related problems or work seen before, and what aspects of their solving process and solutions might be useful in the present context? (may involve reviewing literature and/or reflecting on experience) 8)
Potential solutions?
What are potential solutions? (Based on experience and fitting some criteria for solution they have for a problem having general key features identified.) 9)
Is problem solvable?
Is the problem plausibly solvable and is solution worth pursuing given the difficulties, constraints, risks, and uncertainties? C. Plan Process for Solving
Approximations and simplifications.
What approximations or simplifications are appropriate? How to simplify the problem to make it easier to solve? Test possible simplifications/approximations against established criteria. 11)
Decompose into sub-problems.
How to decompose the problem into more tractable sub-problems? (Independently solvable pieces with their own sub-goals.) 12)
Most difficult or uncertain areas?
2 Which are areas of particular difficulty and/or uncertainty in plan of solving process? Could include deciding: a.
What are acceptable levels of uncertainty with which to proceed at various stages? 13)
What info needed?
What information is needed to solve the problem? Could include: a.
What will be sufficient to test and distinguish between potential solutions? 14)
Priorities.
What to prioritize among many competing considerations? What to do first and how to obtain necessary resources? Considerations could include: What’s most important? Most difficult? Addressing uncertainties? Easiest? Constraints (time, materials, etc.)? Cost? Optimization and trade-offs? Availability of resources? (facilities/materials, funding sources, personnel) 15)
Specific plan for getting information . What is the specific plan for getting additional information? Includes: a.
What are the general requirements of a problem-solving approach, and what general approach will they pursue? (often decided early in problem-solving process as part of framing) b.
How to obtain needed information? Then carry out those plans (b.2). (This could involve many discipline and problem-specific investigation possibilities such as: designing and conducting experiments, making observations, talking to experts, consulting the literature, doing calculations, building models, or using simulations.) c.
What are achievable milestones, and what are metrics for evaluating progress? d.
What are possible alternative outcomes and paths that may arise during problem-solving process, both consistent with predictive framework and not, and what would be paths to follow for the different outcomes? D. Interpret Information and Choose Solutions
Calculations and data analysis.
What calculations and data analysis are needed? Then to carry those out. 17)
Represent and organize information.
What is the best way to represent and organize available information to provide clarity and insights? (usually this will involve specialized & technical representations related to key features of predictive framework) 18)
How believable is information?
Is information valid, reliable, and believable (includes recognizing potential biases)? 19)
Compare to predictions.
As new information comes in, particularly from experiments or calculations, how does it compare with expected results (based on their predictive framework)? 20)
Any significant anomalies?
If a result is different than expected, how should they follow up? (Requires first noticing the potential anomaly). Could involve deciding: a.
Does potential anomaly fit within acceptable range of predictive framework(s) (given limitations of predictive framework and underlying assumptions and approximations)? 3 b.
Is potential anomaly an unusual statistical variation, or relevant data? Is it within acceptable levels of uncertainty? 21)
Appropriate conclusions?
What are appropriate conclusions based on the data? (involves making conclusions and deciding if they’re justified) 22)
What is the best solution?
Deciding on best solution(s) involves evaluating and refining candidate solutions throughout problem-solving process. Not always narrowed down to a single solution. May include deciding: a.
Which of multiple candidate solutions are consistent with all available information and which can be rejected? (could be based on comparing data with predicted results) b.
What refinements need to be made to candidate solutions? E. Reflect (ongoing)
Assumptions + simplifications appropriate?
Are previous decisions about simplifications and predictive frameworks still appropriate? a.
Do the assumptions and simplifications made previously still look appropriate considering new information? (reflect on assumptions) b.
Does predictive framework need to be modified? (Reflect on predictive framework.) 24)
Additional knowledge needed?
Is additional knowledge/information needed? (Based on ongoing review of one’s state of knowledge.) Could involve: a.
Is solver’s relevant knowledge sufficient? b.
Is more information needed and if so, what? c.
Does some information need to be checked? (e.g. need to repeat experiment or check a different source?) 25)
How well is solving approach working?
How well is the problem-solving approach working, and does it need to be modified, including do the goals need to be modified? (Reflect on strategy by evaluating progress toward solution) 26)
How good is solution?
How adequate is the chosen solution? (Reflect on solution) Includes ongoing reflection on potential solutions, as well as final reflection after selecting preferred solution. Can include: a.
Decide by exploring possible failure modes and limitations – “try to break” solution. b.
Does it “make sense” and pass discipline-specific tests for solutions of this type of problem? c.
Does it completely meet the goals/criteria? F. Implications and Communications of Results
Broader implications?
What are the broader implications of the results, including over what range of contexts does the solution apply? What outstanding problems in field might it solve? What novel predictions can it enable? How and why might this be seen as interesting to a broader community? 28)
Audience for communication?
4 What is the audience for communication of work, and what are their important characteristics? 29)
Best way to present work?
What is the best way to present the work to have it understood, and its correctness and importance appreciated? How to make a compelling story of the work? G.
Non-Decision Themes: Ongoing Knowledge and Skill Development (30)
Stay up to date in field
Staying up to date could include: a.
Review literature, which does involve making decisions as to which is important. b.
Learn relevant new knowledge (ideas and technology, from literature, conferences, colleagues, etc.) (31)
Intuition and experience.
Acquiring experience and associated intuition to improve problem-solving. (32)
Interpersonal, teamwork . Includes navigating collaborations, team management, patient interactions, communication skills, etc., particularly as how these apply in the context of the various types of problem-solving processes. (33)
Efficiency.
Time management including learning to complete certain common tasks efficiently and accurately. (34)
Attitude.
Motivation and attitude to the task. Factors such as interest, perseverance, dealing with stress, confidence in decisions, etc.
Notes about Tables S3 and S4: discussion of trends in specific codes.
A few decisions were less likely to be mentioned in interviews, particularly in certain disciplines. See notes in supplemental table 3 for additional details. Decisions 1 and 2 (importance and gaps/opportunities). Mentioned less frequently, particularly where the problem was assigned to the expert (often in engineering or industry) or where the importance was implicit (often in medicine). For example, overall, decision 1 was mentioned in 61% of interviews, but that percentage increased to 94% when medicine and industry were excluded from the tabulation (see table S4). Decisions 27 (broader implications), 28 (audience), and 29 (present). Depended on the scope of the project being described and the expert’s specific role in it, so these decisions had little relevance in some interviews and fields and were not mentioned. 29 was particularly dependent on field, being mentioned in 68% of interviews overall, but increasing to 94% if medicine and industry were excluded (see table S4). Decisions 9 (is the problem solvable). This decision is generally implicit in the fact that the expert picked the problem to describe in the interview (they had decided it was worth solving), but often not mentioned. It was less likely to be coded in interviews in medicine, or other interviews where the problem was assigned to the expert. Often when 9 was mentioned explicitly, it was in 5 the context of deciding not to pursue an approach, or when describing the decision that a specific aspect of a problem or question is not solvable. Decision 11 (decompose). The experts relatively seldom discussed decomposition explicitly, likely because it had become such a fundamental and automatic part of their problem-solving process. This was particularly true in medicine, where deciding how to decompose was rarely mentioned, although it is well established (and supported by our informal interviews) as being fundamental to how the medical diagnostic process is structured (e.g. thinking through organ systems). Overall, decision 11 was mentioned in 68% of interviews, but that increased to 76% if medicine were excluded (see table S4), and likely that is still a significant undercount of the true use judging from the informal interviews. Decision 17 (represent and organize information), and to a lesser extent 16 (calculations and data analysis). These usually came up explicitly in interviews only if/after the expert was asked how they arrived at conclusions. Without such prompting, the expert would typically describe the information they collected, and then what they interpreted or concluded from that information, without elaborating on how the data was analyzed unless asked. Thus 16 and/or 17 must have happened during this process, but we didn’t have enough evidence to code for them. In medicine in particular, 16 and 17 were unlikely to be mentioned, because typically a doctor is provided with test results that are already analyzed by a lab or radiologist, so they do not have to make decisions themselves about how to analyze and represent the data. Overall, 16 was mentioned in 80% of interviews, but that increases to 96% if medicine was excluded (table S4). Decisions 18 (how believable is information?) and 20 (any significant anomalies?) were coded less frequently than some other decisions, probably because of the retrospective nature of the interviews. Depending on the problem context, experts may not have encountered significant anomalies or needed to question the validity of information, or they did not come up in the context of the interview, because by that point they had figured out any such behavior, and so in the retrospective process it no longer stood out as puzzling or unexpected. Reflection decisions 23 (reflect on assumptions) and 24 (reflect on knowledge) were coded less frequently than others because our coding of “reflection” required the expert to remember and relay their thinking process, in contrast to describing their actions which was the usual focus. In addition, to distinguish 24c from 13 (what information is needed?) and 15 (plan), we required some explicit evidence of reflection, such as statements about re-thinking or deciding to collect additional different information than in original plan. We also noted some decisions that were particularly likely to co-occur, or in the context of the interview were completely intertwined. Our initial list had several decisions that we found nearly always co-occurred, and those we consolidated. However, some decisions were particularly likely to co-occur but were still mentioned separately a modest fraction of the time, so we kept those separate. We describe the most common of these below and see the additional notes in table 3. Decisions 23 (reflect on assumptions), 25 (reflect on strategy), 26 (reflect on solution). It was often difficult to distinguish between reflection decisions. In some cases, the method (approach) was the solution to a problem or sub-problem being discussed, so 25 and 26 were identical. In other cases, reflection on solution or approach also required reflecting on assumptions, so 23 often co-occurred with 25 and 26. 6 Decisions 3 (goals), 6 (narrow problem), and 11 (decompose). Each is a more specific aspect of refining the problem, and so they could be indistinguishable in an interview, depending on the coherence and detail of the interview. Decisions 14 (priorities), 3 (goals), 15 (plan), and 12 (particular difficulty). 14 was often co-coded with other decisions, because deciding on priorities involves the weighting of a variety of factors. Decisions about resources were often coded with 3 (regarding criteria or constraints). Decisions about which approaches to try first or which parts of the problem to approach first were often coded with 15 (plan) or 12 (particular difficulty), because the expert often plans to prioritize (or prioritize ways to avoid) the areas of difficulty identified in 12. Decisions 1 (importance), 2 (opportunity fits expertise), and 27 (broader implications). These require very similar cognitive processes, but at different parts of the problem-solving process. Given the structure of our interviews, they were often hard to distinguish. The experts often discussed the importance of the problem as it related to what opportunities there were to make progress that matched with their expertise, so 1 and 2 were frequently coded together. For 27, after discussing their process and solution to a problem, broader implications were often mentioned in the context of discussing their next steps in terms of goals and opportunities, thus 27 would lead to a new problem and a new round of 1 and 2. A subset of interviews had a specific 27 + 2 combination: The expert would describe their development of a new tool or theory, then move on to talking about what problem(s) they could solve using this tool. This also involved 9, in that they were examining what current outstanding problems in the field would now be tractable. A key feature of the interviews that is not captured by the decisions list is the iterative nature of the decisions being made. Most of the decisions were mentioned multiple times in each interview. Sometimes they were separated into discrete cycles of going through a set of decisions to solve one component of the problem, then repeating to solve a different component of the problem. Often, reflection or an unexpected result or difficulty would trigger iteration back to earlier steps, where the expert would try a different approach at solving the problem or to refine the goals of the problem and solve a modified problem. The experts would also describe problems within problems, for example they would have a bigger-picture problem of trying to answer a scientific question or create a tool, but would also describe in detail the problem-solving process involved in troubleshooting a technical aspect of one of the steps needed in the bigger problem. Table S1. (separate file)
Notes on complete problem-solving decisions list, and examples of each decision. 8
Table S2. Field Number Position/level of interviewees
Biology and Biochemistry 2 Academic faculty Medicine 4 Medical faculty (3 internal med/peds, 1 surgery) Physics 2 Academic faculty Electrical Engineering 4 2 academic faculty, 1 lab manager, 1 industry Chemical Engineering 2 1 industry, 1 lecturer with previous industry experience Earth Science 1 Academic/ government faculty Mechanical Engineering 2 1 academic faculty, 1 industry with extensive hiring experience Chemistry 1 Academic faculty Computer Science 2 Academic faculty Biological Engineering 2 1 academic faculty, 1 lecturer with previous industry experience Informal interviews by field Table S3.
Field Number Position/level of interviewees
Biology (cell, molecular) and Biochemistry 7 1 industry, 2 academic faculty, 1 industry/ academic faculty, 1 academic senior staff, 2 senior postdocs (year 5+) Medicine 6 Medical faculty (3 internal medicine, 1 pediatrics, 1 oncology, 1 surgery) Physics 5 Academic faculty (3 theoretical, 2 experimental) Electrical Engineering 3 Industry Chemical Engineering 2 Industry Earth Science 2 1 academic faculty, 1 industry Mechanical Engineering 2 1 academic faculty, 1 industry/ government Chemistry 2 Academic faculty Computer Science 1 Industry Ecology 1 Academic faculty Semi-structured, formal, interviews by field Table S4. (separate file)
Final coding tabulation of decisions that occurred in each semi-structured interview.
Data S1. (separate file)
Collection of semi-structured interview transcripts (with redactions for privacy) from subset of experts who agreed to have their interview transcript published. To view, go to LINK XXX TO BE PROVIDED PER JOURNAL GUIDANCE.
SI References B. Crandall, G. A. Klein, R. R. Hoffman,
Working Minds: A Practitioner’s Guide to Cognitive Task Analysis (MIT Press, 2006).(MIT Press, 2006).