Going beyond the one-off: How can STEM engagement programmes with young people have real lasting impact?
Martin Archer, Jennifer DeWitt, Carol Davenport, Olivia Keenan, Lorraine Coghill, Anna Christodoulou, Samantha Durbin, Heather Campbell, Lewis Hou
GGoing beyond the one-off: How can STEM engagement programmes with young people have real lasting impact?
Manuscript in preparation for Research for All
Authors
Martin Archer*, School of Physics and Astronomy, Queen Mary University of London ([email protected], https://orcid.org/0000-0003-1556-4573) *now at Department of Physics, Imperial College London Jennifer DeWitt, UCL Institute of Education, University College London, London; Independent Research and Evaluation Consultant ([email protected], https://orcid.org/0000-0001-8584-2888) Carol Davenport, NUSTEM, Northumbria University ([email protected], https://orcid.org/0000-0002-8816-3909) Olivia Keenan, South East Physics Network ([email protected]) Lorraine Coghill, Science Outreach, Durham University ([email protected]) Anna Christodoulou*, Department of Physics, Royal Holloway University of London ([email protected]) *now at University of Essex Samantha Durbin, The Royal Institution ([email protected]) Heather Campbell, Department of Physics, University of Surrey ([email protected]) Lewis Hou, Science Ceilidh ([email protected])
Abstract
A major focus in the STEM public engagement sector concerns engaging with young people, typically through schools. The aims of these interventions are often to positively affect students’ aspirations towards continuing STEM education and ultimately into STEM-related careers. Most schools engagement activities take the form of short one-off interventions that, while able to achieve positive outcomes, are limited in the extent to which they can have lasting impacts on aspirations. In this paper we discuss various different emerging programmes of repeated interventions with young people, assessing what impacts can realistically be expected. Short series of interventions appear also to suffer some limitations in the types of impacts achievable. However, deeper programmes that interact with both young people and those that influence them over significant periods of time (months to years) seem to be more effective in influencing aspirations. We discuss how developing a Theory of Change and considering young people’s wider learning ecologies are required in enabling lasting impacts in a range of areas. Finally, we raise several sector-wide challenges to implementing and evaluating these emerging approaches. eywords
Impact, schools, young people, interventions, engagement programmes, Theory of Change, learning ecology, learning ecosystem
Key Messages ● One-off and short-series of STEM interventions with young people don’t appear to have the long-term impacts on aspirations that universities and other practitioners of STEM engagement are often hoping to achieve ● Deeper programmes of engagement are required based around Theories of Change and considering young people’s wider learning ecology ● Many sector-wide challenges exist to implementing and evaluating the long-term impacts of such programmes
Introduction
Engaging schools and young people with Science, Technology, Engineering and Mathematics (STEM) has long been a priority for STEM engagement bodies and practitioners, particularly in the cases of universities/researchers — a survey of UK researchers found that engineering and physical sciences researchers place more importance on engaging school students out of all possible publics than researchers in other areas do (Hamlyn et al., 2015). This is perhaps driven by concerns over the perceived low numbers of young people opting for studies in STEM subjects (e.g. Smith, 2004). Researchers’ and practitioners’ motives to engage young people in order to encourage them to study a particular subject or even to consider a particular institution, often what are meant by the terms “outreach” and “recruitment” respectively, are typically conflated with public engagement, which in its purest sense is formed of genuine two-way interactions for mutual benefit, and likely affects their approach to engagement practice with schools (Hamlyn et al, 2015). For example, Thorley (2016) found that physicists consider engaging young people as typically transmissive (i.e. one-way) in nature and while they place high levels of importance on a range of content types and messages for young people (knowledge, excitement, relevance, careers information etc.) they unilaterally considered their own role in conveying these topics as less important. Therefore, it appears that in general more critical consideration is needed in recognising audiences’ needs and appropriate methods of engagement in order to improve the efficacy of raising STEM aspirations in young people. STEM engagements with young people have generally taken the form of various one-off experiences, such as a school trip, a show, a speaker, a video or some other interaction. While popular, these are realistically limited in the types of impacts they might be able to have and the likelihood that these impacts will last in the longer term. There is evidence that one-off experiences can in the short-term support content learning (likely just in the form of ‘factoids’), as well as lead to affective outcomes, such as interest and inspiration (Bell et al., 2009; DeWitt & Storksdieck, 2008). Likewise, meeting a scientist can certainly provide students with an increased awareness of what a particular career involves or the range of careers that might be available, and even what courses might be required to progress (Woods-Townsend et al., 016). Memory research, however, suggests that without further reinforcement of these learning outcomes they will likely be limited to days or weeks (e.g. Murre and Dros, 2017). Furthermore, such one-off experiences are unlikely to have a lasting measurable effect on aspirations. Research into young people’s aspirations (cf. L. Archer & DeWitt, 2017, among many others) highlights that aspirations are complex and multifaceted and evolve over time. They are influenced by a range of factors, including experiences at home, in the school and in the wider community, as well as background factors that are interwoven with the way young people experience school, engagements with science, and everything else. They are also closely linked to identity - to what or whom young people can see themselves becoming, and what type of person they (and others) perceive them to be. Consequently, it should not be surprising that one-off experiences, while inspirational and having potential to support a range of outcomes, are unlikely to significantly impact aspirations and educational trajectories. At the same time, because aspirations develop and are maintained (or not) over time, there can be a role for such experiences in providing additional support in their maintenance. Moreover, when considered from the perspective of learning ecologies (discussed in more detail later), any potential impact of a one-off experience can be extended by linking to other experiences that young people may have, both shorter term and longer in duration. In other words, while one-off experiences can be worthwhile experiences, it is important to be realistic about what they can achieve by themselves - and thus to maximise the opportunities they offer and their potential impact by linking to other aspects of young people’s lives and experiences. As we aim to move towards more impactful engagement and deeper learning, we have to carefully consider and evaluate the different types of engagement that we develop. As the sector has moved from the more didactic ‘Public Understanding of Science’ towards more egalitarian constructivist approaches, we still see the dominance of traditional interventions such as lectures. We are aware of the limitations of the traditional “listen to me” lecture-style event (e.g. Freeman, 2014; Marbach-Ad, 2000; Marbach-Ad, 2001) and in 2013, The Times Higher Education noted over 700 studies determining that lectures were less effective than other teaching strategies (Gibbs, 2013). As such, there are definite moves in undergraduate teaching away from such traditional lecturing with shifts in: the format of lecturing, such as segmenting and including discussion-based approaches (e.g. Iowa State University, 2020); increased use of active learning strategies (e.g. Durham University, 2019); and even the redesigning of learning spaces or “build pedagogy” (Monahan, 2002; Elkington, 2019). However, learners and teachers will often state a preference for lectures over active learning (Deslauriers, 2019). Students feel that they learn more from lectures, even although the evidence suggests otherwise, perhaps partly caused by the additional cognitive effort required by active learning methods (Deslauriers, 2019). Against this backdrop, in trying to develop STEM public engagement projects that have greater scope to develop and support students’ aspirations, we are also faced with the additional resources, costs, relationship-building and time required to make such programmes happen. Nonetheless, several organisations and practitioners have been moving towards engagement programmes of repeated-interventions with the same group of young people in order to maximise the likelihood of impacts on their aspirations towards STEM. This paper provides a andscape review and commentary aimed at STEM engagement practitioners (be they independent or based in a university or institutional setting) on some of the different approaches that have emerged, predominantly in the United Kingdom. It explores their potential benefits and limitations by drawing from evaluation literature as well as social science and educational research/theory. Current challenges faced by the STEM engagement sector and what might be required in order to move forward effectively are also discussed. These topics arose from discussions at a session on repeated interventions at the BIG STEM Communicators Network’s BIG Event 2019 (M.O. Archer et al., 2019).
Short series
The next logical step on from a one-off intervention is to instead deliver a short series of (typically a few) interventions. Here we discuss some different examples of such series delivered within the academic year and their evaluation thus far. We note that other programmes working on interventions across multiple years (though with only one session per academic year) are starting to emerge, but we are not aware of sufficient published evidence of the impacts of these types of initiatives yet. Activities like summer schools, typically run for late secondary and sixth-form students (aged 15-18), in some sense can be thought of as a short series or “intensive one-off”, since they are highly focused down into often just a single week. It appears that summer schools, while enjoyable to the participants, only cause moderate changes to students’ likelihood to apply to a selective university in surveys immediately afterwards (e.g. Universify Education, 2018) and many summer school programmes show no change in STEM aspirations from before to after (e.g. Bhattacharyya et al., 2011; MacIver and MacIver, 2015). There is also a critical lack of longitudinal studies at present, as highlighted in a recent review of the higher education widening access sector (Robinson and Salvestrini, 2020). Such studies are necessary in demonstrating whether any increased uptake of higher education actually occurs and whether there is causality in summer schools themselves (and not other factors) leading to improved progression. We also note that summer schools are expensive to deliver and often have severely limited places, making them highly competitive. Even with targeting of participants using widening participation criteria, they are likely selecting predominantly those who are already highly bought into the subject of the summer school. This is an aspect that TASO (https://taso.org.uk/) plan to investigate with randomised control trials in the near future. ‘I’m a Scientist, Get Me Out of Here’ (https://imascientist.org.uk/) involves an online chat between students (between ages 9-18) and scientists followed by more extended Q&A over the span of typically 2 weeks. They commissioned an evaluation using the framework of science capital to understand what impact it might be having and what might be contributing to that impact (DeWitt, 2019). Perhaps not surprisingly, one of the biggest impacts was on how young people perceived scientists - as normal, regular people, with hobbies, families and interests outside of science. Such perspectives on scientists are similar to those shared by individuals with higher levels of science social capital - who know people (e.g. family, friends’ parents) who work in science. While there was limited evidence that this awareness completely changed aspirations of young people, there was an increased willingness among many to consider the ossibility of pursuing science further. Of course, the realisation of any longer-term or significant impact on aspirations is dependent on many experiences that may (or may not) happen after. But this increased openness to the possibility of pursuing science, which seems to be influenced by young people realising that scientists are people ‘like me’, can have a role to play. A similar shift in perceptions of scientists was found among primary school pupils participating in ‘Scientist of the Week’, an intervention developed by the NUSTEM team at Northumbria University. This is a five week, teacher-led intervention, in which a new scientist was ‘introduced’ to the students each week using presentation slides, classroom posters and postcards to take home. In the materials provided to teachers there was a short paragraph describing the work and life of each scientist which included three key attributes of that person, attributes which both contrasted with stereotypical views of scientists (e.g. curious, open-minded, creative) and represented characteristics that young children could imagine themselves possessing and often likely already possessed. In other words, these attributes also communicated that these scientists were ‘like me’. Evaluation of this project provided encouraging evidence that young people’s perceptions of scientists were shifting - after the intervention pupils were more likely to use non-stereotypical words than stereotypical words when asked to describe a scientist (Shimwell et al., under review). L. Archer et al. (2014) report on a six-week STEM careers intervention series for Year 9 (13-14 year-old) students at a London girls’ school that combined multiple activities that were co-designed and delivered by classroom teachers. The comprehensive evaluation involved before and after surveys, classroom observations, post-intervention student discussion groups, and teacher interviews. While the series did appear to have a positive effect on broadening students’ understanding of the range of jobs that science can lead to or be useful for, it did not significantly change students’ aspirations or views on science.
Deeper programmes
The big picture
While it appears that long-term interventions have the ability to measurably influence young people’s aspirations and identification with science, such efforts are beyond the scope of many organisations. Such repeated-interventions are also challenging to develop, deliver and maintain in settings outside of school, simply because young people generally are not required to be there in the same way that school attendance is mandatory. Thus, it becomes important to consider how to maximise potential impacts of the many activities that can and do take place in formal and informal settings: one-offs, short series or even deeper programmes of engagement. This is where the notion of learning ecosystems or ecologies can be helpful (e.g. Brofenbrenner, 1979). A learning ecology is the context (physical, social, cultural) in which learning takes place. Ecological perspectives on learning acknowledge that young people (and adults, for that matter) learn across a range of contexts, as well as over time, and all of this is influenced in a complex way by previous experiences, background factors, and what follows any given experience. An example diagram of a learning ecology model is shown in Figure 1, where the individual lies at the centre and the layers surrounding have decreasing amounts of direct interaction/influence with them. Such models serve to remind us that our interventions - whether one-offs, short eries, or longer, in whatever setting - do not operate in isolation. Moreover, they can be more powerful as learning experiences by linking to other elements in the learning ecosystem. That is, by making a conscious and concerted effort to form links among organisations, young people may be guided towards other experiences that reinforce what they have learned in our activities (Bevan, 2016; Traphagen and Traill, 2014). Doing so can also help us refine our efforts - define our niche - so that it better fits what is needed. In the United States, such initiatives are gaining momentum - and, indeed, a STEM Funders Network has formed to support such efforts in a concerted way (https://stemecosystems.org/about-the-stem-funders-network/). Figure 1: The learning ecology model of Brofenbrenner (1979). From Rhodes (2013). It is perhaps tempting for organisations that want to effect behaviour or aspiration change to develop an intervention that is based on ‘common-sense’, but without considering the learning ecology that the child or young person is situated within, without identifying a realistic way in which the desired change could be achieved by the intervention, and with no research base underpinning the intervention. One way to overcome this temptation is to use a Theory of Change approach. This was initially developed to evaluate complex initiatives (Sullivan and Stewart, 2006) and support long-term behaviour change. A key benefit of using a Theory of Change is that it starts with the end goal, and then requires the identification of intermediate outcomes through a process of backwards mapping that will, over time, lead to that goal. In this way causal chains can be identified that link an intervention with evidence-based steps that hould eventually lead to the desired change (Davenport et al., 2020). This process is an iterative one, and allows the production of a Theory of Change diagram, such as that shown in Figure 2, which depicts the changes involved in achieving the goal of an intervention. Figure 2: An example Theory of Change diagram for increasing the number of young people choosing a career in STEM post-18. From Davenport et al. (2020).
Challenges to the sector
Conclusions
Raising and maintaining STEM aspirations are a typical aim of universities’ and other practitioners’ and organisations’ engagements with young people. Aspirations are, however, incredibly complex and difficult to affect, so it is not surprising that any single one-off intervention in isolation, while able to achieve a number of worthy positive outcomes, is unlikely to fundamentally change a young person’s future trajectory. We have presented a landscape review and commentary of different emerging programmes of repeated STEM interventions with young people which therefore aim to maximise the likelihood of raising and supporting young people’s STEM aspirations. While short series of interventions seem to suffer some of the same limitations as one-offs at present, there is emerging evidence that deeper programmes over the course of months to years that interact not only with the young person but also key components within their wider learning ecology are able to measurably impact on STEM aspirations. However, there is no one-size-fits-all solution to raising and subsequently maintaining the STEM pipeline and a variety of different approaches are required throughout a young person’s educational journey. While we believe that having more deeper programmes of STEM engagement would be the most effective way of positively affecting the STEM landscape overall, we realise that this may not be possible for all types of informal STEM engagement practitioners and organisations. One-offs do, however, have inherent value and might be able to collectively within the wider learning ecology contribute towards building and maintaining aspirations through various different interventions, so long as they present coherent messages that reinforce one another. Programmes of repeated interventions may simply enable providers to control that longer-term messaging, giving individual providers more influence in a consistent way over a great stretch of time than would otherwise be possible. They also inherently allow a greater ability to capture the impacts that the interventions have had. Nonetheless, we advocate that those who develop any STEM engagement activities or programmes consider adopting a Theory of Change approach o critically consider whether what they deliver, be they protracted programmes or one-offs, are likely to achieve (or even just contribute towards) the intended aims. Furthermore, we urge organisations where possible to try to involve and influence the wider ecology in an audience-focused way with every activity to increase the likelihood of any impacts lasting. In lieu of a more joined-up collaborative approach across the entire sector, these considerations at the very least should help us all to make more of a difference.
Notes on contributors
Dr Martin Archer is a UKRI Stephen Hawking Fellow in space physics and public engagement at Imperial College London. He has been a leader in developing award-winning innovative and impactful research-based outreach and public engagement activities for over a decade, sharing the excitement and importance of physics in accessible ways to a variety of often underserved audiences.
Dr Jennifer DeWitt is a Senior Research Fellow at the UCL Institute of Education, where she is a member of the team developing and applying the concept of science capital. She is also a research and evaluation consultant, specialising in science learning and engagement, with particular interests in learning in informal settings, including the implications of science capital research for equitable practice in these settings.
Dr Carol Davenport is a Senior Lecturer and Director of NUSTEM at Northumbria University, Newcastle. Her research interests include the impact of STEM engagement activities on young people’s career choices, the implementation of the Gatsby career benchmarks in subject classrooms and the effects of unconscious bias in primary schools.
Dr Olivia Keenan is Director of Outreach and Public Engagement at the South East Physics network. She leads the network’s outreach programme and public engagement work. SEPnet work with schools to improve accessibility to, and uptake of, physics. They support their partner universities to engage diverse publics on the research they are conducting. Olivia is passionate about equality and representation in STEM and enjoys working on projects which help embed social justice by removing barriers to access.
Dr Lorraine Coghill is Science Outreach Coordinator and NE Regional Representative for Durham University and the Ogden Trust. Lorraine creates, develops and cultivates opportunities for people to work together to explore, enquire and engage others. With nearly 20 years engagement experience, she develops and manages creative and award-winning programmes, and is particularly passionate about training and supporting others in advancing their own engagement practice and confidence.
Anna Christodoulou is Collaborative Outreach Officer for Make Happen, the Uni Connect Partnership in Essex. She is working with schools, HE and FE Institutions and charities to raise aspirations and widen participation in higher education throughout Essex. She is a very experienced project manager specialising in outreach and public engagement with research, and an awarded science communicator.
Samantha Durbin has been Clothworkers’ Associate in Mathematics at the Royal Institution since 2012, managing the Ri’s Secondary Mathematics Masterclass network across the UK.. She also co-organises Talking Maths in Public, a skills sharing network and biennial conference for people working in maths outreach/communication.. Samantha was previously a STEM Clubs Advisor at the British Science Association, having completed an MSc in Science Communication at the University of the West of England in 2012 and Master of Mathematics at the University of Bath in 2010.
Dr Heather Campbell completed an Mphys in Astrophysics with Research Placement at the University for Sussex in 2009. Then, a PhD on the explosions of stars, Type Ia Supernovae to explore the accelerated expansion of the Universe, at the University of Portsmouth. Heather worked as a postdoctoral researcher on the Gaia Satellite at the University of Cambridge. For the last 4 years Heather has been the SEPnet/Ogden Public Engagement and Outreach Manager at the University of Surrey.
Lewis Hou is an interdisciplinary education and cultural participation specialist. He is particularly interested in public and community engagement approaches that build meaningful and equitable relationships with diverse groups beyond the “already converted” and was the recipient of the Royal Society of Edinburgh’s Public Engagement Innovator Medal in 2018. He founded and directs the Science Ceilidh, an education and community organisation exploring curiosity, creativity, equity, and health and wellbeing with adults, youth groups and schools across Scotland.
References ‐ American High School Students' Career Choices. School Science and Mathematics, 111: 345-353. doi:10.1111/j.1949-8594.2011.00097.x Braund, M. and M. Reiss (2006). Towards a more authentic science curriculum: The contribution of out-of-school learning. Int. J. Sci. Educ., 65:13731388. doi: 10.1080/09500690500498419. Bright Purpose (2019). The role of informal science in youth work. Findings from Curiosity round one. Wellcome Trust and BBC Children In Need. URL: https://wellcome.ac.uk/sites/default/files/role-informal-science-learning-youth-work.pdf Bronfenbrenner, U. (1979). The Ecology of Human Development: Experiments by nature and design. Cambridge, MA: Harvard University Press, 1-348. ISBN 0674224574 Davenport, C., Dele-Ajayi, O., Emembolu, I. et al. A Theory of Change for Improving Children’s Perceptions, Aspirations and Uptake of STEM Careers. Res Sci Educ (2020). https://doi.org/10.1007/s11165-019-09909-6 Deslandes, R., & Cloutier, R. (2002). Adolescents’ Perception of Parental Involvement in Schooling. School Psychology International, 23(2), 220–232. https://doi.org/10.1177/0143034302023002919 Deslauriers, L., McCarty, L.S., Miller, K., Callaghan, K., Kestin, G., (2019),
Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom , Proceedings of the National Academy of Sciences, 116(39), 19251-19257. DeWitt, J., L Archer, J Moote (2019). 15/16-Year-Old Students’ Reasons for Choosing and Not Choosing Physics at A Level. International Journal of Science and Mathematics Education, 17, 1071-1087. https://doi.org/10.1007/s10763-018-9900-4 DeWitt, J. (2019) I’m a Scientist: Supporting Science Capital. Report https://about.imascientist.org.uk/files/2019/11/IAS-Science-Capital-Main-Report-Sep-2019.pdf Durham University Centre for Academic Excellence, (2019),
DCAD Resource Bank , Retrieved from: https://sites.durham.ac.uk/dcad-resourcebank/ Last accessed 07/02/20. Elkington, S., Bligh, B., (2019),
Future Learning Spaces: Space, Technology and Pedagogy
Using Action Research to Design and Evaluate Sustained and Inclusive Engagement to Improve Children’s Knowledge and Perception of STEM Careers.
International Journal Of Science Education https://doi.org/10.1080/09500693.2020.1729442 Freeman, S., Eddy, S.L., McDonough, M., Smith, M.K., Okoroafor, N., Jordt, H., Wenderoth, M.P., (2014),
Active Learning Increases Student Performance in Science, Engineering and Mathematics , Proceedings of the National Academy of Sciences USA, 111(23), 8410-8415. Gibbs, G., (2013),
Lectures Don’t Work But We Keep Using Them
Effective Lectures
Journal of Research in Science Teaching , 37, 854–870. Marbach-Ad, G., Claassen, L.A., (2001), Improving Students' Questions in Inquiry Labs,
The American Biology Teacher , 63(6), 410-419. Monahan, T. (2002),