PhysPort use and growth: Supporting physics teaching with research-based resources since 2011
Sarah B. McKagan, Linda E. Strubbe, Lyle J. Barbato, Adrian M. Madsen, Eleanor C. Sayre, Bruce A. Mason
PhysPort use and growth: Supporting physics teaching with research-based resources since 2011
Sarah B. McKagan , Linda E. Strubbe , Lyle J. Barbato , Adrian M. Madsen , Eleanor C. Sayre , Bruce A. Mason American Association of Physics Teachers 2.
Kansas State University
Introduction
What is PhysPort?
What can I find on PhysPort?
PhysPort contains many resources for faculty, including overviews of over 50 research-based teaching methods and over 80 research-based assessments, the Virtual New Faculty Workshop and the Periscope collection of video-based TA training and faculty professional development materials, Expert Recommendations with specific guidance on implementing research-based teaching and assessment, and the Data Explorer. Figure 1 and Table I describe the content available on PhysPort. Fig. 1. Content available on PhysPort.
Research, design and development of PhysPort
PhysPort's mission is supported by three kinds of research: (1) Fundamental research on faculty needs around teaching and assessment ; (2) Applied research on user experience with the PhysPort website ; and (3) Synthesis and fundamental research on effective practices in teaching, learning and assessment . This research has been primarily among college / university physics faculty in the U.S. and focused on undergraduate teaching; we discuss other groups of PhysPort users below. Additionally, PhysPort enables research into student learning by creating tools that support developers of assessment instruments and curricula in their research efforts and helping them reach a broad audience. Development of PhysPort is ongoing, as the project team continually strives to improve the site to make it most useful to current and potential users. PhysPort developers are currently being funded to conduct research on how faculty use resources to support their teaching and how PhysPort impacts faculty teaching practices. Usage of PhysPort
How well used is PhysPort?
PhysPort usage is substantial and has increased annually since its release at the end of 2011. Figure 2 shows how key usage measures have increased from 2012 to 2018 (the first and last full years of the site’s existence), determined from Google Analytics. During 2018, PhysPort had 84,000 visits , 56,000 unique visitors, and 1,500 users who visited the site five or more times. All of these measures have been increasingly annually since PhysPort’s release. Figure 2. PhysPort usage over time, 2012-2018. (a) Number of visits to PhysPort each year. A visit to PhysPort (“session”) may include looking at multiple pages within the site. We show total number of visits, number of visits where a user visited five or more pages, and number where the visit lasted longer than three minutes. (b) Number of visitors to PhysPort each year. A unique visitor is a user traceable to a unique IP address, who might visit the site multiple times. “Regular users” visit PhysPort more than three times in a year; “frequent users” visit more than five times in a year. Data is from Google Analytics. Which parts of PhysPort are most used?
Figure 3. Total number of page views each year by site section. The dates in the legend indicate when the section was added to the site. Hatched regions indicate new sections added to the site that year. The Assessments and Teaching Methods sections receive the most page views on PhysPort. Data is from Google Analytics. Figure 3 shows the total number of page views for each section of the site. The Assessments section, where users can learn about and download research-based assessments, currently receives the most page views on PhysPort. Of the 97 assessments published on PhysPort, 65 had more than 100 unique page views and nine had more than 1,000 unique page views in 2018. The most popular assessment, the Force Concept Inventory (FCI), had 11,000 unique page views and was downloaded 1,400 times in 2018. Over the lifetime of PhysPort , users have downloaded almost 36,000 assessments. The site section with the second highest fraction of page views is Expert Recommendations. For the 10 recommendations with over 1,000 unique page views in 2018, the average time spent on the page (for users who later visit another page on PhysPort) is five minutes. PhysPort’s most popular recommendation in 2018, “Effect size: What is it and when and how should I use it?” , had 9,300 unique page views and an average time on page of nine minutes. These times are substantial, since web users usually click between pages very quickly: 80% of users leave a typical page within 70 seconds — and users tend to judge pages related to education and science particularly quickly . We infer that these recommendations are capturing the attention of PhysPort visitors. How engaging is PhysPort?
PhysPort visitors spend more time on the site and view similar or more pages than at comparable sites about physics or about teaching and classroom resources, according to Google Benchmarks; see Figure 4.
Figure 4. PhysPort use in 2018 compared to Google Benchmarks comparable sites: “Physics” and “Teaching & Classroom Resources” for sites with 100-499 daily sessions. See “About Benchmarking”, 2019: https://support.google.com/analytics/answer/6086666?hl=en.
Who is using PhysPort? How well is PhysPort reaching its target audiences?
The target audiences of PhysPort are two overlapping groups: physics/astronomy faculty at U.S. colleges / universities and members of AAPT. These priorities are largely driven by the priorities of our funders, availability of research about these groups and their students, and our ease of access to these groups for needs and usability research. PhysPort has two other major populations of users as well: U.S. high school teachers and educators outside the U.S. We are pleased that these latter groups find PhysPort useful; however, we note that our user-centered research and design process has so far focused almost exclusively on U.S. college faculty—whose needs and contexts may be quite different. If funding were available in the future, we would be delighted to design PhysPort around the needs of U.S. high school teachers and educators outside the U.S. also. Figure 5. Verified Educators on PhysPort, broken down by population. (a) Growth of Verified Educators over time. (b) Verified Educators as of March 2019. We primarily learn about who is using PhysPort by requesting demographic information from users who become “Verified Educators” so they can access restricted resources (download assessments, use the Data Explorer, and access Periscope workshops). Currently there are 6,600 Verified Educators, and the number has been growing steadily; see Figure 5. About one-third of Verified Educators are based outside the U.S., slightly more than one-third are at U.S. colleges, and slightly less than one-third are U.S. high school teachers. One-fifth of Verified Educators are AAPT members (see below for details). Only about 10% of Verified Educators are physics education researchers , indicating that PhysPort use extends well beyond the PER community. In comparison to statistics from American Institute of Physics Statistical Research Center (AIP Statistics) (Table II), PhysPort is reaching a substantial fraction of our (overlapping) target audiences; we are also reaching substantial numbers of U.S. high school teachers and educators outside the U.S. U.S. college physics faculty
U.S. college physics faculty are the main target audience of PhysPort, since nearly all funding has come through NSF lines targeting undergraduate education. According to the most recent studies conducted by AIP Statistics, there were 9,400 full-time equivalent (FTE) faculty members (including those in temporary or non-tenure-track positions) at physics degree-granting departments during the 2009-10 academic year and “almost 3,300 faculty members taught physics courses in two-year colleges during the 2011-12 academic year” , giving about 13,000 U.S. college faculty in total. PhysPort has 2,600 Verified Educators who are U.S. college faculty, and 85% of those who give their department name are in a physics-related department; thus, it is estimated that about 20% of U.S. college physics faculty are Verified Educators. This suggests PhysPort is doing well at reaching its main target audience. U.S. high school physics teachers
While PhysPort does not explicitly target high school physics teachers, many PhysPort resources are applicable to (or even designed for) high school classrooms. Since many AAPT members are high school teachers, advertising PhysPort via AAPT reaches many high school teachers. According to AIP Statistics, approximately 27,000 teachers in U.S. high schools taught at least one physics class during the 2012-13 school year . PhysPort has 1,800 Verified Educators who are U.S. high school physics teachers, thus we estimate about 7% of U.S. high school physics teachers are Verified Educators. A logical future step for PhysPort would be to target high school teachers explicitly by conducting user research with them and creating resources targeted to the needs of high school classrooms. AAPT members
AAPT is the main U.S. professional society for physics educators at all levels; as such, its 7,000 members are a natural target audience for PhysPort. PhysPort is regularly marketed to AAPT members through email announcements and presentations and workshops at meetings. AAPT members have the exclusive benefit of becoming Verified Educators instantly, while other users must wait 1-2 days for someone to verify their account. About 1,300 Verified Educators have used this “instant verification” process; therefore, we estimate that 20% of AAPT members are Verified Educators. This is lower than we would like, so we see opportunities for further marketing to AAPT members. Additionally, because 80% of PhysPort Verified Educators are not AAPT members, there is also an opportunity for PhysPort to recruit new members to AAPT.
What is the international impact of PhysPort?
Figure 6. Unique visitors to PhysPort in 2018, from within and outside the U.S. Data is from Google Analytics. PhysPort has many users outside the U.S: 31% of Verified Educators come from outside the U.S., and 41% of PhysPort visits and 40% of unique visitors were from outside the U.S. in 2018 (Figure 6). It is beyond the scope of this paper to estimate the number of potential users outside the U.S. since data would need to be gathered from many different sources, but we expect that this number is at least an order of magnitude larger than the number of U.S. college faculty and high school teachers. It is important to note that PhysPort was not designed for educators outside the U.S., who often teach in very different educational environments and cultures and speak different languages than in the U.S., and therefore may need different resources. Nevertheless, the PhysPort team is very interested in serving educators outside the U.S. and has taken preliminary steps in this direction. Our team has done some outreach and created several resources for users outside the U.S., including talks, workshops, and collaborations in Canada, Germany, Ghana, Kyrgyzstan, Mexico, Rwanda, and Tajikistan. Although primarily an English-language site, PhysPort includes some resources in other languages as well. In spite of very little effort to solicit translations or advertise the availability of translations on PhysPort, we receive offers each month to translate assessments. Twenty-seven of our 82 assessments are available in at least one language additional to English ; these represent 14% of all assessment downloads. The most translated assessment is the FCI, available in 29 languages additional to English. PhysPort hosts 12 assessments in Spanish, and we have an Expert Recommendation listing Spanish-language physics education resources . Educators outside the U.S. are a large and important potential audience, but also a challenge to serve because their needs and contexts are so diverse, and because there is little U.S.-based funding available for international physics education work. We are very interested in future collaborations with physics educators abroad to expand the use and usefulness of PhysPort beyond the U.S. Conclusions
In the last decade, PhysPort has gone from an idea to a site recognized throughout physics education with over 80,000 visits and over 50,000 unique visitors in 2018, and 6,600 Verified Educators as of March 2019. PhysPort usage has been growing every year by key common measures. PhysPort has substantial usage among U.S. college faculty (its primary target audience), as well as U.S. high school teachers and educators outside the U.S. The Assessments and Expert Recommendations are the most popular sections of the site. Many users come back to PhysPort regularly and spend substantial time there, indicating that PhysPort is making a valued contribution to the field.
Acknowledgments
PhysPort is wholly grant-supported, and has been created through multiple NSF grants: 0840853, 1256352/1256354, 1245490, 1347821/1347728, 1726113/1726479, and portions of 1323699, 1223405, 1323129, 1626496, 1140860, and 1140706. We thank the team that has made PhysPort possible.
Site Section (release date) Description a ) PhysPort Verified Educators from this population b, c Total target population d Percentage of target population using PhysPort (as Verified Educators) U.S. College Physics Instructors 2,600 13,000* 20% AAPT Members 1,300 7,000 20% U.S. High School Physics Teachers 1,800 27,000* 7% Educators Outside the U.S. 2,000 Unknown Unknown Table II. Target populations for PhysPort and percentage of each using PhysPort. a The rationale for these priorities is explained in the main text. b Verified Educators as of March 2019. c Note that these populations overlap: AAPT members are typically also members of the other groups listed here. d Total target population from AIP surveys (refs. 17, 18, 19); see main text.
References Adrian Madsen, Sarah B. McKagan, Martin “Sandy” Martinuk, Alexander Bell, and Eleanor C. Sayre. “Research-based assessment affordances and constraints: Perceptions of physics faculty,” Phys. Rev. Phys. Educ. Res. 12(1), 1–16 (February 2016). 2.
Adrian Madsen, Sarah B. McKagan, Eleanor C. Sayre, Martin Martinuk, and Alexander Bell, “Personas as a Powerful Methodology to Design Targeted Professional Development Resources,” in Polman, J. L., Kyza, E. A., O'Neill, D. K., Tabak, I., Penuel, W. R., Jurow, A. S., O'Connor, K., Lee, T., and D'Amico, L. (Eds.). Learning and becoming in practice: The International Conference of the Learning Sciences (ICLS) 2014, Volume 2. (International Society of the Learning Sciences, Boulder, CO, 2014), p. 1082. 3.
Zohrabi et al. in prep. 4.
We draw from J. G. Redish, Letting go of the words: Writing web content that works. (Morgan Kaufmann, 2007). 5.
Adrian Madsen, Sarah B. McKagan, and Eleanor C. Sayre, “Gender gap on concept inventories in physics: What is consistent, what is in- consistent, and what factors influence the gap?” Phys. Rev. ST Phys. Educ. Res. 9, 020121 (November 2013). 6.
Adrian Madsen, Sarah B. McKagan, and Eleanor C. Sayre, “How physics instruction impacts students’ beliefs about learning physics: A meta-analysis of 24 studies,” Phys. Rev. ST Phys. Educ. Res. 11, 010115 (June 2015). 7.
Adrian Madsen, Sarah McKagan, and Eleanor C. Sayre, “Resource Letter: RBAI-1: Research-based assessment instruments in physics and astronomy,” Am. J. Phys. 85, 245 (March 2017). 8.
Adrian Madsen, Sarah B. McKagan, and Eleanor C. Sayre, “Best Practices for Administering Concept Inventories,” Phys. Teach. 55(9), 530–536 (November 2017). 9.
Adrian Madsen, Sarah B. McKagan, Eleanor C. Sayre, and Cassandra A. Paul, “Resource Letter: RBAI-2: Research-based assessment instruments: Beyond physics topics,” Am. J. Phys 87, 350 (May 2019). 10.
Joshua Von Korff, Benjamin Archibeque, Kathy A. Gomez, Tyler Heckendorf, Sarah B. McKagan, Eleanor C. Sayre, Edward W. Schenk, Chase Shepherd, and Lane Sorell, “Secondary Analysis of Teaching Methods in Introductory Physics: a 50k-Student Study,” Am. J. Phys. 84, 969 (2016). 11.
A visit or session may include looking at multiple pages within the site. Google Analytics does not provide measurement uncertainties, but measuring similar quantities in different ways in Google Analytics suggests that two significant figures is typically appropriate. 12.
Google Analytics distinguishes between “page views” and “unique page views.” The former is any visit to a page; for the latter, if a user visits the same page multiple times within a single visit to the website, this is counted as only one unique page view. Figure 3 shows page views. See https://support.google.com/analytics/answer/1257084?hl=en. 13.
All data in this article is current as of March 2019. 14.
Chao Liu, Ryen W. White, and Susan Dumais, “Understanding Web Browsing Behaviors through Weibull Analysis of Dwell Time,” in Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, (ACM, New York, 2010), p. 379. 16.
Hereafter, we use “college” to refer to all institutions of higher education.
We estimate the fraction of verified educators who are physics education researchers as the fraction who have logged into PER-Central.org as one of the last four ComPADRE libraries they visited while logged in. Logging into PER-Central is required to submit papers or reviews for the Physics Education Research Conference Proceedings, edit a research group profile on the map of PER groups, or perform other common tasks for physics education researchers. 18.
The fraction of verified educators who used instant verification is an imperfect estimate of the fraction who are AAPT members. Some AAPT members may not use instant verification, since it is not available for students, or they may have forgotten their AAPT password. And some PhysPort users who used instant verification in the past may no longer be AAPT members. However, we assume that these effects are small and tend to cancel each other out. 22.