Keith E. Maull

Assessing and tracing the outcomes and impact of research infrastructures

Recent policy shifts on the part of funding agencies and journal publishers are causing changes in the acknowledgment and citation behaviors of scholars. A growing emphasis on open science and reproducibility is changing how authors cite and acknowledge “research infrastructures”—entities that are used as inputs to or as underlying foundations for scholarly research, including data sets, software packages, computational models, observational platforms, and computing facilities. At the same time, stakeholder interest in quantitative understanding of impact is spurring increased collection and analysis of metrics related to use of research infrastructures. This article reviews work spanning several decades on tracing and assessing the outcomes and impacts from these kinds of research infrastructures. We discuss how research infrastructures are identified and referenced by scholars in the research literature and how those references are being collected and analyzed for the purposes of evaluating impact. Synthesizing common features of a wide range of studies, we identify notable challenges that impede the analysis of impact metrics for research infrastructures and outline key open research questions that can guide future research and applications related to such metrics.

A Two-Dimensional Framework for Evaluating Teachers’ Technology Adoption

When describing teachers’ adoption of new technology, existing conceptual models tend to focus on a single dimension, either quantitative, such as the frequency or amount of technology use, or qualitative, such as the way in which a given technology influences teachers’ instructional practices. Such one-dimensional approaches can fail to adequately describe similarities and differences in technology adoption between teachers. In this chapter, we describe a conceptual framework for teacher technology adoption that incorporates two dimensions: a quantitative description of technology adoption and a qualitative description of technology adoption. Then, apply this framework to data from a year-long mixed-methods study of Earth science teachers in the Denver Public Schools district who adopted a new Web-based instructional planning system called the Curriculum Customization Service. Our findings suggest that this two-dimensional framework offers useful insights for those seeking to evaluate not only “how much” or “how frequently” teachers adopt a new technology but also how a new technology relates to teachers’ instructional practices.

Describing Digital Objects: A Tale of Compromise

The Alliance Digital Repository (ADR) is a consortial digital repository service developed by Colorado Alliance of Research Libraries (Alliance). This article details how a standard descriptive metadata policy for repository records developed, and how that policy is currently being implemented. All digital objects in the ADR are required to have MODS and OAI-Dublin Core metadata that conform to certain minimum requirements. To help members meet the requirements, Alliance staff and the ADR Metadata Working Group, using tools available in the Fedora/Fez repository environment, have developed a customized set of core ADR material type templates in XSD form.

Algorithms for robust knowledge extraction in learning environments

This paper presents preliminary results on a generalizability study that was carried out to evaluate the robustness of a knowledge extraction algorithm.

Assessing the uptake of persistent identifiers by research infrastructure users

Significant progress has been made in the past few years in the development of recommendations, policies, and procedures for creating and promoting citations to data sets, software, and other research infrastructures like computing facilities. Open questions remain, however, about the extent to which referencing practices of authors of scholarly publications are changing in ways desired by these initiatives. This paper uses four focused case studies to evaluate whether research infrastructures are being increasingly identified and referenced in the research literature via persistent citable identifiers. The findings of the case studies show that references to such resources are increasing, but that the patterns of these increases are variable. In addition, the study suggests that citation practices for data sets may change more slowly than citation practices for software and research facilities, due to the inertia of existing practices for referencing the use of data. Similarly, existing practices for acknowledging computing support may slow the adoption of formal citations for computing resources.

Teacher sociality and information diffusion in educational digital libraries

Understanding the social aspects of digital resource utilization is an area of active research. In this study, we examine the digital library resource utilization and social behaviors of middle and high school Earth Science teachers of a large United States urban school district. We present the results of three analysis based on teachers using an online curriculum planning tool called the Curriculum Customization Service (CCS), and examine the social networks that emerge among the participating teachers. We explore these networks in the context of the digital library resources that were part of the CCS and the use of socio-centric features around those resources. Our initial findings show promise toward developing a broader understanding of the social networks of teachers, their behaviors around and usage of digital library resources, as well as the diffusion of information through those networks.

Automated approaches to characterizing educational digital library usage: linking computational methods with qualitative analyses

The need for automatic methods capable of characterizing adoption and use has grown in operational digital libraries. This paper describes a computational method for producing two, inter-related, user typologies based on use diffusion. Furthermore, a case study is described that demonstrates the utility and applicability of the method: it is used to understand how middle and high school science teachers participating in an academic year-long field trial adopted and integrated digital library resources into their instructional planning and teaching. Use diffusion theory views technology adoption as a process that can lead to widely different patterns of use across a given population of potential users; these models use measures of frequency and variety to characterize and describe such usage patterns. By using computational techniques such as clickstream entropy and clustering, the method produces both coarse- and fine-grained user typologies. As a part of improving the initial coarse-grain typology, clickstream entropy improvements are described that aim at better separation of users. In addition, a fine-grained user typology is described that identifies five different types of teacher-users, including “interactive resource specialists” and “community seeker specialists.” This typology was validated through comparison with qualitative and quantitative data collected using traditional educational field research methods. Results indicate that qualitative analyses correlate with the computational results, suggesting automatic methods may prove an important tool in discovering valid usage characteristics and user types.

Teaching box builder: customizing pedagogical contexts for use of digital library resources in classrooms

This paper introduces the Teaching Box Builder application that, as being implemented, supports the development of pedagogically rich inquiry-based Earth science lessons using digital library resources

Conceptual personalization technology: promoting effective self-directed, online learning

This paper presents an empirical learning study using a prototype system designed to provide fully automatic, domain-independent conceptual personalization algorithms The prototype system, the Customized Learning Service for Concept Knowledge (CLICK), was implemented as an adaptive essay writing environment in a scientific domain Results demonstrate that conceptual personalization promotes deep metacognitive strategies during online learning, and these strategies correlate with deep domain understanding.

A personalized learning environment

We report on the current research activities and results obtained through the Concept Learning service for Concept Knowledge (CLICK) and present a demonstration of the system. This poster session will focus on a demonstration of the CLICK system and the results of the learning study