Carol Barnes

Assessing the Validity of a Language Arts Instruction Log through Triangulation

In this study we attempted to illuminate why measures of instruction sometimes fail to meet discrete tests of validity. We used a triangulation strategy—multiple methods, data sources, and researchers—to investigate teachers’ and observers’ reports on a daily language arts log. Data came from a pilot study of the log conducted in 8 urban public elementary schools. Statistical results increased our confidence in the log’s ability to measure (a) instruction at grosser levels of detail, (b) instructional activities that occurred more frequently, and (c) word analysis instruction. Some qualitative evidence gave us greater confidence in the instrument, for example, when teachers differed from observers because they possessed background knowledge not available to observers. Other qualitative evidence illustrated dilemmas inherent in measuring instruction. Overall, we believe triangulation strategies provided a more holistic understanding of the validity of teachers’ reports of instruction than past validity studies.

Developing Instructional Leaders: Using Mixed Methods to Explore the Black Box of Planned Change in Principals’ Professional Practice:

Purpose: This study examines learning, and both cognitive and behavioral change among a sample of randomly assigned urban principals, half of whom participated in a sustained, district-based professional development program (DPD). Research Methods: Latent class analyses of daily log data, qualitative typology development, and case studies of change provide a rich portrait of the learning and change process. Findings: Few dramatic transformations of practice. Instead, principals attributed to the DPD a gradual refinement of existing practice through a process that allowed them to “break down” declarative knowledge to better understand its consequences for their work, but also provided knowledge structures, tools, and routines for reintegrating ideas from the program into strategically valuable procedural knowledge. Implications: Results suggest potential for developing principals’ competencies within continuing practice communities, but expectation of incremental rather than a dramatic “turn around” in principals’ leadership through program interventions.

Measuring the Learning-Centered Leadership Expertise of School Principals.

We as a field believe that school principals can acquire new expertise by participating in principal preparation and professional development programs; however, we have few methodologies to measure leadership expertise, especially expertise that links leadership to improved student learning. In this article, we present the results of a study that examines two instruments for measuring leadership expertise, principal surveys and open-ended scenarios. First we make the case regarding the need for measurements of expertise. Next we discuss the conceptual definitions of expertise in general and present the specific domains of leadership expertise we attempt to measure. Finally, we present the results of a study that implemented two measures of leadership expertise: principal surveys and open-ended scenarios. The descriptive statistics, correlations, and examples we present in this article offer mixed results regarding the strengths and weaknesses of the various methods to conceptualize and measure leadership expertise.

State Education Agencies’ Acquisition and Use of Research Knowledge for School Improvement

Over the last two decades, state education agencies (SEAs) have been given considerable responsibilities for improving low-performing schools and for adopting research-based practices in doing so. Yet we know little about how and where these organizations search for, select, and use research and other kinds of evidence. We examined these questions as they relate to school improvement designs and strategies in three SEAs using a combination of surveys and interviews conducted in 2010–11. We found that SEA staffs relied most heavily on their colleagues for information but that information often flowed across departments and offices, contrary to the usual image of the SEA as a segmented and siloed bureaucracy. A large number of external organizations were identified in SEA research advice networks and played a catalyzing role in the design or elaboration of research for policy. Although most sources were named by just one person, each SEA also had central internal staff who played an important role in brokering research on school improvement. Identifying and cultivating such influential actors, and connecting individuals who are now isolated or only weakly engaged in these communication networks, could create a more robust exchange of knowledge around school improvement.

How State Education Agencies Acquire and Use Research Knowledge for School Improvement

This chapter examines the organizing structures and search-incorporation processes by which research and other knowledge are acquired, spread, and used in three SEA’s school improvement strategies. Social network analyses, sociograms, and descriptive statistics using web-based survey data provide insights into broad patterns of knowledge flow, influence, and source-user connections within SEA’s knowledge networks. In-depth cognitive interviews provide rich portraits of how and why these patterns emerged. SEA staff were receptive to research and sought it from multiple external sources as well as through internal lateral connections to use in their school improvement strategies. Network connections brought different perspectives into the search, incorporation, and implementation process and created flexible structures for managing and generating knowledge more responsive to specific school improvement problems. Policy incentives and research providers should encourage more dissemination methods that go beyond transmitting research and, instead, also cultivate “core networks,” varied internal and external sources of expertise working together, to translate generalized findings into knowledge for use in specific contexts. Future research should track how the integrity of research is maintained during the social processes that we found could foster research use, to understand the trade-offs between fidelity to research principles and adaptation to contextual contingencies.