Tsung-Hau Jen

Estimating the accuracies of journal impact factor through bootstrap

The journal impact factor (JIF) reported in journal citation reports has been used to represent the influence and prestige of a journal. Whereas the consideration of the stochastic nature of a statistic is a prerequisite for statistical inference, the estimation of JIF uncertainty is necessary yet unavailable for comparing the impact among journals. Using journals in the Database of Research in Science Education (DoRISE), the current study proposes bootstrap methods to estimate the JIF variability. The paper also provides a comprehensive exposition of the sources of JIF variability. The collections of articles in the year of interest and in the preceding years both contribute to JIF variability. In addition, the variability estimate differs depending on the way a database selects its journals for inclusion. In the bootstrap process, the nested structure of articles in a journal was accounted for to ensure that each bootstrap replication reflects the actual citation characteristics of articles in the journal. In conclusion, the proposed point and interval estimates of the JIF statistic are obtained and more informative inferences on the impact of journals can be drawn.

Development and Validation of a Multimedia-based Assessment of Scientific Inquiry Abilities

The potential of computer-based assessments for capturing complex learning outcomes has been discussed; however, relatively little is understood about how to leverage such potential for summative and accountability purposes. The aim of this study is to develop and validate a multimedia-based assessment of scientific inquiry abilities (MASIA) to cover a more comprehensive construct of inquiry abilities and target secondary school students in different grades while this potential is leveraged. We implemented five steps derived from the construct modeling approach to design MASIA. During the implementation, multiple sources of evidence were collected in the steps of pilot testing and Rasch modeling to support the validity of MASIA. Particularly, through the participation of 1,066 8th and 11th graders, MASIA showed satisfactory psychometric properties to discriminate students with different levels of inquiry abilities in 101 items in 29 tasks when Rasch models were applied. Additionally, the Wright map indicated that MASIA offered accurate information about students’ inquiry abilities because of the comparability of the distributions of student abilities and item difficulties. The analysis results also suggested that MASIA offered precise measures of inquiry abilities when the components (questioning, experimenting, analyzing, and explaining) were regarded as a coherent construct. Finally, the increased mean difficulty thresholds of item responses along with three performance levels across all sub-abilities supported the alignment between our scoring rubrics and our inquiry framework. Together with other sources of validity in the pilot testing, the results offered evidence to support the validity of MASIA.

Classical Test Theory

Classical Test Theory (CTT), also known as the true score theory, refers to the analysis of test results based on test scores. The statistics produced under CTT include measures of item difficulty, item discrimination, measurement error and test reliability. The term “Classical” is used in contrast to “Modern” test theory which usually refers to item response theory (IRT). The fact that CTT was developed before IRT does not mean that CTT is outdated or replaced by IRT. Both CTT and IRT provide useful statistics to help us analyse test data. Generally, CTT and IRT provide complementary results. For many item analyses, CTT may be sufficient to provide the information we need. There are, however, theoretical differences between CTT and IRT, and many researchers prefer IRT because of enhanced measurement properties under IRT. IRT also provides a framework that facilitates test equating, computer adaptive testing and test score interpretation. While this book devotes a large part to IRT, we stress that CTT is an important part of the methodologies for educational and psychological measurement. In particular, the exposition of the concept of reliability in CTT sets the basis for evaluating measuring instruments. A good understanding of CTT lays the foundations for measurement principles. There are other approaches to measurement such as generalizability theory and structural equation modelling, but these are not the focus of attention in this book.

An Ideal Measurement

When one undertakes the measurement of a latent trait, what are the desirable properties one would like to have for the measures?

Test Administration and Data Preparation

This chapter highlights some steps in test administration and the preparation of data for test analysis, including data collection, coding and data cleaning. The key to the success of test administration is careful planning and management. From printing the test booklets to conducting the test, every step needs to be closely managed and nothing should be left to chance. For example, there could be security issues related to test papers, and attendance issues related to participating students. The whole process calls for competent management skills.

Residual-Based Fit Statistics

This chapter discusses the commonly used residual-based fit statistics for the Rasch model.

Rasch Model (The Dichotomous Case)

There are many different IRT models. The simplest model specification is the dichotomous Rasch model. The word “dichotomous” refers to the case where each item is scored as correct or incorrect (0 or 1).

Multidimensional IRT Models

The incorporation of a population model discussed in Chap. 14 leads to an extension where the population distribution is a multivariate distribution rather than a univariate one.

Two-Parameter IRT Models

The Rasch model is sometimes also called the one-parameter IRT model in that the probability of success as a function of the ability \( \theta \) has only one parameter (the item difficulty parameter) estimated for each item, as shown in Eq. (10.1).

What Is Measurement

Most of us are familiar with measurement in the physical world, whether it is measuring today’s maximum temperature, the height of a child or the dimensions of a house, where numbers are given to represent “quantities” of some kind, on some scales, to convey properties of some attributes that are of interest to us.