DATA IN ACTION
Null modeling approaches with applications in international large-scale educational assessments baseline The explored two applications of null baseline model comparisons. The first application is more methodological and focuses on thesis
students were not very consistent in their random response behavior across the questionnaire, suggesting that random responding is likely not a general trait. Ultimately, these studies hopefully increase awareness about the need for data quality monitoring for the survey part of international large-scale educational assessments, while the specific methodology used might offer a useful tool in the context of measurement validity and sensitivity of inferences. ■
model fit evaluation with the comparative fit index. The second application addresses the issue of random response behavior in the TIMSS 2015 student questionnaire throughout four empirical studies. In the context of low-stakes assessments, like TIMSS, where there are no direct consequences for the participating students, the quality of students’ responses and whether these responses still provide valid information about their opinions and beliefs can be a concern. The aim of the studies was to explore random response behavior, a type of invalid response behavior often associated with the low- stakes context, and provide insight into the prevalence, impact, and variability of this behavior across countries and scales, as well as the influence of questionnaire and student characteristics. A mixture model approach was adopted to distinguish between two different types of responders: (1) students who respond consistently to the items on a scale in line with a traditional latent variable measurement model; and (2) students who provide unrelated responses that are more in line with a null baseline model. The results showed that the average prevalence of random responders was estimated at 7.5%, ranging from 0 to 38% across different countries and scales. When comparing the two versions of the eighth-grade student questionnaire, there was no support for an absolute effect of questionnaire length. However, a positive effect for scale position was found, with a 5% increase in random responding for both the shorter and longer version. Yet, scale character turned out to be more important, where scales related to students’ confidence in mathematics and science showed an increase of 9% in random response behavior. Furthermore, it was found that certain groups of students were more likely to be identified as random responders (i.e., students in higher grades or male students, students reporting to have fewer books, or speaking a language different from the test language at home). At the individual level, the results indicated that
Saskia photo credit: Shane Colvin/University of Oslo
Bruce H. Choppin Memorial Award
IEA offers two annual awards to recognize high-quality empirical research that makes use of IEA data. These awards were established in commemoration of the significant contributions that Bruce H. Choppin and Richard M. Wolf made to IEA. The IEA Research Awards are held annually and the deadline for applications is the 31 March of each year.
Find out more via iea.nl/about/opportunities/award
59
Made with FlippingBook Online newsletter maker