The ATLAST teacher assessments have strong evidence of validity and reliability.<\/p>\n
Three lines of evidence support the argument that the assessments are valid measures of teachers\u2019 knowledge of the targeted science concepts. First, cognitive interviews with teachers (see Writing Items and Cognitive Interviews<\/a>) established that teachers interpret the items as intended and that teachers must use their knowledge of content to answer the items correctly. Second, a panel of three content experts (e.g. individuals with a Ph.D. in physics) reviewed the assessment items (see Expert Review<\/a>) at three stages (see Development<\/a>) to ensure content accuracy. A panel also reviewed the final assessment and judged it to be an adequate measure of the content domain. Finally, dimensionality analyses (including both factor analysis and cluster analysis) indicate that all items on the assessment measure a single dominant trait.<\/p>\n HRI conducted a separate\u00a0investigation of teacher assessment validity.\u00a0 Each assessment was administered to approximately 100 middle school science teachers.\u00a0 Teachers with the lowest five scores and teachers with the highest\u00a0five scores were then interviewed by content area experts, without the experts knowing how the individuals scored.\u00a0 Based on the interview, each teacher was categorized as having either extensive or limited content knowledge.\u00a0 We then compared these determinations to the assessment scores as an indication of validity.\u00a0 Across 30 teachers, there was only one instance in which the expert’s categorization of the individual did not agree with the test score.<\/p>\n The ATLAST teacher assessments have strong internal reliability and test-retest reliability.<\/p>\n Using data from the final field test, the internal reliability of each assessment was calculated using both a classical approach (Cronbach’s alpha) and Item Response Theory (IRT).\u00a0 The IRT internal reliability for each teacher assessment is given below:<\/p>\n <\/p>\n The Cronbach alphas were similarly high.<\/p>\n To study test-retest reliability, ATLAST recruited approximately 100 middle grades science teachers for each assessment who took the test twice, two weeks apart with no intervening instruction. \u00a0The<\/span> test-retest reliability calculated for each assessment is given below.<\/p>\n Although no standards exist for minimum reliability, anything approaching 0.9 is considered good.[1]<\/a><\/p>\nReliability<\/h3>\n
\nInternal Reliability<\/h4>\n<\/div>\n
\n
Test-Retest Reliability<\/h4>\n
\n