In developing the ATLAST assessments, we found repeated review of items by content experts to be valuable. The primary function of such reviews is to ensure the content accuracy of items. Each item should be reviewed with respect to:
- Scenarios/examples presented In cases where items present natural or realistic situations, these should accurately reflect the real world (e.g., a polar bear preying on seals rather than penguins). In items featuring hypothetical cases, all appropriate assumptions/instructions should be explicitly stated (e.g.,. air resistance experienced by a moving cart is to be ignored). Review by a content expert can provide assurance that these situations are presented with the accuracy needed to assess the target content. This point applies to graphic elements as well as text.
- Validity of stated/implied generalizations Extensive content knowledge may be helpful in identifying inaccuracies resulting from over-generalization. For instance, in assessing whether students have the misconception that plants do not metabolize sugars as animals do, a distractor might state “Plants and animals use food in different ways to grow.” A precocious student may realize that food is used in plant tissue growth largely in the formation of cellulose whereas animal growth does not include cellulose. Such a student would interpret the intended distractor as a correct statement. Inclusion of a content expert heads off such problems.
- Field-specific use of terms Within the context of a specified content area, particular terms may have significantly specialized or restricted meanings (e.g., “plate” in plate tectonics and “food” in flow of matter and energy). The input of a content expert can clarify subtleties associated with these definitions. This consideration is important to the principle of maximizing comprehension (see Item writing principles) to produce items that best assess the target ideas.
- Prerequisite content knowledge The requirement for prerequisite knowledge in grasping the targeted ideas should be considered during the content clarification stage of item development. However, assessment development also benefits from a review of each item with the intention of determining what prerequisite knowledge is required for correct interpretation. The perspective of a content expert is valuable at both stages because it is shaped by in-depth and repeated examination of interrelationships among the relevant concepts.
Content expert review is also important for providing adequate coverage of the content domain. Associating individual items with corresponding ideas in the content domain is key to determining coverage. Thorough and accurate association of assessment items to the specified ideas is sometimes not straightforward. Feedback from a content expert may be important for a making all appropriate associations.