The MSP Knowledge Management and Dissemination project developed a system for conducting reviews of empirical research intended to ensure a transparent process with integrity and protections against bias in all phases. The process for reviewing empirical research in the literature is outlined below in two parts. First, a search was conducted to identify empirical research studies of teachers’ content knowledge in mathematics or science. Second, studies were summarized and reviewed using standards of evidence as a basis for evaluating the strength of evidence supporting particular findings on teachers’ mathematics and science content knowledge.
Identifying Studies in the Research Literature
The parameters for search and selection of studies were intended to yield a set of studies with a tight focus on the mathematics and science content knowledge of in-service teachers. To be included in this review, each study had to meet all of the following criteria:
- Teachers’ mathematics or science content knowledge was studied empirically, through a specific measure (e.g., multiple choice test, open-response written items, interviews) or through systematic analysis of samples of teachers’ work, including classroom practice;
- The subjects or participants in the study were practicing in-service teachers within grades Pre-Kindergarten through 12; and
- The study was published since 1990.
Search parameters for the review were identified by initially searching on a set of keywords in the Education Resource Information Clearinghouse (ERIC) database, including:
- Teacher knowledge and mathematics/science;
- Teacher content knowledge and mathematics/science; and
- Pedagogical content knowledge and mathematics/science
Results of these searches were examined to identify several studies that met the criteria for inclusion. Once identified, the ERIC descriptors for each of these studies were recorded. From these descriptors, the complete set of search parameters was developed and entered as a keyword search into both ERIC and EBSCO Professional Development Collection. (See Search Parameters.) [PDF 25K]
Between mathematics and science, the searches yielded close to 2,000 articles. A member of the MSP-KMD team read the abstract, and skimmed the study if needed, for each study identified through this search process to determine its initial inclusion based on the criteria for the review. Close to 90 percent of the articles were eliminated in this initial screening, in most cases because the study did not include a specific measure of teacher content knowledge or a systematic analysis of samples of teachers’ work. Others were in fact not studies (e.g., they were advocacy pieces) and/or dealt solely with pre-service teachers. Further screening of the remaining studies resulted in the inclusion of 86 studies in mathematics and 98 in science, some with multiple parts and/or multiple publications.
Summarizing Studies and Applying Standards of Evidence
Each study that was included was summarized and reviewed by a member of the Knowledge Management and Dissemination staff. The summary consisted of:
- An abstract, describing the study, including the methods or measures used to assess teachers’ content knowledge and other key constructs or variables;
- The findings of the study;
- The conclusions and implications of the study;
- A checklist describing the study’s participants and the context in which the study was conducted; and
- The theoretical perspective(s) on teachers’ mathematics/science content knowledge represented in the study, if discernable.
Summarizers also ensured that the studies met the inclusion criteria based on their more complete reading of the articles and reports.
Following the preparation of the summary, each study was reviewed using a set of standards of evidence for empirical research. The KMD project developed the standards of evidence to operationalize principles for conducting empirical research in education and social science. The standards of evidence drew on numerous writings about research rigor, quality, and reporting, including efforts to address quantitative, qualitative, and mixed methodologies. A panel of mathematics and science education researchers, research methodologists, and mathematics and science education leaders assisted the KMD staff in the development of these standards of evidence to help ensure not only their quality, but also their broader utility. (See Codebook for Standards of Evidence for Empirical Research [PDF 188K], which details both the standards and their application.)
The purpose of applying standards of evidence to the studies was to identify the contributions of each study to the field’s knowledge base. Contributions were characterized in terms of what is known from the findings based on the substance of the study, and the confidence that can be placed in the findings based on the quality of the study. Applying the standards of evidence was not intended to make “good/bad” or “in/out” judgments on studies, nor to suggest that all studies should be strong on every standard. Rather the application of standards of evidence was conducted to aid understanding of the strengths and limitations of each study’s contributions to the knowledge base.
The application of standards of evidence was audited by an external evaluation team from the Consortium for Policy Research in Education (CPRE) at the University of Pennsylvania. The CPRE team randomly selected and independently reviewed a subset of 12 research studies examining teachers’ mathematics or science content knowledge using the KMD project’s standards of evidence. The results of the CPRE team’s reviews were compared with the KMD project’s reviews, showing high agreement overall between the two groups’ application of standards of evidence.
The research studies from the literature were characterized into five areas in order to situate their contributions to the knowledge base on teachers’ mathematics and science content knowledge:
- Studies to measure the nature, extent, or elements of teachers’ mathematics/science content knowledge;
- Studies of factors or interventions that contribute to or influence teachers’ mathematics/science content knowledge;
- Studies of how the practice of teaching contributes to or influences teachers’ mathematics/science content knowledge;
- Studies of the relationship between teachers’ mathematics/science content knowledge and their teaching practice, including both classroom instruction and other aspects of teachers’ professional lives; and
- Studies of the relationship between teachers’ mathematics/science content knowledge and outcomes for their students, primarily mathematics/science achievement.
As an external check on the KMD project’s work to situate research on teachers’ mathematics and science content knowledge, the CPRE team audited the KMD project team’s narrative documentation of what is known from empirical research for all studies that examined factors or interventions that contribute to or influence teachers’ mathematics content knowledge. The CPRE team found that the narrative documentation accurately depicted and situated the studies’ findings, and identified the relevant strengths and limitations of each study’s methodology.