The collected MSP research on teacher professional learning communities was reviewed using a system developed by the MSP Knowledge Management and Dissemination (MSP-KMD) project and intended to ensure a transparent process with integrity and protections against bias in all phases. The process included steps for identifying, analyzing, and summarizing studies for the published review on MSP research on STEM PLCs.
Identifying Studies
A multi-step process was conducted to identify MSP-produced empirical research studies on STEM PLCs. MSP-KMD staff searched the electronic database of the MSP intranet (www.mspnet.org), looked through publications on the selected MSP public websites, and queried MSP project leaders.
The community of over 70 MSP projects is joined through a password-protected intranet system, which includes a database of MSP-related papers such as advocacy pieces, press releases, annual reports, conference presentations, and research studies on a variety of topics within K-16 STEM education. The MSPnet.org database of papers serves the purpose of providing ready access to MSP products, but does not include the powerful search functionalities of more robust online article databases (such as EBSCO or ERIC). Therefore, MSP-KMD staff engaged in targeted approaches to finding research on STEM PLCs. As an initial screen, staff scanned the database through searches of words related to STEM PLCs (for example: “professional learning communities,” “inquiry groups,” “teacher teams,”) and reviews of collections of papers on topics related to STEM PLCs.
The search of MSP studies was also guided by the input of NSF-program officers who oversee the MSP program. Seven MSP projects that featured STEM PLCs as part of their work were identified by program officers; all papers in the MSPnet.org database from these seven projects were screened. Next, MSP-KMD staff examined the public websites of 15 MSP projects that had papers identified in the initial screen, and sent direct requests to MSP project leaders for any additional reports on STEM PLC research.
The executive summaries of all papers identified in the initial search were screened using the following criteria:
- At least one of the research questions or topics concerned STEM PLCs
- The study addressed an explicit question or topic concerning STEM PLCs or made explicit reference to STEM PLCs in the findings
- The subjects of study included in-service teachers
- The study took place in the context of K-12 education (although it could also include higher education as long as it primarily focused on K-12 education)
Additional studies were screened out as reviewers read the full-text copies of the studies as part of the review process. Eventually, nine studies were identified for the review.
Summarizing studies and applying standards of evidence
The nine studies that met the inclusion criteria were summarized and reviewed by a member of the MSP-KMD staff. The written summaries consisted of:
- an abstract, describing the study context, intervention (if present), and any measures or instruments used;
- the findings of the study;
- the conclusion and implications of the study; and
- a checklist describing the study participants and the context in which the study was conducted.
After each study was summarized, it was reviewed using a set of standards of evidence for empirical research. The MSP-KMD project developed the standards of evidence to operationalize principles for conducting empirical research and social science. The standards of evidence drew on a number of writings about research rigor, quality, and reporting, including efforts to address quantitative, qualitative, and mixed methodologies. A panel of mathematics and science education researchers, research methodologists, and mathematics and science education leaders assisted the MSP-KMD staff in the development of the standards of evidence to help ensure not only their quality, but their broader utility. (See Codebook for Standards of Evidence for Empirical Research, which details the standards and their application.)
The purpose of applying standards of evidence to the studies was to identify the contributions of each study to the field’s knowledge base. Contributions were characterized in terms of what is known from the findings based on the substance of the study, and the confidence that can be placed in the findings based on the quality of the study. Applying the standards of evidence was not intended to make “good/bad” or “in/out” judgments on studies, or to suggest that all studies should be strong on every standard. Rather the application of standards of evidence was conducted to aid understanding of the strengths and limitations of each study’s contributions to the empirical knowledge base.
The application of standards of evidence was audited by an external evaluator. The evaluator reviewed the set of nine MSP studies examining STEM PLCs. The results of the evaluator’s reviews were compared with the MSP-KMD project’s reviews, showing high agreement overall between the two groups’ application of standards of evidence.
The findings from the MSP studies were categorized into four areas in order to situate their contributions to the knowledge base on STEM PLCs. These areas consisted of studies that examine:
- the nature of the experience and PLC functioning;
- the relationship between an intervention and teachers’ knowledge, attitudes, and/or beliefs;
- the relationship between an intervention and teachers’ practice; and
- the relationship between an intervention and student outcomes.