methodology Science Teachers as Public Health Educators: How Has the COVID-19 Pandemic Reshaped the Roles and Experiences of K-12 Science Teachers? (COVID 2022)
Methodology
The methodology for this study involved developing a teacher survey and interview protocol, recruiting participants, collecting data, and analyzing data. This section provides a description of each of these components of the methodology, as well as important information for interpreting the findings of the study while reading the report.
Instrument Development
Survey
The survey, which included a mix of items that had been used in the previous study and new items, covered a broad range of topics, including how teachers addressed COVID in their instruction, how their teaching about COVID changed over time, and factors that exerted the greatest influence on their teaching about COVID. Additionally, the survey gathered information about the impacts of COVID on science teachers themselves, including the manageability of workload, physical/mental wellness, and job satisfaction. For former teachers (i.e., those who left the profession after the 2019–20 school year), a subset of survey items focused on the ways in which COVID impacted their decision to leave the profession.
Once survey items had been drafted, an abridged version was piloted with a sample of 30 teachers, 10 from each grade band (elementary, middle, high). Pilot survey responses informed the revision of existing survey items and addition of new items. The final version of the survey was programmed into an online administration platform and tested to ensure it functioned according to design specifications, including different pathways for current and former teachers and other skip logic. The final version of the survey is included in Appendix A.
Interview Protocol
The teacher interview protocol focused on many of the same topics as the teacher survey and was intended to elicit additional information about the varied contexts in which teachers worked. The interview protocol was piloted with a small number of teachers
Study Recruitment
HRI recruited teachers for the study from two sources. First, we sent emails to all teachers who participated in the previous COVID study, as well as science teachers subscribed to a mailing list maintained by HRI. We also enlisted the help of the National Science Teaching Association (NSTA), which has a membership of over 55,000 teachers and a mailing list of over 200,000. NSTA sent a description of the study and link to the study registration form to a substantial portion of their members. Between the two recruiting strategies, we registered just under 2,000 current and former K–12 science teachers for the study.
Data Collection
Survey
Administering the survey to teachers before the end of the 2021–22 school year was important for achieving an adequate response rate. The survey was launched in May 2022 and closed at the end of August 2022 with a response rate of 56 percent.2
The study timeline and budget precluded drawing a nationally representative sample for the teacher survey. Instead, HRI attempted to register and survey enough teachers so that a representative group could be constructed from respondents for analysis purposes. We received completed surveys from 1,081 current and former teachers, which was not a large enough sample to exclude any without risking large standard errors. However, the teachers in our sample are quite similar on most demographic factors to the greater population of teachers, according to demographic data from the 2018 National Survey of Science and Mathematics Education.3 These comparisons are included in Appendix B.
HRI segmented the sample of current teachers into elementary, middle, and high school grade bands. A small number of former teachers also completed the survey. The number of teachers in each category is shown in Table 1.
Interviews
Teachers who completed the survey were asked if they were willing to participate in a follow-up interview. HRI drew a purposive sample from those who agreed to participate, with the goal of balancing the sample in terms of grade band, life science/non-life science teaching assignment (at the middle and high school levels), and region of the country. Within these strata, teachers were randomly selected and contacted on a rolling basis throughout the data collection period. When a selected teacher declined or did not respond, a similar backup was contacted as a replacement. Using this approach, 40 of the 80 teachers contacted were interviewed.
Data Analysis
Survey
To facilitate the reporting of large amounts of survey data, and because individual survey items are potentially unreliable, HRI used factor analysis to identify survey items that could be combined into “composites.” Each composite represents an important construct related to COVID in science education and is reported on a scale from 0 to 100. A detailed description of the composite creation and composite definitions are included in Appendix C.
Although not designed primarily as an equity study, the survey also provides some data about the extent to which students across the nation had equitable opportunities to learn about COVID. Data were analyzed by four factors4 historically associated with differences in educational opportunities:
- Percentage of students in the school eligible for free/reduced-price lunch (FRL)
Teachers were grouped into 1 of 4 categories based on the percentage of students in the school eligible for FRL. The categories were defined as quartiles within groups of schools serving the same grades (e.g., schools with grades K–5, schools with grades 6–8). Cut points for these quartiles are included in Appendix C. - Percentage of students in the school from historically underrepresented minority (URM) groups
Teachers were grouped into 1 of 4 quartiles based on the percentage of students in the school from race/ethnicity groups historically underrepresented in STEM (i.e., American Indian or Alaskan Native, Black or African American, Hispanic or Latino, Native Hawaiian or Other Pacific Islander, multi-racial). Cut points for these quartiles are included in Appendix C. - Community type
Teachers were coded into 1 of 3 types of communities:- Urban: central city;
- Suburban: area surrounding a central city, but still located within the counties constituting a Metropolitan Statistical Area (MSA); or
- Rural: area outside any MSA.
- Political leaning
Teachers were coded into 1 of 2 categories based on whether the majority of voters in their school’s county voted for the Democratic presidential candidate or Republican presidential candidate in the 2020 election.
Equity analyses of selected survey items and composites include all current teachers (grades K–12) with available equity data.Teachers were presented with several open-ended items throughout the survey. Responses to these items were analyzed using an emergent coding scheme, where responses were classified into one or more different categories. Responses were then analyzed by frequency within grade bands.
Interviews
Interview data were used to write a vignette report,5 which provide illustrative examples of the interplay among numerous factors that influenced teachers’ response to COVID. Teacher quotes from the interviews are also interspersed throughout this report to supplement the survey findings.
Organization of This Report
The results of the study, like those from any survey based on a sample of a population (rather than on the entire population), are subject to sampling variability. The sampling error (or standard error) provides a measure of the range within which a sample estimate can be expected to fall a certain proportion of the time. For example, survey findings may indicate that 15 percent of elementary teachers gave a lecture when they addressed COVID with their students. If the sampling error for this estimate was 3 percent, then, according to the Central Limit Theorem, 95 percent of all possible samples of that same size selected in the same way would yield estimates between 9 percent and 21 percent (that is, 15 percent ± 2 standard error units). The standard errors for the estimates presented in this report are included in parentheses in the tables (see Figure 1).
A summary of each table highlighting or interpreting the results precedes the table. The summary points out only those differences that are substantial as well as statistically significant at the 0.05 level.6 When full distributions of responses are shown, differences among grade bands or timepoints were tested using the Chi-square test of independence. Post-hoc tests to determine which response option(s) are different were not conducted, but likely reasons for any observed difference are mentioned in the text.
Comparisons were made between groups within each equity factor. For FRL and URM, comparisons were made between the highest and lowest quartiles. For community type, comparisons were made among all three locales (urban vs. suburban, urban vs. rural, and rural vs. suburban). For political leaning, comparisons were made between Democratic- and Republican-leaning counties.
1Throughout the remainder of this report, we will use the term “COVID” to refer to both the virus and the disease. However, we will use the individual terms if we are specifically referring to one or the other.
2 Teachers who registered for the study received an initial email with instructions for accessing and completing the survey. Several email reminders were sent, both during the school year and over the summer, to those who had not yet completed the survey.
3 Banilower, E. R., Smith, P. S., Malzahn, K. A., Plumley, C. L., Gordon, E. M., & Hayes, M. L. (2018). Report of the 2018 NSSME+. Horizon Research, Inc.
4 Three factors—percentage of students eligible for FRL, percentage of students from URM groups, and community type—are school-level factors. The fourth—political leaning—is a county-level factor. For analysis purposes, all factors were assigned to individual teachers’ responses.
5 Trygstad, P. J., Harper, L. A., Bruce, A. D., Safley, S. E., & Smith, P. S. (2023). Teaching Science During the COVID Pandemic: K-12 Teachers Tell Their Stories. Horizon Research, Inc.
6 Given the exploratory nature of this report, all tests of significance were conducted without controlling the Type 1 error rate.