Methodology

Most LEOs Serve In Small Jurisdictions, While Most
Registered Voters Are Served By A Small Percentage Of LEO

Although all LEOs share some basic responsibilities, there is enormous variation in the scope of their roles, the kinds of communities they serve, the size of their operations, and much more. For instance, although there are over 8,000 LEOs in the country, almost half of the registered voters in the country are served by just 2% of LEOs.

These create unique challenges for a creates a unique for conducting a random sample survey of local election officials. The first challenge is sampling: creating a comprehensive list of jurisdictions and LEOs and choosing a sampling method from such a uniquely distributed population. The second challenge is to create a survey questionnaire that captures the collective experience of LEOs across the country but also provides sufficient nuance to adequately reflect the diverse legal, administrative, and political environments that constitute the “complex quilt” of American elections. Our surveys have been iterative, building upon the expertise we have gained in each subsequent year. Over the past four years, EVIC has developed the state of the art expertise for sampling and surveying local election administrators.

Our work rests on the foundations built by scholars and researchers who preceded us, most notably Eric Fischer’s groundbreaking work at the Congressional Research Service in 2004, 2006, and 2008. Other innovators in the field include Dr. David Kimball at the University of Missouri, St Louis; Dr. Barry Burden at the Elections Research Center, University of Wisconsin, Madison; Rebecca Gambler at the Government Accounting Office, and many other researchers, election officials, and allies in the elections and democracy space.

Read More About Sampling

Sampling

We have improved our sampling procedures each year to try and control for the impact of jurisdiction size as measured by the total number of registered voters. Much of the credit for the sampling methodology and sampling weights lies with Jay Lee (Reed ’19), currently a research associate at the Sightline Institute’s  democracy program

The 2018 and 2019 LEO surveys were significant learning opportunities to develop best practices for sampling, surveying, and producing accurate statistical estimates across the “complex quilt” (Brown, Hale, and King 2020) of American elections and election administration.

The 2018 Local Election Official Survey utilized a sampling frame built off of a comprehensive list of all local election officials in the country obtained from the US Vote Foundation. This was matched with registered voter totals from the 2018 Election Administration and Voting Survey (EAVS) administered by the US Election Assistance Commission. We also used local contacts and websites when necessary for sub-county data. We drew a sample of 3,000 jurisdictions, using sampling proportional to the registered voter population – that is, a given small jurisdiction is less likely to be sampled than a given large jurisdiction.

For the 2019 Survey of Local Election Officials, we developed a new sampling frame by building off of the 2018 EAVS and scraping data from Secretary of State websites or similar sites, state by state. We made a series of edits to this list of jurisdictions to create proper entries for each local jurisdiction that included a local administrator responsible for election administration. This resulted in a sampling frame with 7,834 local jurisdictions. We drew a sample of n = 3,000 from this list using the random systematic sampling method, with inclusion probabilities proportional to the number of registered voters in each jurisdiction. This ensured that all of the largest jurisdictions (> 15,000 registered voters) were included in the sample, and we collected a representative sample of jurisdictions of smaller sizes.

In 2020 and 2021, we used the same sampling frame (N = 7,834) and random systematic sampling method to draw new samples of n = 3,000 LEOs (2020) and n=3118 (2022). Sampling was conducted with inclusion probabilities proportional to the number of registered voters in each jurisdiction, with the one change in 2022 of sampling all LEOs serving minority-majority jurisdictions (hence the added 118 in the sample).

Weights are based on the design weights from our sampling method (i.e. smaller jurisdictions, with less chance of being sampled, received higher weights). The weights are trimmed to avoid individual small jurisdictions having inordinately large weights, and post-stratified our responses to ensure the weighted proportion of responses in each size bin (0 to 5,000 registered voters; 5,001 to 25,000; 25,001 to 100,000; 100,001 to 250,000; more than 250,000) was equal to the overall proportion of each size bin in the full LEO population.

A detailed discussion of the sampling and weighting methodology is provided by Lee and Gronke (2020).

Read More About Questionnaires, Survey Mode, and Response Rates

Questionnaire, Survey Mode, and Response Rates

The survey questionnaires were developed at Reed College in collaboration with our partners at Democracy Fund and an Advisory Board of academics and practitioners. In 2022, we also benefited from invaluable feedback from SSRS regarding question-wording, order, clarity, and other issues related to questionnaire quality in order to maximize response rates, reliability, and validity. In each year, the Reed College team also conducted some internal pretesting of questions among our selected election officials. The survey questionnaires are provided in the “Resources” section of this page.

The surveys have used different modes, as we learned how to maximize our response rates and because of the impact of the COVID-19 pandemic. The 2018 survey began as an online survey distributed LEOs after building an email list of 3000 officials (collected after sampling). Approximately half of the responses were collected online. Non-responding LEOs were then sent a paper survey via mail, making up the remaining half of the survey (n=1071). In general, we found that smaller jurisdictions were more likely to respond to paper surveys, while larger jurisdictions responded via online.

In 2019 the LEOS was administered entirely by mail with a paper survey (n=876). Extra outreach was made via email to increase response rates among the larger jurisdictions based on lessons learned in the 2018 survey. Due to the rapidly shifting challenges of the early pandemic, the 2020 LEOS was administered entirely online (n=857).

Finally, in 2022, we returned to the mixed-mode method that we felt worked best in 2018 and 2019–respondents were initially given the opportunity to respond online and a print follow-up was sent to respondents within two weeks (n=855). In all four surveys, the questionnaires were in the field in the middle of the summer to avoid the bulk of Primary Election activity, and well before the General Election.

We also conducted a set of in-depth interviews (IDIs) with LEOs in conjunction with the 2020 survey, to add a wider qualitative component to the open-ended responses that we had collected over multiple years of surveys. These interviews were conducted in September and October 2020 by Fors Marsh Group, with question guidelines developed by EVIC and Democracy Fund. Thirty LEOs were interviewed, selected from survey respondents to achieve a mix of jurisdiction sizes, tenures in office, geographic area, and elective/appointive history. Questions focused on career paths and planning, job satisfaction, compensation, and the impact of various election administration reforms. A report detailing the methods and findings of these IDIs is available below.