There are unique challenges to conducting a random sample survey of local election officials. The Elections & Voting Information Center (EVIC) at Reed College has developed scientifically valid methods for sampling, surveying, and reporting results that are generalizable to the population of local election officials.
Most LEOs serve in small jurisdictions, while most registered voters are served by a small percentage of LEOs
The first challenge is what we call the 75:8 problem. Although all local election officials (LEOs) share some basic responsibilities, there is enormous variation in the scope of their roles, the kinds of communities they serve, the size of their operations, and much more. As shown in the figure, 75% of LEOs service 8.4% of all registered voters, while 8.1% of LEOs serve 75% of all voters. Another way to express this distributional feature: half of the registered voters in the country are served by just 2% of LEOs. Furthermore, 3,400 LEOs work in two states (Michigan and Wisconsin) and another 1,500 administer elections in towns, villages, and municipalities in New England.
This means that simple random sampling cannot be used for a survey of local election officials, or the resulting sample will be heavily skewed toward a non-representative set of states (and in just two regions). As described in more detail below, we on unequal probability sampling to achieve representativeness of the EVIC LEO Surveys.
The second challenge is to create a survey questionnaire that captures the collective experience of LEOs across the country but also provides sufficient nuance to adequately reflect the diverse legal, administrative, and political environments that constitute the “complex quilt” of American elections. In addition, our experiences and past survey reports have shown that the elections community is loathe to admit that they face resource, technology, or staffing limitations that present barriers to administering safe, secure, and accessible elections. We have created over the years a set of innovative questions that provide avenues for administrators to identify areas where procedures and practices can be improved.
Ultimately, the EVIC LEO Surveys have been iterative and evolutionary, building upon the expertise we have gained in each subsequent year.
The result is state-of-the-art expertise for sampling and surveying local election administrators.
Our work rests on the foundations built by scholars and researchers who preceded us, most notably Eric Fischer’s groundbreaking work at the Congressional Research Service in 2004, 2006, and 2008. Other innovators in the field include David Kimball, PhD at the University of Missouri, St Louis; Barry Burden, PhD at the Elections Research Center, University of Wisconsin, Madison; Rebecca Gambler at the Government Accounting Office, and many other researchers, election officials, and allies in the elections and democracy space.
Read More About Sampling
Sampling
We have improved our sampling procedures each year to try and control for the impact of jurisdiction size as measured by the total number of registered voters. Much of the credit for the sampling methodology and sampling weights lies with Jay Lee (Reed ’19), currently a research associate at the Sightline Institute’s democracy program
The 2018 and 2019 LEO Surveys were significant learning opportunities to develop best practices for sampling, surveying, and producing accurate statistical estimates across the “complex quilt” (Brown, Hale, and King 2020) of American elections and election administration.
Since 2019, we have developed our sampling frame by building off of the most recent EAVS survey, and then scraping data from secretary of state websites or similar sites, state by state. We made a series of edits to this list of jurisdictions to create proper entries for each local jurisdiction that included a local administrator responsible for election administration. This resulted in a sampling frame with 7,834 local jurisdictions. We drew a sample of n = 3,000 from this list using the random systematic sampling method, with inclusion probabilities proportional to the number of registered voters in each jurisdiction. This ensured that all of the largest jurisdictions (> 15,000 registered voters) were included in the sample, and we collected a representative sample of jurisdictions of smaller sizes.
In 2022 and 2023, we drew an oversample of majority-minority jurisdictions, which increased our sample by approximately 120.
Final sample weights are based on the design weights from our sampling method (i.e. smaller jurisdictions, with less chance of being sampled, received higher weights). The weights are trimmed to avoid individual small jurisdictions having inordinately large weights and post-stratified our responses to ensure the weighted proportion of responses in each size bin (0 to 5,000 registered voters; 5,001 to 25,000; 25,001 to 100,000; 100,001 to 250,000; more than 250,000) is equal to the overall proportion of each size bin in the full LEO population.
A detailed discussion of the sampling and weighting methodology is provided by Lee and Gronke (2020).
Read More About Questionnaires, Survey Mode, and Response Rates
Questionnaire, Survey Mode, and Response Rates
The survey questionnaires were developed by EVIC at Reed College in collaboration with our survey partners (SSRS in 2023 and 2022) and an advisory panel of academics and practitioners. The survey questionnaires are provided in the “Resources” section of this page.
The surveys have used different modes, as we learned how to maximize our response rates, adjust to meet budgetary limits, and respond to unexpected developments like the COVID-19 pandemic.
2018
2019
2020
2022
2023
Mode
Web, Mail
Mail
Web
Web, Mail
Web, Postcard
n
1071
876
857
856
886
Response Rate
36%
29%
29%
28%
29%
Broadly speaking, we have learned two lessons over the years. First, the response rates of a web-only instrument will be lower among the smallest jurisdictions, which show a reluctance to click on web links contained in emails. Second, larger jurisdictions are less responsive to print communications, we believe because of the difficulties in getting a piece of paper to the chief election officer of a larger jurisdiction. For them, web links in an email work best.
In summary, our experiences indicate that mixed-mode surveys, combining web and print modes, are optimal for local election official surveys. Web-only methods can be used but require substantial follow-ups using postcards and other communication methods.
We also conducted a set of in-depth interviews (IDIs) with LEOs in conjunction with the 2020 LEO Survey, to add a wider qualitative component to the open-ended responses that we had collected over multiple years of surveys. These interviews were conducted in September and October 2020 by Fors Marsh, with question guidelines developed by EVIC and Democracy Fund. Thirty LEOs were interviewed, selected from survey respondents to achieve a mix of jurisdiction sizes, tenures in office, geographic area, and elective/appointive history. Questions focused on career paths and planning, job satisfaction, compensation, and the impact of various election administration reforms. A report detailing the methods and findings of these IDIs is available below.
