By: Gary Segura & Matt Barreto
Latino Decisions estimates differ significantly from the network exit polls and this raises the question of whether we, or they, have a systematic flaw (complete Latino Decisions Election Polls posted here). Examination of the method of selection, clustered interviews, statistical properties of the resulting samples, uneven distribution of minority populations, and the low incidence of bilingual interviewing all suggest that the exit poll estimates for racial and ethnic sub-populations, reported in the National Exit Pool surveys, systematically underestimate Latino and African-American Democratic vote share by over-representing higher income, higher education, and more socially integrated minority voters than their share of the electorate warrants.
The problem of faulty exit poll data for Latinos is not new, yet very few in the media have expertise in polling Latinos and analyzing Latino vote data, and as a result are not in a position to assess on election night the veracity of the Latino results. In January 2005 in the aftermath of the 2004 Bush-Kerry election, Warren Mitofsky, the head of the 2004 National Exit Poll, wrote in a self-assessment of the erroneous exit poll data, “Some estimates differ by several points among certain demographic groups, most noticeably among Hispanics. These differences appear mostly among demographic groups that are both relatively small and those that tend to be geographically concentrated.” While Mitofsky’s NEP data suggested Bush won 44% of the Latino vote, the Latino-only focused exit poll by the William C. Velasquez Institute estimated Bush won just 31% of the Latino vote.
We believe the data convincingly show that the 2010 national exit poll severely misestimated the Latino vote, and almost any expert who studies the Latino vote would agree that the Latino Decisions Election Poll is far more accurate. (more on our methodology here)
I. Selection and Design Effects
Exit polls are designed to capture a sample of the electorate. The selection of precincts is done for the purpose of “representativeness” but, in most instances, violates a key norm of selection—it is not random. That is, a random selection of precincts would use a method called “probability proportionate to size” or PPS. The larger a precinct, the greater chance it will be selected, in proportion to the size of the electorate in that precinct. This is NOT the usual method of exit poll selection. Instead, the network exit polls use a “purposive” sample. That is, they select precincts for specific reasons, including a precinct that has a history of even division between the parties, socially diverse precincts, and others selected by virtue of being deemed particularly informative. This is not an unreasonable method for making outcome predictions, but it in no way guarantees a generally accurate reflection of the overall electorate in the sample. Evenly divided or socially diverse precincts are not the modal circumstances of most minority voters. Voters living in those precincts are not “representative.”
Comparing multiple representative telephone surveys of Latinos to the actual exit poll results in 2004, existing published scholarship suggests the exit poll was not accurate. In 2004, Latino Decisions’ Matt Barreto, co-authored an academic research article with Prof. Rodolfo de la Garza of Columbia University, and Prof. David Leal of the University of Texas, that examined the exit poll data in 2004 as compared to 10 different Latino surveys, stating, “we conclude that the pre-election data provide little evidence that President Bush received the 44% level of support from Latinos estimated by the 2004 exit polls.”
A second problem, related to the first, is that the precincts selected systematically under-represent homogenous minority precincts. Some number of minority voters will appear in the precincts strategically selected, but evenly divided or socially diverse precincts are not the modal circumstances of most minority voters. Voters living in those precincts are not “representative.” Since minority voters comprise a smaller share of the electorate, exit pollsters devote fewer resources to those estimations. This is not to say that no homogenous African American or Latino precincts are chosen, only that their representation in the exit poll precincts is less than would be necessary to capture a fair share of minority voters who live in high-density minority environments. As a consequence of severe residential segregation in the US, a very large share of minority voters live and vote in high-density minority areas. This problem can occur even with PPS selection, but it is guaranteed to occur when an exit pollster is attempting an accurate estimation of overall vote rather than the votes of a particular sub-group.
A third issue, endemic to any clustered sample, is the design effect. The design effect is a statistical estimate of lost “power” or lost precision in estimates that comes from the clustering of interviews rather than a random spread. Put in layman’s terms, we learn more from one interview in 200 different places within the area we are studying than we learn from 200 interviews in only one place. In the case of the exit polls, though the NEP results may report 1100 interviews, for example, those 1100 interviews are not randomly distributed throughout the state but, rather, are concentrated in some number of precincts—say 100. People who live near one another often exhibit similarities on several other dimensions. The design effect measures the degree of within-cluster correlation and uses that information to estimate how much precision we “lost” by speaking to too many similar people and too few different ones (that we’d have gotten in a truly random sample). While the N may have been 1100 in this state, the “effective N” is considerably smaller. The practical effect is that margin-of-error calculations for exit polls that do not correct for the design effect are over-estimating the precision of their results.
Since the NEP does not release its data for some time after an election, we cannot compute the design effect for 2010. For now, it is important only to know that if, for example, NEP interviewed 400 African Americans in a state with a million or more African American members of the electorate, the effective N is significantly smaller and the margin-of error is not ±4.9%, as would be the case with a truly random sample, but something considerably greater. In 2004, Mitofsky admitted that clustering effects can particularly effect Latino data, “A detailed look at the distribution of plurality Hispanic precincts in the National Exit Poll sample demonstrates how this clustering effect can influence the estimate of Hispanic voting in the National Exit Poll.” He concludes, “If we want to improve the National Exit Poll estimate for Hispanic vote we would either need to drastically increase the number of precincts in the National Sample or oversample the number of Hispanic precincts.”
The result of this design and the attendant issues is two-fold. First, the number of interviews for minority groups is generally small and, therefore, the confidence intervals around any estimation tend to be large. Note, as well, that the margins-of-error reported actually understate the breadth of uncertainty in exit polls because of the design effect we mentioned above—the clustering of cases reduces the effective sample size by artificially constraining variation.
Second, those minority voters who are interviewed are more likely to come from low-density, heterogeneous or predominantly white precincts than should be the case. Assimilated, middle-class, and English dominant respondents are over-represented while poorer, socially segregated, and Spanish dominant respondents are underrepresented. The result, inevitably, is a conservative or pro-GOP bias. The size of the bias can vary across states based on the size of the sub-samples and how skewed the precinct selection actually was, but in nearly all cases (given current levels of residential segregation and the patterns of relationships between observable social covariates and two-party vote) the bias will tilt the estimate in a conservative direction. Actually, more specifically, the bias will tilt against the modal Latino position (which has been under-surveyed). Usually that is less liberal and more conservative, but in Florida, due to the GOP identity of the powerful Cuban national-origin population, the tilt is in the other direction (which is exactly what we observe).
II. Spanish Language Interviewing
More than 60% of Latino adults in the US are immigrants and 40% of the citizen population is immigrant. The vast majority of those immigrants are Spanish-dominant, even if they have achieved English fluency as a precursor to naturalization. Political surveys occasionally use esoteric terms and vocabulary that requires more than a functional fluency (this is, indeed, occasionally a problem even from native English speakers). Fully bilingual adults, if Spanish was the language of primary education, are often more comfortable in Spanish.
When Latino Decisions polls registered voters at random, we offer all respondents an immediate opportunity to speak Spanish. Since our interviewers are bilingual, this does not require a call back or even a delay—the interviewer proceeds in Spanish without interruption. Since we use computer-assisted telephone interviewing protocols, we can even switch back and forth if the respondent does not understand a question on one language or the other. While the share varies from state to state and over time, we generally get a volunteered Spanish interview rate of around 40% nationally which, not surprisingly, is very close to the share of the Latino electorate that is foreign born and, considering the number of children born to foreign born parents, is significantly smaller than the share of US Latino citizens whose first language is Spanish.
Exit polls provide no such opportunity. Spanish language interviewing is available only on a limited basis. Specifically, field staff may make Spanish questionnaires available in majority-Latino precincts, which numbered 11 nationwide in 2004; but not available in the thousands of other precincts they select. The result is that the total share of all Latino interviews conducted in Spanish is extremely small. In 2004, for example, 35 interviews were completed in Spanish out of 638 Latinos in the national NEP dataset, or 4.7%. While it is certainly plausible to expect English-dominant citizens to register and vote at a higher rate, the fact that we used registered voter lists and screen for vote propensity eliminates this effect from our numbers.
In short, exit polls are under-surveying Spanish-dominant Latino citizens by almost 10-fold. This result is consistent with our earlier design concerns and reinforces our claim that exit polls over-represent assimilated, middle class, and later generation Latinos, introducing a systematic bias.
III. Evidence in the 2010 Exit Polls
Do we have any evidence that would support our claim that discrepancies between our estimates and the NEP samples are a consequence of their sample and interviewing practices? We are, in fact, awash with them.
Here are several of our favorites, which we will address in some depth:
• The Exit Polls estimate African American support for Meg Whitman in California at 21%.
• The Exit Polls estimate that among non-white voters in California, 40% have college degrees. In Arizona, 45% have college degrees. In Colorado, an impressive 61% of non-whites have finished college. These data are a far cry from the actual numbers which are less than 20%
• In Arizona, only 314 Latino interviews were complete, in Colorado 134, in Illinois 193. The margins-of-error would be 5.5%, 8.5% and 7.1%, respectively, if we don’t correct for the design effect. After correction, they would be far greater.
• No vote breakouts by race are provided for Colorado, even though they estimate Latinos at 14% of the electorate. Why was the data not reported, if it was collected?
Consider for example the African-American estimate of two-party vote in the California Governor’s election. The exit polls peg black support for Meg Whitman at 21%. For context, African American voters break around 9-1 on average in favor of Democrats, and 19-1 in favor of Democrats in 2008. Democratic two-party vote share among African Americans has ranged, in the last three decades, between 87 and 95 percent Democratic in presidential elections. The California Field Poll, the week before the election, estimated black support for Whitman at 8%. Yet, on Election Day, the exit polls arrived at 21% for Whitman against the former Mayor of Oakland. If we assume, for a moment, that the exit polls are correct, we would have to significantly revise decades of research on the political behavior of African Americans to explain why black voters more than doubled their average GOP vote share for a white former CEO than a popular former Governor who had championed civil rights.
Likewise, the exit poll numbers for Sharron Angle in Nevada are mind boggling. Angle, who arguably ran the most offensive campaign against Latino immigrant is estimated to have won 30% of the Latino vote – an 8 percentage point improvement over John McCain in 2008 who won 22% of the Latino vote in Nevada. It is not possible, nor plausible that Angle improved by 8% among Latinos in 2010 given her attacks on Latinos, and Reid’s strong defense of immigration reform. Rather, the Latino Decisions data in the election poll, and our 10 weeks of data from Nevada in our tracking poll both point to very large gains for Democrats in Nevada, and our 90% estimate for Reid matches very close to the vote totals that resulted in a surprise win by Reid that pre-elections polls missed.
In California, the national exit polls would have us believe that Meg Whitman who saw her favorability ratings plummet among Latinos after her contradictory statements supporting Arizona’s SB1070 and her undocumented immigrant housekeeper scandal, also did better among Latinos than John McCain in 2008. In 2008 McCain won an estimated 23% of the California Latino vote, yet the 2010 exit polls suggest Whitman won 30% of the Latino vote, a 7 point improvement? Latino Decisions estimated just 13% for Whitman in 2010.
In Arizona, Jan Brewer, the force behind SB1070 who claimed Mexican immigrants were beheading innocent Arizonans in the southern desert is reported to have won 28% of the Latino vote according to the national exit polls. In contrast Latino Decisions found back in May that Brewer was attracting only 12% of the Latino vote, and that in October our tracking poll estimated 13% vote for Republicans in Arizona, and ultimately only 14% vote for Brewer in 2010, a far cry from 28%.
In these cases, and across the board in 2010, the NEP Latino exit poll results are laughable.
