Page 1

Evaluation of Edison/Mitofsky Election System 2004

prepared by Edison Media Research and Mitofsky International

for the National Election Pool (NEP)

Evaluation of Edison/Mitofsky Election System

page 2

January 19, 2005

Table of Contents

Executive Summary............................................................................................................ 3

Glossary of Terms............................................................................................................... 7

Overview........................................................................................................................... 11

Recommendations............................................................................................................. 15

Edison/Mitofsky Election System Development.............................................................. 18

Accuracy of Exit Poll Estimates....................................................................................... 19

Evaluation of Samples ...................................................................................................... 28

Evaluation of the Within Precinct Error (WPE) ............................................................... 31

Exit Poll Location Coverage............................................................................................. 48

Exit Poll Interviewer Recruitment Process....................................................................... 49

Completion Rates.............................................................................................................. 53

Survey Weighting ............................................................................................................. 56

National Survey Weighting Gender Adjustment.............................................................. 58

Comparison of National Exit Poll with Cross Survey...................................................... 59

Comparison of NEP exit poll results with other exit polls ............................................... 64

Legal and Distance Issues................................................................................................. 70

Absentee/Early Voter Telephone Surveys........................................................................ 72

Areas for Further Investigation in 2005............................................................................ 76

Evaluation of Edison/Mitofsky Election System

page 3

January 19, 2005

Executive Summary

On November 2, 2004, the Election System created by Edison Media Research and

Mitofsky International for the National Election Pool (NEP) produced election estimates

and exit poll data for analysis in 120 races in all 50 states and the District of Columbia.

In addition, between January and March 2004, Edison and Mitofsky conducted exit polls

for 23 Democratic Primaries and Caucuses. For every election, the system delivered on

its main goals: there were no incorrect NEP winner projections, and the exit poll data

produced on election day were used on-air and in print by the six members of the NEP

(AP, ABC, CBS, CNN, FOX and NBC) as well as several dozen media organizations

who subscribed to that data. However, the estimates produced by the exit poll data on

November 2

nd

were not as accurate as we have produced with previous exit polls.

Our investigation of the differences between the exit poll estimates and the actual vote

count point to one primary reason: in a number of precincts a higher than average Within

Precinct Error most likely due to Kerry voters participating in the exit polls at a higher

rate than Bush voters. There have been partisan overstatements in previous elections,

more often overstating the Democrat, but occasionally overstating the Republican. While

the size of the average exit poll error has varied, it was higher in 2004 than in previous

years for which we have data. This report measures the errors in the exit poll estimates

and attempts to identify the factors that contributed to these errors.

The body of this report contains the details of our analysis of the performance of the exit

polls and the election system. In addition to the information included in this report, exit

poll data from this election is being archived at the Roper Center at the University of

Connecticut and at the Institute for Social Research at the University of Michigan and is

available there for review and further analysis. This is the procedure that we have

followed for all previous exit polls, which are also available at the Roper Center and ISR.

The description of the methodology of the exit polls has already been posted on our Web

site – www.exit-poll.net - along with all questionnaires used on election day and the

completion rates nationally and by state.

Here is a brief summary of our findings:

1. Exit Poll Estimates

The exit poll estimates in the 2004 general election overstated John Kerry’s share of the

vote nationally and in many states. There were 26 states in which the estimates produced

by the exit poll data overstated the vote for John Kerry by more than one standard error,

and there were four states in which the exit poll estimates overstated the vote for George

W. Bush by more than one standard error. The inaccuracies in the exit poll estimates

were not due to the sample selection of the polling locations at which the exit polls were

conducted. We have not discovered any systematic problem in how the exit poll data

were collected and processed. Exit polls do not support the allegations of fraud due to

rigging of voting equipment. Our analysis of the difference between the vote count and

Evaluation of Edison/Mitofsky Election System

page 4

January 19, 2005

the exit poll at each polling location in our sample has found no systematic differences

for precincts using touch screen and optical scan voting equipment. We say this because

these differences are similar to the differences for punch card voting equipment, and less

than the difference for mechanical voting equipment.

Our detailed analysis by polling location and by interviewer has identified several factors

that may have contributed to the size of the Within Precinct Error that led to the

inaccuracies in the exit poll estimates. Some of these factors are within our control while

others are not.

It is difficult to pinpoint precisely the reasons that, in general, Kerry voters were more

likely to participate in the exit polls than Bush voters. There were certainly motivational

factors that are impossible to quantify, but which led to Kerry voters being less likely

than Bush voters to refuse to take the survey. In addition there are interactions between

respondents and interviewers that can contribute to differential non-response rates. We

can identify some factors that appear to have contributed, even in a small way, to the

discrepancy. These include:

• Distance restrictions imposed upon our interviewers by election officials at the

state and local level

• Weather conditions which lowered completion rates at certain polling locations

• Multiple precincts voting at the same location as the precinct in our sample

• Polling locations with a large number of total voters where a smaller portion of

voters was selected to be asked to fill out questionnaires

• Interviewer characteristics such as age, which were more often related to precinct

error this year than in past elections

We plan further analysis on the following factors:

• Interviewer training and election day procedures

• Interviewing rate calculations

• Interviewer characteristics

• Precinct characteristics

• Questionnaire length and design

We also suggest the following changes for future exit polls:

• Working to improve cooperation with state and local election officials

Evaluation of Edison/Mitofsky Election System

page 5

January 19, 2005

• Improvements in interviewing training procedures

• Changes in our procedures for hiring, recruiting and monitoring interviewers

Even with these improvements, differences in response rates between Democratic and

Republican voters may still occur in future elections. However, we believe that these

steps will help to minimize the discrepancies.

It is also important to note that the exit poll estimates did not lead to a single incorrect

NEP winner projection on election night. The Election Night System does not rely solely

on exit polls in its computations and estimates. After voting is completed, reported vote

totals are entered into the system. Edison/Mitofsky and the NEP members do not project

the outcome of close races until a significant number of actual votes are counted.

As in past elections, the final exit poll data used for analysis in 2004 was adjusted to

match the actual vote returns by geographic region within each state. Thus, the

discrepancy due to differing response rates was minimized and did not significantly affect

the analysis of the vote. The exit polls reliably describe the composition of the electorate

and how certain demographic subgroups voted.

2. Survey Weighting

Early in the afternoon on November 2

nd

, preliminary weightings for the national exit poll

overstated the proportion of women in the electorate. This problem was caused by a

programming error involving the gender composition that was being used for the

absentee/early voter portion of the national exit poll. This error was discovered after the

first two sets of weighting; subsequent weightings were corrected. This adjustment was

made before NEP members and subscribers used exit poll results on-air or in print.

After election day, we adjusted the exit poll analysis data in three states (Tennessee,

Texas, and Washington) to more accurately reflect the proportion of absentee ballots that

came from each geographic region in those states. We have implemented a change to the

survey weighting program to take into account the geographic distribution of the absentee

votes in the future.

3. Technical Performance

While the computer system performed well for most of the night, a database server

problem led to NEP member and subscriber screens “freezing up” shortly after 10:35 PM

ET election night. This problem caused disruptions in the system until shortly after

midnight when we switched to a backup server for the rest of the night. There was a

second occurrence of this problem at approximately 2:45 AM ET. Details of the data

server problems and other technical issues are outlined in the technical performance

report being distributed to the NEP Technical Committee. We have isolated the reasons

Evaluation of Edison/Mitofsky Election System

page 6

January 19, 2005

behind the database server problem and list several recommended technical changes in

this report to help avoid a repeat of this problem in future elections.

Evaluation of Edison/Mitofsky Election System

page 7

January 19, 2005

Glossary of Terms

Absentee/Early Voter Telephone Surveys

Absentee/Early Voter Telephone Surveys were conducted in 13 states that had a high

proportion of absentee/early voters. The estimates from these surveys were used to adjust

the exit poll estimates from election day to account for the absentee/early voters who can

not be interviewed at the polling location on election day. The questionnaire responses in

these surveys were also incorporated in the survey analysis in the 13 state surveys and the

national survey.

Age-Race-Sex adjustment

An Age-Race-Sex adjustment is performed based upon the refusals and misses from

sample voters that are observed by the interviewers at each polling location. The age,

race and gender compositions in the exit poll results are adjusted to account for the

differing completion rates of these demographic groups.

Best Survey Estimate

The Best Survey Estimate is the computation with the lowest SEDF (Standard Error on

the Difference) using only the exit poll tallies.

Completion Rate

The Completion Rate is the percentage of sample voters who agree to fill out the

questionnaire. The rate equals completed questionnaires divided by completed

questionnaires plus refusals plus missed voters who were in the sample.

Composite Estimate

The Composite Estimate is a weighted average of the Prior Estimate and the Best Survey

Estimate. The Composite Estimate is most often the estimate used in the survey

weighting process to create the exit poll analysis data during election day before the

actual vote is reported.

County Model

The County Model is a set of computations based upon the county data that is being

reported by the Associated Press.

Critical Value (Crit)

The t-score is the ratio of the estimated difference between the two leading candidates

and the standard error on the difference (SEDF). A critical value occurs when this ratio

is 2.6 or more. The critical value increases when there are 40 or fewer sample precincts.

This critical value is the first of several criteria for a “Call Status.” It means there is a

.995 statistical probability that the leader is the winner. It only accounts for sampling

error in the estimate. It does not account for other possible sources of error or statistical

bias.

Evaluation of Edison/Mitofsky Election System

page 8

January 19, 2005

Cross Survey

Cross Survey is a procedure through which state surveys are combined to form estimates

of survey characteristics. When state surveys are combined in this way, the respondent

weights are adjusted so that each state survey is represented in its correct proportion of

the total. The Cross Survey is different from the National Survey. The Cross Survey only

includes questions common to the state surveys.

Decision Screens

These screens provide the details for the 14 different estimates that are computed for each

election day race. In addition, these screens include details on estimated candidate votes

with and without exit poll results, with and without absentee votes factored in, sampling

errors for all estimates, estimates by stratum, and quality control information.

Integrated Model

The Integrated Model is a computation based upon a composite of the estimates from the

Sample Precinct Model and the County Model.

Interviewing Rate

Each exit poll interviewer is assigned an Interviewing Rate that is used to select sample

voters as they leave the polling place. The interviewing rate is defined as the number of

voters that the interviewer counts between sample voters. An interviewing rate of “1”

means that the interviewer will approach every voter; an interviewing rate of “10” means

that the interviewer will approach every 10

th

voter.

Miss Rate

The Miss Rate is the percentage of voters designated to be in the sample that are missed

by the interviewer because the interviewer could not physically approach the voter and

ask them to fill out a questionnaire.

National Exit Poll (National Survey)

The National Exit Poll is based upon the results from a national sample of 250 polling

locations. These 250 locations are a sub-sample of the 1,480 locations that are in the state

samples. In addition, 500 telephone interviews of absentee/early voters in 13 states with

a high proportion of absentee/early voters were included in the National Exit Poll results.

There were four different versions of the national exit poll questionnaire. One–fourth of

the sample at every national exit poll location was asked to complete each version of the

national questionnaire.

Prior Estimate

Prior Estimates are based upon pre-election surveys conducted in each state. The Prior

Estimate is used in combination with the Best Survey Estimate during election day to

create a Composite Estimate.

Projections

A projection is based on an estimate of the vote. The first of many requirements for

projecting a winner is that the leading candidate is estimated to be ahead of his or her

Evaluation of Edison/Mitofsky Election System

page 9

January 19, 2005

nearest challenger by a margin that is sufficiently larger than the standard error. That

margin would have to be 2.6 (at a minimum) times the standard error on the difference

between the two candidates. The probability of incorrectly concluding that the leading

candidate is ahead is .005.

Refusal Rate

The Refusal Rate is the percentage of sampled voters who are approached by the

interviewer, but who refuse to fill out the questionnaire.

Sample Precinct Model

The Sample Precinct Model is a set of computations under different assumptions that use

either precinct level exit poll results or actual vote returns. The exit poll results and actual

vote returns may be used separately or in combination.

Standard Error on the Difference (SEDF)

We select only one sample of precincts per state out of the many different samples that

could have been selected. Each possible sample will have a slightly different estimate of

the election result. A standard error is a measure of the variation in all those possible

results. While most samples have results that are close to the average for all the samples,

it is theoretically possible that the one sample we selected differs from the overall

average. The standard error tells us the likelihood of having a sample that differs from the

overall average by given amounts. For making projections we are interested in the

Standard Error on the Difference (SEDF). It is computed on the difference between the

top two candidates for each estimate.

Survey Call 1, 2, 3

Exit poll interviewers call in the results of their interviews to our telephone centers three

times during election day. The first call – Call 1 – is shortly before noon local time. Call

2 is in the late afternoon. The last call – Call 3 – is during the last hour before the time the

polls close. The exit poll is not complete until the Call 3 interviews are used in the

computations.

Survey Weighting

Survey Weighting is the process by which the respondents in each survey are weighted

for the exit poll analysis. This weighting process takes into account the probabilities of

selection of the precinct and the sample voters within each sample precinct, the age-race-

sex adjustment for non-interviews, the best estimate of the candidate vote percentages

from each geographic region, and if applicable the portion of the vote that is being cast

by absentee/early voters.

t-score

For the value of the “t-score” see the definition of the Critical Value, which is defined

above. The “t” refers to a distribution of probabilities for these scores for small samples.

Evaluation of Edison/Mitofsky Election System

page 10

January 19, 2005

Within Precinct Error (WPE)

Within Precinct Error (WPE) is an average of the difference between the percentage

margin between the leading candidates in the exit poll and the actual vote for all sample

precincts in a state. The signed WPE gives the direction of this error; in this report a

negative WPE represents a Democratic overstatement in the exit poll and a positive WPE

represents a Republican overstatement in the exit poll. The absolute WPE represents the

total error.

Evaluation of Edison/Mitofsky Election System

page 11

January 19, 2005

Overview

On election day this past November, the Election System created by Edison Media

Research and Mitofsky International delivered election estimates and exit poll data for

analysis in 120 races in all 50 states and the District of Columbia. Between January and

March 2004 Edison/Mitofsky conducted exit polls for 23 Democratic Primaries. For each

election, results were delivered in a timely manner and the system successfully delivered

data to the members and the subscribers.

The Edison/Mitofsky Election System delivered on its two main tasks – there were no

incorrect NEP winner projections, and the system delivered exit poll and election

estimates for every state on election day. As with any complex real-time data collection

and computation system of this magnitude there were complications on election day. The

estimates produced by the exit poll data on election day were not as accurate as we have

produced with previous exit polls. This report identifies the factors which contributed to

these errors. We have learned from our experience in 2004 and we will use that

knowledge to help us in the preparation and operation of the system for future elections.

Here we summarize the performance of the Election Day System. Each item is detailed

in later sections of the report.

Accomplishments:

The exit poll location coverage was almost complete. Out of 1,480 exit poll locations, we

gathered data from more than 99% of the locations – seven polling locations had no

interviewer, and four polling locations had no data collected because our interviewers

were unable to conduct interviews due to distance restrictions enforced by local election

officials at the polling place.

The computations operated successfully throughout election day. The sample precinct

estimates with vote returns were accurate, as were the county model estimates and the

Integrated Model. While some estimates using the exit poll data differed from the final

actual vote, the exit poll estimate computations and questionnaire processing worked

according to the specifications. In this report we examine the accuracy of these

estimates, and in the next few months we plan an in-depth evaluation of the various

computations to measure their accuracy throughout election day. These areas of further

investigation are listed later in this report.

With a few exceptions that are detailed in the body of the report, the survey weightings

were delivered on schedule until the database server problem beginning at 10:35 PM ET.

We interviewed absentee and early voters in 13 states for both state and national surveys.

These data were successfully combined with the election day voters interviewed in the

exit poll. We are suggesting improvements to make this process yet better.

Evaluation of Edison/Mitofsky Election System

page 12

January 19, 2005

Communication of data through the feeds to the NEP members was maintained

throughout election day except for the period of time during which we switched over to

the backup system following the database server problem.

Edison/Mitofsky communicated correct race calls through the feeds and the message

system for each race on election night. All races were called correctly including 74

winners projected at the time the polls closed.

The election computer system supported approximately 650 users without seriously

overloading the capacity of the computer hardware assigned to the task.

In addition to the experience of the leaders of this operation, we now have a large trained

staff that we intend to retain through future election cycles.

In short, we have constructed an election system that works and a team dedicated to

running it. We were tasked with delivering an enormous amount of data to a large

number of users on election day, and in that effort we succeeded.

Election Day Problems:

The exit poll estimates in this year’s general election in many states and in the national

survey had a sizeable overstatement of the estimated percentage of the vote for John

Kerry. All evidence is that this is attributable to “Within Precinct Error” (WPE) and not

to any systematic problems in the sampling or the way the data was processed after it was

received from the exit poll interviewers. We report on the initial findings of our

investigation in this report, and we have identified several factors that contributed to the

size of the WPE. Later in this report we examine the history of WPE and the several

possible factors contributing to WPE, including those we can control for in the future,

and those factors that are more difficult to control, especially differential non-response by

Republican vs. Democrat voters. While the estimates in the primaries were more

accurate, we need to do more investigation into the causes of the statistical skew in the

exit poll data for the general election.

Preliminary weightings for the National Exit Poll overstated the proportion of women in

the electorate. This error was discovered and corrected on election day by externally

adjusting the male-female breakout in the National Survey to match the male-female

breakout from the cross survey weighted average of all of the state surveys combined

(54% Female/ 46% Male). We have identified the cause as a problem in the way that the

national survey weighting program dealt with the absentee/early voter telephone portion

of the national survey, and this problem has been corrected.

After election day, we needed to re-weight the state surveys from Tennessee, Texas and

Washington because the exit poll size and vote in geographic areas within a state did not

match the actual final vote returns. Because we had not been certain that the absentee

Evaluation of Edison/Mitofsky Election System

page 13

January 19, 2005

survey respondents would be identified by geographic area, the state survey weighting

spec did not include the respondents from the absentee/early voter telephone surveys in

the geographic forcing portion of the weighting procedure. This caused the state surveys

in certain states, where at least half of the voters cast absentee or early ballots, to deviate

from the actual vote by geographic region. This was an issue only in the Tennessee,

Texas, and Washington surveys. The state survey weighting spec will be adjusted to deal

with this issue in the future.

On the technical side, the biggest problem was a database server problem that led to the

screens “freezing up” shortly after 10:35 PM ET on election night and later at 2:45 AM

ET. In the Technical Performance Report we discuss the changes we will make for the

future. In dealing with the database server problem, the switch to the backup server took

longer than it had during testing. After the switch, one of the backup servers was

requesting data from the original database. As a result, some screens did not appear to be

in sync with other screens. In the Technical Performance Report, we address the

operational and technical changes that will avoid a repeat of these problems.

Items for further investigation:

We are in the process of an in-depth evaluation of the exit poll process in Ohio and

Pennsylvania. We will verify every piece of data that was used in the system for the

estimates in these states – both current data on election day and past data used in the

computations. We will report in detail on the recruiting of each individual exit poll

interviewer. We will follow up with in-depth interviews with the exit poll interviewers in

the precincts in which we saw the largest errors in an attempt to determine if there were

any factors that we have missed thus far in our investigation of Within Precinct Error.

Since election day there has been discussion about the differences between the National

Exit Poll and the estimates from the Cross Survey from all state surveys. Some estimates

differ by several points among certain demographic groups, most noticeably among

Hispanics. These differences appear mostly among demographic groups that are both

relatively small (8% or less of the voting population) and those that tend to be

geographically concentrated. We discuss in detail later in this report possible ways of

dealing with the cluster effects that cause these differences between the National Exit

Poll and Cross Survey estimates.

In this evaluation report, we examine the performance of the system. We believe that the

positives greatly outweigh the negatives. However, we share the members’ desire for an

even more reliable operating system and more accurate data on election day. Now that

we have a fully operational election system in place we will be concentrating during the

next few months on evaluating every aspect of the data collection and data processing

with the aim of implementing changes to improve the accuracy of the data that is

presented on election night.

Evaluation of Edison/Mitofsky Election System

page 14

January 19, 2005

To achieve that goal we have detailed many aspects of the data collection, data

processing, and systems operation for the 2004 Elections. We have noted the areas where

the system can be improved and we have made recommendations for implementing these

improvements. We look forward to the input of the members in this process and we plan

to work together with the members in the next few months to put a plan in place to ensure

that the election system serves the needs of all of the news organizations in future

elections.

Evaluation of Edison/Mitofsky Election System

page 15

January 19, 2005

Recommendations

Our goal is to have smoother and more accurate election nights in the future. After two

months of study we recommend the following changes in what we do and how we

function. We believe these recommendations will result in improvements for NEP and

the subscribers.

IMPROVING OUR ESTIMATES

Exit Poll Errors

One way to reduce error is to take additional steps to keep the interviewers focused on

strictly following their interviewing rates in order to properly sample voters within each

polling location. This will be made an even greater priority in the future. We will

develop additional steps in the recruiting and training process to make certain that the

interviewers are following the detailed instructions that we give them.

Another way to potentially reduce the overall error in the exit poll is to improve the

completion rates. We plan to examine more closely how the size, design and layout of

the questionnaires may affect the percentage of sampled voters who choose to complete

the exit poll questionnaire.

While we have identified factors that we can control in order to lessen Within Precinct

Error, we cannot eliminate the possibility of any statistical bias based upon differential

non-response by Democratic and Republican voters. We plan to identify indicators in the

exit poll data that will give those who are using the exit poll data an early warning that

there may be the possibility of Within Precinct Error in the exit poll estimates.

Recruiting Exit Poll Interviewers

We plan to make enhancements to recruiting exit poll interviewers.

• We will use augmented recruiting methods to reduce the proportion of students

and young adults as interviewers.

• We will add a standardized training script for all individual training phone

conversations that occur prior to our main training/rehearsal call.

• We will evaluate other training techniques such as the video training guide and

interviewer tests.

• We will use the Internet more effectively as an interviewer training tool.

Evaluation of Edison/Mitofsky Election System

page 16

January 19, 2005

Distance Issues

We need to be more proactive in gaining cooperation from state and local election

officials who try to impose distance restrictions of 50 feet or more on exit poll

interviewers. Compared to the 2000 data collected by VNS, more than twice as many of

our exit poll interviewers in 2004 reported that they were forced to stand more than 50

feet away from the polling location. There is convincing evidence that both the response

rates and the accuracy of the exit poll data decrease once an interviewer is forced to stand

more than 50 feet away from the polling location. The priority states in our efforts

should be Arizona, Florida, Minnesota, Nevada, New Mexico, Ohio and South Dakota,

which all tried to impose distance requirements greater than 50 feet on exit poll

interviewers during this election.

IMPROVING OUR COMPUTER SYSTEM

We plan to make the following changes in the operations and technical features of the

computer system:

We will have one dedicated person on both the decision and technical conference bridge

when there are any problems with the system so that there is no confusion in

communication. That person will have no other system responsibilities.

Technical changes will include testing the system with at least twice the number of users

and a compressed time during simulations. Testing will also include twice the expected

data as on election day. This may have exposed the database server problem we

experienced with the screens “freezing up” starting at 10:35 PM.

We plan to use more sophisticated monitoring tools for the database on election night that

will pinpoint hot spots for us prior to any problems occurring.

The only circumstance where untested code will be executed on election night is to

correct a problem causing the system not to function. Otherwise no untested code will be

executed as long as the system is up and running.

During failover we will isolate each data center so that all primary servers point to the

primary database server and all backup servers point to the backup database server. This

eliminates the need for running scripts for this purpose.

Evaluation of Edison/Mitofsky Election System

page 17

January 19, 2005

IMPROVING THE ANALYSIS

National Exit Poll and Cross Survey

We will look for ways to better achieve consistency between the Cross Survey results for

a few selected characteristics and the National Exit Poll.

If we want to improve the National Exit Poll estimates for minority groups or other

characteristics that are highly clustered, we need to increase the number of polling

locations in the National Sample or oversample polling locations with the characteristics

of interest.

Absentee/Early Voter Telephone Surveys

Ideally, the state absentee/early voter telephone survey sample sizes should be increased

in the states where absentee vote is a large proportion of the total vote. With only 500

respondents in 2004 representing nearly one-fifth of all voters nationally who voted early

or by absentee, the results of the national and regional breakouts for some of the smaller

demographics are based upon sample sizes that are too small.

OTHER IMPROVEMENTS

Dealing with the Leaking of NEP Exit Poll Data

The decision by the NEP members to withhold the distribution of exit poll information

within their organizations until 6PM ET on election day will help prevent, or at least

delay, the use of exit poll data before poll closing by those who have not purchased the

data. We will work closely with the NEP members to develop security measures

deemed appropriate to implement this policy.

Subscribers

In the general election several subscribers felt that they were not given the same guidance

about possible inaccuracies in the exit poll estimates that we had given the NEP

members. On Election Day, at 4:30 PM ET, we convened a conference call with the

Decision Teams of the NEP members and cautioned them that we expected sizeable

errors in the exit polls in nine states; in seven states (Connecticut, Delaware, Minnesota,

New Hampshire, Pennsylvania, Vermont and Virginia) we suspected that the exit poll

estimates were overstatement of the vote for Kerry; in two states (South Dakota and West

Virginia) we suspected an overstatement of the Bush vote. We made these warnings

based upon the discrepancies between the exit polls and our prior estimates in these nine

states. We made a mistake in not sharing with the subscribers our concerns about the

accuracy of the exit poll estimates in those nine states. In the future we will need to make

sure that whatever guidance we share with the NEP members is also communicated to the

subscribers so that they can feel comfortable using the data.

Evaluation of Edison/Mitofsky Election System

page 18

January 19, 2005

Edison/Mitofsky Election System Development

In January 2003, the six members of the National Election Pool (NEP) asked Edison

Media Research and Mitofsky International to design and implement a full exit poll and

election projection system that would be operational for the 2004 Presidential Election.

We believe that we have accomplished a lot in a short period of time.

We have designed, developed and implemented a fully operational custom computer

system. This system gathers exit poll and vote count data; it processes exit poll data for

analysis and election computations for estimates and projections; it delivers all of this

data to the six NEP members (and several dozen subscribers) simultaneously; and it

handles the data load and the user load of nearly 650 simultaneous users. In less than

twelve months the system was up and running for 23 Democratic Primaries and Caucuses

during a nine-week period from the Iowa Caucuses on January 19, 2004 through the

primaries on March 9, 2004. On November 2, 2004 the system collected, processed and

distributed data for 120 races in all 50 states. Between the primaries and the general

election more than 140 races were projected without one incorrect winner being declared

by NEP in any race.

In addition, we met all of the development milestones on schedule.

The system, though, is not yet perfect. In this report we detail areas that we intend to

improve for the next election cycle. However, we do not want to lose sight of the fact

that the six NEP members now have what they did not have for the previous election

cycle – a fully operating exit poll and election projection system.

Evaluation of Edison/Mitofsky Election System

page 19

January 19, 2005

Accuracy of Exit Poll Estimates

The exit poll estimates for president on election day overstated the actual share of the

total vote received by the Democratic candidate by more than one standard error in more

than half of the states. Some of these errors were small and some were large, but in a

presidential race as close as 2004, even the smallest overstatement of the Democratic vote

led to an expectation during election day that differed from the actual results.

In order to diagnose the potential causes of these differences, we first need to measure the

error in the exit poll estimates on election day. The tables on the following pages include

the survey estimates for each of the President and Senate races that were calculated at the

time of the Call 3 exit poll weighting. The Best Geo Estimator is the estimate with the

lowest Standard Error on the Difference (SEDF) using the cumulative precinct tallies for

each candidate. This would be the best estimate that was displayed on the Decision

Screens for the members’ decision teams to review.

Comparison of Survey Estimates vs. Actual Results:

Best Geo Survey Estimate vs. Actual Results - Presidential Races (All

States)

Average

Error on

Difference

Average

Absolute

Error on

Difference

# of states

with Dem

overstated by

t-score>1

# of states with

Rep overstated

by t-score>1

# of states

with

Difference of

t-score<=1

Call 3

-5.0

6.1

26

4

20

Call 2

-5.3

6.5

22

3

25

Call 1

-5.2

6.6

19

2

28

Best Geo Survey Estimate vs. Actual Results - Senate Races (32 states)

Average

Error on

Difference

Average

Absolute

Error on

Difference

# of states

with Dem

overstated by

t-score>1

# of states with

Rep overstated

by t-score>1

# of states

with

Difference of

t-score<=1

Call 3

-3.6

5.4

14

3

15

Call 2

-2.8

5.4

13

1

18

Call 1

-2.7

5.9

9

2

20

Note: The above analysis does not include the Oregon president and senate races where

all interviews were done by telephone; the Idaho senate race which was unopposed, and

the California president and senate race for Call 1.

On average, the exit poll estimates demonstrated an overstatement of the Democratic vote

in both the President and Senate races. The average overstatement was slightly larger for

Kerry in the Presidential race than for the Democratic Senate candidates.

Evaluation of Edison/Mitofsky Election System

page 20

January 19, 2005

In the 32 states with exit poll estimates for both a Presidential race and a Senate race the

average error on the difference between the top two candidates was 5.0 points in the

Democratic direction for President and 3.6 points in the Democratic direction for Senate.

Comparison of Composite Estimates vs. Actual Results:

The Composite Estimator includes the Best Survey Estimate and the Prior Estimate based

upon the pre-election polls. The Composite Estimator is the one that was used in the exit

poll weighting and these estimates would represent the statewide numbers that were

being displayed on the exit poll analysis screens that were being used by the NEP

members and the subscribers.

The errors in the Composite Estimates were slightly lower than those in the Best Survey

Estimate since the Composite Estimate includes the Prior Estimate, which is based upon

analysis of the available pre-election surveys in each state. The average weighted state

survey results, available shortly after poll closing, differed from the final actual vote by

3.6 points on the Bush-Kerry difference.

Composite Estimate vs. Actual Results - Presidential Races

Average

Error on

Difference

Average

Absolute

Error on

Difference

# of states

with Dem

overstated

by t-score>1

# of states

with Rep

overstated by

t-score>1

# of states

with

Difference of

t-score<=1

Call 3

-3.6

4.5

29

3

18

Call 2

-4.0

4.4

25

0

25

Call 1

-3.2

3.6

17

0

32

Prior

-2.0

3.1

10

2

39

Composite Estimate vs. Actual Results - Senate Races

Average

Error on

Difference

Average

Absolute

Error on

Difference

# of states

with Dem

overstated by

t-score>1

# of states

with Rep

overstated by

t-score>1

# of states

with

Difference of

t-score<=1

Call 3

-2.5

4.7

18

3

12

Call 2

-1.9

4.5

15

5

13

Call 1

-1.4

5.0

13

6

12

Prior

-0.3

7.5

9

8

16

Comparison of National Exit Poll vs. Actual Results:

The national exit poll had a similar Kerry overstatement. The weighted national survey

numbers showed Kerry with 51% and Bush with 48%. The final national popular vote

margin ended up being 2.5% for Bush. Thus, the national exit poll had an error of 5.5

points on the difference in the Democratic direction which is similar to the 5.0 average

from the state surveys.

Comparison of Best Geo Estimate and Composite with the Final Election Result for President - Call 3

Estimated Difference and Final Margin are positive when Democrat (Kerry) leads and negative when Republican (Bush) leads

Best Geo Estimator:

Composite Estimator:

State

Race

Call

Final

MarginLeaderEstimate

Estimated

Difference

Estimated

Error

(SEDF)

Actual

Error

Absolute

Error

t-

score LeaderEstimate

Estimated

Difference

Estimated

Error

(SEDF)

Actual

Error

Absolute

Error

t-

score

Alabama

President 3 -25.6Bush

57.5

-15.5

8.3 -10.1

10.1 -1.2Bush

58.7

-18.1

4.3

-7.5

7.5 -1.8

Alaska

President 3 -25.6Bush

57.4

-16.2

4.1 -9.4

9.4 -2.3Bush

58.8

-19.8

3.1

-5.8

5.8 -1.9

Arizona

President 3 -10.5Bush

53.5

-7.0

4.5 -3.5

3.5 -0.8Bush

53.2

-6.4

3.7

-4.0

4.0 -1.1

Arkansas

President 3 -9.7Bush

52.4

-5.6

4.6 -4.2

4.2 -0.9Bush

52.2

-5.2

3.0

-4.5

4.5 -1.5

California

President 3 10.0Kery

56.5

13.0

3.1 -3.0

3.0 -1.0Kery

56.5

13.0

4.0

-3.0

3.0 -0.8

Colorado

President 3 -4.7Bush

52.5

-5.5

1.0

0.9

0.9 0.8Bush

51.4

-3.7

2.0

-1.0

1.0 -0.5

Connecticut

President 3 10.4Kery

59.3

19.7

3.7 -9.4

9.4 -2.6Kery

58.1

17.6

3.1

-7.3

7.3 -2.3

Delaware

President 3

7.6Kery

61.5

23.6

5.7 -16.0

16.0 -2.8Kery

57.7

16.5

3.9

-8.9

8.9 -2.3

District of ColumbiaPresident 3 79.8Kery

91.1

83.0

3.4 -3.1

3.1 -0.9Kery

90.2

81.8

2.6

-1.9

1.9 -0.7

Florida

President 3 -5.0Bush

50.3

-1.1

1.6 -3.9

3.9 -2.4Bush

50.1

-0.8

1.6

-4.3

4.3 -2.7

Georgia

President 3 -16.6Bush

56.5

-13.0

4.3 -3.7

3.7 -0.9Bush

57.1

-14.1

3.2

-2.5

2.5 -0.8

Hawaii

President 3

8.7Kery

56.5

13.1

8.2 -4.3

4.3 -0.5Kery

53.6

7.2

4.8

1.6

1.6

0.3

Idaho

President 3 -38.1Bush

69.1

-38.2

3.1

0.0

0.0 0.0Bush

68.3

-36.7

4.3

-1.4

1.4 -0.3

Illinois

President 3 10.1Kery

57.5

14.9

3.9 -4.8

4.8 -1.2Kery

57.0

14.1

3.3

-3.9

3.9 -1.2

Indiana

President 3 -20.7Bush

59.6

-19.1

3.0 -1.6

1.6 -0.5Bush

58.8

-17.5

3.1

-3.2

3.2 -1.0

Iowa

President 3 -0.7Kery

50.0

1.0

2.5 -1.6

1.6 -0.7Kery

50.0

1.0

2.2

-1.6

1.6 -0.7

Kansas

President 3 -25.4Bush

62.8

-26.2

3.8

0.8

0.8 0.2Bush

64.6

-30.2

3.2

4.8

4.8

1.5

Kentucky

President 3 -19.9Bush

58.6

-18.0

2.8 -1.9

1.9 -0.7Bush

58.3

-17.4

2.5

-2.4

2.4 -1.0

Louisiana

President 3 -14.5Bush

56.3

-13.1

6.2 -1.4

1.4 -0.2Bush

54.8

-10.5

3.2

-4.0

4.0 -1.2

Maine

President 3

8.8Kery

54.3

9.7

2.7 -0.9

0.9 -0.3Kery

53.9

9.5

2.1

-0.7

0.7 -0.3

Maryland

President 3 13.0Kery

59.4

19.7

4.3 -6.7

6.7 -1.5Kery

56.6

14.1

3.3

-1.1

1.1 -0.3

Massachusetts

President 3 25.2Kery

66.3

32.7

6.9 -7.5

7.5 -1.1Kery

65.7

31.5

4.4

-6.3

6.3 -1.4

Michigan

President 3

3.4Kery

51.8

4.5

2.1 -1.1

1.1 -0.6Kery

51.9

4.8

1.7

-1.4

1.4 -0.8

Minnesota

President 3

3.5Kery

56.7

14.3

3.6 -10.8

10.8 -3.0Kery

53.7

8.8

2.7

-5.3

5.3 -2.0

Evaluation of Edison/Mitofsky Election System

page 22

January 19, 2005

Mississippi

President 3 -20.0Bush

53.2

-7.0

9.8 -13.1

13.1 -1.3Bush

56.0

-12.6

3.6

-7.4

7.4 -2.0

Missouri

President 3 -7.2Bush

52.2

-4.4

3.3 -2.7

2.7 -0.8Bush

52.1

-4.3

2.5

-2.9

2.9 -1.2

Montana

President 3 -20.5Bush

59.9

-22.1

10.2

1.6

1.6 0.2Bush

60.0

-22.8

5.2

2.3

2.3

0.4

Nebraska

President 3 -33.4Bush

61.7

-24.2

4.8 -9.2

9.2 -1.9Bush

62.6

-26.5

3.8

-6.9

6.9 -1.8

Nevada

President 3 -2.6Kery

49.3

1.4

3.7 -4.0

4.0 -1.1Kery

48.9

0.6

3.0

-3.2

3.2 -1.1

New Hampshire

President 3

1.4Kery

57.1

15.0

2.8 -13.6

13.6 -4.9Kery

55.1

11.2

2.1

-9.9

9.9 -4.6

New Jersey

President 3

6.8Kery

58.4

18.2

5.4 -11.5

11.5 -2.1Kery

55.3

12.5

3.0

-5.7

5.7 -1.9

New Mexico

President 3 -0.8Kery

51.7

4.2

3.0 -5.0

5.0 -1.7Kery

50.8

2.8

2.2

-3.6

3.6 -1.7

New York

President 3 17.4Kery

65.1

31.3

3.7 -13.9

13.9 -3.8Kery

63.1

27.6

2.9 -10.3

10.3 -3.5

North Carolina

President 3 -12.4Bush

51.8

-3.6

3.4 -8.8

8.8 -2.6Bush

51.9

-3.8

2.9

-8.6

8.6 -3.0

North Dakota

President 3 -27.4Bush

66.7

-34.4

4.0

7.1

7.1 1.8Bush

64.9

-31.6

3.3

4.2

4.2

1.3

Ohio

President 3 -2.1Kery

53.2

6.5

3.9 -8.6

8.6 -2.2Kery

51.7

3.4

2.6

-5.5

5.5 -2.2

Oklahoma

President 3 -31.1Bush

65.8

-31.7

1.9

0.5

0.5 0.3Bush

65.4

-30.8

1.6

-0.3

0.3 -0.2

Pennsylvania

President 3

2.3Kery

56.9

13.8

3.6 -11.5

11.5 -3.2Kery

54.2

8.5

2.7

-6.2

6.2 -2.3

Rhode Island

President 3 20.8Kery

62.4

26.1

5.3 -5.3

5.3 -1.0Kery

63.2

28.3

3.2

-7.5

7.5 -2.4

South Carolina

President 3 -17.1Bush

52.4

-6.0

3.9 -11.1

11.1 -2.8Bush

53.8

-8.7

3.3

-8.4

8.4 -2.6

South Dakota

President 3 -21.5Bush

63.2

-28.3

3.3

6.8

6.8 2.0Bush

61.5

-24.7

2.6

3.2

3.2

1.2

Tennessee

President 3 -14.3Bush

58.5

-18.2

3.0

3.9

3.9 1.3Bush

57.6

-16.3

2.4

2.0

2.0

0.8

Texas

President 3 -22.9Bush

63.5

-27.0

3.4

4.1

4.1 1.2Bush

62.9

-25.8

3.5

2.9

2.9

0.8

Utah

President 3 -45.5Bush

69.2

-39.3

3.7 -6.2

6.2 -1.7Bush

68.3

-38.4

3.2

-7.1

7.1 -2.3

Vermont

President 3 20.1Kery

67.0

36.6

3.0 -16.5

16.5 -5.5Kery

64.5

31.7

2.7 -11.6

11.6 -4.3

Virginia

President 3 -8.2Kery

50.2

0.5

5.8 -8.7

8.7 -1.5Bush

51.9

-3.9

3.3

-4.3

4.3 -1.3

Washington

President 3

7.2Kery

54.9

10.7

2.6 -3.5

3.5 -1.4Kery

54.1

9.5

2.9

-2.3

2.3 -0.8

West Virginia

President 3 -12.8Bush

57.4

-15.8

4.9

3.0

3.0 0.6Bush

54.2

-9.3

3.1

-3.5

3.5 -1.1

Wisconsin

President 3

0.4Kery

52.5

5.7

3.2 -5.3

5.3 -1.7Kery

49.6

0.4

2.4

-0.1

0.1

0.0

Wyoming

President 3 -39.7Bush

63.6

-29.1

8.5 -10.6

10.6 -1.3Bush

66.4

-34.8

4.6

-4.9

4.9 -1.1

Average

-5.0

6.1 -1.2

-3.6

4.5 -1.2

St. Dev

1.511

1.308

Senate States

Average

-5.0

6.0 -1.4

-3.7

4.6 -1.3

Evaluation of Edison/Mitofsky Election System

page 23

January 19, 2005

Comparison of Best Geo Estimate and Composite with the Final Election Result for Senate - Call 3

Estimated Difference and Final Margin are positive when Democrat leads and negative when Republican leads

Best Geo Estimator:

Composite Estimator:

State

Race Call

Final

Margin LeaderEstimate

Estimated

Difference

Est

Error

(SEDF)

Actual

Error

Absolute

Error

t-

score LeaderEstimate

Estimated

Difference

Est Error

(SEDF)

Actual

Error

Absolute

Error

t-

score

Alabama

Senate 3

-35.2 Shel

63.3

-26.6

8.3 -8.6

8.6 -1.0 Shel

63.4

-26.7

4.5

-8.5

8.5

-1.9

Alaska

Senate 3

-3.1 Know

50.1

0.1

4.2 -3.2

3.2 -0.8 Know

50.6

1.1

2.9

-4.2

4.2

-1.5

Arizona

Senate 3

-56.1 McCa

77.9

-55.8

3.3 -0.3

0.3 -0.1 McCa

75.0

-50.0

3.2

-6.1

6.1

-1.9

Arkansas

Senate 3

12.0 Linc

55.4

10.8

4.6

1.2

1.2 0.2 Linc

56.8

13.5

3.7

-1.5

1.5

-0.4

California

Senate 3

20.3 Boxr

60.9

21.7

2.7 -1.4

1.4 -0.5 Boxr

59.3

18.5

3.1

1.8

1.8

0.6

Colorado

Senate 3

4.3 Salz

52.2

4.3

1.0

0.0

0.0 0.0 Salz

51.9

3.8

2.0

0.5

0.5

0.2

Connecticut

Senate 3

33.7 Dodd

74.1

48.1

3.9 -14.4

14.4 -3.7 Dodd

72.2

44.4

3.2 -10.7

10.7

-3.3

Florida

Senate 3

-1.1 Cast

51.4

2.8

1.7 -3.9

3.9 -2.3 Cast

51.1

2.1

1.6

-3.2

3.2

-2.0

Georgia

Senate 3

-18.0 Isak

57.5

-14.9

4.4 -3.1

3.1 -0.7 Isak

56.5

-13.0

2.8

-5.0

5.0

-1.8

Hawaii

Senate 3

54.5 Inoy

72.9

45.8

5.7

8.7

8.7 1.5 Inoy

69.6

39.2

4.1

15.3

15.3

3.7

Illinois

Senate 3

42.9 Obam

73.8

47.6

4.5 -4.7

4.7 -1.1 Obam

71.7

43.5

3.1

-0.6

0.6

-0.2

Indiana

Senate 3

24.3 Bayh

62.8

25.6

2.9 -1.3

1.3 -0.5 Bayh

63.6

27.1

3.1

-2.8

2.8

-0.9

Iowa

Senate 3

-42.5 Gras

72.4

-44.8

2.9

2.3

2.3 0.8 Gras

71.0

-42.1

2.5

-0.4

0.4

-0.2

Kansas

Senate 3

-41.9 Brwb

74.6

-49.2

5.4

7.3

7.3 1.3 Brwb

73.1

-46.2

3.5

4.3

4.3

1.2

Kentucky

Senate 3

-1.4 Mong

50.1

0.2

2.5 -1.6

1.6 -0.7 Bung

50.8

-1.5

2.1

0.1

0.1

0.1

Louisiana

Senate 3

-21.7 Vitr

53.8

-28.0

4.5

6.3

6.3 1.4 Vitr

51.3

-26.2

3.3

4.5

4.5

1.3

Maryland

Senate 3

31.1 Mikl

64.9

29.8

5.9

1.3

1.3 0.2 Mikl

63.5

26.9

4.2

4.2

4.2

1.0

Missouri

Senate 3

-13.3 Bond

54.6

-9.2

2.2 -4.1

4.1 -1.8 Bond

54.8

-9.5

2.1

-3.8

3.8

-1.8

Nevada

Senate 3

26.0 Reid

64.2

31.5

3.3 -5.5

5.5 -1.7 Reid

63.5

30.2

3.0

-4.2

4.2

-1.4

New Hampshire Senate 3

-32.6 Greg

60.2

-20.4

3.3 -12.2

12.2 -3.7 Greg

60.6

-21.2

2.9 -11.4

11.4

-4.0

New York

Senate 3

46.0 Shmr

77.9

58.8

3.6 -12.8

12.8 -3.6 Shmr

73.0

51.8

2.4

-5.8

5.8

-2.4

North Carolina Senate 3

-4.6 Bowl

50.4

0.8

3.5 -5.4

5.4 -1.6 Bowl

50.1

0.1

2.6

-4.7

4.7

-1.8

North Dakota

Senate 3

36.4 Dorg

71.2

42.4

4.1 -6.0

6.0 -1.5 Dorg

66.4

32.9

3.4

3.5

3.5

1.0

Evaluation of Edison/Mitofsky Election System

page 24

January 19, 2005

Ohio

Senate 3

-27.8 Voin

56.4

-12.7

4.1 -15.1

15.1 -3.7 Voin

56.7

-13.5

3.2 -14.3

14.3

-4.5

Oklahoma

Senate 3

-11.6 Cobr

56.4

-12.8

2.4

1.2

1.2 0.5 Cobr

54.7

-9.4

2.0

-2.2

2.2

-1.1

Pennsylvania Senate 3

-10.8 Spec

47.2

-0.5

3.9 -10.3

10.3 -2.6 Spec

50.5

-6.6

2.8

-4.2

4.2

-1.5

South Carolina Senate 3

-9.6 DeMn

50.9

-1.8

4.4 -7.8

7.8 -1.8 DeMn

51.1

-2.3

2.8

-7.3

7.3

-2.6

South Dakota Senate 3

-1.2 Thun

51.1

-2.2

3.5

1.0

1.0 0.3 Thun

50.6

-1.3

2.6

0.1

0.1

0.0

Utah

Senate 3

-39.6 Benn

68.0

-36.1

3.4 -3.5

3.5 -1.0 Benn

67.4

-34.7

2.6

-4.9

4.9

-1.9

Vermont

Senate 3

46.1 Leah

78.2

56.5

4.0 -10.4

10.4 -2.6 Leah

75.0

50.1

3.4

-4.0

4.0

-1.2

Washington

Senate 3

12.3 Mury

58.8

17.5

3.0 -5.2

5.2 -1.8 Mury

57.5

15.1

2.5

-2.8

2.8

-1.1

Wisconsin

Senate 3

11.3 Fein

57.4

14.9

3.7 -3.6

3.6 -1.0 Fein

57.0

14.1

3.1

-2.8

2.8

-0.9

Average -3.6

5.4 -1.0

-2.5

4.7

-1.0

St. Dev

1.49

1.657

Evaluation of Edison/Mitofsky Election System

page 26

January 19, 2005

Comparison of Democratic Primary Survey Estimates vs. Actual Results:

On average, we did not see errors of this magnitude in the exit poll estimates from the

2004 Democratic Primaries. The table on the next page shows the best survey estimates

and the actual results for the top three candidates in the 22 Democratic Primary exit polls

that were conducted from January to March 2004. The average error on the candidate in

the primaries was 1.9 points. Three exit polls contributed most to this error – Florida,

Texas and Vermont. In Florida and Texas, the NEP absentee telephone surveys were

cancelled after John Edwards withdrew from the race. Candidates who had withdrawn

did better among the absentee/early voters than among election day voters in these states.

Thus, the election day estimates in Florida and Texas overstate the total vote for Kerry

including the absentees. In Vermont there was a significant (6%) write-in vote for John

Edwards who was not on the ballot and thus was not included in the exit poll estimate

computations. Note that since most primaries had more than two major candidates the

analysis in this section is on the error in the estimate on the candidate. All other analysis

in this report is on the difference between the two major candidates.

The smaller error in the Democratic primaries is yet another indication that the errors in

the 2004 General Election exit polls were due to differences in how Democrats and

Republicans responded to the exit polls in this election.

Exit Poll Results from the 2004 Democratic Presidential Primaries

Comparison of Best Exit Poll Estimate and Actual Result for Top 3 candidates

State

Candidates # of Precincts Best Exit Poll Est Actual Result Difference

Absolute

Difference

New Hampshire Kerry

40

36.0

38.4

-2.4

2.4

Dean

29.4

26.3

3.1

3.1

Edwards

12.4

12.1

0.3

0.3

Arizona

Kerry

35

42.8

42.6

0.2

0.2

Clark

25.9

26.5

-0.6

0.6

Dean

14.7

14.0

0.7

0.7

Delaware

Kerry

30

49.6

50.4

-0.8

0.8

Lieberman

11.7

11.1

0.6

0.6

Edwards

10.4

11.0

-0.6

0.6

Missouri

Kerry

35

50.6

50.6

0.0

0.0

Edwards

24.3

24.6

-0.3

0.3

Dean

10.9

8.7

2.2

2.2

Oklahoma

Clark

35

29.5

29.9

-0.4

0.4

Edwards

30.6

29.5

1.1

1.1

Kerry

28.8

26.8

2.0

2.0

South Carolina Edwards

40

45.1

45.1

0.0

0.0

Kerry

28.8

29.8

-1.0

1.0

Sharpton

9.5

9.7

-0.2

0.2

Tennessee

Kerry

32

42.5

41.0

1.5

1.5

Edwards

28.9

26.5

2.4

2.4

Clark

18.7

23.1

-4.4

4.4

Virginia

Kerry

35

53.0

51.5

1.5

1.5

Edwards

25.6

26.6

-1.0

1.0

Clark

9.5

9.2

0.3

0.3

Evaluation of Edison/Mitofsky Election System

page 27

January 19, 2005

Exit Poll Results from the 2004 Democratic Presidential Primaries

Comparison of Best Exit Poll Estimate and Actual Result for Top 3 candidates

State

Candidates # of Precincts Best Exit Poll Est Actual Result Difference

Absolute

Difference

Wisconsin

Kerry

40

36.9

39.6

-2.7

2.7

Edwards

36.2

34.3

1.9

1.9

Dean

17.8

18.2

-0.4

0.4

California

Kerry

44

63.5

64.4

-0.9

0.9

Edwards

23.8

19.8

4.0

4.0

Dean

5.2

4.2

1.0

1.0

Connecticut

Kerry

20

65.1

58.3

6.8

6.8

Edwards

22.0

23.7

-1.7

1.7

Lieberman

2.3

5.2

-2.9

2.9

Georgia

Kerry

35

50.6

46.8

3.8

3.8

Edwards

39.4

41.4

-2.0

2.0

Sharpton

6.8

6.2

0.6

0.6

Maryland

Kerry

30

59.8

59.6

0.2

0.2

Edwards

26.2

25.6

0.6

0.6

Sharpton

5.7

4.5

1.2

1.2

Massachusetts Kerry

20

71.0

71.7

-0.7

0.7

Edwards

18.7

17.6

1.1

1.1

Kucinich

3.0

4.1

-1.1

1.1

New York

Kerry

41

60.4

60.8

-0.4

0.4

Edwards

19.5

20.0

-0.5

0.5

Sharpton

11.3

8.0

3.3

3.3

Ohio

Kerry

36

53.1

51.8

1.3

1.3

Edwards

34.1

34.1

0.0

0.0

Kucinich

10.2

9.0

1.2

1.2

Rhode Island

Kerry

20

71.2

71.5

-0.3

0.3

Edwards

20.4

18.6

1.8

1.8

Dean

3.7

4.0

-0.3

0.3

Vermont

Dean

20

66.8

53.6

13.2

13.2

Kerry

27.9

31.6

-3.7

3.7

Kucinich

5.3

4.1

1.2

1.2

Florida

Kerry

42

82.1

77.2

4.9

4.9

Edwards

8.5

10.0

-1.5

1.5

Sharpton

2.4

2.8

-0.4

0.4

Louisiana

Kerry

21

71.4

69.7

1.7

1.7

Edwards

17.5

16.1

1.4

1.4

Dean

6.3

4.9

1.4

1.4

Mississippi

Kerry

22

83.0

78.4

4.6

4.6

Edwards

6.1

7.3

-1.2

1.2

Sharpton

3.5

5.2

-1.7

1.7

Texas

Kerry

35

78.5

67.1

11.4

11.4

Edwards

10.2

14.4

-4.2

4.2

Dean

5.0

4.8

0.2

0.2

Average

1.9

Evaluation of Edison/Mitofsky Election System

page 28

January 19, 2005

Evaluation of Samples

To determine the possible sources of the errors in the exit poll estimates, we began by

examining the precinct samples. We conclude that, on average, the precinct samples did

not contribute to the error of the exit poll estimates in the Kerry direction.

The table on the following pages shows the vote estimates computed using the actual vote

returns for each precinct in our samples. The first set of estimates is from the precincts

that were in the exit poll sample. The second set of estimates is from the larger vote

count precinct samples. The exit poll sample is a sub-sample of the precinct samples,

although in some states the two samples were the same.

It should also be noted that these sample precinct model computations include absentee

vote adjustments for the states in which the absentee/early vote is not reported at the

precinct level.

The average error on the vote estimates using actual precinct returns for the full samples

was 0.31 percentage points in the Bush direction. For the exit poll samples the vote

estimates using the actual precinct returns was 0.43 percentage points in the Bush

direction.

Evaluation of Edison/Mitofsky Election System

page 29

January 19, 2005

Comparison of Best SPM Estimate Using Precinct Reported Vote in Exit Poll and Full Samples with the Final Election Result:

Estimated Difference and Final Margin are positive when Democrat (Kerry) leads and negative when Republican (Bush) leads

Presidential Races Only

Best SPM Reported Vote in Exit Poll Sample:

Best SPM Reported Vote in Full Sample:

State

Race

Final

Margin LeaderEstimate

Estimated

Difference

Estimated

Error

(SEDF)

Actual

Error t-score LeaderEstimate

Estimated

Difference

Estimated

Error

(SEDF)

Actual

Error t-score

Alabama

President

-25.6 Bush

63.6

-27.4

1.6

-1.8

-1.1 Bush

62.2

-24.7

1.2

0.9

0.8

Alaska

President

-25.6 Bush

61.8

-25.4

1.3

0.2

0.2 Bush

61.8

-25.4

1.3

0.2

0.2

Arizona

President

-10.5 Bush

54.1

-8.2

2.4

2.3

1.0 Bush

54.1

-8.2

2.4

2.3

1.0

Arkansas

President

-9.7 Bush

53.0

-6.4

1.4

3.3

2.4 Bush

53.7

-7.9

1.0

1.8

1.8

California

President

10.0 Kery

53.6

7.3

1.9

-2.7

-1.5 Kery

55.3

10.5

1.7

0.5

0.3

Colorado

President

-4.7 Bush

52.1

-5.0

2.7

-0.3

-0.1 Bush

51.8

-4.5

2.6

0.1

0.1

Connecticut

President

10.4 Kery

54.9

10.6

2.1

0.3

0.1 Kery

55.6

12.0

1.2

1.7

1.3

Delaware

President

7.6 Kery

53.8

8.2

2.0

0.6

0.3 Kery

52.9

6.3

1.0

-1.2

-1.3

District of Columbia

President

79.8 Kery

90.1

80.8

1.4

1.0

0.7 Kery

90.1

80.8

1.4

1.0

0.7

Florida

President

-5.0 Bush

51.0

-2.4

1.7

2.6

1.6 Bush

51.1

-2.6

1.5

2.4

1.5

Georgia

President

-16.6 Bush

58.1

-16.2

1.3

0.4

0.3 Bush

57.8

-15.5

1.3

1.1

0.9

Hawaii

President

8.7 Kery

55.1

10.3

2.9

1.5

0.5 Kery

55.2

10.4

3.0

1.6

0.6

Idaho

President

-38.1 Bush

71.4

-42.8

1.3

-4.7

-3.5 Bush

71.4

-42.8

1.3

-4.7

-3.5

Illinois

President

10.1 Kery

56.1

12.2

1.8

2.1

1.1 Kery

54.9

9.8

1.2

-0.4

-0.3

Indiana

President

-20.7 Bush

60.8

-21.5

1.6

-0.9

-0.5 Bush

59.6

-19.3

1.3

1.4

1.1

Iowa

President

-0.7 Bush

49.9

-0.7

1.8

-0.1

0.0 Bush

49.6

-0.1

1.7

0.6

0.3

Kansas

President

-25.4 Bush

65.3

-31.4

1.7

-6.0

-3.5 Bush

65.3

-31.4

1.7

-6.0

-3.5

Kentucky

President

-19.9 Bush

59.1

-18.6

1.8

1.3

0.7 Bush

59.5

-19.4

1.4

0.4

0.3

Louisiana

President

-14.5 Bush

56.8

-14.1

1.4

0.4

0.3 Bush

56.6

-13.5

1.1

1.0

0.9

Maine

President

8.8 Kery

52.6

6.5

1.1

-2.3

-2.0 Kery

53.2

7.6

0.7

-1.2

-1.8

Maryland

President

13.0 Kery

54.1

8.7

1.4

-4.3

-3.0 Kery

54.5

9.5

1.7

-3.5

-2.0

Massachusetts

President

25.2 Kery

63.4

26.9

2.1

1.7

0.8 Kery

63.4

26.9

2.1

1.7

0.8

Evaluation of Edison/Mitofsky Election System

page 30

January 19, 2005

Michigan

President

3.4 Kery

50.9

2.6

1.3

-0.9

-0.7 Kery

51.1

3.0

1.2

-0.4

-0.4

Minnesota

President

3.5 Kery

52.0

4.7

1.0

1.2

1.2 Kery

51.9

4.5

0.7

1.1

1.4

Mississippi

President

-20.0 Bush

62.3

-25.0

1.8

-4.9

-2.7 Bush

62.3

-25.0

1.8

-4.9

-2.7

Missouri

President

-7.2 Bush

55.0

-10.0

1.1

-2.8

-2.4 Bush

54.5

-9.0

0.9

-1.8

-1.9

Montana

President

-20.5 Bush

61.6

-24.6

2.8

-4.1

-1.5 Bush

61.6

-24.6

2.8

-4.1

-1.5

Nebraska

President

-33.4 Bush

66.1

-33.0

1.6

0.4

0.2 Bush

66.1

-33.0

1.6

0.4

0.2

Nevada

President

-2.6 Bush

49.9

-1.3

2.5

1.3

0.5 Bush

50.3

-2.1

2.4

0.5

0.2

New Hampshire

President

1.4 Bush

49.9

-0.4

0.9

-1.8

-2.1 Kery

50.4

1.5

0.6

0.1

0.1

New Jersey

President

6.8 Kery

53.4

7.4

1.9

0.7

0.4 Kery

52.4

5.3

1.4

-1.5

-1.0

New Mexico

President

-0.8 Kery

49.9

0.5

2.7

1.3

0.5 Kery

50.5

1.7

2.6

2.5

0.9

New York

President

17.4 Kery

58.0

18.1

2.1

0.7

0.3 Kery

58.1

17.9

1.5

0.5

0.3

North Carolina

President

-12.4 Bush

56.4

-12.8

1.7

-0.4

-0.2 Bush

55.9

-11.9

1.6

0.5

0.3

North Dakota

President

-27.4 Bush

64.1

-29.5

2.3

-2.1

-0.9 Bush

64.1

-29.5

2.3

-2.1

-0.9

Ohio

President

-2.1 Bush

51.6

-3.2

1.1

-1.1

-1.0 Bush

50.8

-1.6

1.0

0.5

0.5

Oklahoma

President

-31.1 Bush

65.0

-30.0

1.1

1.2

1.0 Bush

65.0

-30.0

0.8

1.1

1.3

Pennsylvania

President

2.3 Kery

52.1

4.1

0.8

1.8

2.2 Kery

52.3

4.5

0.7

2.2

3.1

Rhode Island

President

20.8 Kery

59.8

20.5

2.5

-0.2

-0.1 Kery

60.0

20.8

2.4

0.0

0.0

South Carolina

President

-17.1 Bush

57.7

-15.8

1.5

1.3

0.9 Bush

58.4

-17.1

1.0

0.0

0.0

South Dakota

President

-21.5 Bush

61.2

-23.7

1.5

-2.2

-1.5 Bush

60.4

-22.0

1.0

-0.5

-0.5

Tennessee

President

-14.3 Bush

59.2

-19.7

2.7

-5.3

-1.9 Bush

59.3

-19.8

2.6

-5.4

-2.0

Texas

President

-22.9 Bush

64.3

-28.5

2.9

-5.7

-1.9 Bush

64.1

-28.2

2.8

-5.4

-1.9

Utah

President

-45.5 Bush

71.6

-44.5

1.9

1.1

0.6 Bush

71.6

-44.5

1.9

1.1

0.6

Vermont

President

20.1 Kery

60.4

22.4

2.0

2.3

1.1 Kery

59.3

20.0

0.9

-0.1

-0.1

Virginia

President

-8.2 Bush

53.4

-6.8

1.1

1.4

1.3 Bush

54.1

-8.2

0.9

0.0

0.0

Washington

President

7.2 Kery

54.0

9.0

2.8

1.8

0.7 Kery

54.0

8.9

2.8

1.7

0.6

West Virginia

President

-12.8 Bush

54.8

-10.2

1.4

2.7

1.9 Bush

55.2

-11.1

1.0

1.7

1.7

Wisconsin

President

0.4 Kery

50.6

1.7

1.4

1.3

0.9 Kery

49.9

0.2

0.8

-0.2

-0.3

Wyoming

President

-39.7 Bush

71.8

-44.8

2.0

-5.1

-2.6 Bush

71.6

-44.3

2.0

-4.6

-2.3

Actual Error Mean:

-0.43

Actual Error Mean:

-0.31

t-score Mean:

-0.22

t-score Mean: -0.08

t-score Standard Deviation:

1.48

t-score Standard Deviation:

1.38

Evaluation of Edison/Mitofsky Election System

page 31

January 19, 2005

Evaluation of the Within Precinct Error (WPE)

As we have shown in the previous section, most of the error in the exit poll estimates was

not caused by the sample of precincts. The samples produced very good estimates of the

final vote count when the vote returns, rather than exit poll results, were used to make the

estimates. The additional error in the exit polls must be caused by errors that occurred

within the precincts from sampling voters. This is called WPE, Within Precinct Error. It

is not the total error in an estimate from an exit poll. Other parts of the estimating process

may increase or decrease the final statistical error. The WPE is only one component of

the error. This section examines that source of error.

For the 1,460 exit poll precincts where we have both exit poll tallies and final vote

returns, we calculated an average WPE of -6.5 on the difference between Kerry and

Bush. (The sign is “+” when Bush is overstated and “–” when Kerry is overstated.) In

other words, on average the exit poll results from each precinct overstated the Kerry-

Bush difference by -6.5 points. This is the largest WPE that we have observed on a

national level in the last five presidential elections, greater than the next largest WPE,

which occurred in the 1992 VRS exit polls when the average WPE on the Clinton-Bush

difference was -5.0 points.

First, we will describe the process of calculating WPE. Within each precinct a sample of

voters is selected. Theoretically, the within precinct sampling errors across all precincts

should average close to zero if the sample is very large. If it did, what remained would be

mostly a statistical bias. The -6.5 percentage points of WPE in this election is mostly the

statistical bias. There is statistical bias when the exit poll consistently overstates one

candidate.

The absolute value of the WPE does not average out the sampling error. It is included in

the absolute WPE along with the bias. The within precinct sampling error on the

difference averages 10.3 percentage points per precinct. This calculation is based on

114,559 sample voters at the 1,460 precincts in our state and national samples. The large

sampling error per precinct makes comparisons with the official precinct vote for a single

precinct not very useful. However, estimates of a state based on the entire sample can

provide reliable results, provided there is little or no statistical bias. The bias component

in 2004 is most of the -6.5 percentage points cited above. The mean absolute WPE per

precinct is 14.4. It is possible to have a large sampling error and no statistical bias. The

2004 problem is the statistical bias. In other years, when the statistical bias was smaller,

the overall error has been almost as large as it is in 2004.

We also analyze the impact of different factors on the completion rates reported at each

polling location. The average completion rate for all exit poll precincts was 53%. While

we cannot measure the completion rate by Democratic and Republican voters,

hypothetical completion rates of 56% among Kerry voters and 50% among Bush voters

overall would account for the entire Within Precinct Error that we observed in 2004.

Evaluation of Edison/Mitofsky Election System

page 32

January 19, 2005

WPE – 1988-2004

The table below shows the Average WPE by state for the presidential races from 1988 to

2004. There is some correlation of WPE by state in 2004 to the past elections especially

with 2000 and 1992. The past WPE is not always predictive of the direction and size of

the WPE but some states seem to be more consistent in demonstrating an average WPE in

the Democratic direction. Seven of the ten states with the largest WPE in 1992 were also

among the fifteen states with the largest WPE in 2004 (California, Connecticut,

Delaware, Maryland, New Hampshire, New Jersey and Vermont). While much attention

has been paid to the size of the overstatement in Florida, Ohio and Pennsylvania, it is

important to note that there were several other states with a higher WPE – Alabama,

Connecticut, Delaware, Mississippi, New Hampshire, New York, North Carolina and

Vermont.

Correlation of State Presidential WPE

Averages Between Past Years:

2000

1996

1992

1988

2004

0.48

0.19

0.35

0.30

2000

0.05

0.12

0.23

1996

0.15

0.26

1992

0.29

Average Within Precinct Error (WPE) For Presidential Races 1988-2004 (see note below)

2004

2000

1996

1992

1988

st

STATE

Overlay

Model WPE

IM WPE

WPE

Overlay

WPE

Overlay

WPE

Overlay

WPE

Overlay

WPE

1

AL

10

-19.5

-10.0 -11.3

14 -5.5

16

-2.4

15

-1.2

38 0.0

2

AK

25

-10.9

-9.3

-9.6

.

.

.

.

.

.

19 -1.2

3

AZ

30

-6.8

-0.3

-4.6

.

.

19

-7.7

16

-6.6

.

.

4

AR

31

-1.0

-1.3

-0.5

18 -3.2

22

1.5

10

-7.8

49 0.8

5

CA

30

-13.8

-11.6 -10.9

26 -3.8

37

-4.7

41

-8.5

92 -3.8

6

CO

34

-5.3

-6.1

-6.1

13 -5.6

23

-1.5

26

-7.2

46 -3.0

7

CT

11

-15.8

-16.0 -15.7

20 -0.9

20

4.2

25

-8.3

39 -5.2

8

DE

11

-16.3

-15.9 -15.9

21 -7.1

15

-1.3

11

-7.3

23 -1.8

9

DC

11

-2.6

-2.8

-3.4

.

.

.

.

.

.

.

.

10

FL

51

-8.0

-7.8

-7.6

39 -0.6

35

-0.6

25

-5.5

49 -2.4

11 GA

31

-2.2

-1.3

-2.2

27 -5.6

26

3.3

20

-6.5

47 1.8

12

HI

11

-4.2

-8.2

-4.7

.

.

.

.

.

.

17 -0.6

13

ID

11

0.4

-4.0

-1.0

7 2.5

12

-3.5

9

-0.5

.

.

14

IL

26

-4.0

-3.5

-4.4

28 -6.4

23

1.2

34

-6.1

67 -2.6

15

IN

21

-1.8

-2.2

-1.5

10 -3.6

24

-2.0

15

-6.8

26 -9.2

16

IA

41

-2.4

-3.0

-3.0

19 3.0

21

0.3

16

-2.0

40 -0.6

17

KS

11

-1.5

-1.1

-1.7

11 -4.4

19

-3.2

11

-3.4

48 -3.6

18

KY

16

0.5

-0.4

0.1

22 4.4

27

1.0

26

-3.9

19 -1.0

19

LA

35

-3.8

-2.6

-3.8

19 -0.6

18

6.5

26

1.0

59 -2.4

20 ME

31

-3.5

-4.0

-3.8

19 -2.1

26

-4.4

18

-5.1

28 -6.4

Evaluation of Edison/Mitofsky Election System

page 33

January 19, 2005

21 MD

16

-8.8

-7.3

-8.1

19 -4.3

13

-3.6

19

-8.1

38 -5.2

22 MA

11

-6.6

-7.7

-5.8

14 -4.3

36

-3.3

20

-7.1

41 -6.8

23

MI

46

-6.6

-6.4

-6.3

36 -2.2

20

-3.5

36

-4.9

64 -1.2

24 MN

41

-10.8

-9.2

-9.3

28 0.5

26

1.7

23

-6.4

40 1.2

25 MS

11

-15.6

-18.5 -11.3

12 -3.2

14

-0.3

15

-5.1

47 0.2

26 MO

45

-5.2

-5.8

-5.8

28 1.8

18

-5.8

30

-8.6

48 -0.6

27 MT

11

-5.7

2.6

1.8

14 3.2

15

-2.4

10

0.7

31 -4.4

28

NE

11

-8.0

-8.7

-8.1

10 -4.1

14

-6.5

11

-2.8

10 -5.0

29

NV

36

-15.6

-9.9 -10.1

25 -6.0

.

.

2

2.1

28 -3.0

30 NH

33

-15.6

-14.0 -13.6

25 -2.4

19 -12.2

16 -10.1

29 -6.0

31

NJ

25

-10.2

-9.1

-9.7

27 -0.4

26

-1.9

31 -11.2

54 -4.2

32 NM

31

-8.4

-8.0

-7.8

13 5.1

15

-7.0

19

-6.3

28 -6.6

33

NY

25

-11.9

-12.2 -11.4

40 -3.3

25

2.1

35

-4.6

48 -7.2

34 NC

36

-12.0

-11.9 -11.3

25 -9.8

35

-6.5

29

-4.2

46 -0.4

35 ND

11

5.1

1.7

5.2

16 2.0

16

-2.0

11

-4.2

39 -1.6

36 OH

45

-11.2

-10.6 -10.9

26 -1.0

34

-3.1

33

-4.4

44 -1.6

37 OK

26

1.4

1.2

1.9

10 4.7

21

2.0

21

-4.7

20 5.6

38 OR

.

.

.

.

.

28

-2.4

16 -13.6

37 -7.4

39

PA

46

-11.0

-8.4

-8.8

37 -0.8

31

-3.6

34

-2.0

70 -0.8

40

RI

11

-5.3

-5.3

-4.7

21 -0.4

16

-1.1

11

-9.0

9 0.2

41

SC

30

-9.3

-9.7 -10.0

14 -3.5

23

-2.8

21

-2.0

33 -1.4

42

SD

34

4.0

5.1

4.2

12 -0.9

19

-2.3

14

2.8

36 -1.0

43

TN

21

0.0

-1.3

-0.5

19 2.2

18

-3.0

14

-6.8

47 1.0

44

TX

16

-3.0

-7.6

-4.8

13 -0.4

38

-0.6

31

-2.8

93 0.8

45

UT

11

-1.9

-4.3

-6.4

7 1.0

12

-3.5

12

-2.2

.

.

46

VT

11

-17.1

-15.2 -15.0

16 0.4

16

-5.0

15

-8.6

30 -5.8

47

VA

21

-7.0

-8.7

-7.9

36 -2.0

20

-6.5

16

-3.5

50 0.6

48 WA

30

-8.7

-8.0

-8.4

7 -3.7

12

-2.0

12

-5.5

25 -5.4

49 WV

38

4.7

5.9

5.8

20 4.5

16

-2.6

8

-3.2

26 -0.2

50

WI

41

-5.7

-4.8

-4.7

34 2.4

19

-2.8

31

-2.5

60 2.2

51 WY

11

-8.2

-7.0

-4.3

16 -1.0

21

-3.9

11

-5.9

9 1.0

1

Note: WPE values are on the Dem-Rep difference. A negative WPE value indicates that the

Democratic candidate vote was overstated.

The column definitions are:

1. 'WPE' is the statewide WPE computed using the method used at VNS prior to 2002. In this

method, the four extreme precincts were removed and then the precinct WPE values were averaged.

2. ‘Model WPE’ is the statewide WPE computed using the SPM after removing the extreme

precincts (see Statistical Spec for details). This is the WPE value on the Decision Summary screen.

3. ‘IM WPE’ is the statewide WPE computed using the average of each precinct’s WPE. Extreme

precincts have not been removed. This is the WPE value on the Input Management screen.

4. ‘Overlay’ is the number of overlaid precincts (both exit poll and reported vote) after the extreme

precincts have been removed.

Evaluation of Edison/Mitofsky Election System

page 34

January 19, 2005

Since election day we have examined information from all 1,480 exit poll precincts in our

samples, including all of the exit poll data from election day. This includes presidential

vote tallies; questionnaires by demographics; refusals and misses by demographic, etc.

There were only 20 precincts where we were unable to get the vote returns from the

precinct in our exit poll sample.

Note: The WPE values do not measure just the error of the exit poll in precincts that

contain significant absentee vote. When absentees were greater than 15 % statewide, we

removed precincts from this study that had the absentees merged with the precinct vote.

In these precincts we cannot obtain counts of the election day vote separate from the

absentee vote. Also, not included in this study are any precincts with fewer than 20

interviews as well as three additional precincts with large absolute WPE (112, -111, -80)

indicating that the precincts or candidate vote were recorded incorrectly. Out of the

1,480 exit poll precincts, 1,250 were included in the analysis that follows.

COMPARISON TO RECENT ELECTIONS

The following table displays a count of exit poll precincts grouped by the size and

direction of our current computation of the WPE for all 1,460 exit poll precincts:

2004

2000

1996

Number of Precincts With:

N

%

N

%

N

%

WPE < -5 (Dem Direction)

767

53

394

36

465

39

-5 <WPE < 5

341

23

374

34

439

36

WPE > 5 (Rep Direction)

352

24

315

29

305

25

In 2004, the Kerry-Bush difference within precinct was overstated by more than 5

percentage points in 53% of the precincts. More than twice as many precincts overstated

Kerry than overstated Bush.

The following table summarizes the WPE for the 1988-2004 presidential elections:

For the Precinct WPE:

2004 2000 1996 1992 1988

Average

-6.5

-1.8

-2.2

-5.0

-2.2

Average Absolute Value

14.4

11.3

9.9

NA

NA

Standard Deviation

18.2

16.8

13.3

NA

NA

This table shows that the WPE is larger than it had been. The variation of the WPE

(standard deviation) is not nearly as large as the difference in the average WPE. What this

means is that the errors in 1996 and 2000 were more random, while the errors in 2004

were much more in one direction. While WPE is clearly greater in this election, it is

really the direction of the WPE that changed the most. Between 2000 and 2004, the

average WPE increased more than 3 times resulting in a Democratic overstatement but

the average absolute value of the WPE increased 1.3 times.

Evaluation of Edison/Mitofsky Election System

page 35

January 19, 2005

As the WPE was largest in the 1992 and 2004 elections, it is appropriate to analyze the

factors that these two elections have in common. The 1992 and 2004 elections had the

highest voter turnout among the last five elections. Also, pre-election polling showed

that voters reported paying more attention to these presidential campaigns than other

recent elections.

We need to further investigate the possible relationship between increased voter turnout

and high levels of voter interest with the increased size of WPE to see if these factors are

contributing to partisan differences in response rates in the exit polls.

A. NON-INTERVIEWER EFFECTS

1. Interviewing Rate:

The interviewers are instructed to sample the voters as they leave the polling place by

following an interviewing rate. The interviewing rate is defined as the number of voters

that the interviewer counts in order to select a voter to approach. In other words an

interviewing rate of “1” means that the interviewer will approach every voter; and

interviewing rate of “10” means that the interviewer will approach every 10

th

voter.

The statistical bias generally gets worse as the interviewing rate increases. This occurs at

polling locations where a large number of people are voting, either because our sample

precinct is large or because other precincts in addition to our sample precinct may be

voting at the same polling place. The increased WPE in these precincts could suggest

that some interviewers do not follow the interviewing rate exactly. As the interviewing

rate increases so does the potential for interviewers to exercise more of their own

judgment on whom they will approach in order to participate in the exit poll.

However, the data also show a WPE in the Kerry direction still exists even in precincts

where the interviewer was instructed to ask every single voter to participate in the exit

poll (and the interviewer had no option in the selection of the respondent). In precincts

with an interviewing rate of “1”, there was still a WPE in the Kerry direction of almost 4

points. Again, this indicates that a portion of the WPE is coming from differential non-

response.

Election Year WPE

Voter Turnout (%VAP) % paying a lot of attention to campaign*

2004 -6.5

55.3%

66%

2000 -1.8

51.3%

49%

1996 -2.2

49.1%

40%

1992 -5.0

55.2%

68%

1988 -2.2

50.2%

49%

* Source: CBS News national polls from late October of each year

Question: How much attention have you been able to pay to the Presidential campaign - a

lot, some, not much, or no attention so far?

Evaluation of Edison/Mitofsky Election System

page 36

January 19, 2005

Interviewing rate at the beginning of

election day*

mean WPE

median WPE

mean Abs(WPE)

N

1

-3.9

-4.5

14.0

142

2

-3.3

-3.7

11.9

144

3

-6.7

-4.9

14.1

178

4

-7.0

-7.2

13.4

136

5

-6.9

-5.1

13.9

159

6

-8.4

-9.4

15.0

101

7

-7.0

-7.4

12.6

80

8

-7.1

-4.5

13.3

62

9

-5.7

-5.8

11.0

50

10

-10.5

-9.7

15.4

198

*1 = attempt to interview every voter, 2 = every other voter, 3 = every third voter, etc

.

There is no significant relationship between completion rates and interviewing rate. :

Interviewing rate at the beginning of

election day

Completion Rate

Refusal Rate

Miss Rate

1

0.54

0.35

0.12

2

0.49

0.37

0.14

3

0.50

0.38

0.12

4

0.53

0.36

0.11

5

0.54

0.35

0.11

6

0.56

0.35

0.09

7

0.58

0.32

0.11

8

0.54

0.39

0.08

9

0.62

0.29

0.08

10

0.56

0.35

0.09

2. Precinct Partisanship:

When the precincts were grouped based on their vote (high Kerry through high Bush), the

high Bush precincts have the greatest statistical bias. The average signed WPE increases

sharply with the increase in the Bush vote. A small Bush overstatement exists in the

highest Kerry precincts. The analysis is more meaningful if the precincts where Kerry

and Bush received more than 80% of the vote are ignored. In the highest Kerry precincts

there is little room for overstatement of his vote. Similarly the highest Bush precincts

have more freedom to only overstate Kerry rather than Bush. The three middle groups of

precincts show a relatively consistent overstatement of Kerry.

Precinct partisanship

mean WPE

median WPE

mean Abs(WPE)

N

High Dem (Kerry>=.80)

0.3

-0.4

8.8

90

Mod Dem (0.60<=Kerry<.80)

-5.9

-5.5

13.4

165

Even (0.40<=Kerry<.60)

-8.5

-8.3

15.2

540

Mod Rep (0.20<=Kerry<.40)

-6.1

-6.1

13.2

415

High Rep (0.00<=Kerry<.20)

-10.0

-5.8

12.4

40

Evaluation of Edison/Mitofsky Election System

page 37

January 19, 2005

There was no significant difference between the completion rates and the precinct

partisanship:

Precinct partisanship

Completion Rate

Refusal Rate

Miss Rate

High Dem (Kerry>=.80)

0.53

0.35

0.12

Mod Dem (0.60<=Kerry<.80)

0.55

0.33

0.12

Even (0.40<=Kerry<.60)

0.52

0.37

0.11

Mod Rep (0.20<=Kerry<.40)

0.55

0.35

0.10

High Rep (0.00<=Kerry<.20)

0.56

0.33

0.11

3. Interviewer distance from exit:

The average WPE was smaller when the interviewers were able to stand inside the

polling location or within at least 25 feet of the entrance.

Distance

mean WPE

median WPE

mean Abs(WPE)

N

Inside

-5.3

-4.2

11.8

416

Right outside the entrance

-6.4

-7.5

13.4

207

10 to 25 feet away

-5.6

-4.2

14.0

220

25 to 50 feet away

-7.6

-7.3

14.8

150

50 to 100 feet away

-9.6

-10.3

17.7

97

More than 100 feet away

-12.3

-12.1

16.7

37

When the interviewer was forced to stand away from the precinct entrance the

completion rates dropped and both the refusal and miss rates increased:

Distance

Completion Rate

Refusal Rate

Miss Rate

Inside

0.59

0.33

0.09

Right outside the entrance

0.54

0.36

0.10

10 to 25 feet away

0.53

0.36

0.11

25 to 50 feet away

0.51

0.37

0.13

50 to 100 feet away

0.45

0.39

0.16

More than 100 feet away

0.43

0.39

0.18

4. Ability to approach every voter:

As expected, if the interviewers said that they were able to approach all voters, the

average WPE was smaller. The precincts with more difficult to reach voters produced

more error.

Possible to approach

every voter?

Mean WPE

median WPE

mean Abs(WPE)

N

Yes

-6.0

-5.2

13.3

790

No

-8.0

-7.6

14.6

310

The completion rate was slightly lower when the interviewers said that they were not able

to approach all voters. Note that the miss rate in these precincts only includes the voters

Evaluation of Edison/Mitofsky Election System

page 38

January 19, 2005

that the interviewer could see but could not interview. In many cases the distances were

far enough that the interviewer could not see every voter as they exited the polling place.

Possible to approach

all voters?

Completion Rate

Refusal Rate

Miss Rate

Yes

0.55

0.35

0.10

No

0.51

0.36

0.13

5. Cooperation by polling place official:

In precincts where the interviewer said that the polling place official was not cooperative,

the average WPE was greater.

Was the precinct official

cooperative (according to

interviewer)?

mean WPE

median WPE

mean Abs(WPE)

N

Cooperative

-6.4

-6.0

13.5

1017

Not cooperative

-8.0

-7.7

15.6

87

The completion rate was also lower for these precincts:

Was the precinct official

cooperative (according to

interviewer)?

Completion Rate

Refusal Rate

Miss Rate

Cooperative

0.55

0.35

0.10

Not cooperative

0.46

0.38

0.15

6. Cooperation by voters

:

In precincts where the interviewer said that the voters were not cooperative, the WPE was

greater.

Were the voters cooperative

(according to interviewer)?

mean WPE

median WPE

mean Abs(WPE)

N

Cooperative

-6.2

-5.8

13.4

1007

Not cooperative

-10.2

-10.0

16.7

94

The completion rate was also much lower in these precincts:

Were the voters cooperative

(according to interviewer)?

Completion Rate

Refusal Rate

Miss Rate

Cooperative

0.56

0.34

0.10

Not cooperative

0.34

0.51

0.15

7. Interference at the precinct by non-election officials:

There was no significant difference in the WPE for precincts where the interviewer said

that non-election officials (like poll watchers and lawyers) interfered with the exit poll.

Evaluation of Edison/Mitofsky Election System

page 39

January 19, 2005

Did any other people (poll

watchers, lawyers, etc)

interfere with interviewing?

mean WPE

median WPE

mean Abs(WPE)

N

Yes

-5.5

-5.9

14.1

67

No

-6.6

-6.3

13.6

1036

Even though there was not much difference in the average WPE, the completion rate was

lower in the precincts where the interviewer said that a non-election official interfered:

Did any other people (poll

watchers, lawyers, etc)

interfere with interviewing?

Completion Rate

Refusal Rate

Miss Rate

Yes

0.48

0.39

0.12

No

0.54

0.35

0.11

8. Size of place:

Precincts in cities and the suburbs had larger average WPE than precincts in rural areas or

small towns.

Size of Place

mean WPE

median WPE

mean Abs(WPE)

N

Over 500,000

-7.9

-5.9

12.1

105

50,000 to 500,000

-8.5

-7.7

14.3

236

Suburbs

-8.1

-7.9

14.3

487

10,000 to 50,000

-4.9

-5.0

12.8

126

Rural

-3.6

-3.6

13.4

296

The completion rates are only slightly higher for small towns and rural areas:

Size of Place

Completion Rate

Refusal Rate

Miss Rate

Over 500,000

0.52

0.34

0.14

50,000 to 500,000

0.54

0.35

0.11

Suburbs

0.53

0.37

0.11

10,000 to 50,000

0.57

0.33

0.09

Rural

0.55

0.34

0.11

9. Polling place Equipment:

Some have suggested that the exit poll data could be used as evidence of voter fraud in

the 2004 Election by showing error rates were higher in precincts with touch screen and

optical scan voting equipment. Our evaluation does not support this hypothesis. In our

exit poll sample overall, precincts with touch screen and optical scan voting have

essentially the same error rates as those using punch card systems. In the larger urban

areas these systems had lower WPEs than punch card precincts.

Evaluation of Edison/Mitofsky Election System

page 40

January 19, 2005

WPE in precincts with any type of automated voting system is higher than the average

error in paper ballot precincts. These errors are not necessarily a function of the voting

equipment. They appear to be a function of the equipment’s location and the voters’

responses to the exit poll at precincts that use this equipment.

Type of equipment used at

polling place

mean WPE

median WPE

mean Abs(WPE)

N

Paper Ballot

-2.2

-0.9

11.2

40

Mechanical Voting Machine

-10.6

-10.3

16.3

118

Touch Screen

-7.1

-7.0

14.8

360

Punch Cards

-6.6

-7.3

14.2

158

Optical Scan

-6.1

-5.5

12.6

573

The value of the WPE for the different types of equipment may be more a function of

where the equipment is located than of the equipment itself. The larger urban areas had

higher WPEs than the rural/small towns. The low value of the WPE in paper ballot

precincts may be due to the location of those precincts in rural areas, which had a lower

WPE than other places.

Size Of Place

Type of equipment used

at polling place

mean WPE median WPE mean Abs(WPE)

N

Paper Ballot

-6.0

-11.5

15.7

5

Mechanical Voting Machine

-12.7

-12.5

16.8

92

Touch Screen

-7.5

-7.6

14.8

272

Punch Cards

-9.3

-10.0

15.2

108

Urban Areas

(> 50,000)

Optical Scan

-7.2

-5.8

12.3

350

Paper Ballot

-1.6

-0.6

10.5

35

Mechanical Voting Machine

-3.2

-5.4

14.7

26

Touch Screen

-6.0

-4.8

14.8

88

Punch Cards

-0.8

-1.7

12.0

50

Rural/Small

Town Areas

(< 50,000)

Optical Scan

-4.4

-5.0

13.2

223

Type of equipment used at polling place:

Size Of Place:

Paper

Ballot

Mechanical

Voting Machine

Touch

Screen

Punch

Cards

Optical

Scan

Total

Over 500,000

0

11

43

6

45

105

50,000 to 500,000

0

15

76

30

114

235

Suburbs

5

66

153

72

191

487

10,000 to 50,000

2

8

38

19

59

126

Rural

33

18

50

31

164

296

Total

40

118

360

158

573

1249

Evaluation of Edison/Mitofsky Election System

page 41

January 19, 2005

10. Weather:

The WPE was slightly greater in precincts where the interviewer said that the weather

affected the ability to conduct interviews.

Did the weather affect interviewing?

mean WPE

median WPE

mean Abs(WPE)

N

Yes

-7.3

-8.2

14.8

259

No

-6.2

-5.6

13.3

844

The completion rate was lower at precincts with weather issues:

Did the weather affect interviewing? Completion Rate

Refusal Rate

Miss Rate

Yes

0.48

0.41

0.12

No

0.56

0.34

0.10

11. Number of precincts at polling place:

The WPE was greater when there were three or more precincts at the polling place.

Some of the additional error for these precincts could be due to two sources:

i) the interviewer was not able to interview voters only from the sample precinct;

ii) either the reported vote or the exit poll is not only from the sample precinct alone.

Other precincts at the polling place may be included in either the exit poll and/or the vote

returns. We may be measuring a lack of consistency between the exit poll and the vote.

Further study is required to clarify this situation.

Number of precincts at polling place

mean WPE

median WPE

mean Abs(WPE)

N

1 precinct

-6.3

-5.8

13.2

888

2 precincts

-6.1

-6.9

14.0

201

3 precincts

-8.3

-7.8

15.1

95

4 or more precincts

-13.6

-10.8

18.8

66

The completion rate was not affected by the number of precincts at the polling place:

Number of precincts at polling place Completion Rate

Refusal Rate

Miss Rate

1 precinct

0.54

0.36

0.11

2 precincts

0.54

0.35

0.11

3 precincts

0.54

0.35

0.11

4 or more precincts

0.54

0.35

0.11

Evaluation of Edison/Mitofsky Election System

page 42

January 19, 2005

12. Swing states:

The WPE was greater in the more competitive “swing” states. For this analysis, the

following were considered swing states: Colorado, Florida, Iowa, Michigan, Minnesota,

Nevada, New Hampshire, New Mexico, Ohio, Pennsylvania, and Wisconsin.

Swing state

mean WPE

median WPE

mean Abs(WPE)

N

Precinct not in a swing state

-6.1

-5.1

13.1

776

Precinct in a swing state

-7.9

-8.6

14.8

474

Swing state

Completion Rate

Refusal Rate

Miss Rate

Precinct not in a swing state

0.56

0.34

0.10

Precinct in a swing state

0.50

0.38

0.11

This indicates that voters in the swing states (who were exposed to more paid advertising

and media coverage than voters in non-swing states) were less likely to respond to the

exit poll: but among those who did, more likely to be Kerry voters.

B. INTERVIEWER EFFECTS

1. Completion Rates:

Low completion rates have the potential to affect the total error in the exit poll by

producing a statistical bias. The correlation between the overall completion rate and the

signed WPE in a precinct was not significant (0.05), which shows no effect on the

statistical bias from the overall completion rates. There also is a small correlation

between the completion rates and the absolute WPE (-0.15). This does not rule out a

difference in the completion rates for all Kerry voters or all Bush voters. There well may

be a difference in these two rates.

WPE Category Completion Rate

Refusal Rate

Miss Rate

N

WPE < -15

0.51

0.37

0.12

351

-15 < WPE < -5

0.55

0.34

0.10

315

-5 < WPE < 0

0.57

0.33

0.10

163

0 < WPE < 5

0.55

0.35

0.10

138

5 < WPE < 15

0.53

0.36

0.11

187

WPE > 15

0.50

0.38

0.12

94

The correlation between the WPE (both signed and absolute) and the difference in

completion rates for voters under and over 30 and for the difference between males and

females was computed. None of these correlations were significant (all were below

0.06).

Evaluation of Edison/Mitofsky Election System

page 43

January 19, 2005

2. Interviewer Age:

Older interviewers had lower WPE than the youngest interviewers. They also had better

completion rates. This does not necessarily mean that the younger interviewers did

poorly at their task. It does indicate that in this election voters were less likely to

complete questionnaires from younger interviewers.

Interviewer Age

mean WPE

median WPE

mean Abs(WPE)

N

24 and under

-7.4

-8.6

15.0

430

25-34

-8.2

-7.2

13.4

182

35-44

-4.0

-3.9

13.4

167

45-54

-6.3

-4.7

12.5

191

55-64

-7.0

-5.8

12.6

143

65 and over

-3.7

-5.4

12.9

68

The relationship between interviewer age and the WPE holds when controlling for

interviewer gender and education:

Gender

Interviewer Age

mean WPE median WPE mean Abs(WPE)

N

Men

Under 35

-7.7

-8.9

14.6

248

35 or older

-5.9

-4.5

13.4

196

Women

Under 35

-7.7

-7.9

14.5

364

35 or older

-5.3

-4.7

12.6

373

Interviewer Education

Interviewer

Age

mean WPE

median WPE mean Abs(WPE)

N

High school or less

Under 35

-5.9

-8.1

16.0

96

35 or older

-1.6

-1.2

13.0

81

1 to 3 years of college Under 35

-8.1

-8.9

14.8

301

35 or older

-6.2

-3.9

12.9

225

Four year college degree Under 35

-7.0

-6.9

13.2

110

35 or older

-5.7

-4.9

12.3

111

Some graduate credits Under 35

-9.7

-10.5

12.7

29

35 or older

-2.5

-4.0

11.4

42

Advanced Degree

Under 35

-9.1

-5.9

12.9

29

35 or older

-7.6

-5.1

13.2

94

The completion rates were higher for precincts with older interviewers:

Interviewer Age Completion Rate

Refusal Rate

Miss Rate

24 and under

0.50

0.39

0.12

25-34

0.52

0.38

0.11

35-44

0.53

0.35

0.12

45-54

0.57

0.32

0.10

55-64

0.61

0.31

0.08

65 and over

0.61

0.29

0.10

Evaluation of Edison/Mitofsky Election System

page 44

January 19, 2005

Also, the relationship between interviewer age and completion rates is approximately the

same even within the different voter age groups. Younger interviewers tend to have

lower completion rates regardless of the voter’s age:

Interviewer Age

Overall

Completion

Rate

Age 18-29

Completion

Rate

Age 30-59

Completion

Rate

Age 60+

Completion

Rate

24 and under

0.50

0.53

0.52

0.39

25-34

0.52

0.56

0.54

0.42

35-44

0.53

0.55

0.55

0.42

45-54

0.57

0.58

0.60

0.47

55-64

0.61

0.61

0.64

0.53

65 and over

0.61

0.60

0.64

0.54

This information confirms what we found in our post-election survey of our exit poll

interviewers. We asked each interviewer how cooperative they found the local polling

place officials and how cooperative they found the voters at their location. Younger

interviewers were much less likely to report that the voters at their location were very

cooperative. Again, this is an indication that there is an interaction between the voters

and younger interviewers that contribute to lower completion rates and higher WPE.

How cooperative did you find the polling place officials at your location?

Age of Interviewer

Total 18-24 25-34 35-44 45-54 55+

Very cooperative

71% 63% 71% 73% 74% 80%

Somewhat cooperative

21% 28% 19% 18% 19% 12%

Not very cooperative

7%

7%

9%

7%

6%

6%

Actively tried to keep you

2%

2%

1%

2%

1%

2%

from doing your work

How cooperative did you find the voters at your location?

Age of Interviewer

Total 18-24 25-34 35-44 45-54 55+

Very cooperative

44% 27% 36% 46% 56% 69%

Somewhat cooperative

46% 58% 54% 46% 40% 26%

Not very cooperative

9% 14% 10%

8%

4%

5%

Actively tried to keep you

0%

1%

0%

0%

0%

0%

from doing your work

3. Interviewer Gender:

The average WPE was slightly greater in precincts with male interviewers:

Gender

mean WPE

median WPE

mean Abs(WPE)

N

Men

-7.1

-7.2

14.1

466

Women

-6.6

-5.8

13.5

775

Evaluation of Edison/Mitofsky Election System

page 45

January 19, 2005

The relationship between interviewer gender and the WPE does not hold when controlled

for interviewer age. There is little difference between precincts with male or female

interviewers when separated into groups depending on whether the interviewer was over

or under 35:

Interviewer Age

Gender

mean WPE median WPE mean Abs(WPE)

N

Under 35

Men

-7.7

-8.9

14.6

248

Women

-7.7

-7.9

14.5

364

35 or older

Men

-5.9

-4.5

13.4

196

Women

-5.3

-4.7

12.6

373

There was no significant difference between the gender of the interviewer and completion

rates:

Gender

Completion Rate

Refusal Rate

Miss Rate

Men

0.53

0.36

0.11

Women

0.54

0.35

0.11

4. Interviewer education:

The absolute WPE decreased slightly when the interviewer had more education.

However, the statistical bias was highest among those with post-graduate education. They

had a significantly greater overstatement of Kerry than any other group. Those with High

School education or less had a lower overstatement of Kerry but a higher absolute error.

Interviewer Education

mean WPE

median WPE

mean Abs(WPE)

N

High school or less

-3.9

-4.6

14.7

177

One to three years of college

-7.3

-7.0

14.0

526

Four year college degree

-6.3

-6.3

12.8

222

Some graduate credits

-5.4

-5.9

11.9

71

Advanced degree such as MA,

MBA or PhD

-7.9

-5.2

13.1

123

The completion rates tend to be slightly higher in precincts with more educated

interviewers

:

Interviewer Education

Completion Rate

Refusal Rate

Miss Rate

High school or less

0.52

0.36

0.11

One to three years of college

0.53

0.37

0.11

Four year college degree

0.55

0.34

0.11

Some graduate credits

0.57

0.34

0.10

Advanced degree such as MA, MBA or PhD

0.60

0.32

0.08

5. When the interviewer was hired:

Interviewers hired well in advance of the election had lower errors than late hires.

Precincts with an interviewer hired within a few days before the election had a greater

average WPE:

Evaluation of Edison/Mitofsky Election System

page 46

January 19, 2005

When was the

interviewer hired?

mean WPE

median WPE

mean Abs(WPE)

N

At least a week before

the election

-6.5

-5.9

13.5

1154

Within a few days before

election or on election

day

-9.5

-10.1

16.3

82

The completion rate also was lower for these precincts:

When was the interviewer

hired?

Completion Rate

Refusal Rate

Miss Rate

At least a week before the

election

0.54

0.35

0.11

Within a few days before

election or on election day

0.48

0.40

0.13

6. Interviewer training:

Precincts where the interviewer said that they were trained “very well” had less WPE:

How well did the interviewer

say they were trained?

mean WPE

median WPE

mean Abs(WPE)

N

Very well

-6.4

-5.7

13.4

862

Somewhat or not very well

-7.0

-7.8

14.3

239

The completion rate was also slightly lower for those interviewers who said that they

were not very well trained:

How well did the interviewer

say they were trained?

Completion Rate

Refusal Rate

Miss Rate

Very well

0.55

0.35

0.10

Somewhat or not very well

0.52

0.36

0.12

Evaluation of Edison/Mitofsky Election System

page 47

January 19, 2005

Summary of WPE

Our objective is to identify precinct characteristics, interviewer characteristics and other

factors that contributed to higher error in 2004. These findings are not always the same as

those from earlier elections. While we believe we can reduce the statistical bias in the exit

polls we do not believe anything we have identified so far will completely eliminate it.

The initial review of the data show indications that the following factors should be

investigated further:

• WPE and absolute error increase significantly once interviewers are more than 25

feet away from the polling place.

• Precincts with younger interviewers have higher WPE than precincts with older

interviewers although there is evidence that this is caused by the interaction of

voters with younger interviewers.

• Interviewers with less education have lower WPEs but a higher absolute error.

Interviewers with advanced degrees have a higher average WPE than those with

less education.

• Interviewers in large precincts, and where a smaller proportion of all voters are

selected, have a higher WPE.

There are several changes in the exit poll process that might lessen the WPE. Only further

experimentation will confirm this:

First, legal remedies need to be pursued to lessen the distances that voting officials are

enforcing upon our interviewers, especially those greater than 50 feet.

Second, the exit poll interviewer recruiting process needs to take greater account for the

age of the interviewers. Measures also need to be put in place to make sure that

interviewers are hired more than two weeks before Election Day to allow time for

additional training.

Third, the exit poll interviewer training process needs to be enhanced in order to confirm

that the interviewers understand the most important aspects of the selection of

respondents. More effort should be made in ensuring the interviewer follows the assigned

interviewing rate.

Fourth, we should experiment with the use of shorter questionnaires in tandem with long-

form questionnaires. One way would be administering short questionnaires to every

other exit poll voter, thereby improving the tallies. The tallies from these short

questionnaires could then be included with the results from the long-form questionnaires.

Evaluation of Edison/Mitofsky Election System

page 48

January 19, 2005

Exit Poll Location Coverage

The total number of exit poll locations in the 50 samples (49 states plus D.C.) was 1,480.

We received data from all but 11 polling locations. Seven polling locations had no

interviewer and in four polling locations our interviewer was prohibited from conducting

interviews due to legal and distance restrictions:

There were 62 polling locations in which a replacement interviewer was sent on election

day to replace an interviewer who did not show up or to replace an interviewer who had

to leave for some reason. Since some of the replacements arrived late, some exit poll

calls were not made from some of these polling locations.

There were 25 polling locations where one or two calls were missed because interviewers

arrived late or because legal and distance restrictions prohibited interviews from being

conducted:

There were 62 polling locations that had missing data on election day because of

problems that interviewers had with telephone access or the input system. The tallies and

Refusals and Misses from these polling locations are included in the post-election day

exit poll file that was used to evaluate the WPE.

Evaluation of Edison/Mitofsky Election System

page 49

January 19, 2005

Exit Poll Interviewer Recruitment Process

Overview:

Edison/Mitofsky recruited exit poll interviewers in 29 states: AK, AZ, CA, CT, DE, DC,

FL, HI, IL, IA, KY, LA, MD, MA, MS, MO, NH, NJ, NM, NY, OH, OK, RI, SC, TN,

TX, VT, VA, WI. Recruitment for these states was divided among seven State

Coordinators, with one specific coordinator responsible for each state. These same

coordinators had been responsible for research in the states they recruited and in most

cases the same recruiter was used for both the Presidential Primary and the General

Election.

The firm of Blum & Weprin was responsible for recruitment in the remaining 21 states:

AL, AR, CO, GA, ID, IN, KS, ME, MI, MN, MT, NE, NV, NC, ND, PA, SD, UT, WA,

WV, WY. Blum & Weprin had a staff of 14 people recruiting these states. A single

person was responsible for each state, and in many cases those coordinators lived in the

states where they were recruiting.

Among all interviewers, 23% (339 interviewers) had previously worked as an exit

interviewer in one or more previous election. Among that group, 63% (214) had worked

for Edison/Mitofsky during the 2004 Presidential Primaries.

In addition to recruiting former Edison/Mitofsky exit poll interviewers and final vote

count reporters, other common sources for recruitment were (in order of frequency):

Recommendations from current and former interviewers

Recommendations from college professors

Career Centers and Departments of Labor

Former VNS interviewers

Job postings on Craigslist.com

Recommendations from election officials

Hiring Timeline:

According to our post-election exit poll interviewer telephone survey, 86% of

interviewers were hired two weeks or more before the election. The data showed that

7% of the staff had been hired “within a few days of election day.”

Training Process:

Each interviewer went through a multi-step training system:

1)

Hiring call: On the call where the interviewer was officially hired, they were

given a general overview of the job. They were told the hours, the outline of

their day including responsibility for “a check-in and three results calls”.

Evaluation of Edison/Mitofsky Election System

page 50

January 19, 2005

They were also given a general description of both the Interviewing Rate and

the process of tracking Refusals and Misses.

2)

Election Day Mailing: Most interviewers received all their election day

materials (with the exception of final questionnaires in several states and in

National Precincts) one week prior to their scheduled Training/Rehearsal Call.

This package was sent via FedEx and included the Interviewer Manual, Tally

Sheets, Refusals and Misses Sheets and special instructions relating to

election laws in their state. National Precinct interviewers were also provided

additional training materials relating to multiple questionnaire versions and

the “breaking news question.”

3)

Training/Rehearsal Call: All interviewers were required to take part in a

Training/Rehearsal Call. This call took on average 20 minutes to complete

and included a detailed question and answer dialogue on all facets of the job.

It also included a practice run through entering rehearsal tallies, refusals and

misses and total ballots.

Operators at our two telephone call centers answered Training/Rehearsal calls

using a Computer Assisted Telephone Interviewing (CATI) programmed

dialogue. A staff of Edison/Mitofsky supervisors was on hand and in most

cases got on the phone with each interviewer before the call was ended. In

addition, the call dialogue instructed the operator to alert a supervisor

whenever a question or problem arose.

These calls were scheduled for October 25

th

and 26

th

, 2004 with each

interviewer assigned a particular day and given an 8-hour window to place the

call. October 27

th

was reserved as a third “back-up” rehearsal day for any

interviewers hired too late to participate over the first two days, or for those

who missed their scheduled calls the previous two days.

According to our survey of exit poll interviewers, 80% of exit poll

interviewers participated in one of these formal rehearsal calls between

October 25

th

and October 27

th

. The remainder was trained on a one-on-one

basis with their recruiter.

4)

Pre-Election reminder call: All interviewers received a final reminder call

some time between October 29

th

and November 1

st

. Their intention to work

election day was confirmed and they were given a chance to ask any questions

they had relating to the materials they had received or their election day tasks.

5)

National Precinct Interviewer’s “Breaking News Question” Call: All National

Precinct interviewers (as well as any back-up who might fill that roll) were

Called on October 31

st

or November 1

st

to be told the “Breaking News”

question that they would need to fill in on their blank questionnaires.

Evaluation of Edison/Mitofsky Election System

page 51

January 19, 2005

Back-up Interviewers

A back-up interviewer was assigned to each exit poll location. In some case it was a one-

for-one back-up system; in others, one back up “covered” several potential polling

locations within a close geographic area. In general, these back-ups were hired with the

understanding that they would work election day in some capacity and were paid a

substantial base pay to be on call all day. They were promised an increase in pay if they

were assigned a polling location to survey.

On election day, 62 of these back-ups were assigned to replace or to relieve the

“frontline” exit poll interviewer for a polling location – representing 4% of all exit poll

polling locations covered.

Evaluation of Edison/Mitofsky Election System

page 52

January 19, 2005

Post-election evaluation:

Exit poll interviewers were instructed to return their completed questionnaires and tally

sheets. We received questionnaires from 1,389 exit poll polling locations – 95% of those

covered. We have determined that the 80 sets of questionnaires that were not returned

have either been misplaced by the interviewer, lost by the postal service, or in some cases

destroyed.

In addition we conducted a follow-up telephone interview with our exit poll interviewers

in the three weeks following election day. In total we were able to contact 1,350

interviewers (92%). For the remaining interviewers we have filled in as much

information as we could from the records of the recruiting process – age, gender, date

hired etc.

The following is the profile of our exit poll interviewers based upon the information that

we have gathered:

Male

37%

Female

63%

Age

18-24

35%

25-34

15%

35-44

14%

45-54

16%

55-64

13%

65+

6%

Median Age: 34 years old

Race:

White

72%

African-American

7%

Hispanic

3%

Asian-American

1%

Other

4%

No Answer

12%

Education:

High School or less

15%

1-3 years of college

41%

College graduate

17%

Some graduate school

6%

Advanced degree

10%

No Answer

11%

Evaluation of Edison/Mitofsky Election System

page 53

January 19, 2005

Suggested Changes for Future Recruiting of Exit Poll Interviewers:

The procedures we used for recruiting and training succeeded in achieving a 100%

location coverage rate in the primaries and a 99% location coverage rate in the general

election. We also had no evidence of any systematic errors in the exit poll data from the

primaries in 2004 nor in any elections we covered in 2002 and 2003.

However, based upon the Within Precinct Error that was observed in the 2004 general

election we plan to make some enhancements to the exit poll interviewer recruiting

process.

• We will use recruiting methods that reduce the number of students and young

adults we use as interviewers.

• In addition to the standardized rehearsal and training dialog, we will add a

standardized pre-rehearsal training script for all individual phone training

conversations.

• We will evaluate other training techniques such as a video training guide and

interviewer tests and use the Internet more effectively as an interviewer training

tool.

Completion Rates

In the table on the next pages, the completion, refusal, and miss rates as well as the

demographic completion rates are displayed for each state and nationally.

The overall completion rate was computed by a weighted average of the precinct rates.

The precinct completion rate is equal to the number completed questionnaires divided by

the number of attempts (completed questionnaires + total refusals and misses). The

refusal and miss rates were computed in a similar manner.

Note that because the questionnaires are subsampled, the age, race, and gender

completion rates may be slightly inconsistent with the overall completion rate.

Evaluation of Edison/Mitofsky Election System

page 54

January 19, 2005

State

Overall

Completion

Rate

Refusal

Rate

Miss Rate

Completion

rate: 18-29

Completion

rate: 30-59

Completion

rate: 60+

Completion

rate: Non-

black

Completion

rate: Male

Completion

rate:

Female

All Precincts

53.2

35.8

11.0

55.3

55.6

43.0

53.2

51.4

54.7

National Survey

52.8

37.2

10.1

55.2

55.2

41.7

53.1

51.0

54.6

Alabama

58.3

31.9

9.8

52.6

62.8

50.6

57.0

57.2

60.0

Alaska

53.2

36.9

9.9

51.9

55.1

48.3

53.0

51.0

55.0

Arizona

57.3

28.4

14.3

61.2

60.5

46.0

57.2

57.0

58.5

Arkansas

60.2

31.1

8.7

68.7

59.5

55.4

56.6

55.2

61.7

California

50.5

35.9

13.5

51.2

54.1

43.1

51.8

51.2

52.4

Colorado

55.5

34.9

9.6

56.5

58.0

46.4

55.0

54.8

55.8

Connecticut

51.0

37.5

11.5

49.9

51.9

48.7

51.2

48.5

53.3

Delaware

57.5

25.8

16.7

59.5

61.4

48.4

55.7

54.4

60.1

D.C.

53.5

30.8

15.7

46.3

58.7

48.3

59.4

50.7

55.7

Florida

49.0

40.3

10.8

50.0

51.4

39.0

50.2

47.9

50.2

Georgia

63.9

27.8

8.3

63.2

65.3

51.6

64.1

61.3

63.3

Hawaii

53.4

34.5

12.1

58.7

55.0

42.9

53.0

50.1

56.3

Idaho

63.2

30.8

6.0

64.9

65.3

53.8

62.9

58.2

67.8

Illinois

51.9

37.8

10.3

54.0

53.6

44.4

52.8

49.6

55.1

Indiana

38.6

41.3

20.1

41.9

41.5

28.2

40.5

37.0

40.1

Iowa

52.6

38.2

9.1

55.6

56.8

41.1

52.0

50.6

54.4

Kansas

64.5

27.6

7.9

68.8

64.3

57.4

64.2

61.2

66.9

Kentucky

52.6

38.6

8.8

58.0

54.8

44.5

53.3

51.4

53.8

Louisiana

47.8

40.1

12.1

54.8

49.9

36.7

46.2

45.0

48.5

Maine

61.3

30.8

8.0

63.0

65.2

53.4

62.4

59.0

66.5

Maryland

59.4

33.0

7.6

56.6

62.1

51.1

63.5

56.3

61.7

Massachusetts

56.5

30.6

12.9

55.0

59.7

50.9

56.7

54.3

58.4

Michigan

50.2

39.4

10.4

51.3

52.2

38.1

51.7

50.3

50.9

Minnesota

45.3

40.0

14.7

47.3

47.3

37.0

44.9

44.0

46.6

Mississippi

49.6

38.9

11.5

47.4

55.4

35.5

53.6

51.2

48.9

Evaluation of Edison/Mitofsky Election System

page 55

January 19, 2005

Missouri

47.0

44.1

8.8

53.0

48.8

36.4

46.8

46.4

48.0

Montana

63.0

29.7

7.3

63.6

64.6

55.9

62.2

61.8

64.7

Nebraska

66.5

25.1

8.4

69.9

67.7

59.0

66.3

63.8

69.2

Nevada

49.1

34.9

16.0

52.8

53.4

36.1

48.3

47.2

50.5

New Hampshire

44.0

46.0

9.9

44.4

47.7

32.7

43.9

42.6

45.8

New Jersey

59.7

29.8

10.5

65.9

61.6

52.8

61.3

55.9

62.9

New Mexico

56.9

32.3

10.8

60.0

58.7

42.0

56.1

56.1

57.1

New York

57.9

33.5

8.6

59.5

58.7

53.1

57.2

55.8

60.2

North Carolina

52.6

36.5

10.9

54.8

56.9

39.0

51.0

51.0

53.7

North Dakota

63.0

27.0

10.0

63.0

66.6

48.6

63.0

58.5

66.6

Ohio

44.1

42.5

13.3

48.1

47.3

31.6

45.8

42.9

45.7

Oklahoma

53.2

34.5

12.3

54.1

57.2

42.3

53.7

50.7

55.6

Pennsylvania

46.8

41.3

11.9

53.3

47.6

36.0

48.4

46.0

47.3

Rhode Island

44.2

42.2

13.6

42.3

47.5

34.6

43.5

41.7

44.9

South Carolina

59.4

28.8

11.8

61.9

62.0

47.2

57.3

56.2

61.6

South Dakota

42.7

35.6

21.7

43.6

46.5

31.6

41.9

41.1

43.8

Tennessee

66.7

24.6

8.7

71.6

67.0

48.6

67.8

65.5

67.5

Texas

58.3

28.6

13.1

65.8

58.3

35.6

58.1

57.1

59.4

Utah

59.6

32.0

8.4

58.1

61.9

50.3

59.3

58.5

60.0

Vermont

53.1

38.8

8.1

47.6

56.8

44.2

53.4

51.6

54.8

Virginia

56.4

35.3

8.3

54.2

60.1

44.4

56.8

53.6

59.1

Washington

53.8

38.8

7.3

57.4

54.4

45.5

53.3

50.4

57.2

West Virginia

48.7

43.0

8.3

51.7

51.8

41.7

48.3

48.4

49.4

Wisconsin

55.3

35.6

9.1

57.0

56.5

46.7

55.1

53.2

57.1

Wyoming

66.0

26.1

7.9

63.7

68.7

52.2

65.0

65.1

65.5

Evaluation of Edison/Mitofsky Election System

page 56

January 19, 2005

Survey Weighting

State crosstab releases on election day

For the most part survey weighting ran smoothly on election day. Overall, out of 151

scheduled state crosstab releases, 136 – or 90 percent – were released early, on-time or

within ten minutes of the scheduled time.

State crosstabs released later than scheduled on election day

On election day, 15 state crosstabs were released ten or more minutes after their

scheduled release time. Of these, six were between ten minutes and thirty minutes late,

and nine were over one hour late. These late releases were not evenly spread throughout

the day, ten of them occurred during Call 1, one during Call 2, and the remaining seven

were during Call 3.

A brief telephone service disruption at our Los Angeles phone room at approximately

11:30AM ET/8:30 AM PT led to most of the late Call 1 releases. The four states most

significantly affected were Illinois, Pennsylvania, Texas and Vermont where the

interviewers were scheduled to make their first data call to the L.A. phone room at that

time. Crosstabs in these four states were released over an hour later than their scheduled

time while the data entry caught up in these states. Edison/Mitofsky sent an e-mail

warning that the Call 1 crosstabs for these four states would be released late to survey

committee members at 12:59 PM ET. There were too few respondents in the system to

release these crosstabs at their scheduled time. In all cases, the extra time allowed the

data entry for these four states to catch up and nearly doubled the number of respondents

in the system when the first call weighting was completed.

Call 1 data releases were delayed in four other states (CA, MN, NV and NM) while the

survey weighters were making sure that subsequent states calling in to the L.A. phone

room were back on schedule. While these states were not as seriously influenced and

included enough data to be released, survey weighters were more cautious in reviewing

the number of questionnaires in these states before releasing them. The longest delay in

this group was 16 minutes for New Mexico. A small number of respondents (only 367 at

the scheduled weighting time) and a large number of races used in the forcing caused the

sole late release for Call 2, in Vermont. The survey weighters took additional time to

review the weighted data for all four statewide races.

During Call 3, five late releases were due to the database server problem that occurred at

10:35 PM ET. Data for Hawaii, Washington, California and Idaho were in the system at

the time of the problem, but crosstabs could not be released until the system returned,

causing delays of approximately ninety minutes. Call 3 data for Alaska were not entered

until the system came back up, so even though the system was available by the time the

Alaska crosstabs were due to be released, the data were unavailable for another ninety

minutes.

Evaluation of Edison/Mitofsky Election System

page 57

January 19, 2005

An issue with the weighting program caused attempts to weight the Call 3 Florida data to

bypass all forcing, and the survey data release was delayed until the problem was

corrected.

National crosstab delays on election day

There were eight scheduled national crosstab releases for election day. Of these eight,

one (scheduled for 10:50 PM ET) was not delivered at all due to the database server

problem, and three were delivered more than ten minutes late.

All three late releases were largely due to additional adjustments made to the data to

adjust the gender distribution after the normal weighting procedure. Taking this step

added 15 minutes to the weighting process, although scheduled call times were

unchanged.

Survey weightings after poll closing

The database server problem caused a delay in the survey weightings that were to be

done to adjust the exit poll estimates once estimates based upon the sample precincts and

county vote were available. These weightings would have begun once the remaining Call

3 weightings had been completed but the database server problem at 10:35 PM ET

delayed these weightings until after 12:30 AM ET.

Evaluation of Edison/Mitofsky Election System

page 58

January 19, 2005

National Survey Weighting Gender Adjustment

Early in the afternoon on November 2

nd

, preliminary weightings for the national exit poll

overstated the proportion of women in the electorate. The problem was caused by a

programming error involving the gender composition that was being used for the

absentee/early voter portion of the national exit poll. The target proportion for the

national absentee age-race-sex responses was not being computed correctly. The

weighting problem that led to the original waves of the national exit poll showing 58% of

the national voters were female.

To correct this problem beginning with the 7:30 PM ET national survey weighting on

election day, the gender split in the national survey was forced outside the election

system to a male-female distribution of 46% / 54%. This correction was based on the

male/female distribution in the cross-survey of all of the state surveys.

At this point, the male-female distribution from the system weighting (prior to the

external forcing) was 44%/56% male/female. The weights from the election system were

exported to an external weighting program. This program had been fully tested and used

in prior elections. The respondent weights from the system were forced to the target

46/54 male/female ratio while keeping constant the regional sizes and presidential vote

distribution from the survey.

All of the national weightings after the 7:30 PM ET weighting were also forced

externally to cross-survey male-female distributions.

Evaluation of Edison/Mitofsky Election System

page 59

January 19, 2005

Comparison of National Exit Poll with Cross Survey

The NEP National Exit Poll is based upon a total of 12,219 respondents - 11,719 election

day interviews at 250 polling locations and an additional 500 telephone interviews of

absentee/early voters.

The Election System also allows the members to combine the results of all state surveys

through a process called “Cross Survey.” Cross Survey gives a survey analyst the ability

to combine the results of common questions across different state surveys by adjusting

the relative sizes of the state samples to represent the total number of voters in each state.

By using Cross Survey for all 51 state surveys, the total sample size increases to 75,537

respondents – 69,719 election day questionnaires at 1,469 polling locations and an

additional 5,818 telephone interviews of absentee/early voters.

For the 12 questions that were on all 51 state questionnaires, we have compared the

results of the National Exit Poll with the Cross Survey weighted average of all 51 state

surveys. That table is on the next two pages of this report.

Between these two methods of estimating national numbers, the results of each question

differ by one point or less except for two groups: the National Exit Poll estimates that the

65-74 age group comprised 11% of the total voters and the Cross Survey has that group at

9%; in the National Exit Poll 25% of the respondents said that their vote for President

was a vote against the opponent while 28% of the respondents in Cross Survey gave that

answer.

While the estimates from the two sources differ somewhat, it is incorrect to conclude that

one estimate is correct and the other estimate is incorrect. All estimates are within

sampling error of each other.

In comparing how each group voted using these two methods, we observe consistency on

most items. Even the responses for non-demographic items such as Party ID, Political

Philosophy, Bush Approval and Iraq Approval show a very consistent vote for President

in both the National Exit Poll and the Cross Survey.

The characteristics that differ most are more highly clustered in a few precincts and have

much larger sampling errors than most other characteristics.

% voting for Bush

National Exit Poll

Cross Survey

Hispanic

44%

40%

Asian

44%

39%

Age 75+

45%

48%

Jewish

25%

22%

Mormon

80%

76%

Muslim

6%

13%

Income >$200,000 63%

60%

Evaluation of Edison/Mitofsky Election System

page 60

January 19, 2005

Comparison of exit poll results from the National Sample and the Cross Survey weighted average of the

State Surveys

November 2004 - NEP Exit Polls conducted by Edison Media Research and Mitofsky International

National Exit Poll

Cross Survey Results

Question

Total

Kerry Bush

Total Kerry Bush

Gender

Male

46

44

55

46

46

53

Female

54

51

48

54

50

49

Race

White

77

41

58

78

41

58

Black

11

88

11

11

86

13

Hispanic

8

53

44

7

58

40

Asian

2

56

44

2

61

39

Other

2

54

40

2

57

40

Age

18-29

17

54

45

18

55

44

30-44

29

46

53

29

45

53

45-49

30

48

51

30

47

52

60+

24

46

54

22

47

52

Age

18-24

9

56

43

10

56

42

25-29

8

51

48

8

53

45

30-39

18

47

51

18

47

52

40-44

11

44

55

11

43

56

45-49

11

46

53

11

46

53

50-59

19

49

50

19

48

51

60-64

8

42

57

8

45

54

65-74

11

44

55

9

46

53

75+

5

54

45

5

51

48

Income

<$15k

8

63

36

8

61

37

$15-$29k

15

57

42

14

56

43

$30-$49k

22

50

49

22

49

49

$50-$74k

23

43

56

23

44

55

$75-$99k

14

45

55

14

44

54

$100-$149k

11

42

57

11

46

54

$150-$199k

4

42

58

4

43

56

$200k+

3

35

63

4

39

60

Religion

Protestant

31

37

62

32

36

63

Catholic

27

47

52

26

49

50

Mormon

2

19

80

2

23

76

Other Christian

21

47

52

21

46

53

Jewish

3

74

25

3

77

22

Muslim

1

92

6

1

86

13

Something Else

6

72

25

6

70

27

None

10

67

31

10

69

29

Evaluation of Edison/Mitofsky Election System

page 61

January 19, 2005

Party Identification Democrat

37

89

11

37

88

12

Republican

37

6

93

37

7

93

Independent

22

50

47

21

51

46

Something Else

4

47

49

5

47

48

Political Philosophy Liberal

21

85

13

21

85

14

Moderate

45

54

45

46

55

44

Conservative

34

15

84

34

16

83

Bush Job Approval Approve

53

9

90

52

8

92

Disapprove

46

93

6

47

93

5

Bush Job Approval Strongly Approve

33

5

94

32

4

96

Somewhat Approve

20

15

83

20

14

85

Somewhat Disapprove

12

80

18

12

81

16

Strongly Disapprove

34

97

2

34

97

1

Iraq

Approve

51

14

85

51

13

86

Disapprove

45

87

12

45

87

11

Financial Situation Better

32

19

80

32

19

80

Worse

28

79

20

28

81

18

Same

39

50

49

38

49

50

Vote For

Candidate

For Candidate

69

40

59

68

39

60

Against Opponent

25

70

30

28

69

30

Time of Decision* Today

5

52

45

5

53

43

In the last 3 days

4

55

42

4

54

43

Sometime last week

2

48

51

3

53

45

During the last month

10

54

44

11

55

42

Before that

78

46

53

77

46

53

Time of Decision* Within last 3 days

9

53

44

9

54

43

One week before or earlier

91

47

52

91

48

51

* Note: Time of Decision question was only asked to election day voters.

Evaluation of Edison/Mitofsky Election System

page 62

January 19, 2005

These demographic groups have two things in common: each group represents 8% or less

of the total number of voters, and each group tends to be concentrated geographically and

thus would be more affected by any “clustering effects” in the precincts selected for the

National Exit Poll sample. A National Sample of 250 precincts can do a good job

estimating all of the broad characteristics of the electorate, but it is not designed to yield

very reliable estimates of the characteristics of small, geographically clustered

demographic groups. These groups have much larger design effects and thus larger

sampling errors.

A detailed look at the distribution of plurality Hispanic precincts in the National Exit Poll

Sample demonstrates how this clustering effect can influence the estimate of Hispanic

voting in the National Exit Poll. Out of the 250 precincts in the national sample, 11 were

plurality Hispanic precincts representing about 4% of the sample. This seems to be a

reasonable number of precincts based upon the total population of Hispanics and the

percentage of Hispanics that would live in predominantly Hispanic areas.

The issue is the distribution of these 11 precincts – four in Florida and one each in

California, Colorado, Illinois, New Jersey, New York, Ohio and Texas – three of the four

in Florida are in Miami-Dade County and two of them appear to be majority Cuban based

upon the questionnaire responses from those precincts. Cuban voters tend to vote more

Republican than other Hispanic voters.

Hispanic Vote by Region

National Exit Poll vs. Cross Survey

National Exit Poll

Cross Survey Results

Region

Total Kerry Bush

Total

Kerry

Bush

National

8

53

44

7

58

40

Northeast

7

68

28

6

70

29

Midwest

5

64

32

3

67

31

South

9

35

64

8

46

53

West

13

58

39

14

62

35

Comparing the regional breakout of the Hispanic vote, one can see the “clustering effect”

that these two majority Cuban precincts have on the estimate of the Hispanic vote in the

South in the National Exit Poll versus the Cross Survey Results. Since the National Exit

Poll sample has only five plurality Hispanic precincts in the South, the number of Cuban

precincts can make a difference in the estimate of the Hispanic vote in the region. If we

want to improve the National Exit Poll estimate for Hispanic vote (or Asian vote, Jewish

vote or Mormon vote etc.) we would either need to drastically increase the number of

precincts in the National Sample or oversample the number of Hispanic precincts.

There is another difference between the National Exit Poll and the Cross Survey in the

measurement of Hispanics. In 24 states and the national survey, respondents are first

asked a race question with Hispanic as a choice along with White, Black, Asian and

Other. In addition a separate “Hispanic or Latino” question is asked and any respondent

who selects Hispanic for either or both questions is coded as Hispanic. Note that in

Evaluation of Edison/Mitofsky Election System

page 63

January 19, 2005

Florida the question asks “Hispanic or Cuban” and in some other states it asks “Hispanic

or Mexican.”

In the remaining 27 states only the race question is asked. Thus, in the Cross Survey

analysis we are comparing a different definition of Hispanic in these 27 states from the

National Survey. Examining the National Survey for these 27 states the Hispanic voters

as defined by the race question alone comprise 2% of the vote in these states and voted

42% for George W. Bush. When the Hispanic voters are defined by using both questions,

the Hispanic composition increases to 4% of the vote in these states and Hispanics voted

49% for George W. Bush. Hispanics who select “White” for the race question but also

answer that they are of Hispanic descent are more likely to have voted for Bush and this

accounts for some of the overall difference that we see in the Hispanic vote in the

National Survey vs. the Cross Survey.

There are three other sources on the national Hispanic vote. The L.A. Times National

Exit Poll showed that 45% of Hispanics voted for Bush. The National Annenberg

Election Survey in October and November 2004 showed 41% of Hispanics voting for

Bush up from the 35% they reported in 2000.

The William C. Velasquez exit poll concluded that 31.4% of Hispanics voted for Bush.

However, the sampling for this exit poll was limited to 56 predominantly Hispanic

precincts in 14 states. The NEP National Exit Poll shows that Bush only received 29% of

the vote among Hispanics living in urban areas with populations over 500,000. This

makes us suspect that the Velasquez exit poll sample over-represents Hispanics who live

and vote in predominantly Hispanic areas.

The bigger question is how to handle the differences between the National Exit Poll and

the Cross Survey results. As the Cross Survey results are based upon five times the

number of interviews it would appear that the Cross Survey results are the better

estimate. However, it is not that simple. The distribution of interviews by state in the

Cross Survey analysis is based upon the precinct sample sizes allocated to the states

based upon their competitiveness, which does not necessarily correlate to the size of the

state. When Cross Survey is adjusted for the total vote in each state, the respondents in

large states like California and Texas can be weighted up by a magnitude of 4 or 5 and

respondents in small states like New Mexico and West Virginia can be weighted down by

a magnitude of 4 or 5. This disparity in the weights per respondent does add to the

sampling error of the estimates.

Regardless of this effect, there are many operational issues to be resolved before we can

design a system to match the results from the results from the National Survey and the

Cross Survey on election day. We will explore this problem further.

Evaluation of Edison/Mitofsky Election System

page 64

January 19, 2005

Comparison of NEP exit poll results with other exit polls

As another validation of the demographic distribution in the NEP exit poll, we compared

the NEP exit polls with other exit polls that were conducted on the same day. The L.A.

Times conducted a national exit poll and an exit poll in California. Market Shares

conducted an exit poll in Wisconsin.

For the common questions we have compared the results of the NEP polls with the other

exit polls that were conducted. Please note that in some cases the question wordings are

slightly different and this can account for some of the differences in the results. This is

especially true with how the L.A. Times measures the number of Hispanics. The L.A.

Times just asks one race question. The NEP questionnaires have an additional question

for Hispanic, and the results combine the responses to the race question and the Hispanic

question to measure the total number of Hispanics.

There are other differences between the survey methodologies that may account for some

of the differences in results. The NEP surveys have an age-race-sex non-response

adjustment. This could account for the fact that the NEP surveys have an age distribution

that is slightly older than the other exit polls.

Even accounting for these differences in question wording and sampling methodology,

comparing the results of the surveys show that they are consistent.

It is worth noting the L.A. Times reported that 45% of Hispanics nationally voted for

George W. Bush, which is almost exactly the percentage (44%) that the NEP National

Exit Poll showed. The L.A. Times National Exit Poll is based upon a sample of 135

precincts nationally so it likely has even more of the clustering effects that were

discussed in the previous section comparing the NEP National Exit Poll with the Cross

Survey results.

Evaluation of Edison/Mitofsky Election System

page 65

January 19, 2005

Comparison of NEP National Exit Poll and Los Angeles Times Exit Poll November 2, 2004

NEP National Exit Poll

L.A. Times National Exit

Poll

Gender

% of voters Bush Kerry

% of voters Bush Kerry

Men

46

55

44

49

53

46

Women

54

48

51

51

49

50

Gender & Marital Status

Married Men

30

60

39

31

59

40

Single Men

16

45

53

16

40

58

Married Women

32

55

44

30

57

42

Single Women

22

37

62

19

35

64

Age

18-29

17

45

54

20

43

55

30-44

29

53

46

32

52

47

45-64

38

52

47

36

54

45

65 or older

16

52

47

12

55

45

Race *

White

77

58

41

79

57

42

Black

11

11

88

10

14

86

Latino

8

44

53

5

45

54

Asian

2

44

56

3

34

64

Religion

Protestant

54

59

40

51

61

38

Catholic

27

52

47

25

55

44

Jewish

3

25

74

4

26

74

Religious Attendance

Weekly or more

41

61

39

42

65

34

Less than that

54

44

55

58

42

57

Gun ownership

Own guns

41

63

37

36

65

34

Don't own any

59

43

57

64

43

56

Voting status

First-time voter

11

46

53

11

42

57

Voted before

89

51

48

89

53

46

Union household

Union households

24

40

59

27

43

56

Non-union households

76

55

44

73

54

45

Evaluation of Edison/Mitofsky Election System

page 66

January 19, 2005

Sexuality

Heterosexual

96

53

46

96

53

46

Gay/lesbian/bisexual

4

23

77

4

17

81

* Note: L.A. Times asks just one race/ethnic question; NEP combines this question with a separate

"Are you Hispanic?" question

Evaluation of Edison/Mitofsky Election System

page 67

January 19, 2005

Comparison of NEP California Exit Poll and Los Angeles Times California Exit Poll November 2, 2004

NEP California Exit

Poll

L.A. Times California

Exit Poll

NEP California

Exit Poll

L.A. Times

California Exit Poll

NEP California

Exit Poll

L.A. Times

California Exit Poll

President

President

Senate

Senate

Prop 71

Prop 71

Gender

% of voters Bush Kerry % of voters Bush Kerry

Boxer Jones

Boxer Jones

Yes No

Yes No

Men

49

47

50

48

45

53

53

44

55

41

58

42

60

40

Women

51

41

57

52

42

57

65

34

62

34

61

39

59

41

Age

18-29

22

39

58

20

38

61

64

33

63

30

63

37

64

36

30-44

28

46

52

29

44

54

58

40

56

39

46

44

58

42

45-64

34

46

51

39

47

52

56

39

56

41

59

41

56

44

65 or older

16

43

55

12

42

57

57

40

63

35

61

39

62

38

Race *

White

65

51

47

65

52

47

51

47

51

45

58

42

56

44

Black

6

18

81

7

14

84

86

14

83

13

61

39

68

32

Latino

21

32

63

14

31

68

73

23

71

23

63

37

61

39

Asian

4

34

66

9

35

64

76

24

66

29

64

36

72

28

Religion

Protestant

44

60

38

46

59

40

46

53

43

53

50

50

47

53

Catholic

28

35

63

26

41

59

67

29

64

32

63

37

61

39

Jewish

4

19

78

5

20

80

79

20

87

12

86

14

77

23

Voting status

First-time voter

13

34

63

11

40

59

70

27

63

31

68

32

61

39

Voted before

87

46

52

89

45

54

57

41

57

39

58

42

58

42

Union household

Evaluation of Edison/Mitofsky Election System

page 68

January 19, 2005

Union households

27

39

58

31

39

60

63

35

63

33

60

40

60

40

Non-union households

73

47

51

69

47

51

56

41

56

40

59

41

59

41

* Note: L.A. Times asks just one race/ethnic question; NEP combines this question with a separate "Are

you Hispanic?" question

Evaluation of Edison/Mitofsky Election System

page 69

January 19, 2005

Comparison of NEP Wisconsin Exit Poll and Market Shares Wisconsin Exit Poll November 2, 2004

NEP Wisconsin Exit Poll

Market Shares Wisconsin

Exit Poll

NEP Wisconsin

Exit Poll

Market Shares

Wisconsin Exit

Poll

President

President

Senate

Senate

Gender

% of voters Kerry Bush

% of voters Kerry Bush

Feingold Michels

Feingold Michels

Men

47

46

52

45

48

50

52

48

51

47

Women

53

53

46

55

53

45

58

42

58

40

Age

18-29

20

57

41

21

58

39

56

42

61

37

30-39

18

44

54

20

44

54

50

48

49

49

40-49

25

47

53

22

46

52

53

46

51

48

50-64

24

50

49

25

53

45

57

42

57

42

65 or older

14

54

46

13

54

44

57

43

54

42

Race

White

90

47

52

91

50

48

53

47

53

44

Black

5

86

14

5

85

13

83

17

84

14

Party Identification

Democrat

35

93

7

43

94

5

93

7

92

6

Independent

27

53

45

16

53

40

62

37

60

34

Republican

38

8

91

41

5

94

14

86

11

87

Voting status

First-time voter

10

58

41

10

61

34

57

41

64

31

Voted before

90

49

50

90

51

47

54

45

54

44

Evaluation of Edison/Mitofsky Election System

page 70

January 19, 2005

Legal and Distance Issues

Legal and distance issues were a major concern in several states.

The most important state was Ohio in which we were forced to file suit against the

Secretary of State of Ohio the day before the election. Five days before election day the

Ohio Secretary of State told county election officials to keep all exit poll interviewers at

the 100-foot electioneering distance. He had previously stated in writing that exit polling

was not electioneering and that no Ohio statute regulated the distance from the polling

place for conducting exit polls. During the summer the Ohio director of elections had

assured us there was no change in the Secretary of State’s position.

Our suit was filed the day before the election. We were successful in overturning the

Secretary of State’s ruling, but the court ruling did not occur until 10:30 PM on the night

before the election. Although we were able to contact all of our interviewers before the

polls opened, many Ohio election officials at our polling places did not know of this

ruling when the polls opened and many of our interviewers in Ohio were delayed in

starting their interviews until the local election official was informed of this ruling. The

last local election official did not permit our interviewer to begin work until close to 5

PM.

There were 69 polling locations in which our legal team dealt with legal and distance

issues on election day. The highest number was in Ohio (14) but there were several other

states with a significant number of legal issues – Arizona (3); Colorado (3); Illinois (4);

Iowa (5); Minnesota (5); New Jersey (8); South Dakota (3); West Virginia (4). It is

important to note that in several states with very restrictive exit polling laws (particularly

Minnesota and South Dakota), many interviewers did not report their situation as a “legal

problem” because they had been trained to anticipate the distance requirement and to

attempt to do their jobs as best they could.

From our post-election survey, the exit poll interviewers reported the distances that they

were forced to stand from the polling location on election day:

Location of Interviewer

# of polling locations mean WPE *

Miss Rate

Inside the Building

506 38%

-5.3

9%

Right outside the entrance 235 17%

-6.4

10

10-25 feet away

239 18%

-5.6

11

25-50 feet away

165 12%

-7.6

13

50-100 feet away

148 11%

-9.6

16

More than 100 feet away

53

4%

-12.3

18

* mean WPE based upon analysis of 1,250 exit poll precincts included in evaluation of

WPE earlier in this report

Evaluation of Edison/Mitofsky Election System

page 71

January 19, 2005

The percentage of polling locations in which our exit poll interviewers were forced to

stand 50 or more feet away was more than double the number that VNS experienced in

2000. According to the VNS records only 6% of their interviewers were forced to stand

50 feet or more from the polling place in 200 as opposed to 15% in 2004.

There were several states, before election day, that we knew were planning on enforcing

an interviewing distance greater than 50 feet. In these states and several others, a large

portion of our interviewers were forced to stand 50 feet or further from the polling place:

State

# of interviewers standing more than

50 feet from the polling location

Arizona

29 polling locations

Colorado

6 polling locations

Florida

14 polling locations

Indiana

6 polling locations

Louisiana

9 polling locations

Minnesota

18 polling locations

Nevada

24 polling locations

New Mexico

13 polling locations

Ohio

8 polling locations

South Dakota

23 polling locations

More than one-quarter of all interviewers (27%) reported that they were not able to be in

a position to approach every voter as they were exiting the polling location. Among

those who were standing more than 50 feet away from the polling location, more than

half (53%) told us that they were not in a position to approach all voters as they left the

polling place.

In addition to problems with local election officials, about 5% (79) of our exit poll

interviewers reported that they experienced interference from other people at the polling

place such as poll watchers, lawyers or electioneers that limited their ability to conduct

surveys on election day. A handful of our interviewers experienced some sort of

sabotage including voters who stole questionnaires, voters who spoiled questionnaires by

spilling liquids on them, and in a few instances our interviewers were escorted from the

polling place by police officers.

With the information that we have gathered on election day and since, we have

convincing evidence that can be used to demonstrate that both the response rates and the

accuracy of the exit poll data decrease once an interviewer is forced to stand more than

25 feet away from the polling location.

Evaluation of Edison/Mitofsky Election System

page 72

January 19, 2005

Absentee/Early Voter Telephone Surveys

As many states have not yet reported the vote among absentee and early voters, it is not

possible to evaluate all of the estimates produced by the absentee telephone surveys.

Once that vote data is available we will do a full evaluation of the estimates produced by

the absentee/early voter telephone surveys.

In this report we discuss our preliminary evaluation of the absentee/early voter telephone

surveys. It appears that the sample sizes of these surveys were in most cases sufficient to

make statewide estimates of the vote that were used to adjust the vote estimates.

However, in comparison to the sample sizes of the election day exit polls, the sample

sizes of the absentee/early vote telephone surveys are probably not sufficient to

accurately measure many of the demographic subgroups in the analysis. In that section

of the report we recommend increasing the sample sizes of these telephone surveys.

We can compare the survey estimates with the actual absentee vote returns in three states

(North Carolina, Oregon and Texas).

Absentee Telephone

Absentee

North Carolina

Survey Estimate

Final Vote

Diff

President

Bush (R)

51%

Bush (R)

53.2%

-2.2%

Kerry (D)

49%

Kerry (D)

46.5%

+2.5%

Other

0.3%

Senate

Burr (R)

48%

Burr (R)

49.8%

-1.8%

Bowles (D) 52%

Bowles (D) 49.2%

+2.8%

Other

1.0%

Governor

Easley (D)

58%

Easley (D)

55.6%

+2.4%

Ballantine (R) 42%

Ballantine (R) 43.2%

-1.2%

Other

1.2%

Oregon

President

Kerry (D)

53%

Kerry (D)

51.4%

+1.6%

Bush (R)

47%

Bush (R)

47.2%

-0.2%

Other

1.2%

Senate

Wyden (D) 69%

Wyden (D) 63.4%

+5.6%

Kin (R)

31%

King (R)

31.7%

-0.7%

Other

4.9%

Measure 33

No

56%

No

57.2%

-1.2%

Yes

44%

Yes

42.8%

+1.2%

Measure 36

Yes

54%

Yes

56.6%

-2.6%

Evaluation of Edison/Mitofsky Election System

page 73

January 19, 2005

No

46%

No

43.4%

+2.6%

Texas

President

Bush (R)

65%

Bush (R)

62.7%

+2.3%

Kerry (D)

35%

Kerry (D)

36.8%

-1.8%

Other

0.5%

Evaluation of Edison/Mitofsky Election System

page 74

January 19, 2005

Optimum size of absentee phone samples:

The sample sizes for the absentee telephone survey samples seem to be sufficient to make

relatively accurate estimates of the vote among absentee/early voters that were used in

the computations.

We suggest the size of the telephone samples be increased for the states where the

absentee vote is a significant share of the total vote. The rationale is as follows: When the

absentee telephone survey respondents are merged in with the election day exit poll

respondents the absentee telephone survey respondents in every state except California

had to be weighted up by a factor of between 2 and 3. This increased the sampling error

over what it would have been if the size of the telephone surveys had been proportional to

the size of the exit poll interviews. This affected both the analysis tabulations and the

estimates. If the telephone respondents are a small proportion of the total voters the

additional weighting factor is probably not significant. However, when the telephone

survey represents a significant share of the total vote the increased weighting factor will

increase the total sampling error substantially.

Ideally, the absentee telephone survey sample sizes should be roughly doubled in the

states where absentees are a large proportion of the total vote.

Additionally, the decision to administer all of the questions from the four versions of the

national survey to all of the respondents in the national absentee telephone survey added

a complication to the processing of the exit poll. Each national absentee telephone

survey respondent had to be entered as four respondents – once for each version. This

meant that in states with absentee surveys the unweighted sample sizes shown in the

crosstab screens counted these respondents four times. The correct sample sizes were

available on the methodology pages of each survey. In retrospect, it would have been

better to have conducted 2,000 interviews, with one-quarter of the respondents getting

just the questions from each version of the national questionnaire.

With only 500 respondents representing 16% of all voters nationally, the results of the

national and regional breakouts for some of the smaller demographics are based upon

some very small sample sizes. For instance, the number of Hispanic absentee telephone

interviews conducted in the southern region was 14, and these 14 interviews were

weighted to reflect approximately 20% of the Hispanic vote in the South.

Evaluation of Edison/Mitofsky Election System

page 75

January 19, 2005

This table shows that in most states in which we conducted absentee/early voter

telephone surveys, the respondents in the telephone surveys needed to be weighted up

often by a factor of 2 or 3. We are recommending that in the future the sample sizes for

the state absentee/early voter telephone surveys be increased by 50% and that the

National absentee/early vote telephone survey sample size be at least 1500.

State

Absentee

N (incl.

Nat’l q’s)

Exit Poll

Election

Day N

% absentee

(unweighted)

% absentee

(weighted)

ARIZONA

381

1500

20%

40%

CALIFORNIA

474

1541

24%

30%

COLORADO

477

2024

19%

55%

FLORIDA

376

2384

14%

27%

IOWA

345

2146

14%

30%

MICHIGAN

324

2198

13%

20%

NEVADA

465

1716

21%

50%

NEW MEXICO

371

1609

19%

50%

NORTH

CAROLINA

299

1800

14%

25%

TENNESSEE

333

1352

20%

47%

TEXAS

346

1237

22%

51%

WASHINGTON 676

1387

33%

73%

NATIONAL

500

11719

4%

16%

Evaluation of Edison/Mitofsky Election System

page 76

January 19, 2005

Areas for Further Investigation in 2005

We have compiled a list of further areas of investigation that we will be pursuing in-

depth during 2005 in preparation for the next round of national elections in 2006 and

2008. Some of these will be continued analysis of items that we have already

investigated for this report. We will be evaluating many areas of the computations with

the goal of proposing adjustments to improve both the accuracy of the estimates and, as

importantly, the accurate computation of the estimated error in each computation. In

addition, we plan to conduct complete studies on the absentee vote estimates and the

demographics measured in the exit poll surveys.

Computations:

1. Evaluate WPE computation methods used in the 2004 election (average precinct

WPE values compared to using the Simple Geo to estimate the WPE).

2. Evaluate the Best WPE adjusted estimator in order to determine at what point the

WPE adjustment is improving the Best SPM estimate.

3. Evaluate the Missing Data Factors (MDF) using the 2004 data. Also, evaluate the

Best SPM estimates using different amounts of reported vote.

4. Evaluate the Best SPM stratum estimates using the precinct vote compared to the

final results.

5. Re-evaluate the County Model SEDF.

6. Evaluate turnout estimates – add screens to summarize state turnout estimates and

add the ability to send approved turnout estimates in the member data feed

Absentee/Early Voter Telephone Surveys (13 states plus the national):

1. Compare vote estimates from telephone surveys with actual reported

absentee/early vote returns.

2. Redesign survey weighting spec to include forcing by candidate by geo stratum

including absentee survey respondents.

3. Evaluate Oregon telephone sample design – Comparing RDD and RBS portions.

4. Evaluate the effect of the large weights needed in most states to adjust the size of

the absentee portion of the surveys and make recommendations for future

telephone sample sizes.

5. Evaluate demographics of absentee surveys with election day surveys and

compare to the demographics of absentee voters from actual voter lists where

possible.

Evaluation of Edison/Mitofsky Election System

page 77

January 19, 2005

National Survey:

1. Evaluate regional estimates used for forcing.

2. Compare demographics in national survey with cross-survey results.

3. Evaluate the effect of computing a national composite estimate with a prior and a

national absentee adjustment.

4. Evaluate age-race-sex non-response adjustment for the national survey.

5. Evaluate national survey precinct sample.

State Surveys:

1. Continue evaluation of Within Precinct Error.

2. Evaluate age-race-sex non-response adjustment.

3. Evaluate demographics (especially Black and Hispanic compositions) and

compare them with past VNS surveys.

4. Compare demographics from state surveys against vote files in the states where

age, race and sex distributions of actual voters are available.

5. Evaluate the correlations between exit poll tallies and past vote data as a potential

indicator of candidate bias in the exit polls.

6. Evaluate interviewing rate and subsampling calculations.

7. Evaluate use of Spanish language surveys in each state.