Attempts to discredit the Klein
RAND report or my ad or me
What is covered here:
to properly challenge scientific data
improper attacks the Bush campaign has waged and why they fail
A proper attack
If you want to refute a scientific study like that these (the RAND
study was done by 4 experts with a combined 50 years of experience in the
field), you must show (in a way that passes a peer-review test) at least
one of four things: (1) an error in the assumptions, (2) an error in the data,
(3) an error in the methodology, or (4) an error in interpreting what the
results mean. And nobody has challenged any of these factors in the new
national test data used was complete and accurate. There is no dispute. Nobody
has brought forward any new data to dispute the consistent conclusions of these
two papers. They just claim there is a conflict with the earlier RAND study
that claimed "significant improvement using national test data."
No attacker I've heard from has not bothered to read the 12 page Klein paper. Nor has anyone
done the calculation themselves. It's not hard.
You look at Table 1 and Table 2 in the paper and you can see what is going on
(you must also know that non-whites in Texas have scored higher than the national average
for a decade, something Bush can't take any credit for).
It is pretty obvious that it would be virtually impossible to challenge the
Klein paper. There was no "fuzzy math" going on as in the July RAND
study which used "adjusted" scores. And the new Klein paper covered data from 1994 onward in Table 2. If
something was going on in Texas, it would have shown up in Table 2. It
didn't even show up in Table 1 which looked at math from 1992 to 1996.
Summary of typical bogus attacks used in the RAND study
- Timing: The Klein report was supposed
to come out earlier. It was delayed because RAND management thought it was
so explosive that they put it through 5 independent peer reviews, far more
than normal. It is also irresponsible to attack a scientific paper on the
basis of the date it was published. The only way to discredit a peer
reviewed scientific paper is to challenge the data, the methodology, the
interpretation, or the assumptions.
- Conflicts with an earlier more comprehensive report: The problem with using the earlier RAND study is that the older
study mixed in data before 1994 when TAAS began. So the old study failed to
make a distinction between states who coached kids (“high stakes
testing”) and states without special coaching. Test scores always go up
when you teach for the test. That is well established. When you re-analyze the data used in the
earlier report, you find that for all tests after 1994, there was no improvement. If
there really was something going on in Texas, you'd have seen the set of
scores after 1994 skyrocket just like the Texas state scores did. They were
flat. And the author of the earlier report (Grissmer) admitted that freely.
So Grissmer's conclusions were exactly correct. He just didn't fully
explain the cause of the gains. He should have added something to the
effect of, "we note that in 1994, Texas instituted high-stakes testing.
This alone may have caused all the gains we saw. Whether there were gains
beyond what would be introduced by high-stakes testing would be the subject
of another research report." And that second research report is Klein's
where Klein showed that the gains that Grissmer observed was all due to the
introduction of high-stakes testing rather than any improvement in real
student proficiency. The mystery of the conflict is solved.
In addition, the
argument about using the earlier study to discredit the new study is
also somewhat of a red-herring. The “Texas Miracle” was based on state
TAAS scores. The earlier study didn’t use TAAS scores.
- Incomplete data means incomplete report: The argument about missing
state data is a red-herring. Klein showed that all the data Texas presented
on their website was garbage. The only possible way the state can produce
credible data is if it corresponds with NAEP testing. And if they did that,
they’d be validating that there were no gains! It’s a no-win situation
for Bush. Chester Finn wrote in The New York Times that the data was
incomplete but he failed to tell us what data was missing, he didn't produce
the missing data, and mentioned nothing that would dispute the conclusion of
the paper other than the "conflict" which we've just explained
(that he couldn't figure out). Remember, RAND said both reports were
correct. Finn needs to come up with an explanation that fits the facts.
- Texas has the highest NAEP scores for people of color: The
argument that Texas NAEP scores are above the US average especially
for Blacks and Latinos is also bogus. It's been that way for a long time
(for at least a decade).
Invalid attacks on the RAND study [Klein]
The "show them conflicting credible data" attack (a.k.a, the
"it's not consistent so the new data can't be right") attack
They point to the first RAND report (which used the "time before
governor" attack below) to create a conflict. Defense: You explain
first how the original report doesn't apply because it mixed apples and oranges.
They misinterpreted the data in the first report because the first report looked
at the trend from 1990 to 1996 and you are more interested in the period from
1994 onward. So if the only gain occurred in 1993, it would show up in the first
report. The first report didn't pinpoint what years the gains occurred. Plus, it
is likely the gain is false since it occurred only on one exam in the lower
grades that is similar to the TAAS exam. Next, you ask them to challenge the
Klein report on it's merits using any of the valid methods above. You ask them
specifically, "if there were gains, why didn't they show up?"
The "explanation that doesn't quite fit the facts" attack a.k.a.
the "partial explanation that seems to fit the facts"
They try to attack the SAT statistic by saying it fell because more students are taking the SAT now in Texas.
You are supposed to think, "more students" means more minorities. Defense:
You find out that the reason the numbers went up is due to population growth,
and that the % of total students taking the SAT hasn't changed, and neither has
the racial mix. Here are a few more "tricks" (as an example so you
don't get fooled):
When Bush says, "Our Minority Students... Are Making Tremendous
Improvement on the SAT" he's only partially telling the story
Bush forgets to say, "... but not as much as the
rest of the country so the fact is that Gore's plans are working and mine are
not. Our minorities showed less improvement on SAT than the rest of the
Texas African-American Students Show Less Improvement in SAT Scores.
While Texas African- American students have shown improvements on the
Scholastic Aptitude Test every year since 1994, the improvements were not as
great as African American students across the nation. [Dallas Morning News,
Texas Hispanic Students Fare Poorly on SAT. Unlike their
counterparts across the country, Texas Hispanic students actually lost
ground on the math section of the Scholastic Aptitude Test and made only
slight gains on the verbal section over the last five years. [Dallas Morning
In Texas, Minority SAT Scores Dropped in 1998. Average combined
SAT scores for minorities in the class of 1998 in Texas schools were lower
than those posted by minorities in the class of 1997. African-Americans in
the class of 1998 scored a combined average of 848 versus a score of 849 in
1997. Hispanics dropped from a score of 907 in 1997 to a score of 904 in
1998. Overall, scores for African-Americans and Hispanics were still
significantly lower than scores for whites. White students in 1998 scored an
average of 1045 on the SAT, up one point from the class of 1997. [1998-1999
Texas State AEIS Report, www.tea.state.tx.us/perfreport/aeis/99/state.html]
They do this "partial story" technique a lot on the campaign web
site. Here's the full story on Bush's
The "taking credit for a pre-existing condition without acknowledging
They point out that Blacks and Hispanics performance in Texas is among the
highest in the country. Defense: Someone has to be first. Nobody knows
why this happens. It's been that way for a decade. Besides, it isn't where you
are currently. It is how much you can show that students under your watch improve faster
than students in other states!
The "bad timing" attack
They say the timing proves it is a biased study. Defense: This is
about data, not timing. The report was delayed for 5 rounds of peer review. Ask
them to show you where one of the four factors is incorrect. As far as the timing
of my ad campaign, the press forced me to do this by not picking up on the
story because it was "too late in the campaign" and because
"they've already covered it." The press has known about my
investigation for a long time. And leading Republican supporters with close ties
to Bush were e-mailed the content of the ad on November 2, five days before the
election. They knew what was coming. I even linked up the final version on my
public website on November 5.
The political motive attack
See the "bad timing" attack.
The "you have your web site on your corporate site" attack
They will claim it is a violation of Federal election law to have my
personal website hosted by my company. Defense: What does that have to do
with the RAND report? Just for your info, I paid my company in advance, to host
my site. It was originally a backup site. It became my primary site because the
ISP I have been using for over a year (FirstWorld), didn't have it's act
together and the server that my website was on, star1.sirius.com, has been down
more than it's been up in the past week. I even called the CEO to complain and
got no where. So I moved the site at the last minute and wrote a check to my
company to pay for it. Furthermore, Propel's business is includes hosting other
people's web sites so this is not at all unreasonable.
The "you are a major donor to the DNC" attack
Defense: What does that have to do with the RAND report? Yes, I am a major
DNC contributor. I admit that and will tell you why
too. But I'm also a registered Republican and have publicly endorsed
Republican candidates for Congress, such as Jim Cunneen. All of this is a
smokescreen. Let's get back to the data, ok?
The "using stats that include the time before you became
This is a variant of the "pre-existing condition" attack. Basically,
you wave a document like the 1999 National Education Goals Report http://www.negp.gov/issues/publication/99statefact/tx.htm
which if you ignore the dates, sounds great. Defense: You point out the
date range of interest here is 1994 and later and bring the focus back to
challenging them to attack Table 2. You also tell them again how
"misleading" it is to refer to this report on your campaign website as
"Greatest Progress in the Nation: Texas is one of two states that
has made the greatest recent progress in education, according to the
Congressionally-mandated National Education Goals Panel" since this covers
a 10 year period, not the last few years. That convenient little fact is left
out of the claim.
The "incomplete data means incomplete report" attack
They say there is more data you didn't have so your conclusions can't be
right. Defense: There is always more data. In this case, since there was
sufficient data to determine the data can't be trusted, bringing out more
untrusted data is a waste of everyone's time.
The "you aren't an expert" or "you've ignored the
expert's analysis" attack
They will point to Bush and the Wall St Journal and say you are wrong. Defense:
I worked with several experts in the field who confirmed what I did was correct. In fact, all I did
was verify the data in Table 1 and Table 2 in Klein and talk to the authors to
explain the 1996 4th grade math score jump (a skill set overlap with the TAAS is
the most likely explanation, but there are others and it doesn't really matter
too much what the real explanation is since Klein showed that the gain was not
sustainable). I also verified that the Tables in Klein show no
true gain. You ask them to show
you where the Wall St Journal and Bush explain the "conflict" between
the RAND reports. Then you ask those experts to attack Klein's report in a way
that would pass a scientific peer review.
The "personal credibility" attack
They found out that you had an unpaid parking ticket or a DUI arrest. Defense:
What the hell does that have to do with challenging the data in Klein? Let's
stick to challenges of the four types at the top of this page, please!
The "you have a typo on your website" attack
By pointing to a typo, they can discredit your argument. Defense: They
have to discredit Klein, not me.
The "the argument on your website is inconsistent" attack
By pointing to a inconsistency, they can discredit your argument. Defense: They
have to discredit Klein, not me. I wrote the website over the course of two
weeks so I'd be surprised if there weren't inconsistencies. And there are many
ways to explain the story. But the stories all are consistent about one thing:
if you look only at the data after 1994, you see no gain in both RAND studies,
so the RAND studies are consistent.
The "this is illegal because you've been talking to the DNC"
True, I talked to the DNC about my donation. That's legal. I didn't talk to
them about my independent expenditures. They have no idea what I have done. I've
talked only briefly with one DNC campaign strategist and that was on Oct 22. I
had not planned to run any ads myself. But because the press was busy with DUI
stories about Bush and "had already covered" the RAND report, I had no
choice but to advertise to get the word out to people. Here's the story on the timing
of the ads.
The "you don't have enough NAEP data to prove it conclusively"
Defense: Funny. When the data showed what you wanted to show, even though
it was at odds with SAT, ACT, TASP, etc. data, it was never questioned
that it was sufficient. When the independent test data confirms the incredible
array of other independent test data (SAT, ACT, Texas' own TASP), you say
"it's not enough." Come on. That's not credible. With Haney and Klein
(and common sense!) showing TAAS scores are bogus, you are only left with data
that is flat to down. Unless you can show how it is possible for all that data
to be wrong with a great explanation or stand-up comedy routine, you lose.
The RAND CEO "obfuscation" attack
They replied by pointing to the letter
in the LA Times from the RAND CEO which didn't address the issue and
explain the conflicts between the reports. In particular, while he did mention
that the adjusted scores were adjusted for the demographic factors, he should
have mentioned that "the scores were not adjusted for the initiation of
high-stakes testing." Nor did he mention that because the earlier report
covered a period from 1990 to 1996, that the only explanation that fits the
facts of both reports are that the educational gains (if any remaining after
compensating for the high-stakes testing intro) in Texas happened before 1994
(when Bush was elected). By continuing to obfuscate the issue, they are being
partisan. Defense: They are hiding material data from the American
public. The facts are unambiguous: if you look at the NAEP data starting from
1994 onwards, Texas has shown no improvement relative to the rest of the
country. Both reports authors confirm that. The reason the earlier report showed
gains is that the gains happened prior to 1994 and the earlier report said there
were gains in the period from 1990 to 1996. This fits the facts and there is no
other explanation that does. So ask them for an explanation that fits the facts.
The "show them some good data to re-direct attention away from the
The attempt here is to shift focus away from the reliable third party national
test score onto some other statistic that looks good to prove the progress
you've made. You try to distract the listener to focus on those great things
instead of measures of academic proficiency. Defense(s): (a) spit back a
horrendous list of negative stats and say this isn't about stats, it is about
measuring academic performance. (b) Get the focus back on telling you how Klein
Table 2 is wrong.
The "discredit the NAEP test" attack
You argue here that the NAEP data doesn't truly measure proficiency because it
is so broad. Defense: Of course, that's the point. And this falls very
flat in this case since the Republicans earlier waved the July RAND report
(which is soley based on NAEP data) as "proof" that there were gains.
Suddenly changing your story on this is not credible. The NAEP test is the
"gold standard." The attack falls very flat.
The "my bluffs should be assumed true until proven otherwise"
This is a very clever strategy. The purpose here is to shift the focus of
attention away from the reliable data. Defense: What should be happening
is that a reviewer should be looking at the reliable data and asking Bush,
"if there are gains, how can you explain why they didn't show up in the
NAEP data." Instead, Bush tries to turn it around his way, "I have an
overwhelming amount of (unreliable) data that shows there is progress. Everyone
knows it. Your new data is suspect because it disagrees with the facts everyone
knows." That's very effective attack because you really want to believe he
is telling you the truth. It's very easy to lose focus. You must concentrate on
remembering that the trusted data is the NAEP data and the burden of proof is on
him, not you, to prove the NAEP data is wrong and prove his data is reliable. If
he doesn't do either, he has no basis for making the claim of educational
The "education track record isn't the issue" attack
They will claim Bush isn't a single issue candidate and what counts is his
philosophy looking forward. Defense: Get back to the issue. We are just
focused on whether his record is correct. Show me the data that invalidates
The "they aren't the same students" attack
You can try to attack Klein's Tables by claiming they aren't the same
students being sampled. Defense: In no case do you ever have the same
students. In comparing 4th grade in 1992 to 4th grade in 1996, you have no
student overlap whatsoever. And they never complained about that before (in
Grissmer's study). At any rate, you can calculate the improvement the other
direction and you get the same answer.
Point out that if education is so good in Texas, then...
- how do you explain Klein (Table 2)?
- why are all the counter-attacks coming from the Bush campaign.and not from
- why are the ACT, SAT scores flat to declining when the % of students (and
the racial mix) haven't changed?
- how do you explain it when The New York Times
that in February 1999, officials with the University of Texas system presented
a report to a Texas House subcommittee complaining of "marked declines in
the number of students who are prepared academically for higher
- why are the teacher turnover
stats so bad?
- why did Bush veto a bill to study hunger in Texas that wouldn't have cost
anything (he doesn't remember)?
For additional attacks, see Feedback I've gotten on the ad.
Steve Kirsch Political Home Page