Attempts to discredit me or the new RAND report by various attacks

Attempts to discredit the Klein RAND report or my ad or me

What is covered here:

How to properly challenge scientific data
The improper attacks the Bush campaign has waged and why they fail

A proper attack

If you want to refute a scientific study like that these (the RAND study was done by 4 experts with a combined 50 years of experience in the field), you must show (in a way that passes a peer-review test) at least one of four things: (1) an error in the assumptions, (2) an error in the data, (3) an error in the methodology, or (4) an error in interpreting what the results mean. And nobody has challenged any of these factors in the new report. The national test data used was complete and accurate. There is no dispute. Nobody has brought forward any new data to dispute the consistent conclusions of these two papers. They just claim there is a conflict with the earlier RAND study that claimed "significant improvement using national test data."

No attacker I've heard from has not bothered to read the 12 page Klein paper. Nor has anyone done the calculation themselves. It's not hard. You look at Table 1 and Table 2 in the paper and you can see what is going on (you must also know that non-whites in Texas have scored higher than the national average for a decade, something Bush can't take any credit for).

It is pretty obvious that it would be virtually impossible to challenge the Klein paper. There was no "fuzzy math" going on as in the July RAND study which used "adjusted" scores. And the new Klein paper covered data from 1994 onward in Table 2. If something was going on in Texas, it would have shown up in Table 2. It didn't even show up in Table 1 which looked at math from 1992 to 1996.

Summary of typical bogus attacks used in the RAND study

Timing: The Klein report was supposed to come out earlier. It was delayed because RAND management thought it was so explosive that they put it through 5 independent peer reviews, far more than normal. It is also irresponsible to attack a scientific paper on the basis of the date it was published. The only way to discredit a peer reviewed scientific paper is to challenge the data, the methodology, the interpretation, or the assumptions.
Conflicts with an earlier more comprehensive report: The problem with using the earlier RAND study is that the older study mixed in data before 1994 when TAAS began. So the old study failed to make a distinction between states who coached kids (“high stakes testing”) and states without special coaching. Test scores always go up when you teach for the test. That is well established. When you re-analyze the data used in the earlier report, you find that for all tests after 1994, there was no improvement. If there really was something going on in Texas, you'd have seen the set of scores after 1994 skyrocket just like the Texas state scores did. They were flat. And the author of the earlier report (Grissmer) admitted that freely. So Grissmer's conclusions were exactly correct. He just didn't fully explain the cause of the gains. He should have added something to the effect of, "we note that in 1994, Texas instituted high-stakes testing. This alone may have caused all the gains we saw. Whether there were gains beyond what would be introduced by high-stakes testing would be the subject of another research report." And that second research report is Klein's where Klein showed that the gains that Grissmer observed was all due to the introduction of high-stakes testing rather than any improvement in real student proficiency. The mystery of the conflict is solved. In addition, the argument about using the earlier study to discredit the new study is also somewhat of a red-herring. The “Texas Miracle” was based on state TAAS scores. The earlier study didn’t use TAAS scores.
Incomplete data means incomplete report: The argument about missing state data is a red-herring. Klein showed that all the data Texas presented on their website was garbage. The only possible way the state can produce credible data is if it corresponds with NAEP testing. And if they did that, they’d be validating that there were no gains! It’s a no-win situation for Bush. Chester Finn wrote in The New York Times that the data was incomplete but he failed to tell us what data was missing, he didn't produce the missing data, and mentioned nothing that would dispute the conclusion of the paper other than the "conflict" which we've just explained (that he couldn't figure out). Remember, RAND said both reports were correct. Finn needs to come up with an explanation that fits the facts.
Texas has the highest NAEP scores for people of color: The argument that Texas NAEP scores are above the US average especially for Blacks and Latinos is also bogus. It's been that way for a long time (for at least a decade).

Invalid attacks on the RAND study [Klein]

The "show them conflicting credible data" attack (a.k.a, the "it's not consistent so the new data can't be right") attack
They point to the first RAND report (which used the "time before governor" attack below) to create a conflict. Defense: You explain first how the original report doesn't apply because it mixed apples and oranges. They misinterpreted the data in the first report because the first report looked at the trend from 1990 to 1996 and you are more interested in the period from 1994 onward. So if the only gain occurred in 1993, it would show up in the first report. The first report didn't pinpoint what years the gains occurred. Plus, it is likely the gain is false since it occurred only on one exam in the lower grades that is similar to the TAAS exam. Next, you ask them to challenge the Klein report on it's merits using any of the valid methods above. You ask them specifically, "if there were gains, why didn't they show up?"

The "explanation that doesn't quite fit the facts" attack a.k.a. the "partial explanation that seems to fit the facts" attack
They try to attack the SAT statistic by saying it fell because more students are taking the SAT now in Texas. You are supposed to think, "more students" means more minorities. Defense: You find out that the reason the numbers went up is due to population growth, and that the % of total students taking the SAT hasn't changed, and neither has the racial mix. Here are a few more "tricks" (as an example so you don't get fooled):

When Bush says, "Our Minority Students... Are Making Tremendous Improvement on the SAT" he's only partially telling the story

Bush forgets to say, "... but not as much as the rest of the country so the fact is that Gore's plans are working and mine are not. Our minorities showed less improvement on SAT than the rest of the nation."

Texas African-American Students Show Less Improvement in SAT Scores. While Texas African- American students have shown improvements on the Scholastic Aptitude Test every year since 1994, the improvements were not as great as African American students across the nation. [Dallas Morning News, 5/25/99]

Texas Hispanic Students Fare Poorly on SAT. Unlike their counterparts across the country, Texas Hispanic students actually lost ground on the math section of the Scholastic Aptitude Test and made only slight gains on the verbal section over the last five years. [Dallas Morning News, 5/25/99]

In Texas, Minority SAT Scores Dropped in 1998. Average combined SAT scores for minorities in the class of 1998 in Texas schools were lower than those posted by minorities in the class of 1997. African-Americans in the class of 1998 scored a combined average of 848 versus a score of 849 in 1997. Hispanics dropped from a score of 907 in 1997 to a score of 904 in 1998. Overall, scores for African-Americans and Hispanics were still significantly lower than scores for whites. White students in 1998 scored an average of 1045 on the SAT, up one point from the class of 1997. [1998-1999 Texas State AEIS Report, www.tea.state.tx.us/perfreport/aeis/99/state.html]

They do this "partial story" technique a lot on the campaign web site. Here's the full story on Bush's educational claims.

The "taking credit for a pre-existing condition without acknowledging it" attack
They point out that Blacks and Hispanics performance in Texas is among the highest in the country. Defense: Someone has to be first. Nobody knows why this happens. It's been that way for a decade. Besides, it isn't where you are currently. It is how much you can show that students under your watch improve faster than students in other states!

The "bad timing" attack
They say the timing proves it is a biased study. Defense: This is about data, not timing. The report was delayed for 5 rounds of peer review. Ask them to show you where one of the four factors is incorrect. As far as the timing of my ad campaign, the press forced me to do this by not picking up on the story because it was "too late in the campaign" and because "they've already covered it." The press has known about my investigation for a long time. And leading Republican supporters with close ties to Bush were e-mailed the content of the ad on November 2, five days before the election. They knew what was coming. I even linked up the final version on my public website on November 5.

The political motive attack
See the "bad timing" attack.

The "you have your web site on your corporate site" attack
They will claim it is a violation of Federal election law to have my personal website hosted by my company. Defense: What does that have to do with the RAND report? Just for your info, I paid my company in advance, to host my site. It was originally a backup site. It became my primary site because the ISP I have been using for over a year (FirstWorld), didn't have it's act together and the server that my website was on, star1.sirius.com, has been down more than it's been up in the past week. I even called the CEO to complain and got no where. So I moved the site at the last minute and wrote a check to my company to pay for it. Furthermore, Propel's business is includes hosting other people's web sites so this is not at all unreasonable.

The "you are a major donor to the DNC" attack
Defense: What does that have to do with the RAND report? Yes, I am a major DNC contributor. I admit that and will tell you why too. But I'm also a registered Republican and have publicly endorsed Republican candidates for Congress, such as Jim Cunneen. All of this is a smokescreen. Let's get back to the data, ok?

The "using stats that include the time before you became governor" attack
This is a variant of the "pre-existing condition" attack. Basically, you wave a document like the 1999 National Education Goals Report http://www.negp.gov/issues/publication/99statefact/tx.htm which if you ignore the dates, sounds great. Defense: You point out the date range of interest here is 1994 and later and bring the focus back to challenging them to attack Table 2. You also tell them again how "misleading" it is to refer to this report on your campaign website as "Greatest Progress in the Nation: Texas is one of two states that has made the greatest recent progress in education, according to the Congressionally-mandated National Education Goals Panel" since this covers a 10 year period, not the last few years. That convenient little fact is left out of the claim.

The "incomplete data means incomplete report" attack
They say there is more data you didn't have so your conclusions can't be right. Defense: There is always more data. In this case, since there was sufficient data to determine the data can't be trusted, bringing out more untrusted data is a waste of everyone's time.

The "you aren't an expert" or "you've ignored the expert's analysis" attack
They will point to Bush and the Wall St Journal and say you are wrong. Defense: I worked with several experts in the field who confirmed what I did was correct. In fact, all I did was verify the data in Table 1 and Table 2 in Klein and talk to the authors to explain the 1996 4th grade math score jump (a skill set overlap with the TAAS is the most likely explanation, but there are others and it doesn't really matter too much what the real explanation is since Klein showed that the gain was not sustainable). I also verified that the Tables in Klein show no true gain. You ask them to show you where the Wall St Journal and Bush explain the "conflict" between the RAND reports. Then you ask those experts to attack Klein's report in a way that would pass a scientific peer review.

The "personal credibility" attack
They found out that you had an unpaid parking ticket or a DUI arrest. Defense: What the hell does that have to do with challenging the data in Klein? Let's stick to challenges of the four types at the top of this page, please!

The "you have a typo on your website" attack
By pointing to a typo, they can discredit your argument. Defense: They have to discredit Klein, not me.

The "the argument on your website is inconsistent" attack
By pointing to a inconsistency, they can discredit your argument. Defense: They have to discredit Klein, not me. I wrote the website over the course of two weeks so I'd be surprised if there weren't inconsistencies. And there are many ways to explain the story. But the stories all are consistent about one thing: if you look only at the data after 1994, you see no gain in both RAND studies, so the RAND studies are consistent.

The "this is illegal because you've been talking to the DNC" attack
True, I talked to the DNC about my donation. That's legal. I didn't talk to them about my independent expenditures. They have no idea what I have done. I've talked only briefly with one DNC campaign strategist and that was on Oct 22. I had not planned to run any ads myself. But because the press was busy with DUI stories about Bush and "had already covered" the RAND report, I had no choice but to advertise to get the word out to people. Here's the story on the timing of the ads.

The "you don't have enough NAEP data to prove it conclusively"
Defense: Funny. When the data showed what you wanted to show, even though it was at odds with SAT, ACT, TASP, etc. data, it was never questioned that it was sufficient. When the independent test data confirms the incredible array of other independent test data (SAT, ACT, Texas' own TASP), you say "it's not enough." Come on. That's not credible. With Haney and Klein (and common sense!) showing TAAS scores are bogus, you are only left with data that is flat to down. Unless you can show how it is possible for all that data to be wrong with a great explanation or stand-up comedy routine, you lose.

The RAND CEO "obfuscation" attack
They replied by pointing to the letter in the LA Times from the RAND CEO which didn't address the issue and explain the conflicts between the reports. In particular, while he did mention that the adjusted scores were adjusted for the demographic factors, he should have mentioned that "the scores were not adjusted for the initiation of high-stakes testing." Nor did he mention that because the earlier report covered a period from 1990 to 1996, that the only explanation that fits the facts of both reports are that the educational gains (if any remaining after compensating for the high-stakes testing intro) in Texas happened before 1994 (when Bush was elected). By continuing to obfuscate the issue, they are being partisan. Defense: They are hiding material data from the American public. The facts are unambiguous: if you look at the NAEP data starting from 1994 onwards, Texas has shown no improvement relative to the rest of the country. Both reports authors confirm that. The reason the earlier report showed gains is that the gains happened prior to 1994 and the earlier report said there were gains in the period from 1990 to 1996. This fits the facts and there is no other explanation that does. So ask them for an explanation that fits the facts.

The "show them some good data to re-direct attention away from the data" attack
The attempt here is to shift focus away from the reliable third party national test score onto some other statistic that looks good to prove the progress you've made. You try to distract the listener to focus on those great things instead of measures of academic proficiency. Defense(s): (a) spit back a horrendous list of negative stats and say this isn't about stats, it is about measuring academic performance. (b) Get the focus back on telling you how Klein Table 2 is wrong.

The "discredit the NAEP test" attack
You argue here that the NAEP data doesn't truly measure proficiency because it is so broad. Defense: Of course, that's the point. And this falls very flat in this case since the Republicans earlier waved the July RAND report (which is soley based on NAEP data) as "proof" that there were gains. Suddenly changing your story on this is not credible. The NAEP test is the "gold standard." The attack falls very flat.

The "my bluffs should be assumed true until proven otherwise" attack
This is a very clever strategy. The purpose here is to shift the focus of attention away from the reliable data. Defense: What should be happening is that a reviewer should be looking at the reliable data and asking Bush, "if there are gains, how can you explain why they didn't show up in the NAEP data." Instead, Bush tries to turn it around his way, "I have an overwhelming amount of (unreliable) data that shows there is progress. Everyone knows it. Your new data is suspect because it disagrees with the facts everyone knows." That's very effective attack because you really want to believe he is telling you the truth. It's very easy to lose focus. You must concentrate on remembering that the trusted data is the NAEP data and the burden of proof is on him, not you, to prove the NAEP data is wrong and prove his data is reliable. If he doesn't do either, he has no basis for making the claim of educational improvement.

The "education track record isn't the issue" attack
They will claim Bush isn't a single issue candidate and what counts is his philosophy looking forward. Defense: Get back to the issue. We are just focused on whether his record is correct. Show me the data that invalidates Table 2.

The "they aren't the same students" attack
You can try to attack Klein's Tables by claiming they aren't the same students being sampled. Defense: In no case do you ever have the same students. In comparing 4th grade in 1992 to 4th grade in 1996, you have no student overlap whatsoever. And they never complained about that before (in Grissmer's study). At any rate, you can calculate the improvement the other direction and you get the same answer.

Counter-attacks

Point out that if education is so good in Texas, then...

how do you explain Klein (Table 2)?
why are all the counter-attacks coming from the Bush campaign.and not from teachers?
why are the ACT, SAT scores flat to declining when the % of students (and the racial mix) haven't changed?
how do you explain it when The New York Times reported that in February 1999, officials with the University of Texas system presented a report to a Texas House subcommittee complaining of "marked declines in the number of students who are prepared academically for higher education?"
why are the t eacher turnover stats so bad?
why did Bush veto a bill to study hunger in Texas that wouldn't have cost anything (he doesn't remember)?

For additional attacks, see Feedback I've gotten on the ad.

Steve Kirsch Political Home Page