Texas Emperor has No Clothes

IMPORTANT
If you find the material on this page useful, please hyperlink to it from a prominent link on your own page, and e-mail the URL on to all your friends ASAP and urge them to check this out!

Summary
Education reform is the cornerstone to Bush's campaign. Bush has touted the supposed massive gains in Texas as proof that he can reform education if he is elected President. It is the #1 issue of his Presidential campaign. There may be a miracle, but there is no reliable data that indicates it. there are no explanations that are consistent with educational improvement that can explain both sets of data. to show that that we know of so far. For example, there may be a test

Commentary by professional reviewers
RAND policy prohibits authors from comparing each others work. That is why I have written this report. Because it's important for the American people to know the truth about the test scores. Stephen Kline, author of the recently released RAND report has read this report and has indicated to me that it is accurate as far as he knows.

Why this is extremely significant to the Bush campaign
Education is Bush's only major accomplishment as Governor of Texas.

Background
The Texas Miracle refers to two things: (1) that Texas has a higher mean score on the national exams (NAEP) than the rest of the nation in each racial group and that (2) that Texas has had tremendous score gains on state exams (TAAS) and a narrowing of the achievement gap between whites and minorities.

Upon closer examination, claim (1) is meaningless because Texas has ranked highly for each ethnic group for long before Bush became governor. The best Bush can claim for gain (1) is that things have not gotten worse.

Claim (2) was discredited by a 2 year study at Boston College [Haney] released in August, 2000 which showed that TAAS scores could not be trusted. A RAND research report [Klein] subsequently confirmed the earlier Haney result that TAAS scores were could not be trusted. Klein went further and determined that since 1994, when Bush took office, academic performance has not improved relative to the rest of the country and that the the achievement gap actually widened, rather than narrowed as the untrustworthy state data had shown.

Two studies. RAND says both right.
Bush has cited Texas test scores as evidence he is an experienced leader and a "compassionate conservative." His supporters frequently call minorities' test-score gains "the Texas miracle."

"It's not a miracle," said Stephen P. Klein, the RAND testing expert who led the study. "We think these scores are misleading and biased because they're inflated. They're improvements in scores, but not in proficiency."

The Bush campaign condemned the report as mistaken and suggested that its release was politically inspired.

The study "contradicts every credible, nonpartisan scientific study that verifies the success in Texas," said Karen Hughes, the campaign's communications director. "Texas consistently ranks at the top in every category of student achievement--all students, all races, all income groups. The timing of this new opinion paper is highly suspect."

In July 2000, RAND issued a report which cited Texas is closer look at the two RAND studies shows that here was "no improvement" in Texas schools under George W. Bush. This is confirmed by all independent test data. There is no data that
Arguments presented here show clearly that the latest RAND study conclusions are consistent with two earlier peer-reviewed independent studies

The new RAND Corporation study confirms an earlier two year study by Walt Haney of Boston College which was published earlier this year in a highly respected peer reviewed journal. Both studies confirm that TAAS scores are not reliable and that there have been no meaningful educational gains in Texas other than artifically boosting scores of the few tests that it is possible to specifically practice for.

About the test data
TAAS is the Texas state data that Klein and Haney showed was untrustworthy. NAEP is national "gold standard" test data.

About the RAND Grissmer report
Focus area was how Texas performed relative to other states. Used "adjusted scores" to account for all sorts of factors. Covered period from 1990 to 1996.

About the RAND Klein report
The purpose of the Klein RAND report was not to discredit Bush. The purpose of the report was to investigate whether high-stakes testing (where the test results are used to determine things like funding, raises, etc.) produces valid test scores. The accepted wisdom was that it does not. The purpose of the report was to validate this. Texas was used as the state that was studied.

Further confirmations which would support the conclusions reached here
SAT scores, the Boston College Haney report

Hi. My name is Steve Kirsch. I’m the founder of Internet search engine Infoseek and a Silicon Valley philanthropist. Worth magazine recently named me one of the 100 Most Generous Americans. You can learn more about me here if you are interested. But I didn’t write this web page to talk about me.

I created this web page because you get to decide the outcome of this election and I’d like to share with you some brand new facts and reasoning that may help you decide which candidate to support. In 1960, John F. Kennedy was elected by a margin of only 1 vote per precinct. This election may be just as close. So your vote counts!

And up until now, it’s also been a confusing decision because on a lot of issues, both candidates are saying the same things. Both candidates are now in favor of a smaller government, a stronger military, tax cuts for the middle class, expanding choices in education, and so on.

And there are good arguments on both sides and there is so much conflicting data that it is impossible for any of us to thoroughly analyze. For example, some experts say Bush was responsible for a “Texas miracle” in education. Other experts say it’s hogwash. Who do you believe? How can anybody sort this out without months of study?

Our newspapers can't agree out which candidate is better
Even our newspapers, with all their resources, haven’t been able to sort this out for us as to who to believe. At the time I’m writing this, about equal numbers of newspapers have endorsed each candidate! The New York Times and Washington Post endorse Gore. The New York Post and the Washington Times support Bush. Talk about a confusing race. Even the names of the endorsers sound confusing!

One thing we know for sure: there can be no doubt whatsoever that education is the cornerstone of the Bush campaign. The home page of Bush’s website on October 28, 2000 had right at the top:

He has called himself a “Reformer With Results” in his press releases and TV ads. But is he really? The answer is important because it is the cornerstone of the Bush campaign that he has delivered on education, his #1 promise when he was elected Governor of Texas in 1994. If you decide he’s not telling you the truth, then you must hold him accountable for that and vote for Al Gore. He’d want you to do that. He’s said it over and over to you…you must hold people accountable… you must hold people accountable.

And it’s important too because if he is misleading you about the gains in Texas education, it means you can’t trust him to deliver on his top priority.

And it’s also important because Bush selected to focus on this issue specifically because he felt it best demonstrates how capable he’d be if he is elected President. He has waved it in front of us as proof of his effectiveness, as proof that he can bring Democrats and Republicans together to achieve real results. It is his shining accomplishment, the “jewel in his crown.” [Time, Feb 21, 2000]. Let there be no doubt. It is his "signature issue" [WSJ, Oct 27].

The best predictor of future performance is past performance. What really matters is what gets done, not what gets said. So if he can’t deliver on his top priority, what makes us think he can deliver on his other promises?

Sen. Joseph I. Lieberman said that "the RAND report totally undercuts what Governor Bush has said about his success in education."

Civil rights leader Jesse L. Jackson said, "Bush is not telling the truth when he says the gap between white and minority kids is shrinking."

The press (who is still as confused as you are about the whole thing) missed four key things about the RAND study that was just released. Four things that you can independently verify for yourself. Four key things that prove there is no conflict. Four key things that show that everything is consistent. Four things that show you that the Bush challenges to the data are hogwash.

How do I know these things? What makes me so smart? Because I called RAND and spoke to the author of the report for an hour about the data. And he said he got lots of calls but all people wanted to talk about was the timing of the report and not the data. I wanted to talk about the data and the Bush claims of inconsistency. And in that call I learned a few things that resolve all the conflicts that I'll share with you here. Then I researched them extensively. And it all fit together. And it was 100% consistent with common sense. One nice consistent result. No loose ends.

The implications of this are enormous. Because if you, like many Americans, thought the candidates were about the same before you read this, this is a devastating discovery that tilts the scales big time in Gore’s favor.

What the press missed
Here’s what happened and the four things the press missed.

When the RAND report entitled “What Do Test Scores in Texas Tell Us?”was announced on October 24, the story reported in the press looked damaging for Bush, but the Bush campaign was able to cast enough doubt on the report, citing its timing and its apparent conflict with other supposedly reliable national test (NAEP) data that showed remarkable progress in Texas, albeit in 4^th grade math scores only (the July RAND report).

We’re about to show you who you should believe. You’re going to see the actual data and we’re going to show you that there was no conflict. No conflict at all. For the first time, it will all make perfect sense to you as to what was going on.

The RAND report
The purpose of the Klein RAND report was not to discredit Bush. The purpose of the report was to investigate whether high-stakes testing (where the test results are used to determine things like funding, raises, etc.) produces valid test scores. The accepted wisdom was that it does not. The purpose of the report was to validate this. Texas was used as the state that was studied.

The only remaining loose end is “why did 4^th grade math scores go up on the NAEP tests?” as reported in both studies.

Both reports agreed that 4th grade math scores went up dramatically. And both conclusions in the table above are right. Yeah, you heard that right. They were referencing the same data (4th grade NAEP math scores) when they made those statements. Both conclusions in Table A are technically absolutely correct.

So I know you're confused right now. After all, either Texas improved or they didn't. They can't both be right!

OK, so now you see why the press wouldn't have been able to figure this out and just reported it as a conflict and let the story die: Two credible sources, from the same institution, both telling the truth, say two opposite things. Cancellation!. No story. Next...?

But hang on...we're almost there...I'm about to tell you how they can both be telling you the truth!

So here's the scoop. You have to know three things to resolve all the conflicts. But the press didn't look hard enough. They got snowed by the Bush campaign rhetoric. All of them got snowed.

Because standing alone, each report has everything you need to interpret them. It's only when you try to compare two reports that you need some additional data. Not surprising...in essence, comparing two reports is like a study on its own!

So based on all that, you don't need me anymore. You can take it from there. But let me help with some observations explaining our apparent conflict, just to make sure:

July report

Texas "showed dramatic improvement" over the period from 1990 to 1996. But it was a bit unfair of us not to account for the introduction of high-stakes testing in 1994. if we had confined our analysis to test data from 1994 onwards, we wouldn't have seen any improvement though we only have reading scores to verify that since the 2000 math scores aren't released yet.. We saw a spike only in a single test: 4th grade math in 1996 relative to the 1992 test. We don't have the right data to tell you whether it is a true gain in proficiency, or just a distorted score due to "test preparation" drilling started in 1994. We tried to fit a line to the two (in most cases) or three (for 8th grade math) data points we had to determine a slope. That slope was the improvement. Thus when we averaged in this score along with other scores, Texas always ranked at or near the top due to this one single score anomaly. That's the same thing that the Klein report noticed. The 4th grade math spike occurred most likely due to the overlap with the TAAS exam content, showing that drilling can sometimes affect the tests in the lower grades. Whether or not this "spike" is really learning or not wasn't the subject of this report. It was covered in Klein's report which showed it was not true learning, i.e., there was no evidence of a true proficiency gain. We only demonstrated that they can increase their 4th grade math scores via intensive drilling, not that they are better mathematicians. Had they been better mathematicians, Klein's report would have picked that up.

Oct report

Texas "showed no improvement." We did see a dramatic improvement in 4th grade math scores from 1994 to 1998, but the data showed the improvement wasn't sustained over time. Nor was there improvement on any other test. Due to the testing coaching with TAAS and the NAEP/TAAS similarity for this particular test, the significant improvement isn't surprising. Students were able to take exams better. So no real learning took place. No improvement.

So there you have it. The apparent "conflict" wasn't really a conflict at all once you wrapped a few more words around it.

If you are still skeptical, read on and we'll explain more tricks like this...like why the report came out right before the election. It's fun reading.

So now you have the entire overview explanation. Let’s dive into more details. The more you dive, the more you believe. Or you can skim over this part and move on to the next section. Your choice.

Four things the press forgot to tell you
First, let’s diffuse the timing issue raised by the Bush campaign (even though you can’t really attack a scientific paper by attacking its timing). What the press didn’t tell you is that the report was supposed to come out much earlier. It was delayed because RAND management, not the authors, wanted to make absolutely sure that the report was without error. So it went through 5 peer reviews before it was released, far more than the normal review process. The CEO of RAND stands 100% behind the accuracy of the data. And RAND is scrupulously non-partisan. And very well respected. So well respected that the Republicans themselves have touted the July RAND study. And the CEO of RAND says this new report is just as reliable.

Secondly, what the press didn’t tell you is how to sort out which report you should believe—the new October report that concluded there were no real gains in proficiency or the July RAND report that concluded that Texas showed dramatic gains. When you talk to RAND, they say both reports are correct. Huh? If you hadn't read all the stuff above, you should be confused. I was when I heard this! So was all the press.

Whoa! Talk about misleading the public! The new RAND report did much more than raise some questions! It absolutely, scientifically, and beyond any doubt established that the “Texas Miracle” is a "Myth." There can be no question whatsoever about that. It absolutely pulled the rug right out from under old Dubya. Nobody has produced any data to the contrary. Nor are they likely to. There is no missing data.

In fact, to give Bush some credit, he wasn't lying when he said that phrase associated with him above. He was stating the facts. It's just that he accidentally forgot to tell everyone that the Texas state scores were so unreliable that the gains that were reported were meaningless.

Thirdly, what the press also completely failed to tell you is that the new RAND report also confirmed an earlier paper done at the Center for the Study of Testing, Evaluation and Educational Policy at Boston College describing the results of a 2 year study that was published in a widely respected peer-reviewed journal in the educational area. They both reached exactly the same conclusion using different methodologies, namely that the Texas state tests (TAAS) are unreliable and you must use the national tests (NAEP). And they both found, using different methodologies, that the national test data shows that he’s has not delivered any meaningful progress in education achievement in Texas in the 6 years he’s been governor! And furthermore, the gap between white and minorities has actually widened, contrary to Bush’s claims! And you’re going to see the proof for yourself in crystal clarity if you continue to read on…

Fourthly, and most importantly, what the press also completely failed to tell you is that neither of the two papers which confirmed that lack of progress in Texas have ever been challenged in any scientifically valid way. If you want to refute a scientific study like that these (the RAND study was done by 4 experts with a combined 50 years of experience in the field), you must show (in a way that passes a peer-review test) at least one of four things: (1) an error in the assumptions, (2) an error in the data, (3) an error in the methodology, or (4) an error in interpreting what the results mean. And nobody has challenged any of these factors. The national test data used was complete and accurate. There is no dispute. Nobody has brought forward any new data to dispute the consistent conclusions of these two papers. They just claim there is a conflict with the earlier RAND study that claimed significant improvement on national test data. And the press just reports it as a conflict and leaves it as that. They drop it. So you’re confused.

You can verify is that there is no credible scientifically valid argument to dispute either study. You know that because if there was, you can bet that the opposition would make it widely available to the press. It would make it right to Bush’s home page, right at the very top. You wouldn't be able to miss it. We’ve seen nothing other than just a claim of a conflict with other data and a claim that there was more data that wasn’t used (which doesn’t matter because the report showed that all the state data was inaccurate, so what good is more inaccurate data).

Which expert should you believe?
Well, for the first time ever, we’re going to resolve that conflict right now so you can see it for yourself. And you’re going to see that there is no conflict at all. It’s all been a mirage. A “Texas tall tale” as they say.

But first a word about quick word about statistics. It’s quite possible for one expert to say “test scores improved” while another expert says “there was no improvement” while looking at the same data. And they can both be right! Now virtually all of us would think that’s impossible—one of them has to be wrong. And we’d be wrong. Let’s look at our first piece of test data and I’ll show you how both can be right and how to tell who is telling you the real story.

The tables basically look at the same group of students as they move from 4^th grade to 8^th grade. Table 1 is Math. Table 2 is reading. Guess what? Both tables show dramatic improvement in Texas for all race groups in both Math and Reading. Looks like a total victory for Bush, right? A 50 point gain!

Not at all! Because the same table also shows that the US as a whole increased by exactly the same amount. In fact, during Bush’s first 4 years, Table 2 shows that Texas scores went up less than the US as whole. In other words, Table 2 shows that for all race groups, Texas actually lost ground slightly in Bush’s first four years relative to the rest of the country.

So now you’ve just seen how one expert can tell you that scores in Texas went up significantly (50 points) over 4 years, and the other expert can tell you that “there was no improvement” and that they’d both be telling you the truth! Obviously, from this chart, it’s clear that the students in Texas are no more special than students anywhere else. The first expert would be misleading you, even though, strictly speaking, he’s telling the truth. The press would just say the two experts disagreed. You’d say, it’s very clear: no “Texas Miracle!”

One other point. You’ll notice from the two Tables that the scores in Texas on NAEP are above the US average (right half of the table) especially for Blacks and Latinos. A victory for Bush? Nope! It's been that way for a long time (for at least a decade). This was not pointed out in the paper. I learned this from one of my calls with the RAND author. Again, the press had absolutely no way to know that significant fact either. That’s why they discounted the study. They had too much data that appeared to be in conflict.

Want to look at some more data? I’ve collected a guide to Interpreting the Texas test data so you can read all the other scientific, peer-reviewed studies, read how the Bush campaign has attacked them, and read why the attacks fall flat.

You don’t need to be a rocket scientist to figure out what to believe here. In short, if Texas’ focus on testing using state tests (TAAS) really produced superior learning, it would have shown up on the national test scores. It didn’t. The only thing Bush gained in Texas was better test-takers, not students that are better educated.

We don’t have to look at test scores at all to figure out whether education in Texas has improved. Consider these other well-established facts from the two reports, a recently released video featuring Texas educators (or read the transcript), and from other news reports:

Now ask yourself this: Are these the type of educational results you’d like to see in your community? Does this sound like an infrastructure where a miracle could occur? So we really didn’t need to look at the test data at all to figure out who was telling the truth!

To be perfectly fair, however, in the course of my research, I did discover two “Texas Miracles” in education that I believe that Bush can take the credit for:

Leading educators overwhelming support Gore

You don’t have to believe me about any of this. Ask around. Ask the educators you trust which candidate they are voting for. We’ve talked to over 75 educators we know and none of them were Bush supporters. The board of the 2.5 million National Education Association unanimously voted to endorse Gore along with 89% (voting via secret ballot) of the 9,000 representative members at their annual meeting. And remember, education was Bush’s top accomplishment, the “jewel in his crown.” [Time, Feb 21, 2000]

Bush talks about “personal responsibility,” “high standards,” holding people accountable for their actions, and “in government that is responsible to the people.” I agree. I’m sure Al Gore does too. But is Bush willing to be judged by his own rhetoric?

Will you hold Bush accountable for not delivering anything on his top priority and for continuing to deceive you about the gains in Texas in spite of overwhelming scientific evidence to the contrary? .

Now that you have the facts, the only question left is: Do you trust him? Will you hold him accountable?

'Texas Miracle' Proves to be a 'Mirage'

Leading educators overwhelming support Gore