Yet Another Source Distressed By How the NYT’s Presented Its Data In A Story About the Dartmouth Research — Part 2

Yesterday, I commented on a New York Times story that appeared Wednesday, June 2, attacking the Dartmouth Research.  The work that Dartmouth has done over the past two decades suggests that hospitals in some parts of the country are over-treating patients. Overtreatment means that patients who didn’t need to be in the hospital in the first place are exposed to the side effects of treatment as well as gruesome hospital- acquired infections, medication mix-ups and a host of other medical errors. Thus unnecessary care puts patients at risk while helping to drive health care bills heavenward— and suggests that we could rein in Medicare spending by squeezing some of that hazardous waste out of the system.  But according to the Times: “Data [from Dartmouth] Used to Justify Health Savings Effort is Sometimes Shaky.”

In Part 1 of this post I discussed what two of the Times’ sources told me about how the Times’ reporters misrepresented what they said. Both Harvard economist David Cutler and Yale’s Dr. Harlan M. Krumholz complained that the story made it seem that they are critics of the research, when in fact they agree with Dartmouth on the basic message of the data, and see the work as, in Krumholz’ words “pivotal to moving us forward  . . . we all agree that there is lots of waste and it is unevenly distributed across the country.”

A third source in Washington D.C. who talked to the Times reporters confided that they seemed to have a clear agenda: “to take down Dartmouth.” 

Today, I received evidence from yet another unhappy source—the Wisconsin Collaborative for HealthCare Quality, a voluntary consortium of organizations working to improve the quality and cost-effectiveness of healthcare in Wisconsin. Chris Queram, the Collaborative’s president, and Jack Bowhan, who guides the development of value metrics for the group, report that they tried to caution New York Times reporter Gardiner Harris that he was misusing their data,   “comparing apples to grapefruits,” and “jumping to a conclusions that  you just can’t make.”  Harris ignored their warnings.

As proof, they produced a series of e-mails that they sent to Harris, and with their permission, I’m quoting from those messages. But first, an excerpt from the Times’ story talking about the Collaborative’s data.

“Last June, as Mr. Obama campaigned for his health care overhaul, he visited Green Bay, Wis., praising the city for getting “more quality out of fewer health care dollars than many other communities. Last June, as Mr. Obama campaigned for his health care overhaul, he visited Green Bay, Wis., praising the city for getting “more quality out of fewer health care dollars than many

“Two of Green Bay’s hospitals, Bellin and St. Mary’s Hospital Medical Center, rank fourth and 11th within Wisconsin on the Dartmouth list.

“But again, Dartmouth ranks hospitals only by costs and number of treatments and procedures. A different picture emerges from work done by the Wisconsin Collaborative for Healthcare Quality, a voluntary group of health care organizations that uses both price and quality of care measures. In an analysis of heart attack care, for example, it ranks Bellin second, and St. Mary’s 15th, among the 22 hospitals in the state.

“And a Medicare ranking based on its own data that shows how many people die after treatment for certain conditions — statistics that exclude costs entirely — puts Bellin fifth, but drops St. Mary’s to second-to-last: 67th of the 68 hospitals statewide that were measured by both Dartmouth and Medicare.

“Do the Green Bay hospitals favored by Dartmouth really offer better care? Maybe not.”

The e-mail Trail

Here is the statement from the Collaborative that I received today:

The Wisconsin Collaborative for Healthcare Quality was contacted by Gardiner Harris (GH) on March 30th seeking information about the comparison of Dartmouth Atlas rankings versus WCHQ quality rankings. That is, “- do the same systems that show positive performance on Dartmouth data show the same performance on your data? Put another way; are these systems as good as Dartmouth says they are using other measures?”

There are two series of email exchanges with GH [Gardiner Harris] ; one set on March 30-31st and again on April 19-20th. Throughout the emails, GH was cautioned not to use WCHQ’s data and methodology for comparison to ranking results generated by the Dartmouth Atlas. Examples of those cautions from the Collaborative’s Chris Queram (CQ)  and Jack Bowhan (JB) include:

CQ, 3/31/10, 10:47AM – “our data relate to physician groups, Dartmouth's relates to hospitals. And, the conditions being measured are different in many cases. So, the comparisons are very limited and should not be used to cast aspersions on the Dartmouth data.”

JB, 3/31/10, 1:10 PM –”There really is no way to reasonably compare the WCHQ metrics against Dartmouth and its process. “

CQ, 3/31/10, 1:53 PM – “I think you are raising an important issue, but want to be sure not to cast the [Dartmouth] Atlas in an unfair light.”

JB, 3/31/10, 5:33 PM – “I think you are jumping to a conclusion you cannot make. Here is why – I don’t think you can say Dartmouth is, or is not, a good proxy when you are trying to compare apples to grapefruits. We need an apples-to-apples comparison…”

CQ, 3/31/10, 6:59 PM – “It does make me nervous that decades worth of research by the team at Dartmouth might be impugned by the differences show in our hospital quadrants. Our methods while sufficient to enable our members to feel comfortable reporting this data and using it to guide internal improvement efforts — has not been the subject of or withstood rigorous scientific evaluation to confirm the association / correlation between the data reported on the two axis.”

12 thoughts on “Yet Another Source Distressed By How the NYT’s Presented Its Data In A Story About the Dartmouth Research — Part 2

  1. Thanks Maggie-
    Dr. George Lundberg-former editor of JAMA-very much agreed with me that local media is often in unholy economic alliance with the bloated, redundant, unnecessarily high-cost-high-tech hospital industry in cities whose economies are dominated by the medical care sector.
    It is sad to see the status quo in both media and medicine in these cities attempt to save each other from inevitable and necessary change.
    Gardiner Harris is a pathetic example of that phenomenum in NYC but I see this in Philly also. I suspect examples could be found throughout the nation.
    Dr. Rick Lippin

  2. Dr. Rick,
    it makes sense that there is a symbiotic relationships between newspapers and hospitals in many cities.
    Sadly, newspapers are under so much financial pressure these days that they are going to be even more reluctant to take a risk in terms of losing ad revenues.
    As the same time, as we begin to implement health care reform, we really need newspapers to help the public understand how hospitals that over-treat are hurting patients–while also boosting the nation’s health care bill.

  3. In the mid-90s I was editor of a small, weekly paper that served a small, exurban town of about 10,000 people when I hired a reporter whose only experience was in small-market TV. She was good and had enthusiasm, no question, but had picked up some bad habits.
    In the first staff meeting she attended, she politely asked if we covered health stories? Not particularly, because we had no hospital in town, and were focused on local issues. If it wasn’t local, we had other uses for the resources. But I’ll never forget her rationale that we should consider adding some health stories. “They’re easy to do. You just call the hospital PR department and they practically write them for you.”
    But it’s one thing for a small weekly staffed by reporters in their first or second job to get sloppy or lose sight of the mission.
    It’s quite another for the same thing to happen to the country’s paper of record.

  4. How timely, just as a number of high cost New York hospitals were reported to receive unexpectedly high Medicare quality payments.

  5. P. Samuel, Rick and Ron
    P. Samuel– Yes, bloggers can correct misinformation in the mainstream media.
    But this isn’t to say that bloggers don’t make mistakes. The mainstream media could correct our errors as well.
    Rick– I totally agree. This shouldn’t happen at our paper of record. Everyone makes mistakes but this was more than one mistake . .
    Ron– Medicare payments to NY’s teaching hospitals have always been high. I’m told that Moynihan cut a sweet deal for NYC years ago.

  6. Rick – I have always suspected that many health stories get written straight off the copy a reporter has received from a hospital, drug or device manufacturer. They are unfailingly un-analytic in presenting whatever finding is being reported.
    I am not surprised to hear your story as I have long suspected that reporters in the mainstream media parrot whatever they receive.
    It’s my suspicion that good science writers find more lucrative employment elsewhere and those left at newspapers just don’t have the background or time to investigate the issues.

  7. Ginger G–
    You are right about some reporters, but not all of them.
    There truly are many reporters out there who actually dig to find the facts, and throw press releases in their wastebaskets.
    But I’m afraid that more and more of them are losing their jobs.

  8. Ginger G–
    You are right about some reporters, but not all of them.
    There truly are many reporters out there who actually dig to find the facts, and throw press releases in their wastebaskets.
    But I’m afraid that more and more of them are losing their jobs. They are too expensive. What they do takes time.

  9. Ed–
    Yes a good number of letters on that NYT thread were skeptical abougt over-testing in a very healthy way.
    But it is going ot take years to get the message out to the majority of the public. We just have to keep repeating it.

  10. Maggie,
    I thought I’d point you to our rebuttal in The Times to the claims by the Dartmouth researchers (which you repeated) that our June 3 piece was inaccurate. (
    In addition to this, I’d like to make a couple of points about the kind of journalism we practice at The Times. First, we quote people in some stories even when those quotes make them uncomfortable. For instance, we used the data from the Wisconsin Collaborative for Healthcare Quality even though the collaborative was uncomfortable about that. Clearly, hospitals in Wisconsin would benefit enormously if the Dartmouth work was used — as Rep. Ron Kind of Wisconsin and others have advocated — to guide Medicare payment decisions. That does not mean our use of the data was inappropriate. Did we make an “apples to grapefruit” comparison, as the collaborative claimed and you have criticized us for? Absolutely. That was the point. We showed what would happen to the ranking of hospitals in Wisconsin if quality measures were included. Dartmouth’s hospital rankings only use cost data. So we cited the Collaborative’s numbers that combine both price and quality. And then we made another “apples and grapefruit” comparison and cited CMS’s quality rankings, with no cost numbers. All three rankings offered wildly different outcomes.
    The point here was to show that Dartmouth’s rankings should not be seen as definitive. Put some qualitative measures in there, and the rankings change dramatically.
    It is interesting that you did not quote my own responses to the collaborative, in which I explained all this. Why didn’t you?
    You have a habit of claiming that we misquote people or quote them out of context. After my Feb. 18 piece on Dartmouth, you quoted Dr. Elliott Fisher as saying that I had misquoted him. But in that story, Dr. Fisher wrote his own quote via email. I still have the emails. How can you misquote someone when they wrote their own quote themselves?
    Similarly with Dr. David Cutler and Dr. Harlan Krumholz, we clearly did not misquote these guys. What was interesting about Cutler is that, when he sent you his letter, he still was unaware that the Atlas was not adjusted for price. Did we include everything from the interviews we had of them? No, and we never do. Did we take their quotes “out of context?” Not at all. We pointed out in the story that Dr. Cutler supported the Dartmouth research but said he does not support the use of unadjusted data. He still doesn’t. He simply was unaware even at the time he sent you his letter that the Atlas data are unadjusted.
    Finally, we point out in our most recent rebuttal that Dr. Fisher and his colleague, Jon Skinner, substantially mischaracterized the results of their own 2003 Annals piece. How is this possible? How can academics fail to understand or accurately describe their own work?
    –Gardiner Harris