Why in hospital deaths are not a good quality measure

You may be tired of hearing about the Surgeon Scorecard—the surgeon rating system that was recently released by an organization called ProPublica. Like many others, I have pointed out some flaws in it. You can read my previous posts here and here.

I had decided to stop commenting about it because enough is enough, but a recent paper in the BMJ raises a question about one of the criteria ProPublica used to formulate its ratings.

ProPublica defined complications 1) as any patient readmission within 30 days and 2) "any patient deaths during the initial surgical stay."

The authors of the BMJ paper randomly selected 100 records of patients who died at each of 34 hospitals in the United Kingdom. The 3400 records were reviewed by experts to determine whether a death could have been avoided if the quality of care had been better.

The number of patient records in which a death was at least 50% likely to have been avoidable was 123 or 3.6%.

There was a very weak association between the number of preventable deaths and the overall number of deaths occurring at each hospital. By two measures of overall hospital deaths, the hospital standardized mortality ratio and the summary hospital level mortality indicator, the correlation coefficient between avoidable deaths and all deaths was 0.3, not statistically significant.

From the paper: "The absence of even a moderately strong association is a reflection of the small proportion of deaths (3.6%) judged likely to be avoidable and of the relatively small variation in avoidable death proportions between trusts [hospitals]. This confirms what others have demonstrated theoretically—that is, no matter how large the study the signal (avoidable deaths) to noise (all deaths) ratio means that detection of significant differences between trusts is unlikely."

The Surgeon Scorecard was derived from administrative data. No individual analysis of patient deaths was undertaken. According to a ProPublica article discussing some key questions about their methodology, "As for deaths, we took a conservative approach and only included those that occurred in the hospital within the initial stay."

Maybe that wasnt such a conservative approach after all.

And maybe we need to rethink that 2013 paper claiming that medical error caused up to 440,000 deaths per year.

Read More..

Big data is not big enough

Today ProPublica released its “Surgeon Scorecard” touting it as the best way to pick the right surgeon.

It took me less than a minute to discover some interesting omissions from the application.

For laparoscopic cholecystectomy, the only general surgery procedure listed, the app omits approximately one-third of the hospitals in my state including two where I have practiced.

It looks like the problem is that using Medicare fee-for-service data does not yield enough surgeons performing 20 or more cases in some categories such as laparoscopic cholecystectomy for the five years included in the database.

At one of the biggest hospitals in my state, apparently only one surgeon performed 20 laparoscopic cholecystectomies on fee-for-service Medicare patients in the five years studied; 23 other surgeons were listed as having performed fewer than 20 laparoscopic cholecystectomies on patients in the target population. I don’t see how patients who want to use that hospital for their gallbladder surgery will benefit from the Surgeon Scorecard.

In general, the complication rate for laparoscopic cholecystectomy is low, but I think I understand why ProPublica chose that procedure to review. They needed to select a procedure that was done frequently enough to yield a sufficient number of cases for analysis. Unfortunately, because of the limitations of the Medicare fee-for-service data and the low complication rate of the procedure, the Surgeon Scorecard is useless for anyone looking to compare general surgeons.

Similar problems with the scorecard may be in play for prostate surgery. Again, the procedure was chosen because of its high frequency, but in quickly looking through some searches in that area, I note that a number of urologists I know also did not perform 20 cases on fee-for-service Medicare patients.

Perhaps the next iteration of the scorecard will utilize a data set that contains enough patient and surgeon records to make a meaningful comparison.

Until then, general surgeons can relax. They will not have to explain away their complications but will simply have to explain why they aren’t listed in the Surgeon Scorecard.

Read More..