What can be done about letters of recommendation

Many surgical educators feel that letters of recommendation are not particularly helpful in evaluating applicants or predicting eventual resident performance.

Among the issues are lack of uniformity of content, excessive use of superlatives, reliability—if the writer is not known to the recipient—and more.

Even if the writers are well-known academic surgeons, the degree of their personal knowledge of the applicants is not always clear.

During an extensive Twitter discussion over the weekend, someone mentioned that in an attempt to deal with this problem, emergency medicine had developed a standardized letter of recommendation.

A recent paper from the EM Standardized Letter of Recommendation Task Force shows that there is still work to be done. From the abstract:

For the question on "global assessment," students were scored in the top 10% in 234 of 583 of applications (40.1%), and 485 of 583 (83.2%) of the applicants were ranked above the level of their peers. Similarly, >95% of all applicants were ranked in the top third compared to peers, for all but one section under "qualifications for emergency medicine."

Ive written before that deans letters are more like public relations press releases than accurate assessments of a students performance. You will rarely find negative comments in them. But another recent paper by a group of psychiatrists found that The presence of any negative comments in the deans letter yielded significant correlations with future problems. Further, those applicants with future major problems had significantly more negative comments in the deans letter than did those with future minor problems. Other factors such as USMLE scores, failed courses, letters of recommendation, and interviewer ratings and comments did not predict future problems.

These problems are not new. A 1983 New England Journal opinion piece about recommendation letters entitled "Fantasy Land" is remarkable for its validity even today. Here are a few choice quotes.

Its a land where everyone is "a pleasure to work with," has "excellent initiative," is "enthusiastic and conscientious," and possesses and "above-average fund of knowledge."

No one is ever poor, fair, or average; they are all "very good" or "excellent."

The author, Dr. Richard B. Friedman, said letters of recommendation were useless and advocated doing away with them.

A brief JAMA essay by Dr. Henry Schneiderman in 1988 called for more openness in describing students but acknowledged that negative comments were often "the kiss of death."

He proposed a new system of categorizing medical student performance. Here are just a few examples.



@AmirGharferi suggested this:

"Dr.G, do you feel comfortable writing me a strong letter?"
"No."
"Ok, Ill find someone who is."

That works if the student is aware enough to ask, and the faculty member is honest enough to say no. In my experience, even the most marginal of students can find someone—in addition to the dean, of course—to write a good letter.

I am no longer involved in the process of selecting residents. I have no suggestions.

What is your solution to this problem?

Read More..

Chance can turn a surgeon into a killer

Risk-adjusted 30- to 90-day outcome data for selected types of operations done by specific surgeons and hospitals are now being publicly posted online by Englands National Health Service.

According to the site, "Any hospital or consultant [attending surgeon in the UK] identified as an outlier will be investigated and action taken to improve data quality and/or patient care."

After cardiac surgery outcomes data were made public in New York, some interesting unexpected consequences were noted.

Surgeons and hospitals resorted to "gaming the system" by declining to operate on patients who were high-risk and tinkering with patient charts to make those they did operate on seem sicker. This can be done by scouring the charts for all co-morbidities and making sure none are overlooked when they are coded. An article from New York Magazine explains it in more detail.

Interpreting outcomes data can be tricky.

In a post three years ago about a report that nine Maryland hospitals had higher-than-average complication rates, I pointed out that whenever you have averages, some hospitals are going to be worse than average unless all hospitals perform exactly the same way or, like medical students, are all above average.

A much more sophisticated way of looking at this subject appeared in a fascinating 2010 BBC News piece by Michael Blastland, who is the Nate Silver of England [or maybe Nate Silver is the Michael Blastland of the US], called "Can chance make you a killer?"

Blastland set up a statistical chance calculator for a hypothetical set of 100 hospitals or 100 surgeons performing 100 operations each. The model assumes that every patient has the same chance of dying and that every surgeon is equally competent. The standard is that a mortality rate 60% worse than the norm set by the government for any hospital or surgeon is not acceptable.

You are assigned one hospital. Using a slider, you may choose an operative mortality rate anywhere from 1% to 15%. After you do this a number of times and recalculate for each mortality rate, you will notice that the number of unacceptably performing hospitals or surgeons changes randomly for each percent mortality and your hospital may appear in the underperforming group strictly by chance alone.

The whole concept is explained in more detail on the site. I encourage you to try it for yourself. The link is here.

So it may be difficult for the NHS to separate the true outliers from the unlucky surgeons who happened to fall outside the established norms.

What do you think about this?

Read More..