A few days ago, some surgeons on Twitter discussed the role of the American Board of Surgery In-Training Examination, a test which is given every year in January.
The test was designed to assess residents knowledge and give them an idea of where their studying should be focused. However, many general surgery program directors (PDs) use the test results in other ways. Some impose remediation programs on residents with low scores and even base resident promotion or retention on them. Some even demand that all residents in their programs maintain scores above the 50th percentile.
The Residency Review Committee (RRC) for Surgery frowns upon these practices and states in its program requirements (Section V.A.2.e) that residents knowledge should be monitored "by use of a formal exam such as the American Board of Surgery In Training Examination (ABSITE) or other cognitive exams. Test results should not be the sole criterion of resident knowledge, and should not be used as the sole criterion for promotion to a subsequent PG [postgraduate year] level."
The problem for program directors is that the RRC also mandates (Section V.C.2.c) that "as one measure of evaluating program effectiveness" 65% of a residency programs graduates must pass both the American Board of Surgerys Qualifying Examination (written) and Certifying Examination (oral) on their first attempts. I have said before that the "65% on the first attempt rule" does not seem evidence-based.
Does performance on the ABSITE predict performance on the boards examinations?
A recent paper by the staff of the American Board of Surgery states, "Although the ABSITE does not have a direct effect on board certification, it has been shown to be predictive of ABS Qualifying Examination performance." The authors cited three references.
The best is a 2010 Archives of Surgery paper that analyzed 607 graduates of 17 programs from the western US. It found "On multivariable analysis, scoring below the 35th percentile on the ABSITE at any time during residency was associated with an increased risk of failing both examinations (odds ratio, 0.23 [95% confidence interval, 0.08-0.68] for the qualifying examination and 0.35 [0.20-0.61] for the certifying examination)."
Note: The boards paper found that ABSITE scores do not correlate with passing the certifying (oral) exam. This makes sense because the oral exam is more about judgment and situational thinking than recall of facts.
A systematic review of 26 papers, published online in the Journal of Surgical Education, showed that "Structured reading programs and setting clear expectations with mandatory remedial programs were consistently effective in improving ABSITE performance, whereas the effect of didactic teaching conferences and problem-based learning groups was mixed."
However, its not so simple. Structured reading and mandatory remedial programs will only work if the deficient resident is committed to succeeding, an attitude that is not always present. [See "grit."]
A brilliant post of mine from two years ago pointed out that program size has a lot to do with being able to maintain a better than 65% board passage rate on the first attempt. Using a simple statistical fact, I explained why smaller programs may be much more likely to fail to meet the 65% standard.
A resident who, despite attempts at remediation, has single digit ABSITE percentile scores over two or three years creates a serious dilemma for the director of a small program. Should the PD keep the resident in the program which can ill afford to graduate a resident with a high risk of failure on the written board examination or dismiss the resident and try to find a competent replacement from a very small pool of available candidates?
Having been there, I can tell you its not an easy decision.
Read More..
The test was designed to assess residents knowledge and give them an idea of where their studying should be focused. However, many general surgery program directors (PDs) use the test results in other ways. Some impose remediation programs on residents with low scores and even base resident promotion or retention on them. Some even demand that all residents in their programs maintain scores above the 50th percentile.
The Residency Review Committee (RRC) for Surgery frowns upon these practices and states in its program requirements (Section V.A.2.e) that residents knowledge should be monitored "by use of a formal exam such as the American Board of Surgery In Training Examination (ABSITE) or other cognitive exams. Test results should not be the sole criterion of resident knowledge, and should not be used as the sole criterion for promotion to a subsequent PG [postgraduate year] level."
The problem for program directors is that the RRC also mandates (Section V.C.2.c) that "as one measure of evaluating program effectiveness" 65% of a residency programs graduates must pass both the American Board of Surgerys Qualifying Examination (written) and Certifying Examination (oral) on their first attempts. I have said before that the "65% on the first attempt rule" does not seem evidence-based.
Does performance on the ABSITE predict performance on the boards examinations?
A recent paper by the staff of the American Board of Surgery states, "Although the ABSITE does not have a direct effect on board certification, it has been shown to be predictive of ABS Qualifying Examination performance." The authors cited three references.
The best is a 2010 Archives of Surgery paper that analyzed 607 graduates of 17 programs from the western US. It found "On multivariable analysis, scoring below the 35th percentile on the ABSITE at any time during residency was associated with an increased risk of failing both examinations (odds ratio, 0.23 [95% confidence interval, 0.08-0.68] for the qualifying examination and 0.35 [0.20-0.61] for the certifying examination)."
Note: The boards paper found that ABSITE scores do not correlate with passing the certifying (oral) exam. This makes sense because the oral exam is more about judgment and situational thinking than recall of facts.
A systematic review of 26 papers, published online in the Journal of Surgical Education, showed that "Structured reading programs and setting clear expectations with mandatory remedial programs were consistently effective in improving ABSITE performance, whereas the effect of didactic teaching conferences and problem-based learning groups was mixed."
However, its not so simple. Structured reading and mandatory remedial programs will only work if the deficient resident is committed to succeeding, an attitude that is not always present. [See "grit."]
A brilliant post of mine from two years ago pointed out that program size has a lot to do with being able to maintain a better than 65% board passage rate on the first attempt. Using a simple statistical fact, I explained why smaller programs may be much more likely to fail to meet the 65% standard.
A resident who, despite attempts at remediation, has single digit ABSITE percentile scores over two or three years creates a serious dilemma for the director of a small program. Should the PD keep the resident in the program which can ill afford to graduate a resident with a high risk of failure on the written board examination or dismiss the resident and try to find a competent replacement from a very small pool of available candidates?
Having been there, I can tell you its not an easy decision.
