Earlier studies of supervision in anesthesiology focused on how to evaluate the quality of individual anesthesiologist's clinical supervision of trainees. What is unknown is how to evaluate clinical supervision collectively, as provided by the department's faculty anesthesiologists. This information can be a metric that departments report annually or use to evaluate the effect of programs on the quality of clinical supervision over time.

This retrospective cohort study used all 48,788 evaluations of the 115 faculty anesthesiologists using the De Oliveira Filho supervision scale completed by 202 residents and fellows over nine academic years at one department.

The distributions of mean scores among raters had marked negative skewness and were inconsistent with normal distributions. Consequently, accurate confidence intervals were impracticably wide, and their interpretation suggested lack of validity. In contrast, the logits of the proportions of scores equaling the maximum possible value, calculated for each rater, followed distributions sufficiently close to normal for statistically reliable use in random effects modeling. Parameters and confidence intervals were estimated using the intercept only random effects models, and then inverses computed to convert results from the logit scale to proportions. That approach is analogous to random effect meta-analysis of proportional incidence (or prevalence). Departments that chose to use semi-annual or annual surveys of trainees regarding supervision quality, and report those raw counts, will have far lower estimates of supervision quality versus when calculated accurately using daily evaluations of individual anesthesiologists.

Random effects meta-analysis of percentage incidences of maximum scores is a suitable statistical approach to analyze the daily supervision scores of individual anesthesiologists to evaluate the overall quality of clinical supervision provided to the trainees by the department over a year.