There are several methods for assessing when a person becomes proficient in a given task. These methods are crucial in medicine, particularly in training students in certain technical procedures (surgery, imaging, etc...). In this study, we aim to compare two main techniques: the moving average and the cumulative summation test for learning curve (LC-CUSUM).

The moving average method calculates the failure rate of the last n attempts (size of block). If this rate reaches the given acceptable failure rate (p0), the trainee is considered competent. The LC-CUSUM method uses a cumulative score. For each attempt, a weight is added to the score, whose amplitude and sign depends on the success of the attempt, and on prespecified parameters (p0, and p1: the inacceptable rate of failure). The trainee is declared competent once the score reaches a predetermined limit H. The performances of both methods were evaluated based on their ability to detect when a person becomes competent (the Probability of Successful Detection: PSD) and the probability of mistakenly considering an incompetent trainee as competent person (Probability of False Alarm: PFA). Through simulations involving datasets of both competent and non-competent students, we calculated these PSD and PFA values to identify under which parameters both methods exhibited similar performance. Subsequently, we applied these methods to a simulated dataset of learning students.

For every pairs of p0 and p1 parameters, we evaluated the parameter of LC-CUSUM (H) and the moving average (size of bloc) to establish the best performances. To achieve similar performance, it appears that the moving average method requires a substantial size of bloc in order to have performances similar to the LC-CUSUM ones (example in Fig. 1). This minimal number of observations to detect when a trainee becomes competent implies potentially unnecessary attempts, longer training durations and less sensitivity to competency. However, these results depends on the p0 and p1 parameters chosen according to the setting: in some situation, the moving average method could be pertinent in competency evaluation.

Although less immediately intuitive, the LC-CUSUM method appears preferable due to its greater sensitivity to change while maintaining similar performances compared to the moving average in detecting when a trainee becomes competent in a given task. The choice between these two techniques should consider these results along with the practicality of using either method.