·Clinical Research· Current Issue· ·Achieve· ·Search Articles· ·Online Submission· ·About IJO· PMC

Citation: Vedana G, Cardoso FG, Marcon AS, Araújo LEK, Zanon M, Birriel DC, Watte G, Jun AS. Cumulative sum analysis score and phacoemulsification competency learning curve. Int J Ophthalmol 2017;10(7):1088-1093

Cumulative sum analysis score and phacoemulsification competency learning curve

Gustavo Vedana^1,2, Filipe G. Cardoso², Alexandre S. Marcon², Licio E. K. Araújo², Matheus Zanon², Daniella C. Birriel², Guilherme Watte³, Albert S. Jun¹

¹Wilmer Eye Institute, Johns Hopkins Medical Institutions, Baltimore 21287, USA

²Irmandade Santa Casa, Misericórdia, Porto Alegre 90020160, Brazil

³Federal University of Rio Grande do Sul, Porto Alegre 90035003, Brazil

Correspondence to: Matheus Zanon. Pavilhão Pereira Filho, Irmandade Santa Casa, Misericórdia, Porto Alegre 90020160, Brazil. mhgzanon@hotmail.com

Received: 2016-01-07 Accepted: 2017-05-11

Abstract

AIM: To use the cumulative sum analysis score (CUSUM) to construct objectively the learning curve of phacoemulsification competency.

METHODS: Three second-year residents and an experienced consultant were monitored for a series of 70 phacoemulsification cases each and had their series analysed by CUSUM regarding posterior capsule rupture (PCR) and best-corrected visual acuity. The acceptable rate for PCR was <5% (lower limit h) and the unacceptable rate was >10% (upper limit h). The acceptable rate for best-corrected visual acuity worse than 20/40 was <10% (lower limit h) and the unacceptable rate was >20% (upper limit h). The area between lower limit h and upper limit h is called the decision interval.

RESULTS: There was no statistically significant difference in the mean age, sex or cataract grades between groups. The first trainee achieved PCR CUSUM competency at his 22^nd case. His best-corrected visual acuity CUSUM was in the decision interval from his third case and stayed there until the end, never reaching competency. The second trainee achieved PCR CUSUM competency at his 39^th case. He could reach best-corrected visual acuity CUSUM competency at his 22^nd case. The third trainee achieved PCR CUSUM competency at his 41^st case. He reached best-corrected visual acuity CUSUM competency at his 14^th case.

CONCLUSION: The learning curve of competency in phacoemulsification is constructed by CUSUM and in average took 38 cases for each trainee to achieve it.

KEYWORDS: phacoemulsification learning curve; cumulative sum analysis; score; posterior capsule rupture; best corrected visual acuity; cataract surgery

DOI:10.18240/ijo.2017.07.11

INTRODUCTION

Phacoemulsification (phaco), despite being one of the most performed surgeries in the world is one of the hardest to master and the learning process is well discussed elsewhere^[1-6]. Multiple studies have addressed the best way to learn this procedure^[7-9] and some describe a learning curve based on complications^[4-5,10-11]. Few try to answer the question that arises from residency training: how many surgeries are necessary for a resident to become proficient in phaco?

Salowi et al^[12] showed that the cumulative sum analysis score (CUSUM) can be an objective way to measure competency in cataract surgery through a graphical representation, but they had a small number of surgeries and trainees at different stages in order to construct a learning curve. Thus, use of this objective score to construct the learning curve and give a competency achievement score for different trainees with the same background experience in cataract surgery since their first phaco case is lacking.

Our goal in this study is to prospectively analyse this method of graphical representation and objectively construct the learning curve of phaco competency for three residents showing how many surgeries they have to do before achieving proficiency.

SUBJECTS AND METHODS

Study Design Three second-year residents with 30 or more cases of extracapsular cataract extraction (ECCE) and an experienced consultant were monitored for a series of 70 phaco cases each. The two outcomes analysed were posterior capsule rupture (PCR) and best corrected visual acuity (BCVA) worse than 20/40 within 1-month post-operative follow-up.

Two-hundred-and-eighty sequenced surgeries were done from 2012 to 2013. The patients were from the Ophthalmology Service of Irmandade Santa Casa de Misericórdia de Porto Alegre. The study was reviewed and approved by the Institutional Review Board of that institution, and informed consent was given by all patients who underwent a complete ophthalmologic examination including manifest refraction, slit-lamp evaluation, tonometry, and fundoscopy before surgery. Cataract was the only reason for impaired vision and if any other causes were found the patient was excluded from the analyses.

Statistical Procedure Data were presented as mean±SD or frequency and percentage. The normality of distribution assumption was checked by Shapiro-Wilk test. We performed associations between variables with the Pearson’s χ² test with standardized residuals. For comparing multiple group comparisons, we used ANOVA test with Bonferroni correction. A P value of less than 0.05 was considered statistically significant. Data were analysed using Stata software, version 11 (StataCorp, College Station, TX, USA).

Cumulative Sum Analysis The CUSUM scores were used as a quantitative measurement. CUSUM scores of consecutives performances of an individual are displayed with the x-axis representing the consecutive series of procedures and the y-axis representing the CUSUM score^[13]. It was applied in ophthalmology to assess the learning curve in phacoemulsification (Salowi et al^[12]) and has been used for clinical performances in surgical procedures to define competency^[14-16].

Mathematically, the CUSUM score is determined using the formula: CUSUM C_n=max (0, C_{n-1 +}X_n – k).

Where C=case; n=n^th phaco case in the series; X_n=outcome measure for the n^th phaco case considering binary outcomes X_n=0 (success) and X_n=1 (failure). For continuous measurement, X_nis standardized to have a zero mean and unit standard deviation (SD). k=reference value defined for a standard performance in terms of acceptable and unacceptable failure rates, which was calculated on basis of π₁ and π₂ using methods described by Hawkins and Olwell^[17] and cataract outcomes of PCR and BCVA from a 12 798 case series reported by the National Cataract Surgery Registry^[18].

The acceptable rate for PCR was less than 5% and the unacceptable rate was more than 10% of surgeries performed. For a visual outcome worse than 20/40, the acceptable rate was less than 10% and the unacceptable rate was more than 20% of patients operated (Table 1).

Table 1 Cumulative sum charting design for monitoring cataract surgery performance

Parameters

Acceptable rate of performance, p1

Unacceptable rate of performance, p2

Decision interval

(upper limit h)

Decision interval

(lower limit h)

IC-ARL

OC-ARL

PCR

0.072

1.0

0.5

BCVA

10%

0.145

2.0

1.0

PCR: Posterior capsule rupture; BCVA: Best corrected visual acuity; IC-ARL: In control-average run length; OC-ARL: Out of control-average run length. h and the ARLs determine the degree of sensitivity of the chart and are selected by the user. Reference value k, h, IC-ARL and OC-ARL were described by Hawkins & Olwell^[17].

At the start, CUSUM C0=0. At the n^th procedure, X_n is the outcome measure for the n^th procedure. Performance with an acceptable standard has a negative score, and the chart is either flat or slopes downwards. Performance with an unacceptable standard has a positive score, and the CUSUM chart slopes upwards. Because the equation has a maximum value between zero and the score of sum, the graph will not slope downward below zero in our series (examples 1 and 5 of Figures 1 and 2, respectively). When consecutive phacos performed by the same surgeon are of an unacceptable standard, the graph will continue to slope upward until it crosses the first line drawn across the graph, lower limit h (LLh), of an area called the decision interval (h) (Figures 1 and 2). When this occurs for the first time, the CUSUM chart is said to signal unsatisfactory performance, but expected, and continued observation is still necessary (in the learning curve)-examples 2 and 6 of Figures 1 and 2, respectively. If the decision interval (h) is crossed through its second line, upper limit h (ULh) (Figure 1), a deteriorating or substandard performance is shown and an implementation of corrective actions to prevent subsequent patients being harmed is advised (revise the learning method or technique). After crossing the ULh, the CUSUM chart is restarted. Restart should theoretically be at zero or the x-axis. However, to obtain a chart that looks like a learning curve, it restarts at the ULh, which now acts as the new x-axis (examples 3 and 7 of Figures 1 and 2, respectively). We considered achievement of competency when the CUSUM curve crosses the LLh from above and never crosses the ULh again, meaning that the failure rates of that surgeon are within an acceptable rate (examples 4 and 8 of Figures 1 and 2, respectively). The decision interval (h) is determined by specifying the in-control average run length and out-of-control average run length. In-control average run length is the average number of consecutive performance required for a CUSUM chart to cross a decision interval or signal during the period when the operator is performing at an acceptable level. This is akin to significance or false positive (type I error) in hypothesis testing. On the other hand, out-of-control average run length is the average number of procedures performed before the CUSUM chart signals, during the period when an individual is performing at an unacceptable level. It is a measure of sensitivity and is akin to power (type II error) or false negative error in hypothesis testing.

Gustavo Vedana1

Figure 1 Examples of CUSUM analysis curves and parameters for occurrence of posterior capsule rupture A: Acceptable standard only with negative scores; B: Unsatisfactory performance, but expected (in the learning curve); C: Deteriorating or substandard performance-implementation of corrective actions to prevent subsequent patients being harmed is advised (revise the learning method or technique); D: Achievement of competency.

Gustavo Vedana2

Figure 2 Examples of CUSUM analysis curves and parameters for best-corrected visual acuity A: Acceptable standard only with negative scores; B: Unsatisfactory performance, but expected (in the learning curve); C: Deteriorating or substandard performance-implementation of corrective actions to prevent subsequent patients being harmed is advised (revise the learning method or technique); D: Achievement of competency.

Procedure The procedures were done with Alcon INFINITI^®Vision system machine (Alcon Surgical, Fort Worth, Texas, USA) by the divide and conquer technique after peribulbar anaesthesia. The intraocular lens (IOL) was calculated with SRK/T formula and an Alcon MA60AC IOL (Alcon Surgical) was implanted. Two self-sealing incisions (2.75 mm and 1.1 mm), a 5.5 mm curvilinear capsulorhexis, and in the bag IOL implantation at the end were always attempted. Senior residents were the auxiliary surgeons through the whole procedure in the first 10 cases, except in the occurrence of PCR where they became primary surgeon until the end of the case. The need for continuous supervision through the whole procedure after the tenth case was determined if the CUSUM chart signalled unsatisfactory, but expected performance in the learning curve. If deteriorating or substandard performance was shown by the CUSUM chart the senior resident was responsible for identifying the steps the trainee was failing and do them on the next case while instructing the trainee. This corrective action was taken before the trainee could attempt the entire procedure again.

RESULTS

Table 2 shows demographic data and the Lens Opacities Classification System 3 (LOCS3) for cataracts among groups. There was no statistically significant difference in the mean age, sex or cataract grades between groups.

Table 2 Clinical characteristics of the study baseline

Characteristics	Female	Age (a)	Cataract type^b:
Characteristics	Female	Age (a)	Nuclear ^[4-6]	Cortical^[4-6]	Subcapsular ^[4-6]
Total	187 (67)	71±9	93 (33)	13 (5)	59 (21)
Consultant	43 (61)	71±10	23 (33)	3 (4)	15 (21)
Trainee 1	44 (63)	69±9	20 (29)	4 (6)	17 (24)
Trainee 2	49 (70)	72±10	28 (40)	2 (3)	14 (20)
Trainee 3	51 (73)	72±8	22 (31)	4 (6)	13 (19)

Sample size: 280 patients; Data are presented as No. (%) or mean±SD. ^bCataract type, as defined by LOCS3^[19]. Comparison by Pearson’s χ² test (standardized residuals) or ANOVA test (Bonferroni correction) between groups: P>0.05 for all comparisons.

Table 3 lists the other complications done by all the residents. The procedures performed by the main surgeon presented no complications.

Table 3 Summary of other complications

Complications	n (%)
Conversion to ECCE due to incomplete capsulorhexis	6 (2.1)
Dropped nucleus and vitreous loss	1 (0.4)
IOL displacement to anterior chamber	1 (0.4)
Sutured incision after incision burn	1 (0.4)
Corneal decompensation	1 (0.4)
IOL reposition	2 (0.7)
Aphakia	1 (0.4)
Total	13 (4.6)

Data are presented as No. (%). ECCE: Extracapsular cataract extraction; IOL: Intraocular lens.

Figures 3 and 4 show the PCR and BCVA CUSUM charts, respectively, for the consultant and the learning curve of the three trainees (residents) since their first phaco case. The consultant had a flat line underneath the LLh, without PCR cases in the 70^th phaco series and all patients with BCVA better than 20/40. The first trainee had PCR in his fourth case and crossed the LLh, demanding continuous observation (in learning curve). In his 11^thcase, another PCR happened and the ULh was crossed. Implementation of corrective actions was necessary. From his 16-21^st surgeries he was in the decision interval (h) and continuous observation was still necessary. Considering the PCR CUSUM alone competency was achieved in his 22^nd case when LLh was crossed from above though another PCR occurred in his 67^th case. On the other hand, his BCVA CUSUM was in decision interval (h) since his third case and stayed there until the end, never reaching competency. Trainee 1 despite being at an acceptable level of PCR at the end of his training program did not achieve competency in BCVA. The most common cause for that was persistent corneal oedema at 1-month post-operative and high astigmatism following phaco conversion to ECCE due complication at capsulorhexis performance which was not included in this CUSUM analysis.

Gustavo Vedana3

Figure 3 The CUSUM scores of consecutives performances of the consultant (A) and three trainees (residents) (B, C, D) since their first phaco case showing posterior capsule rupture and their learning curve The x-axis represents the consecutive series of procedures and the y-axis representing the CUSUM score. LLh: Lower limit h of 5%; ULh: Upper limit h of 10%.

Gustavo Vedana4

Figure 4 The CUSUM scores of consecutives performances of the consultant (A) and three trainees (B, C, D) since their first phaco case showing BCVA outcomes and their learning curve The x-axis represents the consecutive series of procedures and the y-axis representing the CUSUM score. LLh: Lower limit h of 10% for BCVA worse than 20/40. ULh: Upper limit h of 20% for BCVA worse than 20/40.

The second trainee (Figure 3) had PCR in his 11^th case and crossed the LLh, demanding continuous observation afterwards (in learning curve). In his 17^thcase, another PCR happened and the ULh was crossed. Implementation of corrective action was necessary. Despite that, he had another PCR in his 23^rd case and another corrective action was done. From his 33-38^th surgeries he was in the decision interval (h) and continuous observation was still necessary. Considering the PCR CUSUM alone, competency was achieved in his 39^th case when LLh was crossed from above though another PCR occurred in his 55^th case. On the other hand, his BCVA CUSUM (Figure 4) was in decision interval (h) from his 6-21^st case, when he could reach the BCVA CUSUM competency at his 22^nd case.

The third trainee had PCR in his fourth case and crossed the LLh, demanding continuous observation afterwards (in learning curve). In his ninth case, another PCR happened and the ULh was crossed. Implementation of corrective action was necessary. Despite that, he had another PCR in his 14^th and 20^th cases and additional corrective action occurred. From his 35-40^th surgeries he was in the decision interval (h) and continuous observation was still necessary.

Considering the PCR CUSUM alone competency was achieved in his 41^st case. On the other hand, his BCVA CUSUM was in decision interval (h) from his 8-13^th case, when he could reach competency at his 14^th case.

DISCUSSION

As shown by Salowi et al^[12], CUSUM evaluation is faster in detecting trends of unacceptable performance. It is objective and dynamic, tracks performance over time with benchmarks and easily displays a graphic for the phacoemulsification learning curve. As in other precision microsurgeries with desirable low failure rates, phaco has a learning process that pursues a nearly perfect outcome at the expense of repetition^{[15-16,19-23]}. When this point is reached in our training program, the resident has the ability to perform this procedure with competency without supervision, achieving a constant rate of acceptable failure. This is the goal of ophthalmology residency training programs in phaco, but it would better if they also could promptly identify unacceptable patterns of the learning curve to take corrective measures earlier, enhancing the learning process with fewer surgeries necessary to become proficient. This is a potential improvement in residency training programs that CUSUM analysis could help with, together with its objective measurement of competency for compliance and certification processes.

In this study, we could demonstrate for the first time the amount of surgeries necessary for each trainee achieve competency, since their first phaco case considering two major markers for phacoemulsification success^[2,5,10-11]. In average, it took 38 cases to achieve an acceptable 10% rate of PCR and 20% rate of BCVA worse than 20/40 at one-month post-operative follow-up as standardized by the National Cataract Surgery Registry^[18]. In addition, their learning curves were constructed from comparable cataract grade surgeries, which could be a determining factor in their performances^[24-25]. The average of 38 cases to achieve competency in our series likely reflects each resident’s experience with 30 or more ECCE procedures prior to starting phaco training.

Proficiency in phacoemulsification cannot take in consideration only the PCR CUSUM chart, despite it being one of the most important factors for surgical success. The BCVA CUSUM chart is also very important and signals to others intraoperative mistakes that could be leading to a poor visual outcome. In our series, the second and third trainees achieved BCVA CUSUM competency before PCR CUSUM competency, which did not occur with the first trainee. In this series, we considered the residents competent in phacoemulsification only if they achieved both CUSUM chart competency. We believe that PCR CUSUM chart competency showed technical dexterity intrinsic to the procedure, but if the BCVA CUSUM chart competency was not achieved phacoemulsification proficiency was incomplete.

Our study has some limitations. We only had three trainees that did the surgeries at same surgical centre. It would be better if we could have a larger sample size and different training programs to test it, enabling us to show an average number necessary to achieve competency in phaco for a broader range of residency training programs.

Also, there are other parameters that can be studied for competency in phaco, like surgery time, that should be analysed by CUSUM. In addition, the standards we applied in our study for trainees are likely to be different from standards applied to experienced surgeons. In our series, only one trainee was able to achieve competency comparable to that of an experienced surgeon^{[2,4-5,11,18]}. However, our goal was not to show when the resident is comparable to an experienced surgeon, but when he reaches the point in his own learning curve that doing phaco without assistance will respect the basic principal of medicine: “first, do no harm.”

ACKNOWLEDGEMENTS

Conflicts of Interest: Vedana G, None; Cardoso FG, None; Marcon AS, None; Araújo LEK, None; Zanon M, None; Birriel DC, None; Watte G, None; Jun AS, None.