Validation of Peek Acuity application in pediatric screening programs in Paraguay

Bryce de Venecia1, Yasmin Bradfield1, Ralph Møller Trane1, Alicia Bareiro2, Miguel Scalamogna2

1Department of Ophthalmology and Visual Sciences, University of Wisconsin Madison, Madison 53792, Wisconsin, USA

2Fundación Visión, Calle Ingavi y Coronel Cazal Departamento,San Lorenzo 2300, Central, Paraguay

INTRODUCTION

The most recent World Health Organization estimates suggest 19 million children are affected by visual impairments. Of these impairments, refractive errors constitute the majority, especially for children in low income households and developing countries[1]. A poor visual acuity has been shown to contribute to lower average test scores and development of social or emotional problems; however,correcting these errors with spectacles resulted in improved academic attainment[2-4]. Tests for visual acuity using the Snellen Chart are the gold standard visual acuity evaluation in primary care offices in the United States where there are education screening programs in place for 42 of 50 states[5-6].

These programs either provide a screening service at the schools prior to enrollment or require the student to undergo an evaluation by an outside provider[4]. Without access to primary care physicians or education-based screening programs experienced by children of low income households,visual impairments can remain undiagnosed, leading to poor visual acuity during critical schooling years. Within Latin America, uncorrected refractive errors are found at rates of around 1 in 10[7-8].

An answer to this discrepancy in visual acuity lies in mobile phone screening tests. The most recent prediction of global smartphone ownership is 1 in 5[9]. Increased smartphones access is expected in low-income settings, which increases the access for mobile health screening applications[10].Through earlier intervention with a mobile phone visual acuity screening test, children can be referred to an ophthalmologist for adequate follow-up and improved outcomes.

A risk associated for any screening protocol is a high false positive referral rate leading to inappropriate labeling of the subject, resulting in undue anxiety surrounding the result and consumption of resources through inappropriate specialist intervention. To reduce this risk of high false positive rates in large screening operations, it is beneficial to use technology that is easy to implement and requires minimal training while maintaining accuracy. This furthers the argument for use of mobile phones to act as health intervention technologies, since they are easy to operate and understand[11]. Mobile phones have been a promising area for improved ophthalmic evaluation by non-professionals in screening settings and the emergency department when a patient requires an ophthalmology consultation[12].

Peek Acuity can fulfill the niche of a mobile phone screening modality that is accessible in low income areas and carries a low false positive referral rate. This free-to-download application has been shown to have 85% sensitivity and 98%specificity when used to test visual acuity by a community member in a patient’s home when compared to the gold standard of an ophthalmic clinician using the logMAR ETDRS chart[13]. Peek Acuity aims to bring visual acuity screening to populations devoid of such opportunities, which has enormous potential in school-based screening programs. School based programs are less expensive to implement and reduce disability secondary to refractive errors than relying on a primary eye care setting[14].

Peek Acuity, to our knowledge, has not be implemented in a screening setting that involves children; therefore, our study aims to validate the application in pediatric populations in Fernando de la Mora, Paraguay. Additionally, the study attempts to analyze the application’s utility as a screening device when administered by a layperson against the tumbling E chart and Spot Vision Screener (Welch Allyn). The Spot Vision screener is an automated vision refactor that has been extensively studied to screen for amblyopia[15-17]. It has been utilized in community level screening programs both domestically and abroad using non-medical personnel,where the autorefractor indicated a significant percent of the population requiring follow-up[17-18].

We hypothesize that Peek Acuity will be a beneficial tool for screening pediatric populations for visual acuity errors,comparable to existing school screening protocols including the tumbling E chart (used in global health settings) and Spot Vision Screener (commonly used in the United States) in both detection of disease and cost-effective implementation.

SUBJECTS AND METHODS

ParticipantsThis study took place from July 3, 2017 to October 31, 2017 within a joint screening effort by Combat Blindness International (Madison, Wisconsin) and Fundación Visión (Asuncion, Paraguay). A total of 393 subjects were enrolled in the study. Age of the children ranged from 6-16y.

Ethics ApprovalThe study adhered to the tenets of the Declaration of Helsinki and was approved by the Institutional Review Board of the University of Wisconsin Health Sciences Ethic Committee. Approval to perform the study in the public schools of Fernando de la Mora was obtained from the mayor of Fernando de la Mora and the school superintendent.

Informed consent was obtained from all the participants of the study. The subjects were given an informed consent form to be fulfilled by their parent or guardian. The risks and benefits for participation in the study were described in the informed consent form in Spanish, the prevalent language of Fernando de la Mora, Paraguay.

ScreeningThe study aimed to test Peek Acuity against existing screening modalities. Three screening modalities were incorporated into the screening portion of the experiment: Peek Acuity, Tumbling E single line of 6 optotypes set at 20/40, and Spot Vision Screener (Welch Allyn). The parameters of Peek Acuity are laid out in Bastawrous et al[13]. The Peek Acuity program was run on a Samsung Galaxy A3 running Android Version 6.0.1, with the screen illuminated at 100% for the screening process. Subjects were seated at 2 m away from the phone, were asked to cover one eye, and progressed through the series of optotypes presented. The examiner watched the child being tested to make sure they were not peeking from behind the palm of their hand. The subjects were asked to cover the other eye and the test was repeated. For Tumbling E, the subjects sat 6 m away, were asked to cover one eye, and progressed through six optotypes. The subjects were asked to cover the other eye and the test was repeated. For Spot Vision Screener, the subjects were screened according to manufacturer guidelines: the subjects sat 1 m away from the device and the refraction data taken. The Spot Vision Screener was installed with the latest Association of Pediatric Ophthalmology and Strabismus (AAPOS) recommendations for improved diagnostic accuracy[15,17,19].

Subjects were randomly assigned one of the three screening modalities to begin the screening process. The subject was then directed to complete the other two screening modalities.

Figure 1 The visual acuity agreement between Peek Acuity and the gold standard Snellen chart conducted by a pediatric ophthalmologist A: N=95/393 subjects (190 eyes) were referred and examined by a pediatric ophthalmologist. Visual acuity was analyzed for each eye, comparing the results of Peek Acuity screening and the Snellen Chart examination by a pediatric ophthalmologist the data suggests an overestimation by Peek Acuity, represented by a “left shift”; B: Heat map of the visual acuity agreement between Peek Acuity and the gold standard suggests an exact agreement of 31% (n=58/190), within one step 59% (n=110/190), and within two steps 71% (n=133/187)with an apparent overestimation of visual acuity by Peek Acuity. Weighted kappa was calculated to be 0.18, suggesting a low agreement between the two.

Subjects were selected for further diagnostic evaluation by a pediatric ophthalmologist if they did not pass any one of the three screening modalities cut-offs. The cut-off for Peek Acuity was set at 20/40 and was obtained according to the program’s calculations[13]. For the Tumbling E single line of optotypes,the size of the optotypes was set at 20/40, so an indication for referral was achieved if the subject did not correctly identify more than 2 out of 6 optotypes. For the Spot Vision Screener,the parameters were set according to the most recent AAPOS recommendations for referral indication, which resulted in the device presenting “complete eye exam recommended” to prompt a referral to an ophthalmologist or “all measurements within range” for no referral (Welch Allyn).

One quarter of the subjects who did not have an indication for referral according to the cut-offs established in the previous paragraph were selected for further evaluation by a pediatric ophthalmologist. This cohort was considered the control cohort for screening modality referral in statistical analysis.

Diagnostic EvaluationOphthalmic examination was considered to be the gold standard for determining refractive errors and functioned as a diagnostic screening method. A pediatric ophthalmologist, blinded to the visual acuity status of the student, performed visual acuity using a Snellen Chart,stereopsis and motility evaluation. The subject’s eyes were then dilated for cycloplegic refraction using Proparacaine hydrochloride ophthalmic solution USP 0.5% followed by tropicamide 1%, phenylephrine 2.5%, and cyclopentolate 1%was administered 30 to 40min prior to evaluation. Dilated eye examination by a pediatric ophthalmologist consisted of a cycloplegic retinoscopy and retinal examination. Diagnosis was entered into a database for analysis.

Statistical AnalysisFollowing completion of the subject screening, the screening results and the diagnosis were matched using the identification number. Subjects were stratified according to their age and analyzed within their age group. The sensitivity, specificity, positive predictive value, and negative predictive value were calculated for Peek Acuity comparing the accuracy of Peek Acuity when used by untrained personnel against the gold standard Snellen Chart when used by an ophthalmologist. Data from the control cohort was corrected using the “missing at random” protocol outlined in Xue et al[20].The referral agreement between the methods was compared using the kappa statistic. The same statistical analysis was performed for the other diagnostic tests as well (Spot Vision Screener and Tumbling E Chart). The utility of each test was suggested when comparing across the screening modalities using the sensitivity, specificity, and cost of implementing equipment.

RESULTS

PopulationThe validation study took place from July 2017 to October 2017. Of the 393 subjects enrolled in the study, 97(25% of subjects) completed both the screen and evaluation by a pediatric ophthalmologist, with n=22 subjects referred by Peek Acuity and n=73 subjects randomly selected for the control group. Two subjects referred by Peek Acuity were excluded as they were not able to complete all three screening tests.

Visual Acuity AgreementThe visual acuity supplied by Peek Acuity was analyzed for each eye (n=190) in the subjects referred to the pediatric ophthalmologist for follow up.The visual acuity obtained from the screening method was compared to the visual acuity obtained by the “gold standard”evaluation by the pediatric ophthalmologist using a Snellen chart (Figure 1).

Figure 2 Referral agreement among Peek Acuity, Spot Vision Screener, tumbling E optotypes, and the gold standard evaluation by a pediatric ophthalmologist A: Referral criteria for the screening techniques were set according to AAPOS guidelines and at 20/40 for visual acuity. The indication for referral was compared between the screening and the gold standard evaluation. Peek Acuity compared to gold standard evaluation resulted in 76% agreement and a kappa of 0.30; B: Spot Vision Screener compared to gold standard evaluation resulted in 78%exact agreement and a kappa of 0.38; C: Tumbling E single line of optotypes set at 20/40 compared to gold standard evaluation resulted in 77%exact agreement and a kappa of 0.39. No single screening technique achieved a significant advantage over the other when compared to the gold standard evaluation.

Table 1 Duration and cost comparison for each screening technique

Referrals AgreementSubjects were indicated for referral if they did not meet the cut-off set in the study. Additionally,one out of four subjects were randomly selected for analysis by the pediatric ophthalmologist. The agreement between the indication for referral was compared between each screening device and the indication for referral according to the ophthalmologist evaluation, and the data was analyzed using kappa statistic (Figure 2). When compared to the gold standard analysis, Peek Acuity was determined to have 47%sensitivity, 83% specificity, 43% positive predictive value and 85% negative predictive value for referral using 20/40 as a cut-off.

Utility of TestsThe total time collected from each screening modality was tested using a GEE procedure to estimate the mean time taken by each procedure (Table 1). The cost of implementing each screening modality was analyzed by determining the cost of each device. The cost associated with Peek Acuity was determined to be the market value cost of a Samsung Galaxy A3 (Table 1).

Subjects were screened with one of three randomly assigned modalities (Peek Acuity, Spot Vision Screener, or Tumbling E optotypes), completing the other two randomly assigned to prevent learner’s bias. The time, visual acuity, indication for referral, and age and sex of the subject were collected.The average duration to run each screening technique favors Spot Vision Screener (average time 5.10s) over Peek Acuity(average time 56.24s). The utility of the screening techniques;however, does not favor Spot Vision Screener in terms of cost.The Peek Acuity application is free-to-download. The cost reflected above signifying the price of the Samsung Galaxy A3, which was the smartphone used in this study to run the Peek Acuity app.

DISCUSSION

Pediatric visual acuity errors can be detrimental to a child’s wellbeing and education, but most cases are correctable,refractive errors that are treated using correction lenses[4].Implementation of a screening technique that is cost-effective and scalable in low income settings is needed to detect uncorrected refractive errors and improve the education of the subject. Currently, there is an estimated net loss in Gross Domestic Product of $202 000 million annually due to uncorrected refractive errors, signaling a need to detect these errors earlier and in an efficient manner[21].

This study attempted to determine the utility of Peek Acuity in school screenings and compared this technology with existing screening techniques. Peek Acuity was designed to provide cost-effective screening to populations with limited access to an optometrist or ophthalmologist. The high prevalence of cell phone use makes this screening device appealing. Peek Acuity has been validated in adult populations, but not in the pediatric age group[13]. Without access to screening, undetected and untreated vision abnormalities in children can lead to permanent vision loss (amblyopia), and hinder progress in school. Detecting visual problems earlier through school based screening and refractive error correction can result in decreased disability adjusted life years[14].

In this study, the Peek Acuity application performed by an untrained medical professional demonstrated moderate utility.When compared to the gold standard analysis of evaluation by a pediatric ophthalmologist using a Snellen Chart, Peek Acuity was determined to have a 47% sensitivity, an 83% specificity, a 43% positive predictive value and an 85% negative predictive value for referral using 20/40 as a cut-off. Peek Acuity tended to overestimate the subject’s visual acuity, more likely to give the patient a visual acuity that does not warrant follow-up.This low sensitivity would negatively affect school screening programs as the invested resources would not be utilized to their potential if the application fails to recognize a child that requires further evaluation by a medical professional.Because of the low sensitivity, the application did not correctly identify the subjects that required glasses. The pediatric ophthalmologist identified the need for corrective lenses or further intervention in 56 of the 95 subjects enrolled in the study, whereas Peek Acuity only identified 15 of these subjects.However, Peek Acuity had high specificity. The screener accurately identified children not needing glasses or a referral for a comprehensive eye exam. By correctly identifying children who do not need intervention through specialty care,there would be a decreased burden on screening programs in resource limited areas.

The discrepancy between the visual acuity provided by Peek Acuity and the gold standard evaluation could be due to a shortfall in the application or a diminished comprehension in a reduced-literacy population for the Snellen Chart. Peek Acuity was developed for better comprehension in low-literacy and non-English-speaking populations, utilizing the tumbling E optotype instead of the letters of the modern English alphabet.In pediatric populations where there is a lower patient reliability, a subject can randomly select an orientation of the tumbling E optotype and correctly identify the orientation a quarter of the time[5]. When paired with an algorithm that does not consider the time it takes to complete each optotype, which might suggest a “guess” if the time elapsed on a correctly identified optotype is longer, the Peek Acuity application would overestimate the visual acuity, reducing the test’s sensitivity.To rectify this issue, Peek Acuity could limit identification time within the algorithm or use more complicated optotypes in this population such as images similar to the Allen picture chart. Conversely, the population could have better understood the Peek Acuity tumbling E optotypes compared to the Snellen chart English alphabet, which would result in Peek Acuity“overestimating” the visual acuity status of the subject. Snellen is recognized to be a more rigorous test compared to pictures or tumbling E. This hypothesis is further supported by the similar “overestimation” seen by the single row of tumbling E optotypes set at 20/40.

When compared to the other screening techniques used in the study, Peek Acuity performed similarly in the ability to refer patients according to the cut-offs. The three techniques were within 1% agreement with each other when compared to the gold standard evaluation by a pediatric ophthalmologist.Since the Spot Vision Screener had been previously validated in similar populations, the degree of agreement between the technologies warrants further research into the utility and implementation of Peek Acuity in a pediatric age group.

With regards to utility in screening programs, Peek Acuity had the longest duration of evaluation (mean of 56.24s) compared to the single line of optotypes (mean of 21.63s) and Spot Vision Screener (mean of 5.10s). However, the cost of Peek Acuity is much less than that of Spot Vision Screener ($140 and $6490 respectively). When compared to the single line of optotypes set at 20/40, another cost-effective method, Peek Acuity provides the visual acuity for the subject instead of a binary indication for referral. Additionally, this method is more subjective, requiring more training, and is performed at six meters, a distance that is difficult to achieve in standard classrooms. Peek Acuity outperformed data from Snellen Chart analysis of visual acuity errors by trained teachers in India using a Snellen Chart. These teachers correctly identified true ocular pathology, with the majority of subjects positive for refractive error, in 24.33% of referred subjects[22]. Although different populations were analyzed and the false positive rate not captured in the India study, the data suggests Peek Acuity is competitive compared to other cost-effective screening measures. The duration of screening for Peek Acuity compared to the other technologies in this study is overcome by the application’s low cost, high specificity, and its flexibility of implementation.

Peek Acuity and similar mobile phone concepts remain a key resource for expanding screening capacities in a sustainable and wide scale manner for geographic locations with limited resources of eye care providers. Peek Acuity is a free-to-download application, which makes the only cost to implementation being the upfront cost of a smartphone,a cost that is commonly already covered for personal use.Peek Acuity’s high specificity and low implementation cost makes it a useful tool in school screening programs, and a greater impact can be expected if modifications are made to account for a pediatric population to increase the application’s sensitivity.

ACKNOWLEDGEMENTS

Author’s contributions:We wish to thank the Municipality and Schools of Fernando de la Mora, the staff of Fundación Visión, and Combat Blindness International. A special thanks to Patricia Blaires, Vanesa Rojas, Zunilda Caballero, Sandra Garay, Marianita Vester, Romina Ayala, Cynthia Falcon,Nicolás González, and Javier González.

Foundation:Supported by the Dan and Ellie Albert Student Vision Research Award, Doolittle Professorship Fund, Research to Prevent Blindness, Inc., Combat Blindness International,and Fundación Visión.

Conflicts of Interest:de Venecia B, None; Bradfield Y,None; Moeller R, None; Bareiro A, None; Scalamogna M,None.

1 Solebo AL, Rahi J. Epidemiology, aetiology and management of visual impairment in children. Arch Dis Child 2014;99(4):375-379.

2 Marshall EC, Meetz RE, Harmon LL. Through our children’s eyesthe public health impact of the vision screening requirements for Indiana school children. Optometry 2010;81(2):71-82.

3 Mathers M, Keyes M, Wright M. A review of the evidence on the effectiveness of children’s vision screening. Child Care Health Dev 2010;36(6):756-780.

4 Basch CE. Vision and the achievement gap among urban minority youth. J Sch Health 2011;81(10):599-605.

5 Sharma A, Congdon N, Patel M, Gilbert C. School-based approaches to the correction of refractive error in children. Surv Ophthalmol 2012;57(3):272-283.

6 American Association for Pediatric Ophthalmology and Strabismus.State-by-state Screening Requirements, 2011. Available at: https://aapos.org/resources/. Accessed on 2/10/2017.

7 Latorre-Arteaga S, Gil-González D, Bascarán C, Núñez RH, Morales MD, Orihuela GC. Visual health screening by schoolteachers in remote communities of Peru: implementation research. Bull World Health Organ 2016;94(9):652-659.

8 Ferraz FH, Corrente JE, Opromolla P, Schellini SA. Influence of uncorrected refractive error and unmet refractive error on visual impairment in a Brazilian population. BMC Ophthalmol 2014;14:84.

9 Heggestuen J. “One in Every 5 People in the World Own a Smartphone,One in Every 17 Own a Tablet [CHART]”. Business Insider. 2013.Accessed on: 10 March 2018.

10 Derenzi B, Borriello G, Jackson J, Kumar VS, Parikh TS, Virk P, Lesh N. Mobile phone tools for field-basedhealth care workers in low-income countries. Mt Sinai J Med 2011;78(3):406-418.

11 Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematicliterature review. J Med Syst 2015;39(2):1.

12 Pathipati AS, Wood EH, Lam CK, Sáles CS, Moshfeghi DM. Visual acuity measured with a smartphone app is more accurate than Snellen testing by emergency department providers. Graefes Arch Clin Exp Ophthalmol 2016;254(6):1175-1180.

13 Bastawrous A, Rono HK, Livingstone IA, Weiss HA, Jordan S, Kuper H, Burton MJ. Development and validation of a smartphone-based visual acuity test (peek acuity) for clinical practice and community-based fieldwork. JAMA Ophthalmol 2015;133(8):930-937.

14 Frick KD, Riva-Clement L, Shankar MB. Screening for refractive error and fitting with spectacles in rural and urban India: cost-effectiveness.Ophthalmic Epidemiol 2009;16(6):378-387.

15 Garry GA, Donahue SP. Validation of Spot screening device for amblyopia risk factors. J AAPOS 2014;18(5):476-480.

16 Peterseim MM, Papa CE, Wilson ME, Davidson JD, Shtessel M,Husain M, Cheeseman EW, Wolf BJ, Trivedi R. The effectiveness of the Spot Vision Screener in detecting amblyopia risk factors. J AAPOS 2014;18(6):539-542.

17 Arana Mendez M, Arguello L, Martinez J, Salas Vargas M, Alvarado Rodriguez AM, Papa CE, Peterseim CM, Otárola Víquez M, Trivedi RH,Peterseim MM. Evaluation of the Spot Vision Screener in young children in Costa Rica. J AAPOS 2015;19(5):441-444.

18 Terveen DC, Moser JM, Spencer TS. Results of a pediatric vision screening program in western South Dakota. S D Med 2015;68(3):111-113.

19 Donahue SP, Arthur B, Neely DE, Arnold RW, Silbert D, Ruben JB; POS Vision Screening Committee. Guidelines for automated preschool vision screening: a 10-year, evidence-based update. J AAPOS 2013;17(1):4-8.

20 Xue X, Kim MY, Castle PE, Strickler HD. A new method to address verification bias in studies of clinical screening tests: cervical cancer screening assays as an example. J Clin Epidemiol 2014;67(3):343-353.

21 Fricke TR, Holden BA, Wilson DA, Schlenther G, Naidoo KS,Resnikoff S, Frick KD. Global cost of correcting vision impairment from uncorrected refractive error. Bull World Health Organ 2012;90(10):728-738.

22 Priya A, Veena K, Thulasiraj R, Fredrick M, Venkatesh R, Sengupta S,Bassett K. Vision screening by teachersin southern Indian schools: testing a new "all class teacher" model. Ophthalmic Epidemiol 2015;22(1):60-65.

Correspondenceto:Yasmin Bradfield. Department of Ophthalmology and Visual Sciences, University of Wisconsin Madison, Madison 53792, Wisconsin, USA. ysbradfield@wisc.edu

Abstract● AlM: To validate the Peek Acuity mobile phone application in pediatric populations and compare its utility, both economic and diagnostic, against conventional screening methods using a pediatric ophthalmologist examination as the gold standard.● METHODS: A cohort of 393 subjects from Fernando de la Mora, Paraguay (ages 6-16y) were enrolled in the study.Subjects were randomly assigned a starting screening modality among: Peek Acuity, a single line of tumbling E optotypes set at 20/40, and Spot Vision Screener. Once completing the first screening modality, the subjects completed the two remaining techniques. Referral criteria were established based on the most current American Association of Pediatric Ophthalmology and Strabismus(AAPOS) recommendations: 20/40 for Peek Acuity and the tumbling E, and refractive error detection for the Spot Vision Screener. Subjects that failed to achieve the cutoff for any of the three screening techniques or subjects that passed the screening but were randomly selected to perform a comprehensive eye exam to determine the false negative rate, were evaluated by a pediatric ophthalmologist. This evaluation was considered the gold standard, and included vision assessment by a Snellen chart, strabismus evaluation, and cycloplegic refraction with dilated fundoscopy.● RESULTS: We obtained 48% sensitivity, 83% specificity,43% positive predictive value, and 86% negative predictive value for Peek Acuity's ability to refer compared to evaluation by a pediatric ophthalmologist, failing to achieve a desired sensitivity for implementation. Peek Acuity trended to overestimate the subject's visual acuity,providing a higher visual acuity that would not indicate referral for a comprehensive eye examination. However,its high specificity accurately predicted a significant number of children who did not need further evaluation.When comparing the three screening methods, no single screening modality outperformed the others. Peek Acuity represented a technology that was economically feasible compared to other screening modalities in low income settings, due to the prevalence of cell phone use.● CONCLUSlON: Peek Acuity represents an efficient tool that has potential for implementation in school screenings with different strategies aimed at pediatric populations due to its low cost and high specificity. An increase in sensitivity would improve detection of children with refractive errors.

● KEYWORDS:pediatric ophthalmology; validation study;screening; mobile phone application

Citation:de Venecia B, Bradfield Y, Trane RM, Bareiro A, Scalamogna M. Validation of Peek Acuity application in pediatric screening programs in Paraguay. Int J Ophthalmol 2018;11(8):1384-1389