Abstract

Materials and Methods

K.I.D.R.O.P. currently provides ROP screening for 81 neonatal intensive care units (NICU's) across 18 districts of Karnataka State in Southern India. There are 3 dedicated teams (‘Team A's’) and 12 ‘Team B’s’ for 12 of the district headquarters who were trained and validated in Bangalore. The program was initiated by Narayana Nethralaya Postgraduate Institute of Ophthalmology (NNPIO), Bangalore as a stand-alone program, initially as a pilot for 6 districts in 2008. The NNPIO has undertaken all the training and validation studies.

The program has expanded as a ‘public private partnership’ under the aegis of the National Rural Health Mission, Ministry of Health and Family Welfare, Government of Karnataka (since 2009). All images, data and reports analyzed in this manuscript are restricted to the 6 districts not covered under the PPP and owned by NNPIO. The Institution Review Board, the Research Committee and the Ethics Committee of NNPIO have approved this program.

Training, validation and accreditation of technicians: (The “STAT” score)

Training and validating a technician is germane to the KIDROP program. We have evolved a training methodology with a scoring system, which we use to accredit a novice technician through levels of expertise [Table 1]. This has resulted from the collective experience of training several private and government teams and is called the “KIDROP STAT” (S core for T raining and A ccreditation of T echnicians) and comprises 3 levels (Level I, II and III) in a 20 point score. The scored parameters include basic knowledge of the disease and the program (1-3), imaging related (4-13), which scores the ability of the technician to obtain well focused, oriented images in a proper sequence which includes the temporal ora or at least zone 3, and speed of acquisition. Image grading and reporting (14-16) relate to his or her ability to diagnose and make a clinical management decision. Decision triage is performed using a template created by KIDROP and includes a 3-way triage using images from both eyes [Table 2]: RED - (Type 1 ROP or serious disease in one or both eyes probably needing treatment or at least urgent evaluation by the expert), ORANGE (Type 2 ROP in at least one eye where follow up is needed; and GREEN (can be discharged). Post imaging (17-19) parameters test the ability of the technician to upload the images, use the Tele-Care software, record the details and aid the project manager in scheduling subsequent follow-up visits using images to educate the mothers. The last parameter on the score is management of complications (20) and is assessed for level II and III technicians.

Table 1

KIDROP's Score for training and accreditation of technicians (copyright)

Table 2

Decision Aiding Algorithm (‘Red-Orange-Green Triage’) developed by KIDROP

The ‘levels’ in the STAT score are created to be water-tight compartments rather than an ‘across-the-board’ point system used in other scoring systems.[24] This ensures stricter entry criteria into each level, necessitating all tasks to be completed before accreditation. For example, to qualify as level I, simply having good knowledge of the program and data management will not suffice, if the technician possesses poor skill of image capture or grading and vice-versa. Furthermore, as the expertise increases, speed must improve (6 minutes to 2 minutes per eye), grading scores must improve (60 to > 90%), and the number of routine uploads must decrease from all images to only RED's. This ensures that once a technician reaches Level III, he or she is required to upload only images of RED infants, so that the decision of referral can be verified by an expert. Objective scores are obtained for parameters described in 5,6,7,1216 and 19. Other parameters are assessed using pre and post-tests, theory and practical ‘live’ assessments. In our program it takes approximately 30 working days to ‘create’ a Level I and 90 days for a Level III technician. Trainees progress from observation and practicing on a mannequin (Retcam Imaging Practice Kit (Clarity MSI, USA) to live sessions first under supervision and then independently until they are adept in image acquisition and decision making.

Results

Demographics and distribution

At the time of submission, 3 KIDROP ‘A teams’ servicing 81 enrolled NICU's in 18 districts in Karnataka have performed 41,237 imaging sessions (babies). Of these, 8503 sessions were performed during 2008-2009 and 22,596 performed during 2011-2102. Sessions for analysis of this manuscript were derived from these two timed cohorts. In 2010-2011, 227 sessions were included for the iPhone app validation analysis.

In 2008, the first 4,422 sessions were used for training two primary KIDROP technicians (Level 1). The subsequent 1462 sessions in 2008-09 and 455 sessions by the same technicians upgraded to Level III in 2011-12 were included for kappa correlation and agreement analysis between the technician and ROP expert. The distribution of these 6339 sessions (1601 infants) is summarized in Table 3 and shows that 36.4% of babies had 3 or less imaging sessions before discharge, the remaining (63.6%) had 4 or more sessions.

Table 3

Distribution of 6339 screening sessions by number of episodes

The distribution of the birth-weight plotted along the gestational age for screened and treated babies is depicted in [Fig. 1] and emphasizes that 28.5% of all infants treated had birth weights of > 1501gm and 6.3% of these ‘outliers’ had birth weights of > 2000gm.

Figure 1

Graphic representation and correlation of birth weight against gestational age of 1601 infants screened and 175 treated

Technician Validation

Level I technicians agreed with 85.9% of all management (red, orange and green) of the expert. This improved to 94.3% of all decisions when they upgraded to Level III. The proportion of sessions where the technicians’ management decisions were non-referral when the expert thought this to be indicated dropped from 4.7 to 0.9% (Level I to III). This clinical utility score is summarized in Table 4. Of the 4 infants who were false negatives, two were called ‘red’ by the specialist and ‘orange’ (“follow up within a week”) by the technician and could have potentially been lost. The other 2 were marked ‘orange’ (zone 3 avascular) by the specialist and ‘green’ (mature retina) by the technician and arguably would not acquire ‘blinding’ disease even if missed. Hence, a Level III technician would miss only 0.4% (and not 0.9%) of infants with treatable lesions. Applying this correction to Level I, only 13 (and not 68 infants) [0.9% (13/1462)] would risk missing treatment.

Table 4

Clinical Utility of Technicians making a diagnosis and calling for follow-up (Level I vs Level III technician)

The sensitivity, specificity, positive predictive value and negative predictive values for diagnosing ‘any stage of ROP’ and ‘treatment grade ROP’ during their transition from Level I to III are summarized in Table 5. The discriminatory index [(sensitivity x specificity)/100)] improved from 79.5 to 92.7 and 71.6 to 89.2 for ‘any stage’ and ‘treatment grade ROP’ respectively. The measurement of agreement (kappa) comparing 1) treatment vs no treatment 2) mild vs severe ROP and 3) discharge vs no discharge is summarized in Table 6, which again demonstrates significant improvement as their skills improved.

Table 5

Sensitivity and specificity of diagnosis of retinopathy of prematurity: Technician vs specialist (Level I vs Level III technician)

Table 6

Measurement of agreement (Kappa) between technician and ROP specialist

iPhone validation

The overall agreement for all 10 categories of diagnoses was 96.3%. The measurement of overall agreement for detecting ROP was kappa 0.96 (SE 0.014, P< 0.0001), for type 1 ROP was kappa 0.96, (SE 0.024, P< 0.0001) and for mild, Type 2 ROP, was kappa 0.94, (SE 0.037, P< 0.0001). The sensitivity, specificity, positive and negative predictive values for severe and mild disease detections are summarized in Table 7.

Table 7

iPhone App Image reading and reporting agreement with Retcam laptop analysis

Discussion

India, like many other middle-income countries is facing the ‘third epidemic’ of ROP.[1,2] Over the past two decades, ROP has been reported from major cities of the country with an incidence ranging between 37-54%.[28,29,30] Infants blind from ROP continue to present to tertiary level facilities, many of whom have not been screened. Significantly, over the past decade, India has made steady progress in child health care indices and the infant mortality rate (IMR) has dropped from 81/1,000 live births in 1990 to 47/1,000 live births in 2011.[31] This has led to an increase in surviving neonates, many of whom are preterm, even in rural centers[7,8,9,10] which adds to the unmet challenge of ROP screening compounded by inadequate awareness among pediatricians, gynecologists and lack of sufficiently trained ROP specialists.[9,16]

Telemedicine for ROP using pediatric wide-field digital retinal imaging with ‘remote experts’ who grade images is a successful means of bridging this gap.[2,32] In most ‘real-world’ programs,[2] images captured by nurses,[33,34,35,36,37] ophthalmic photographers[38,39,40,41] or ophthalmologists[42,43,44,45] are read onsite or remotely by ‘ROP experts’ using a direct, ‘store and forward’ or image transfer platforms within a defined time period which could vary from a day to a week. However, in India, finding the ‘ROP expert’ albeit for remote reading is a huge challenge. With less than 500 registered vitreo-retinal surgeons, it is believed that there are less than 30-35 ROP experts who could provide comprehensive ROP care, and mostly reside in urban areas.

We began KIDROP with the belief that a telemedicine program will not solve our problem unless non-physicians can be trained not only to take images, but also to reliably grade them and make management decisions at the time of image capture. This is particularly important in India where larger, more mature infants develop severe ROP, many of whom have already been discharged from neonatal care. A delay in receiving the management decision from experts would pose an enormous administrative challenge in transmitting the management decision to parents, many of whom are poor, uneducated and who live at a distance from the neonatal center. To our best knowledge, in 2008 this was the first program to train technicians to capture and interpret retinal images.[2,9,10,11] We evolved a training and accreditation score (STAT, Table 1) for this purpose. The aim was to provide the mother with a definitive diagnosis and date for the next visit even before she left the neonatal unit. Simultaneously, we tested remote reading by the ROP expert at our headquarters on an indigenously created tele-ROP platform that was enabled on the laptop and smart phone (iPhone) with ‘near-live’ reporting.

KIDROP was built on a “Triple-T” principle, namely T elemedicine, T raining of peripheral ophthalmologists and ophthalmic assistants and T aking the support of neonatologists, pediatricians and gynecologists. The program expanded from 6 pilot districts of Southern Karnataka to 18 districts in all, 12 of these supported by the National Rural Health Mission, Ministry of Health and Family Welfare, Government of Karnataka in the nation's first public private partnership in infant blindness prevention. At the time of submission, 81 neonatal units are serviced by 3 ‘A’ teams and 12 ‘B’ teams (local districts). Over 41,000 screening sessions have been performed using 4 Retcam shuttles and over 860 (10.2% of infants screened) infants have received treatment for ROP.

Previous ‘real-world telemedicine programs’[46] from Canada,[42] United States,[34] and Germany[43] have used trained ophthalmologists or experts to remotely read and interpret images with different terminologies such as referral warranted ROP (RW-ROP)[42] or clinically significant ROP (CS-ROP)[34] for decision algorithms. We chose a new triage graded protocol based on the new ICROP classification[25] which places emphasis on the management decision using a colour coding of red, orange and green [Table 2] keeping in mind the nature of demands on a non-physician grader in a remote area. This was conceived so as to create a wider safety net wherein, for example, Type 1 ROP is marked red, stage 1 is followed up (orange) and only mature retina, or definitely regressing ROP is discharged (green) [Fig. 2]. Another layer in the safety net is that no infant in our program is discharged until a mature retina is imaged on two consecutive visits, one of these between 41-45 weeks of postmenstrual age.

Figure 2

Retcam images captured during Level I technician training (top panel, red, orange and green) and improvement in capturing subtler disease including excellent images of the ora serrata (lower panel, red, orange and green) when they attain Level III status ...

Our results demonstrate that Level I technicians have an overall agreement of 85.9% with the expert and this improves to 94.3% when they upgrade to Level III. In our setting, it takes an average of 30 and 90 working days to achieve a skill level of Level I and III respectively. In a screening program of this nature, missing treatable disease could be dangerous, as the child may never have access to a clinical exam by an ROP specialist. Our results show that this occurrence is indeed very low. A Level I technician would miss 0.9% of infants needing treatment and a Level III only 0.4%. The corollary would be the ‘false positives’, which could add the burden of another visit for the baby and the family. This nearly halves from 9.4% to 4.8% as the skill improves from Level I to III. This can be attributed to their ability to image the temporal periphery up to and including the ora serrata in an increasing proportion of babies, enhancing their confidence of ‘discharging’ the baby [Fig. 2]. Initially Level I technicians begin by uploading all cases (green, orange and red) for remote expert tele-reporting. With increased expertise, Level III technicians are able to triage better, with the more urgent ‘red’ cases being uploaded for the ROP specialist to confirm if treatment is required on the live tele-ROP platform. Despite high grades of agreement with the specialist, it is recommended that before a baby is discharged from the screening network, the ROP specialist ratifies this decision by viewing the images or examines the baby on site wherever possible.

In the United Kingdom, under the NHS Diabetic Eye Screening Programme, technicians from designated central reading centers are employed to interpret downloaded images for diabetic retinopathy changes.[47] In the KIDROP program, our technicians capture, read and report ROP images ‘on-site’ and real-time. In future, with increasing volumes of images coming from different parts of the country, central-reading centers may become necessary.

We developed the smart phone app on the iPhone (Apple, CA, USA) for rapid review and reporting. In 2009, this was the first smart phone to provide touch technology, ‘pinch and drag’, and PDF printing compatibility.[9,11,19,20,21] This app allowed the expert anytime access to the images, less dependence on the computer, a user friendly interface to report the diagnosis and suggest follow-up. This is tightly integrated with the database maintained on a cloud-based server and is compliant with regulatory standards. With the proliferation and improvement of smart phone technology, future research would include imaging using the smart phone, App based data management systems and integration with video conferencing.

A limitation of this study program is that it judged the validity of non-physicians to be graders with the assumption that the ‘gold standard’ for comparison of wide field digital imaging for ROP is dilated indirect ophthalmoscopy. This concept has already been questioned.[48] Studies suggest that photographic documentation may ‘inadvertently’ detect mild disease that was ‘missed’ on ophthalmoscopic examination.[35] The ability of images to document, review, store and compare disease far outweighs routine indirect ophthalmoscopy, even when performed by an expert. In our country, the current limited expertise to use ophthalmoscopy, particularly in rural areas is a case in favor of non-physician grading. The importance of training and validating the technicians cannot be overemphasized. Another limitation is that the cost-utility of this program has not been formally analyzed. Furthermore, the expertise of other Level III technicians has not been compared against our primary technicians who enjoy over 15,000 sessions each, hence making generalizability an unmeasured entity.

The KIDROP model is now being replicated in two more states and training is completed in two more. Level III technicians are the first line decision makers in the rural periphery who are backed by remote experts who read these images on smart phones. The outcome from these centers is awaited. However, the applicability of this program in other developed countries would attract unaddressed medico-legal constraints and needs evaluation.

In India, as ROP care, awareness, training and expertise among general ophthalmologists and specialists improves, it is likely that there would be less dependence on a program built on the tenants of KIDROP. Until then, the societal impact of missed screening due to lack of trained resources in peripheral areas leading to increased infant blindness would certainly be a factor for considering a ‘KIDROP like program’ as a model for ROP screening in underserved areas.

Acknowledgements

Footnotes

References