27505
2014 et al
CPJXXX10.1177/0009922814527505Dombkowski et alClinical Pediatrics
Brief Report
Feasibility of Automated Appointment Reminders Using Email
Clinical Pediatrics 2014, Vol. 53(10) 1004–1007 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0009922814527505 cpj.sagepub.com
Kevin J. Dombkowski, DrPH, MS1, Anne E. Cowan, MPH1, Lauren E. Costello, MSW1, Allison M. Fisher, MPH2, and Sarah J. Clark, MPH1
Background
Methods
Patient reminder/recall has shown to be effective in improving childhood immunization rates.1-4 Pediatric practices have traditionally relied on mailed or voicebased reminders for appointments and vaccinations even though email use is commonplace among US adults.5 Although prior studies have found reminder/ recall via postal mail to be effective, substantial barriers exist. Parental contact information may be out of date and letters may be forwarded, returned to the sender, or discarded—but the disposition of a reminder letter through postal mail is rarely known. Even if a letter is successfully delivered, it may not be in a timely manner; at best, postal mail typically takes 1 to 3 days to reach intended targets. In contrast, the use of newer technologies, such as email and/or text messaging, represents an option for low-cost targeted reminder/recalls. These technologies allow for nearly immediate contact with parents and offer increased opportunity for providers to verify reminder/recall receipt. The household use of email and text-messaging continues to grow as smartphones and Internet access have become widely available for all age groups.5,6 Additionally, a recent study found that email was the most stable form of parental contact compared with cell phone and mailing address.7 The adoption of electronic health record (EHR) systems in response to the federal Meaningful Use program is rapidly changing the technology options available to practices. Incentives are tied to contacting patients per their preference, which may include email;8 despite this, very little is known about the feasibility of pediatric practices using newer technologies such as email for reminder notices. Previously, we evaluated the degree to which perceived barriers existed among private practices regarding the use of text or email messages for patient reminders.9 The objective of this study was to pilot test the feasibility of using email for reminders to parents of pediatric patients.
Study Design To pilot test the feasibility of using email for notifications to parents of pediatric patients, we used a convenience sample of 3 primary care sites in Michigan (2 pediatric practices and 1 family medicine practice) that serve children and expressed interest in expanding their notification practices. The study was approved by the University of Michigan Medical School Institutional Review Board.
Email Administration Business practices for email address collection were documented, and email addresses were obtained from administrative records. Current reminder/recall practices and strategies for obtaining and updating parental contact information were also identified for each provider site. Email addresses included parents of active patients ≤18 years (pediatric practices) and active adult patients, or parents of active patients ≤18 years (family practice). Email addresses were entered into electronic files and de-duplicated. Messages were sent from December 2011 through May 2012 to parents via a commercial email vendor to (a) confirm that email was acceptable for future reminders from practices or (b) notify that influenza vaccine was available at the practice (flu season only; Figure 1). Email messages were generic, contained the respective practices’ logos and contact information, and did not include identifiable patient or parent information. Both message types also included an option to opt-out of further 1
University of Michigan, Ann Arbor, MI, USA Centers for Disease Control and Prevention, Atlanta, GA, USA
2
Corresponding Author: Kevin J. Dombkowski, University of Michigan, 300 N. Ingalls, Ann Arbor, MI 48109-5456, USA. Email: [email protected]
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
1005
Dombkowski et al
Example of Email Confirmation Script Dear Parent / Guardian, You have previously provided this email to XXX Pediatrics. We are considering email as another option for sending appointment and vaccination reminders to parents and are in the process of updating our files. If you are not interested in receiving these messages from us at this email address, please click here: OPT OUT. Sincerely, XXX Pediatrics Please do not reply to this email address as we will not receive your response. If you wish to contact our office, please call XXXX. Example of Availability of Flu Vaccine Script Dear Parent / Guardian, Influenza vaccine is now available in our offices. Because we support the American Academy of Pediatrics recommendation that every child over the age of 6 months receive an influenza vaccine, we are sending you this notification. Please call us at either of the office numbers above to schedule an appointment. We hope to see you soon! If your family has already made or had an appointment for influenza vaccine, please disregard this message. Sincerely, XXX Pediatrics
Results Standard procedures had not been established to acquire and utilize parent email addresses in practice management systems. Each practice collected email addresses on patient intake forms; however, only 1 practice with a newly acquired EHR had the capability of storing email addresses within the practice management module of the system. Routine reminder/recall procedures were not established in the practices and were not conducted on a scheduled basis. One pediatric site did conduct telephone-based reminder/recall occasionally for children aged 9 to 19 months; the other pediatric site used recall in the past but had no routine process. The family practice site had no reminder/recall process. A total of 5191 email addresses were obtained; overall, 1027 (20%) were duplicates and removed. Of the resultant 4164 email addresses, 3670 confirmatory messages and 494 flu reminders were sent over a 6-month period in 9 successive mailings. Preliminary rounds of notifications revealed that opt-out links were not functional (first wave) and undeliverable messages were not being forwarded (first 2 waves). Adjustments were made to correct these problems and notifications were re-sent to those individuals in subsequent waves. The findings reported here include only those for the waves in which opt-out links were functional and email messages were being properly forwarded. Overall, 18% of email messages sent were found to be undeliverable and 3% of message recipients opted out of future email notifications (Table 1).
Discussion
Please do not reply to this email address as we will not receive your response. To contact our office, please call XXX. Thank you. To opt-out of receiving emails please click here: OPT-OUT.
Figure 1. Example of email reminder messages.
notices. Email messages were tested with study personnel and practice managers, including undeliverable and opt-out processes, prior to sending to parents. Messages were sent in waves of approximately 500 messages at least 1 week apart to minimize potential burden on the clinic, in the event that receipt of an email prompted parents to contact the clinic. A mechanism to track undeliverable email messages was not routinely provided by the vendor; email accounts were consequently established for each practice to capture messages returned as undeliverable. Our primary outcomes were the proportion of email messages returned as undeliverable and the proportion of opt-outs.
Our findings demonstrate that it is feasible to send practice-based reminder/recall notifications to parents using a commercial email vendor service. We found that optouts were not common for any of the provider sites, potentially indicating that email may be an acceptable form of communication among parents who provide accurate email addresses to practices. However, undeliverable notifications were relatively common, and generally consistent with other studies that have examined undeliverable rates using postal mail.10,11 A fundamental requirement for successful reminder/recall is the availability of accurate and complete contact information, including email addresses; this finding suggests the need for improved business processes to ensure the quality of email contact information. Although a prior study found that email was the most stable form of contact among parents,7 the results of this study indicate that many email addresses were incorrect. Invalid email addresses could be the result of incorrectly transcribed information from patient intake forms or may not have been kept up-to-date in practice records.
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
1006
Clinical Pediatrics 53(10)
Table 1. Email Reminders by Provider Site. Site No.
Provider Type
Email Messages (n)
Undeliverable (%)
Opt-Out (%)
1 2 3 Overall
Pediatric Pediatric Family medicine
1983 1356 825 4164
22 15 12 18
2 4 2 3
Although new technologies could potentially improve the ease and timeliness of reminder/recall efforts, barriers to broader implementation on a sustained basis may be encountered. Practices in our pilot study recognized the importance of tracking parental email addresses and did so by collecting email addresses on patient intake forms. However, at the time of this study only one of the practices was able to store this information in their practice management systems; another practice has subsequently upgraded its systems to support this function. In addition, if a practice’s EHR system does not support the capability of sending email reminders, establishing new business procedures to coordinate with an external email vendor could require additional staff time and may result in additional costs. Irrespective of the method by which email reminders are sent by practices, parent contact information must be kept current. Consequently, practices will require clearly defined business procedures that include the monitoring of undeliverable emails and update contact information on an ongoing basis. The tracking of undeliverable email notifications is crucial for providers to identify those that did not reach the intended targets, but this may not necessarily be an available feature in EHR systems. In our study, we established email accounts expressly for the purpose of capturing notifications that were not delivered. The feasibility and utility of automated reminders using email will continue to be influenced by the evolving information management environment at practices. The adoption of EHR systems in response to the federal Meaningful Use program incentives is rapidly changing the technology options available to practices that may affect email options. These incentives are tied to practices achieving capabilities specified by the program, which includes taking patient contact preference into account for reminders of preventive or follow-up care.8 Since there is marked variability in patient contact preference,7 the likely progression to new technologies will require EHRs to track different modes of contact, including cell phone, text messaging, and email. Practices may also have increased opportunities to use patient contact information such as email addresses for immunization reminder/recall. Improved interoperability between EHRs and immunization registries through real-time electronic messaging may enable more timely, accurate,
and complete contact information to be shared with immunization registries.12 Consequently, practices might have the option of conducting reminder/recall with their EHR or through their local jurisdiction’s immunization registry, using up-to-date contact information originating from their own practice systems. There are several limitations to this study. First, this pilot study represents a small convenience sample of practices using one email vendor; these results might not be generalizable to other pediatric practices. Second, although we found that a small percentage of our sample opted-out of receiving email notifications, we are unable to confirm that emails were read by parents. Some notifications in our pilot study could have been delivered to valid, but unmonitored, email addresses. Finally, patient characteristics were not available from participating practices and therefore we were unable to summarize our findings by demographic characteristics. Our pilot study demonstrates that practice-based email notifications to parents are feasible using commercially available vendor services. The frequency of undeliverable email messages might indicate the need for improved business processes to ensure timely, accurate, and complete parent contact information. Consistent with Meaningful Use criteria for EHR systems, practices should explore the capacity of their EHR system to indicate patients’ preferences for the specific mode of reminder notifications. The degree to which these procedures can be efficiently integrated with current patient registration and outreach activities (eg, appointment reminders) remains to be determined. In addition, the effectiveness and cost-effectiveness of email reminders at increasing vaccination coverage among pediatric patients affiliated with practices warrant further study. Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Cooperative Agreement Number 1U01IP0000316 from the Centers for Disease Control and Prevention.
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
1007
Dombkowski et al References 1. Szilagyi PG, Bordley C, Vann JC, et al. Effect of patient reminder/recall interventions on immunization rates: a review. JAMA. 2000;284:1820-1827. 2. Daley MF, Barrow J, Pearson K, et al. Identification and recall of children with chronic medical conditions for influenza vaccination. Pediatrics. 2004;113:26-33. 3. Kempe A, Daley MF, Barrow J, et al. Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall. Pediatrics. 2005;115:146-154. 4. Dombkowski KJ, Harrington LB, Dong S, Clark SJ. Seasonal influenza vaccination reminders for children with high-risk conditions a registry-based randomized trial. Am J Prev Med. 2012;42:71-75. 5. The Pew Internet & American Life Project. Trend data (adults). Pew Internet & American Life Project Tracking surveys 2012; http://pewinternet.org/Static-Pages/Trend-Data-(Adults)/ Online-Activites-Total.aspx. Accessed March 1, 2014. 6. Smith A. Smartphone Ownership—2013 Update. Washington, DC: Pew Research Center’s Internet and American Life Project; 2013.
7. Clark SJ, Butchart A, Kennedy A, Dombkowski KJ. Parents’ experiences with and preferences for immunization reminder/recall technologies. Pediatrics. 2011;128: e1100-e1105. 8. American Academy of Pediatrics. Meaningful use of electronic health records (EHRs). http://www.aap.org/ informatics/aapoverview.html. Published 2010. Accessed March 1, 2014. 9. Dombkowski KJ, Harrington L, Hanauer D, Kennedy A, Clark S. Current and potential use of new technologies for reminder notifications. Clin Pediatr (Phila). 2012;51:394-397. 10. Dombkowski KJ, Reeves SL, Dong S, Stevenson J, Clark SJ. Assessing the burden of undeliverable immunization reminder and recall notifications. Prev Med. 2011;53: 424-426. 11. Daley MF, Steiner JF, Kempe A, et al. Quality improvement in immunization delivery following an unsuccessful immunization recall. Ambul Pediatr. 2004;4:217-223. 12. Dombkowski KJ, Clark SJ. Redefining meaningful use: achieving interoperability with immunization registries. Am J Prev Med. 2012;42(4):e33-e35.
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
2014 et al
CPJXXX10.1177/0009922814527505Dombkowski et alClinical Pediatrics
Brief Report
Feasibility of Automated Appointment Reminders Using Email
Clinical Pediatrics 2014, Vol. 53(10) 1004–1007 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0009922814527505 cpj.sagepub.com
Kevin J. Dombkowski, DrPH, MS1, Anne E. Cowan, MPH1, Lauren E. Costello, MSW1, Allison M. Fisher, MPH2, and Sarah J. Clark, MPH1
Background
Methods
Patient reminder/recall has shown to be effective in improving childhood immunization rates.1-4 Pediatric practices have traditionally relied on mailed or voicebased reminders for appointments and vaccinations even though email use is commonplace among US adults.5 Although prior studies have found reminder/ recall via postal mail to be effective, substantial barriers exist. Parental contact information may be out of date and letters may be forwarded, returned to the sender, or discarded—but the disposition of a reminder letter through postal mail is rarely known. Even if a letter is successfully delivered, it may not be in a timely manner; at best, postal mail typically takes 1 to 3 days to reach intended targets. In contrast, the use of newer technologies, such as email and/or text messaging, represents an option for low-cost targeted reminder/recalls. These technologies allow for nearly immediate contact with parents and offer increased opportunity for providers to verify reminder/recall receipt. The household use of email and text-messaging continues to grow as smartphones and Internet access have become widely available for all age groups.5,6 Additionally, a recent study found that email was the most stable form of parental contact compared with cell phone and mailing address.7 The adoption of electronic health record (EHR) systems in response to the federal Meaningful Use program is rapidly changing the technology options available to practices. Incentives are tied to contacting patients per their preference, which may include email;8 despite this, very little is known about the feasibility of pediatric practices using newer technologies such as email for reminder notices. Previously, we evaluated the degree to which perceived barriers existed among private practices regarding the use of text or email messages for patient reminders.9 The objective of this study was to pilot test the feasibility of using email for reminders to parents of pediatric patients.
Study Design To pilot test the feasibility of using email for notifications to parents of pediatric patients, we used a convenience sample of 3 primary care sites in Michigan (2 pediatric practices and 1 family medicine practice) that serve children and expressed interest in expanding their notification practices. The study was approved by the University of Michigan Medical School Institutional Review Board.
Email Administration Business practices for email address collection were documented, and email addresses were obtained from administrative records. Current reminder/recall practices and strategies for obtaining and updating parental contact information were also identified for each provider site. Email addresses included parents of active patients ≤18 years (pediatric practices) and active adult patients, or parents of active patients ≤18 years (family practice). Email addresses were entered into electronic files and de-duplicated. Messages were sent from December 2011 through May 2012 to parents via a commercial email vendor to (a) confirm that email was acceptable for future reminders from practices or (b) notify that influenza vaccine was available at the practice (flu season only; Figure 1). Email messages were generic, contained the respective practices’ logos and contact information, and did not include identifiable patient or parent information. Both message types also included an option to opt-out of further 1
University of Michigan, Ann Arbor, MI, USA Centers for Disease Control and Prevention, Atlanta, GA, USA
2
Corresponding Author: Kevin J. Dombkowski, University of Michigan, 300 N. Ingalls, Ann Arbor, MI 48109-5456, USA. Email: [email protected]
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
1005
Dombkowski et al
Example of Email Confirmation Script Dear Parent / Guardian, You have previously provided this email to XXX Pediatrics. We are considering email as another option for sending appointment and vaccination reminders to parents and are in the process of updating our files. If you are not interested in receiving these messages from us at this email address, please click here: OPT OUT. Sincerely, XXX Pediatrics Please do not reply to this email address as we will not receive your response. If you wish to contact our office, please call XXXX. Example of Availability of Flu Vaccine Script Dear Parent / Guardian, Influenza vaccine is now available in our offices. Because we support the American Academy of Pediatrics recommendation that every child over the age of 6 months receive an influenza vaccine, we are sending you this notification. Please call us at either of the office numbers above to schedule an appointment. We hope to see you soon! If your family has already made or had an appointment for influenza vaccine, please disregard this message. Sincerely, XXX Pediatrics
Results Standard procedures had not been established to acquire and utilize parent email addresses in practice management systems. Each practice collected email addresses on patient intake forms; however, only 1 practice with a newly acquired EHR had the capability of storing email addresses within the practice management module of the system. Routine reminder/recall procedures were not established in the practices and were not conducted on a scheduled basis. One pediatric site did conduct telephone-based reminder/recall occasionally for children aged 9 to 19 months; the other pediatric site used recall in the past but had no routine process. The family practice site had no reminder/recall process. A total of 5191 email addresses were obtained; overall, 1027 (20%) were duplicates and removed. Of the resultant 4164 email addresses, 3670 confirmatory messages and 494 flu reminders were sent over a 6-month period in 9 successive mailings. Preliminary rounds of notifications revealed that opt-out links were not functional (first wave) and undeliverable messages were not being forwarded (first 2 waves). Adjustments were made to correct these problems and notifications were re-sent to those individuals in subsequent waves. The findings reported here include only those for the waves in which opt-out links were functional and email messages were being properly forwarded. Overall, 18% of email messages sent were found to be undeliverable and 3% of message recipients opted out of future email notifications (Table 1).
Discussion
Please do not reply to this email address as we will not receive your response. To contact our office, please call XXX. Thank you. To opt-out of receiving emails please click here: OPT-OUT.
Figure 1. Example of email reminder messages.
notices. Email messages were tested with study personnel and practice managers, including undeliverable and opt-out processes, prior to sending to parents. Messages were sent in waves of approximately 500 messages at least 1 week apart to minimize potential burden on the clinic, in the event that receipt of an email prompted parents to contact the clinic. A mechanism to track undeliverable email messages was not routinely provided by the vendor; email accounts were consequently established for each practice to capture messages returned as undeliverable. Our primary outcomes were the proportion of email messages returned as undeliverable and the proportion of opt-outs.
Our findings demonstrate that it is feasible to send practice-based reminder/recall notifications to parents using a commercial email vendor service. We found that optouts were not common for any of the provider sites, potentially indicating that email may be an acceptable form of communication among parents who provide accurate email addresses to practices. However, undeliverable notifications were relatively common, and generally consistent with other studies that have examined undeliverable rates using postal mail.10,11 A fundamental requirement for successful reminder/recall is the availability of accurate and complete contact information, including email addresses; this finding suggests the need for improved business processes to ensure the quality of email contact information. Although a prior study found that email was the most stable form of contact among parents,7 the results of this study indicate that many email addresses were incorrect. Invalid email addresses could be the result of incorrectly transcribed information from patient intake forms or may not have been kept up-to-date in practice records.
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
1006
Clinical Pediatrics 53(10)
Table 1. Email Reminders by Provider Site. Site No.
Provider Type
Email Messages (n)
Undeliverable (%)
Opt-Out (%)
1 2 3 Overall
Pediatric Pediatric Family medicine
1983 1356 825 4164
22 15 12 18
2 4 2 3
Although new technologies could potentially improve the ease and timeliness of reminder/recall efforts, barriers to broader implementation on a sustained basis may be encountered. Practices in our pilot study recognized the importance of tracking parental email addresses and did so by collecting email addresses on patient intake forms. However, at the time of this study only one of the practices was able to store this information in their practice management systems; another practice has subsequently upgraded its systems to support this function. In addition, if a practice’s EHR system does not support the capability of sending email reminders, establishing new business procedures to coordinate with an external email vendor could require additional staff time and may result in additional costs. Irrespective of the method by which email reminders are sent by practices, parent contact information must be kept current. Consequently, practices will require clearly defined business procedures that include the monitoring of undeliverable emails and update contact information on an ongoing basis. The tracking of undeliverable email notifications is crucial for providers to identify those that did not reach the intended targets, but this may not necessarily be an available feature in EHR systems. In our study, we established email accounts expressly for the purpose of capturing notifications that were not delivered. The feasibility and utility of automated reminders using email will continue to be influenced by the evolving information management environment at practices. The adoption of EHR systems in response to the federal Meaningful Use program incentives is rapidly changing the technology options available to practices that may affect email options. These incentives are tied to practices achieving capabilities specified by the program, which includes taking patient contact preference into account for reminders of preventive or follow-up care.8 Since there is marked variability in patient contact preference,7 the likely progression to new technologies will require EHRs to track different modes of contact, including cell phone, text messaging, and email. Practices may also have increased opportunities to use patient contact information such as email addresses for immunization reminder/recall. Improved interoperability between EHRs and immunization registries through real-time electronic messaging may enable more timely, accurate,
and complete contact information to be shared with immunization registries.12 Consequently, practices might have the option of conducting reminder/recall with their EHR or through their local jurisdiction’s immunization registry, using up-to-date contact information originating from their own practice systems. There are several limitations to this study. First, this pilot study represents a small convenience sample of practices using one email vendor; these results might not be generalizable to other pediatric practices. Second, although we found that a small percentage of our sample opted-out of receiving email notifications, we are unable to confirm that emails were read by parents. Some notifications in our pilot study could have been delivered to valid, but unmonitored, email addresses. Finally, patient characteristics were not available from participating practices and therefore we were unable to summarize our findings by demographic characteristics. Our pilot study demonstrates that practice-based email notifications to parents are feasible using commercially available vendor services. The frequency of undeliverable email messages might indicate the need for improved business processes to ensure timely, accurate, and complete parent contact information. Consistent with Meaningful Use criteria for EHR systems, practices should explore the capacity of their EHR system to indicate patients’ preferences for the specific mode of reminder notifications. The degree to which these procedures can be efficiently integrated with current patient registration and outreach activities (eg, appointment reminders) remains to be determined. In addition, the effectiveness and cost-effectiveness of email reminders at increasing vaccination coverage among pediatric patients affiliated with practices warrant further study. Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Cooperative Agreement Number 1U01IP0000316 from the Centers for Disease Control and Prevention.
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
1007
Dombkowski et al References 1. Szilagyi PG, Bordley C, Vann JC, et al. Effect of patient reminder/recall interventions on immunization rates: a review. JAMA. 2000;284:1820-1827. 2. Daley MF, Barrow J, Pearson K, et al. Identification and recall of children with chronic medical conditions for influenza vaccination. Pediatrics. 2004;113:26-33. 3. Kempe A, Daley MF, Barrow J, et al. Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall. Pediatrics. 2005;115:146-154. 4. Dombkowski KJ, Harrington LB, Dong S, Clark SJ. Seasonal influenza vaccination reminders for children with high-risk conditions a registry-based randomized trial. Am J Prev Med. 2012;42:71-75. 5. The Pew Internet & American Life Project. Trend data (adults). Pew Internet & American Life Project Tracking surveys 2012; http://pewinternet.org/Static-Pages/Trend-Data-(Adults)/ Online-Activites-Total.aspx. Accessed March 1, 2014. 6. Smith A. Smartphone Ownership—2013 Update. Washington, DC: Pew Research Center’s Internet and American Life Project; 2013.
7. Clark SJ, Butchart A, Kennedy A, Dombkowski KJ. Parents’ experiences with and preferences for immunization reminder/recall technologies. Pediatrics. 2011;128: e1100-e1105. 8. American Academy of Pediatrics. Meaningful use of electronic health records (EHRs). http://www.aap.org/ informatics/aapoverview.html. Published 2010. Accessed March 1, 2014. 9. Dombkowski KJ, Harrington L, Hanauer D, Kennedy A, Clark S. Current and potential use of new technologies for reminder notifications. Clin Pediatr (Phila). 2012;51:394-397. 10. Dombkowski KJ, Reeves SL, Dong S, Stevenson J, Clark SJ. Assessing the burden of undeliverable immunization reminder and recall notifications. Prev Med. 2011;53: 424-426. 11. Daley MF, Steiner JF, Kempe A, et al. Quality improvement in immunization delivery following an unsuccessful immunization recall. Ambul Pediatr. 2004;4:217-223. 12. Dombkowski KJ, Clark SJ. Redefining meaningful use: achieving interoperability with immunization registries. Am J Prev Med. 2012;42(4):e33-e35.
Downloaded from cpj.sagepub.com at UCSF LIBRARY & CKM on December 13, 2014
