Journal of Microbiological Methods 106 (2014) 8–15

Contents lists available at ScienceDirect

Journal of Microbiological Methods journal homepage: www.elsevier.com/locate/jmicmeth

Development of a multiplex real-time PCR assay for the rapid diagnosis of neonatal late onset sepsis Marre van den Brand a,⁎, Remco P.H. Peters b, Arnold Catsburg c, Anna Rubenjan a,1, Ferdi J. Broeke a,1, Frank A.M. van den Dungen d, Mirjam M. van Weissenbruch d, A. Marceline van Furth d, Triinu Kõressaar e, Maido Remm e, Paul H.M. Savelkoul a,b,c,1, Martine P. Bos c a

Department of Medical Microbiology and Infection Control, VU University Medical Center, De Boelelaan 1118, 1081HZ Amsterdam, The Netherlands Department of Medical Microbiology, Maastricht University Medical Centre, P. Debyelaan 25, 6229HX Maastricht, The Netherlands c Microbiome, Troubadoursborgh 59, 3992BE Houten, The Netherlands d Department of Pediatrics, VU University Medical Center, De Boelelaan 1118, 1081HZ Amsterdam, The Netherlands e Department of Bioinformatics, University of Tartu, Ülikooli 18, 50090 Tartu, Estonia b

a r t i c l e

i n f o

Article history: Received 17 March 2014 Received in revised form 25 July 2014 Accepted 25 July 2014 Available online 4 August 2014 Keywords: Molecular diagnosis Multiplex PCR Late onset sepsis Neonatal sepsis Rapid diagnosis Real-time PCR

a b s t r a c t The diagnosis of late onset sepsis (LOS), a severe condition with high prevalence in preterm infants, is hampered by the suboptimal sensitivity and long turnaround time of blood culture. Detection of the infecting pathogen directly in blood by PCR would provide a much more timely result. Unfortunately, PCR-based assays reported so far are labor intensive and often lack direct species identification. Therefore we developed a real-time multiplex PCR assay tailored to LOS diagnosis which is easy-to-use, is applicable on small blood volumes and provides speciesspecific results within 4 h. Species-specific PCR assays were selected from literature or developed using bioinformatic tools for the detection of the most prevalent etiologic pathogens: Enterococcus faecalis, Staphylococcus aureus, Staphylococcus spp., Streptococcus agalactiae, Escherichia coli, Pseudomonas aeruginosa, Klebsiella spp. and Serratia marcescens. The PCR assays showed 100% specificity, full coverage of the target pathogens and a limit of detection (LOD) of ≤ 10 CFU eq./reaction. These LOD values were maintained in the multiplex format or when bacterial DNA was isolated from blood. Clinical evaluation showed high concordance between the multiplex PCR and blood culture. In conclusion, we developed a multiplex PCR that allows the direct detection of the most important bacterial pathogens causing LOS in preterm infants. © 2014 Elsevier B.V. All rights reserved.

1. Introduction Neonatal late onset sepsis (LOS) is a serious condition with a high morbidity and mortality. In particular, preterm and very low birth weight (VLBW, b1500 g) infants admitted to the neonatal intensive

Abbreviations: AFLP, amplified fragment length polymorphism; ATCC, American Type Culture Collection; BLB, bacterial lysis buffer; CDC, Center for Disease Control; CFU, colony forming unit; CoNS, coagulase negative staphylococci; Cq, quantification cycle; DSMZ, Deutsche Sammlung von Mikroorganismen und Zellkulturen; eq., equivalent; LOD, limit of detection; LOS, late onset sepsis; NICU, neonatal intensive care unit; NTC, negative template control; OD, optical density; PBS, phosphate-buffered saline; PhHV-1, Phocine Herpesvirus 1; VLBW, very low birth weight. ⁎ Corresponding author. Tel.: +31 20 444 4444. E-mail addresses: [email protected] (M. van den Brand), [email protected] (R.P.H. Peters), [email protected] (A. Catsburg), [email protected] (A. Rubenjan), [email protected] (F.J. Broeke), [email protected] (F.A.M. van den Dungen), [email protected] (M.M. van Weissenbruch), [email protected] (A.M. van Furth), [email protected] (T. Kõressaar), [email protected] (M. Remm), [email protected] (P.H.M. Savelkoul), [email protected] (M.P. Bos). 1 Tel.: +31 20 444 4444.

http://dx.doi.org/10.1016/j.mimet.2014.07.034 0167-7012/© 2014 Elsevier B.V. All rights reserved.

care unit (NICU) are prone to infection and approximately 20% of these infants experience one or more episodes of proven LOS (Stoll et al., 2002). Blood culture is the gold standard for diagnosis but its diagnostic impact is negatively affected by considerable turnaround time and a suboptimal sensitivity (Squire et al., 1979; Pierce et al., 1984). The sensitivity is mainly affected by the small volume of blood inoculated in blood cultures, previous administration of antibiotics and presence of low or intermittent bacteremia (Schelonka et al., 1996; Connell et al., 2007). The effect of blood volume was clearly demonstrated in a study which showed that blood cultures containing an adequate blood volume were twice more likely to yield a positive result compared to blood cultures with an inadequate blood volume (Connell et al., 2007). Since half of all deaths occur within the first three days after blood cultures are obtained, faster and more sensitive identification of the causative pathogen could be of great value in neonatal care to allow an early diagnosis and faster guidance of therapy (Stoll et al., 2002). The molecular detection of bacterial DNA by real-time PCR is an attractive diagnostic option as it has proven to be fast, sensitive and specific for identification of various infectious diseases (Peters et al., 2004a; Espy et al., 2006).

M. van den Brand et al. / Journal of Microbiological Methods 106 (2014) 8–15

Efforts to introduce molecular methods for diagnosis of neonatal sepsis have evolved around broad-range PCR targeting the 16S rRNA gene for universal bacterial or Gram-specific detection (McCabe et al., 1995; Jordan et al., 2006; Wu et al., 2008; Chan et al., 2009). A recent systematic review with meta-analysis evaluating these assays showed a mean sensitivity of 90% and specificity of 96% compared to blood culture indicating that such assays currently offer a promising potential as add-on tests (Pammi et al., 2011). However, an important limitation of these broad-range assays is that additional processing steps such as sequencing or hybridization are required for species identification thereby prolonging the turnaround time significantly. Also, they are particularly prone to false-positive results coming from exogenous bacterial DNA present in PCR reaction components (e.g. Taq polymerase preparation) or from contamination during processing steps. These issues reduce both sensitivity and applicability of these assays (Corless et al., 2000). A species-specific PCR assay could overcome these drawbacks, but should be applied in a multiplexed format to provide sufficient coverage of etiologic pathogens. The implementation of real-time multiplex assays in the diagnostic process is rapidly growing and recent studies report sensitivities equal to monoplex assays (Wittwer et al., 2001; Molenkamp et al., 2007; Bahrdt et al., 2010). As such, these types of assays may as well provide a good option for diagnosis of neonatal bacteremia. Therefore, we set out to develop an easy-to-use, fast and sensitive multiplex PCR assay that detects the most prevalent bacterial pathogens in neonatal LOS in a species-specific manner and which requires only small input blood volumes. 2. Materials and methods 2.1. Bacterial strains A panel of 47 bacterial and 3 fungal species that are either known to cause neonatal sepsis or are regular contaminants of blood culture was selected (Table 1). Strains were obtained from the American Type Culture Collection (ATCC), Deutsche Sammlung von Mikroorganismen und Zellkulturen (DSMZ), Center for Disease Control (CDC) and the culture collections of the microbiology laboratories of the VU University Medical Center (Amsterdam, The Netherlands), University Medical Center Utrecht (Utrecht, The Netherlands) and Radboud University Nijmegen Medical Centre (Nijmegen, The Netherlands). Clinical strains were identified using standard phenotypic and genotypic laboratory procedures, 16S rDNA sequencing (Nijmegen), Phoenix Automated Microbiology System (Utrecht) and MALDI-TOF VitekMS (bioMérieux) or amplified fragment length polymorphism (AFLP) (Amsterdam). 2.2. DNA isolation from pure cultures and blood To obtain microbial DNA from pure cultures, microorganisms from fresh plate cultures were suspended into tryptic soy culture broth (enriched with 10% horse serum and 6 μg/ml nicotinamide adenine dinucleotide) and grown at 37 °C to an optical density (OD) of 0.5 McFarland or directly suspended in phosphate-buffered saline (PBS) to an OD of 0.5 McFarland. Cells were collected from 200 μl of the 0.5 McFarland suspension after centrifugation (10 min at 14,000 ×g) and subsequently lysed by incubation for 10 min at 95 °C whilst shaking (800 rpm) in 200 μl of bacterial lysis buffer (BLB) (Biocartis, Mechelen, Belgium). After addition of 20 μl neutralization buffer (Biocartis, Mechelen, Belgium), microbial DNA was extracted with the QIAamp DNA mini kit (Qiagen, Venlo, The Netherlands). DNA quantity is given as colony forming unit equivalents (cfu eq.), e.g. 10 cfu eq. represents DNA isolated from 10 cfu. Measurement of OD and bacterial plate counts were used to determine cfu. For isolation of microbial DNA from blood, 200 μl of EDTA anticoagulated-blood was treated twice with 1 ml of TTE (1% Triton X-100, 20 mM Tris–HCl pH 8.3 and 1 mM EDTA) to lyse erythrocytes and remove hemoglobin as previously described (Peters et al., 2007).

9

Table 1 List of bacterial strains used to evaluate the specificity of the PCR assays. Species

Source

Sepsis pathogens Actinomyces spp. Acinetobacter baumannii Bacteroides fragilis Bacillus cereus Candida albicans Candida glabrata Candida parapsilosis Citrobacter freundii Eikenella corrodens Enterobacter aerogenes Enterobacter cloacae Enterobacter asburiae Enterococcus faecalis Enterococcus faecium Escherichia coli Escherichia coli Gardnerella vaginalis Haemophilus influenzae Klebsiella pneumoniae Klebsiella oxytoca Listeria monocytogenes Morganella morganii Moraxella catarrhalis Neisseria meningitidis Pseudomonas aeruginosa Pseudomonas aeruginosa Serratia marcescens Serratia marcescens Staphylococcus aureus Staphylococcus aureus Staphylococcus epidermidis Stenotrophomonas maltophilia Streptococcus agalactiae Streptococcus pneumoniae Streptococcus pyogenes Streptococcus oralis Streptococcus parasanguinis

Clinical straina ATCC 19606 ATCC 25282 ATCC 11145 ATCC 90028 ATCC 15545 ATCC 22019 ATCC 8090 DSM 8340 ATCC 13048 ATCC 13047 Clinical strainb ATCC 29212 CDC NY2 ATCC 11775 ATCC 25922 ATCC 14018 ATCC 49247 ATCC 13883 ATCC 13182 ATCC 15313 Clinical straina Clinical straina Clinical straina ATCC 27853 ATCC 10145 ATCC 13880 DSM 12481 ATCC 12600 ATCC 25923 ATCC 14990 ATCC 13637 DSM 2134 ATCC 49619 ATCC 19615 DSM 206267 DSM 6778

Contaminants of blood culture and skin microbiota Clostridium bifermentans Corynebacterium xerosis Lactobacillus acidophilus Propionibacterium acnes Veillonella parvula

DSM 14994 ATCC 373 DSM 20079 DSM 16379 DSM 2008

Additional strains used to evaluate the Streptococcus agalactiae PCR Abiotrofia defectiva Lactococcus lactis Streptococcus dysgalactiae Streptococcus bovis Streptococcus salivarius

Clinical strainb DSM 20481 Clinical strainc Clinical strainc Clinical strainc

Additional strains used to evaluate the E. coli, Klebsiella spp., P. aeruginosa, S. marcescens PCR Cronobacter spp. Clinical straina Proteus mirabilis DSM 4479 Pseudomonas fluorescens ATCC 13525 Pseudomonas putida ATCC 12633 Salmonella enterica ATCC13076 Shigella flexneri ATCC 12022 Yersinia enterocolitica ATCC 23715 Additional strains used to evaluate the Serratia marcescens PCR Serratia odorifera

DSM 4582

a

Species identification with standard laboratory methods and confirmed with MALDITOF VitekMS. b Kindly provided by dr. A.C. Fluit from the Department of Medical Microbiology, University Medical Center Utrecht. Identification of species was performed as described (Paauw et al., 2008). c Kindly provided by Prof. Peter W.M. Hermans, Radboud University Nijmegen Medical Centre. Identification of species was performed by 16S rDNA sequencing.

The resulting bacterial pellet was lysed with BLB as described earlier. Subsequently, DNA was purified with the NucliSENSEasyMAG device (bioMérieux, Zaltbommel, The Netherlands) using protocol “Specific A”

10

M. van den Brand et al. / Journal of Microbiological Methods 106 (2014) 8–15

in an elution volume of 110 μl. The manufacturer's instructions were followed except that 1 ml AL buffer (Qiagen, Venlo, The Netherlands) plus 1 ml easyMAG Lysis buffer (bioMérieux, Zaltbommel, The Netherlands) were used instead of 2 ml easyMAG Lysis buffer during DNA binding to the beads. The AL buffer was spiked with Phocine Herpesvirus 1 (PhHV-1) as extraction and amplification control (internal control) (van Doornum et al., 2003). Successful bacterial lysis and subsequent DNA extraction was verified with a broad-range or pathogenspecific PCR.

The analytical performance of the assays was further explored by calculating the efficiency and linearity (R2) from these dilution series with the LightCycler software. To evaluate the performance of the assays for blood-derived bacterial DNA, bacteria were spiked in 200 μl EDTA anticoagulated-blood from healthy adult volunteers such that 1000, 100, 10 and 1 cfu eq. per PCR could be evaluated (this equals 50,000, 5000, 500 and 50 cfu/ml blood). Results were compared to DNA isolated from bacteria suspended in PBS for 1000 and 100 cfu eq. per PCR.

2.3. Primer and probe design

2.6. Analytical performance characteristics of the multiplex assay

A literature search was conducted to identify species- and genusspecific real-time PCR assays. These assays were re-evaluated for coverage and specificity in silico using the amplicons as queries in BLAST searches in the nucleotide collection (nr/nt) and whole genome shotgun (wgs) databases at NCBI. PCRs were discarded when homologies at the probe sequence or at the 3′ end of the primer sequences were N 90% in a non-target organism. Acceptable PCRs were then tested for coverage of all available sequences of the target organism. New PCR targets were selected using MultiMPrimer3 (http://bioinfo.ut.ee/multimprimer3/) or from alignments of gene sequences retrieved from GenBank (Koressaar et al., 2009). Sequences with adequate in silico coverage and specificity were used for primer and probe design for real-time PCR assays using Primer Express Software 3.0 (Life Technologies, Bleiswijk, The Netherlands).

The sensitivity of the assays in multiplex and monoplex format was compared using serial dilutions of genomic DNA from pure cultures (1000, 100, 50, 10, 5, 1 cfu eq./reaction) performed in triplicate (fivefold at 1 cfu eq./reaction). Pooled blood samples (200 μl EDTA anticoagulatedblood) from 10 healthy adult volunteers were subjected to the multiplex PCR assay to test for nonspecific amplification signals.

2.4. Real-time PCR assay All PCR reactions were performed on a LightCycler 480II (Roche Diagnostics, Almere, The Netherlands). Reaction mixtures contained 12.5 μl of 2× LightCycler 480 Probes Master (Roche Diagnostics, Almere, The Netherlands), 2.5 μl primers and probe(s) and 10 μl DNA input, resulting in a final reaction volume of 25 μl. The optimal primer and probe concentrations were found to range between 300 nM to 900 nM for the primers and 150 and 200 nM for the probes. For the Escherichia coli specific assay the LightCycler Master mix was replaced by the 2 × QuantiFast Multiplex PCR Master Mix (Qiagen, Venlo, The Netherlands) since this mix resulted in lower background signals [unpublished data]. Cycling conditions were as follows: 10 min at 95 °C followed by 45 cycles of 95 °C for 15 s and 60 °C for 1 min. Results were analyzed with the LightCycler Software version 1.5.0. For the multiplex assays, color compensation was applied to each analysis as described in the LightCycler Operator Manual to correct for fluorescent bleed-through. Each experiment included a no-template control (NTC) (PCR grade water) and a positive control (1000 cfu eq. of DNA) for control of contamination and amplification. The primer sets and dual labeled hydrolysis probes were obtained from Isogen (De Meern, The Netherlands) and Macrogen (Amsterdam, The Netherlands). 2.5. Analytical performance characteristics of the monoplex assays The analytical specificity of the monoplex assays was verified using genomic DNA from a panel of bacteria and fungi obtained from pure cultures (Table 1) at a concentration of approximately 1000 cfu eq. per reaction for bacteria and 10,000 cfu eq. for fungi. A total of 15 strains of each target species or genus were used to evaluate the coverage of the assays. The limit of detection (LOD) was determined by testing serial dilutions (10,000, 1000, 100, 50, 10, 1 cfu eq./reaction) of purified genomic DNA from two isolates, each performed in duplicate (i.e. 4 reactions per concentration). The LOD was defined as the lowest concentration that was detected in all four reactions. For E. coli a sample was regarded positive when the Cq value (quantification cycle) of the sample was at least two values lower than that of the NTC.

2.7. Performance of the multiplex assay on clinical samples Whole blood samples (200 μl collected aseptically in EDTA-tubes) obtained together with blood culture from 20 consecutive episodes of suspected LOS in preterm infants admitted to the NICU of our hospital were tested with the multiplex assay. Results were compared to blood culture. Approval was obtained from the local medical ethics committee and written informed consent was obtained from the parents or legal guardians of the neonates. 3. Results 3.1. Design of the pathogen specific assays The most prevalent bacterial pathogens causing LOS in VLBW neonates admitted to the NICU were selected for inclusion in the multiplex assay (Table 2), designated the NeoSep-ID. Based on epidemiological data this selection covers almost 90% of sepsis cases caused by bacterial infections (Stoll et al., 1996, 2002; van den Hoogen et al., 2010; Hornik et al., 2012). Available real-time PCR assays reported in the literature were used for species-specific detection of Enterococcus faecalis, Staphylococcus aureus and E. coli because re-evaluation in silico of these assays yielded high coverage and specificity (Huijsdens et al., 2002; Santo Domingo et al., 2003; Peters et al., 2007). Searches with the Sa442 sequence (Martineau et al., 1998) which has been extensively used for detection of S. aureus yielded only a few strains that would remain undetected due to sequence divergence (Klaassen et al., 2003). Since more than 350 strains showed perfect matches for primers and probe we decided to use the SA442 target for detection. Re-evaluation of the assay used for detection of E. coli revealed a 100% match with Shigella spp. Shigella spp. is rarely encountered in neonatal sepsis and therefore not likely to cause misidentification. A genus specific PCR to identify staphylococci at the genus level based on a previously published assay was adjusted to Table 2 Design of the NeoSep-ID multiplex PCR assay. Dye

Reaction 1

Reaction 2

Reaction 3

ATTO 425

Staphylococcus aureus Enterococcus faecalis Klebsiella spp.

Escherichia coli

Staphylococcus spp.

Fam Yakima Yellow ROX ATTO 647 N

Streptococcus agalactiae Internal control (PhHV)

Pseudomonas aeruginosa Serratia marcescens

M. van den Brand et al. / Journal of Microbiological Methods 106 (2014) 8–15

11

Table 3 Oligonucleotides used in this study. Species

Primer/probe

Sequence

Target sequence

Amplicon size (bp)

Reference

Enterococcus faecalis

Forward primer Reverse primer Probe Forward primer Reverse primer Probe Forward primer Reverse primer Probe Forward primer Reverse primer Probe Forward primer Reverse primers

5′-CGCTTCTTTCCTCCCGAGT-3′ 5′-GCCATGCGGCATAAACTG-3′ 5′-CAATTGGAAAGAGGAGTGGCGGACG-3′ 5′-CATGCCGCGTGTATGAAGAA-3′ 5′-CGGGTAACGTCAATGAGCAAA-3′ 5′-TATTAACTTTACTCCCTTCCTCCCCGCTGAA-3′ 5′-AACCAGGCGTCGATAAT-3′ 5′-GTTTACGGCGCAATCC-3′ 5′-ACAGGAAAGACAAGACTATGCAGACC-3′ 5′-GCCGAGGTCATGGAATTC-3′ 5′-ATCCGCGCCATCATCTTC-3′ 5′-CGACAACCGCAAGGAAGCCGA-3′ 5′-GACCGTGAAGACCACTTCCATTAC-3′ 5′-ACGCCGATGTCGTCTTTCAC-3′ 5′-CACGCCGATATCGTCTTTCAC-3′ 5′-CGATCCACCCGAACGTGTTCTACTTCTC-3′ 5′-CCAACTCCAGAACGTGATTCTG-3′ 5′-CCAACTCCAGAACGTGACTCTG-3′ 5′-CCAACACCAGAACGTGATTCTG-3′ 5′-GTTGTCACCAGCTTCAGCGTAGT-3′ 5′-GTTATCACCAGCTTCAGCGTAAT-3′ 5′-GTTGTCACCAGCTTCAGCATAGT-3′ 5′-ACAGGCCGTGTTGAACGTGGKCAAATCAA-3′ 5′-CATCGGAAACATTGTGTTCTGTATG-3′ 5′-TTTGGCTGGAAAATATAACTCTCGTA-3′ 5′-AAGCCGTCTTGATAATCTTTAGTAGTACCGAAGCTGGT-3′ 5′-TTCACCAGCTGTATTAGAAGTACATGC-3′ 5′-CCCTGAACATTATCTTTGATATTTCTCA-3′ 5′-CAAGCCCAGCAAATGGCTCAAAAGCT-3′ 5′-GGGCGAATCACAGATTGAATC-3′ 5′-GCGGTTCCAAACGTACCAA-3′ 5′-TTTTTATGTGTCCGCCACCATCTGGATC-3′

16S rRNA

143

Santo Domingo et al. (2003)

16S rRNA

96

Huijsdens et al. (2002)

rhaA–rhaD operon

107/108

This study

phzE gene

89

This study

gyrB gene

125

This study

tuf gene

222

Adjusted from Rood et al. (2011)

Sa442

94

Peters et al. (2007)

cfb gene

150

Adjusted from Ke et al. (2000)

gB gene

89

van Doornum et al. (2003)

Escherichia coli

Klebsiella spp.

Pseudomonas aeruginosa

Serratia marcescens

Staphylococcus spp.

Probe Forward primers

Reverse primers

Staphylococcus aureus

Streptococcus agalactiae

Internal Control PhHV-1

Probe Forward primer Reverse primer Probe Forward primer Reverse primer Probe Forward primer Reverse primer Probe

detect the most important species (Staphylococcus epidermidis, Staphylococcus hominis, Staphylococcus haemolyticus, Staphylococcus saprophyticus, Staphylococcus capitis) (Rood et al., 2011). Samples positive with the genus PCR and negative for S. aureus specific testing were considered positive for coagulase negative staphylococci (CoNS). For Streptococcus agalactiae the highly specific cfb gene was selected and primers and probes were modified to suit our PCR protocol (Ke et al., 2000). MultiMprimer3 was employed to identify novel targets for Pseudomonas aeruginosa and Klebsiella spp. (using genome sequences of Klebsiella pneumoniae and Klebsiella oxytoca) as no existing PCR assay with adequate in silico performance was available from literature. MultiMPrimer3 searches for multi-copy species-specific PCR targets by genome comparisons among assembled sequenced genomes. Since insufficient numbers of sequenced and assembled genomes were available for Serratia marcescens to use the MultiMPrimer3 tool, we designed specific primers and probes based on multiple alignments of gyrB

sequences deposited in GenBank. Selected targets with their primer and probe sequences are presented in Table 3. 3.2. Analytical performance of the monoplex assays The specificity of the monoplex assays was assessed using a panel of bacteria and fungi that cause neonatal sepsis or that are regular contaminants of blood culture or skin bacteria (Table 1). In addition, phylogenetically related species and species with target sequence similarity observed with BLAST were tested. Amplification signals were observed only for the target species except in the case of the E. coli specific PCR. In the E. coli assay amplification signals were observed for Shigella flexneri as anticipated from the in silico analysis. NTC signals were only present in the E. coli specific PCR probably resulting from residual E. coli DNA in the Taq polymerase preparation. This signal was reduced significantly (3 Cq values) when using the QuantiFast Multiplex PCR Master Mix while the E. coli specific signal was not negatively affected [data not

Table 4 Performance characteristics of the monoplex assays. Assay

E. faecalis E. coli Klebsiella spp. P. aeruginosa S. marcescens Staphylococcus spp. S. aureus S. agalactiae a b

Analytical sensitivity LOD in cfu eq./reaction (median Cq value)

Coverage (%)

10 (37.6) 10 (33.5) 10 (36.9) 10 (36.7) 10 (36.2) 10 (40.0) 10 (35.9) 1 (38.1)

100 100 100 100 100 100 100 100

Calculated from 15 strains. Median calculated from a dilution series of two strains.

a

Analytical specificity

Accuracy

Cross reactivity

Linearityb (R2)

Efficiencyb

– Shigella flexneri – – – – – –

0.98 0.99 0.98 0.99 0.92 0.98 0.98 0.94

2.02 1.97 1.99 1.81 2.07 1.98 2.07 1.78

12

a) Enterococcus faecalis

1000 cfu eq

100 cfu eq

b) Streptococcus agalactiae

1000 cfu eq

g) Pseudomonas aeruginosa

1000 cfu eq

100 cfu eq

d) Staphylococcus genus

1000 cfu eq

100 cfu eq

e) Klebsiella genus

1000 cfu eq

100 cfu eq

f) Serratia marcescens

1000 cfu eq

100 cfu eq

100 cfu eq

h) Escherichia coli

1000 cfu eq

100 cfu eq

100 cfu eq

Fig. 1. Performance of monoplex PCR assays, indicated above the panels (a–h), for detection of DNA from individual bacteria as spiked to blood samples and derived from pure culture (PBS). Bacteria were spiked in two concentrations (1000 and 100 cfu/reaction). Gray curves = bacterial DNA derived from blood. Black curves = bacterial DNA derived from pure culture.

M. van den Brand et al. / Journal of Microbiological Methods 106 (2014) 8–15

1000 cfu eq

c) Staphylococcus aureus

M. van den Brand et al. / Journal of Microbiological Methods 106 (2014) 8–15

shown]. The observed Cq values of the NTC ranged from ±36 to undetectable. Hence all monoplex assays were regarded as specific. Clinical strains, selected on the basis of differences in AFLP typing patterns and thus representing a set of highly divergent strains, were used to determine the coverage of the assays. For each assay 15 tested strains were successfully detected, resulting in 100% coverage. The LOD ranged between 1 and 10 genome copies per reaction for all assays and linearity was observed in all assays up to 10 cfu eq./reaction (Table 4). The performance of the assays was further verified with bacteria spiked in whole blood at different concentrations. The observed detection limits were similar to those obtained with DNA obtained from pure cultures. The PCR curves of bacterial DNA isolated from PBS versus blood coincided virtually completely, demonstrating that no significant PCR-inhibiting factors are co-purified in the current procedure (Fig. 1).

3.3. Design & validation of the multiplex assay Since most real-time PCR platforms are able to detect fluorescence at 5 different emission wavelength windows (channels), we combined monoplex PCRs to a maximum of five. The sensitivity of the Staphylococcus genus assay was greatly reduced in any multiplexed format. This assay therefore was placed in a dedicated reaction, resulting in a total of three reactions (Table 2). The internal control (IC) probe, for detection of PhHV-1 DNA, was designed to emit in the channel with the highest wavelength, as fluorescence is usually lower at highest wavelengths. This ensures that the IC assay will suffer most from potential inhibition in the reaction. In the same vein, the probes detecting the most prevalent bacteria (E. coli, S. aureus, CoNS) causing LOS were designed to emit in the lowest wavelength channels. This also positions them in separate reactions (Table 2) reducing the chances that multiple targets will be amplified in a single reaction. In the multiplex assays primers were used at 900 nM and probes at 200 nM since this resulted in highest fluorescence. Color compensation was applied to correct for fluorescent bleed-through. Comparison of monoplex and multiplex assays showed lower fluorescence levels and slightly more variable Cq values for the multiplex assays, but sensitivities were not negatively affected (Table 5). The LODs ranged between 1 and 10 cfu eq./reaction for all multiplex assays (except for E. coli). The presence of aspecific amplification signals coming from the blood (e.g. proteins, human leucocyte DNA) was evaluated using 10 pooled blood samples from healthy volunteers. None of the samples yielded positive PCR signals in the multiplex assay except for the IC signal. Table 5 Comparison between multiplex and monoplex PCR assays. Target

Assay format

50 cfu eq.

10 cfu eq.

5 cfu eq.

1 cfu eq.

E. faecalis

Multiplex Monoplex Multiplex Monoplex Multiplex

+++ +++ +++ +++ +++

+++ +++ ++* +++ +++

+++ +++ *** *** +++

+++++ +++++ ***** ***** +++++

Monoplex Multiplex Monoplex Multiplex Monoplex Multiplex Monoplex Multiplex Monoplex

+++ +++ +++ +++ +++ +++ +++ +++ +++

+++ +++ +++ +++ +++ +++ +++ +++ ++−

+++ +++ +++ +++ +++ +++ +++ +++ ++−

++++− ++−−− ++++− ++++− +++++ +−−−− −−−−− +−−−− ++−−−

E. coli Klebsiella spp. P. aeruginosa S. marcescens S. aureus S. agalactiae

Different amounts of DNA, indicated on the top row in cfu eq./reaction, were tested in monoplex and multiplex assays. The outcome of individual assays is indicated by + (positive PCR signal); − (no PCR signal) or * (difference with negative control b2 Cq values).

13

Table 6 Comparison between multiplex PCR assay and blood culture for 20 episodes of suspected LOS. Blood culture

Multiplex PCR

No.

Result

Assay

11

CoNS

1 1 7

E. coli CoNS + K. pneumoniae negative

8 positive 3 negative 1 positive CoNS 7 negative

3.4. Evaluation of the multiplex assay in clinical samples As a proof of principle, EDTA blood samples obtained from 20 consecutive episodes of suspected LOS occurring at the NICU were analyzed by PCR and compared to blood culture. Results are shown in Table 6. For 9 out of 12 monomicrobial positive blood cultures (11 CoNs, 1 E. coli), concomitant PCR results were identical (8 CoNS, 1 E. coli). Three CoNS positive blood cultures were associated with negative PCR results. For one multimicrobial positive blood culture (CoNS + K. pneumoniae), only CoNS was detected by PCR. The Cq values ranged between 28.6 and 37.1. Seven negative PCR results were concordant with negative blood cultures. Overall, concordance was 80% between blood culture and PCR, and there was no significant difference in yield between techniques (McNemar test p-value N 0.05).

4. Discussion In this study, we designed and evaluated a real-time multiplex PCR assay, designated the NeoSep-ID, for molecular diagnosis of neonatal sepsis. The assay detects the eight most prevalent bacterial pathogens that cause LOS in preterm and VLBW infants in our setting. We showed that this assay has a high analytical sensitivity and specificity for detection of bacterial DNA in pure culture and blood. Various approaches have been used in order to improve the diagnosis of neonatal sepsis, but to our knowledge we are the first to develop a multiplex PCR assay specifically designed to diagnose LOS. Enomoto et al. (2009) developed a conventional multiplex PCR for detection of any type of neonatal infection including those due to viral pathogens and Candida albicans. Unfortunately, CoNS, the most prevalent pathogens causing LOS, were not included in this assay. Other studies used broad-range PCR for detection of bacterial DNA. For example, Chan et al. (2009) used a Gram-specific PCR to differentiate between Grampositive and Gram-negative sepsis to allow the neonatologist to anticipate on the more severe course in the case of Gram-negative sepsis. Consequently, this assay is unable to discriminate between CoNS and S. aureus bacteremia of which the clinical course and outcome are dissimilar. Jordan et al. conducted several large sample size studies using various broad-range 16S rRNA gene PCR assays (sometimes with additional steps for species identification) resulting in both high and low sensitivities (Jordan and Durso, 2000, 2005; Jordan et al., 2005, 2006). Other studies of broad-range PCR show similar results with a wide variety in sensitivity of the molecular assays compared to blood culture ranging from 40% to 100%. A recent systematic review, analyzing these studies, concluded that the sensitivity of molecular assays is not yet sufficient for “ruling out” sepsis but they should rather be used as an additional tool for “ruling in”, i.e. diagnosis, of sepsis (Pammi et al., 2011). Ruling out sepsis requires a nearly full coverage of causative agents in the assay, as can only be done in broad-range assays. In contrast, ruling in sepsis requires only the detection of the most prevalent pathogens, favoring a species-specific approach. Currently, several commercial multiplex PCR assays (i.e. LightCycler® SeptiFast, Vyoo™) are available for species-specific molecular diagnosis of sepsis in adults. These assays show promising results in clinical trials (Lebovitz and Burbelo, 2013).

14

M. van den Brand et al. / Journal of Microbiological Methods 106 (2014) 8–15

However, as they require an input volume of at least 1.5 ml of blood, usage of these assays is precluded in preterm infants. Therefore, we decided to develop a species-specific assay on a real-time multiplex PCR platform that can be used in neonates. One advantage of a real-time species-specific approach is that it allows direct identification of the pathogen without further processing steps, reducing the turnaround time and hands-on time. In this respect target selection is a crucial element for sensitive and specific species identification. As the information on microbial DNA sequences has grown exponentially in recent years we were able to identify new DNA sequences for the specific detection of P. aeruginosa, Klebsiella spp. and S. marcescens with use of bioinformatic tools. MultiMPrimer proved to be a useful tool to search for DNA sequences with full coverage that are specific, which is crucial to prevent misidentification. Only for the E. coli assay we could not identify a DNA sequence discriminating between E. coli and Shigella species which is not unexpected given their close phylogenetic relationship. This is not likely to pose a problem in our setting as bacteremia due to Shigella is rare and occurs mainly in developing countries (Ashkenazi, 2004). Another advantage of the species-specific approach is that it is not as much affected by contamination of PCR reagents with bacterial DNA. Many different decontamination procedures have been tested in broad-range applications, some with success but usually with the unwanted consequence of a compromised sensitivity of the PCR assay (Corless et al., 2000; Klaschik et al., 2002; Peters et al., 2004b; Chang et al., 2011). This especially becomes a concern with detection of low concentrations of DNA, as is often the case for neonatal sepsis. Indeed with our E. coli assay, a background signal was observed which at first negatively affected the limit of detection. For this reason a different PCR mix was used which reduced the background signal so that as little as 10 cfu eq./reaction could be detected. Utilization of a real-time platform that obviates the need for post-PCR processing steps like sequencing or hybridization for species identification also reduces the risk of laboratory contamination, a major concern for diagnostic laboratories. Combined in a multiplex assay it also enhances the applicability of the assay in everyday practice as it is less time consuming and more cost effective. Inhibiting substances in blood, e.g. leucocyte DNA and hemoglobin, are known to negatively affect the sensitivity of PCR assays (Akane et al., 1994). We demonstrate that the sensitivity of our assays in blood is equal to detection of bacterial DNA in pure culture (Fig. 1). As such our sample processing method consisting of the removal of hemoglobin (TTE), lysis of bacteria (BLB) and an automatic DNA purification step results in adequate removal of inhibiting substances and lysis of both Gram-positive and Gram-negative bacteria. This is especially important as the bacterial load in blood may be low. Furthermore, the combination of the individual assays in a multiplex format had no influence on sensitivity. This was achieved by evaluating different combinations of PCRs. One important outcome of this evaluation was that the performance of the Staphylococcus genus assay evidently deteriorated in a multiplex format and was therefore placed in a dedicated reaction. Our experience is that the performance of a certain combination of primers and probes is difficult to predict from in silico analysis and laboratory experiments proved to be more useful in our evaluations. The turnaround time of the NeoSep-ID is short compared to blood culture allowing an impact on patient management at an early stage. Preliminary results of blood culture are usually not available until after 24–48 h, and require an additional 1–2 days to finalize results on pathogen identification and antibiotic susceptibility testing. Therefore blood culture has a limited value during the initial treatment of the neonate suspected for sepsis, but the NeoSepID assay could provide a worthy diagnostic alternative or addition as results can be available in less than 4 h, which potentially enables adjustments in (empirical) treatment at an early stage. Moreover, PCR may reveal additional cases of sepsis as it is less affected by the prior use of antibiotics. Since PCR detects the presence of bacterial DNA, nonviable bacteria, intracellular bacteria

and even bacterial DNA circulating in the blood may be detected. Various studies have shown indeed that molecular methods reveal additional cases of probable bacteremia that remained undetected by blood culture (Laforgia et al., 1997; Wu et al., 2008; Fujimori et al., 2010). Besides convenience and rapidity a real-time format also offers the possibility to detect bacterial DNA in a quantitative manner. Evidence is accumulating that the bacterial DNA load is related to severity of disease and is likely to provide additional valuable information (Carrol et al., 2007; Darton et al., 2009; Peters et al., 2009). To conclude, the NeoSep-ID is an easy-to-use pathogen-specific assay that could enable a faster and accurate diagnosis of neonatal late onset sepsis as a supplementary to blood culture. With this tool clinical management could be optimized at an earlier stage improving outcome and reducing the use of broad-spectrum antibiotics in a high-risk group like preterm and VLBW infants. Acknowledgments We want to acknowledge Dr. A.C. Fluit from the Department of Medical Microbiology, University Medical Center Utrecht and Prof. Peter W.M. Hermans, Radboud University Nijmegen Medical Centre for the provision of bacterial strains. Part of this research was funded by ZonMw (project number 205 100 007) and Fonds Nuts Ohra (Grant number 1101-093). Part of this research was performed within the framework of the Center for Translational Molecular Medicine (CTMM) (http://www.ctmm.nl/pro1/general/home.asp), project MARS (grant 04I-\201) TK and MR were supported by Estonian Ministry of Education and Research (grant SF0180026s09) and EU ERDF (Grant number 3.2.0101.08-0011) through the Estonian Centre of Excellence in Genomics. Biocartis kindly provided the reagents BLB and NB2. The sponsors were not involved in the design and execution of the study. References Akane, A.,Matsubara, K.,Nakamura, H.,Takahashi, S.,Kimura, K., 1994. Identification of the heme compound copurified with deoxyribonucleic acid (DNA) from bloodstains, a major inhibitor of polymerase chain reaction (PCR) amplification. J. Forensic Sci. 39 (2), 362–372. Ashkenazi, S., 2004. Shigella infections in children: new insights. Semin. Pediatr. Infect. Dis. 15 (4), 246–252. Bahrdt, C., Krech, A.B.,Wurz, A.,Wulff, D., 2010. Validation of a newly developed hexaplex real-time PCR assay for screening for presence of GMOs in food, feed and seed. Anal. Bioanal. Chem. 396 (6), 2103–2112. Carrol, E.D.,Guiver, M.,Nkhoma, S.,Mankhambo, L.A.,Marsh, J.,Balmer, P., et al., 2007. High pneumococcal DNA loads are associated with mortality in Malawian children with invasive pneumococcal disease. Pediatr. Infect. Dis. J. 26 (5), 416–422. Chan, K.Y.Y.,Lam, H.S., Cheung, H.M., Chan, A.K.C., Li, K., Fok, T.F., et al., 2009. Rapid identification and differentiation of Gram-negative and Gram-positive bacterial bloodstream infections by quantitative polymerase chain reaction in preterm infants. Crit. Care Med. 37 (8), 2441–2447. Chang, S.S.,Hsu, H.L.,Cheng, J.C.,Tseng, C.P., 2011. An efficient strategy for broad-range detection of low abundance bacteria without DNA decontamination of PCR reagents. PLoS One 6 (5), e20303. Connell, T.G., Rele, M., Cowley, D., Buttery, J.P., Curtis, N., 2007. How reliable is a negative blood culture result? Volume of blood submitted for culture in routine practice in a children's hospital. Pediatrics 119 (5), 891–896. Corless, C.E., Guiver, M., Borrow, R., Edwards-Jones, V., Kaczmarski, E.B., Fox, A.J., 2000. Contamination and sensitivity issues with a real-time universal 16S rRNA PCR. J. Clin. Microbiol. 38 (5), 1747–1752. Darton, T.,Guiver, M.,Naylor, S.,Jack, D.L.,Kaczmarski, E.B.,Borrow, R., et al., 2009. Severity of meningococcal disease associated with genomic bacterial load. Clin. Infect. Dis. 48 (5), 587–594. Enomoto, M.,Morioka, I.,Morisawa, T.,Yokoyama, N.,Matsuo, M., 2009. A novel diagnostic tool for detecting neonatal infections using multiplex polymerase chain reaction. Neonatology 96 (2), 102–108. Espy, M.J., Uhl, J.R., Sloan, L.M., Buckwalter, S.P., Jones, M.F., Vetter, E.A., et al., 2006. Realtime PCR in clinical microbiology: applications for routine laboratory testing. Clin. Microbiol. Rev. 19 (1), 165–256. Fujimori, M.,Hisata, K.,Nagata, S.,Matsunaga, N.,Komatsu, M.,Shoji, H., et al., 2010. Efficacy of bacterial ribosomal RNA-targeted reverse transcription-quantitative PCR for detecting neonatal sepsis: a case control study. BMC Pediatr. 10, 53. Hornik, C.P.,Fort, P.,Clark, R.H.,Watt, K.,Benjamin Jr., D.K.,Smith, P.B., et al., 2012. Early and late onset sepsis in very-low-birth-weight infants from a large group of neonatal intensive care units. Early Hum. Dev. 88 (Suppl. 2), S69–S74.

M. van den Brand et al. / Journal of Microbiological Methods 106 (2014) 8–15 Huijsdens, X.W., Linskens, R.K., Mak, M., Meuwissen, S.G.M., Vandenbroucke-Grauls, C.M.J. E.,Savelkoul, P.H.M., 2002. Quantification of bacteria adherent to gastrointestinal mucosa by real-time PCR. J. Clin. Microbiol. 40 (12), 4423–4427. Jordan, J.A.,Durso, M.B., 2000. Comparison of 16S rRNA gene PCR and BACTEC 9240 for detection of neonatal bacteremia. J. Clin. Microbiol. 38 (7), 2574–2578. Jordan, J.A.,Durso, M.B., 2005. Real-time polymerase chain reaction for detecting bacterial DNA directly from blood of neonates being evaluated for sepsis. J. Mol. Diagn. 7 (5), 575–581. Jordan, J.A.,Butchko, A.R.,Durso, M.B., 2005. Use of pyrosequencing of 16S rRNA fragments to differentiate between bacteria responsible for neonatal sepsis. J. Mol. Diagn. 7 (1), 105–110. Jordan, J.A.,Durso, M.B.,Butchko, A.R.,Jones, J.G.,Brozanski, B.S., 2006. Evaluating the nearterm infant for early onset sepsis: progress and challenges to consider with 16S rDNA polymerase chain reaction testing. J. Mol. Diagn. 8 (3), 357–363. Ke, D., Menard, C., Picard, F.J., Boissinot, M., Ouellette, M., Roy, P.H., et al., 2000. Development of conventional and real-time PCR assays for the rapid detection of group B streptococci. Clin. Chem. 46 (3), 324–331. Klaassen, C.H., de Valk, H.A., Horrevorts, A.M., 2003. Clinical Staphylococcus aureus isolate negative for the Sa442 fragment. J. Clin. Microbiol. 41 (9), 4493. Klaschik, S., Lehmann, L.E., Raadts, A., Hoeft, A., Stuber, F., 2002. Comparison of different decontamination methods for reagents to detect low concentrations of bacterial 16S DNA by real-time-PCR. Mol. Biotechnol. 22 (3), 231–242. Koressaar, T.,Joers, K.,Remm, M., 2009. Automatic identification of species-specific repetitive DNA sequences and their utilization for detecting microbial organisms. Bioinformatics 25 (11), 1349–1355. Laforgia, N.,Coppola, B.,Carbone, R.,Grassi, A.,Mautone, A.,Iolascon, A., 1997. Rapid detection of neonatal sepsis using polymerase chain reaction. Acta Paediatr. 86 (10), 1097–1099. Lebovitz, E.E.,Burbelo, P.D., 2013. Commercial multiplex technologies for the microbiological diagnosis of sepsis. Mol. Diagn. Ther. 17 (4), 221–231. Martineau, F.,Picard, F.J.,Roy, P.H.,Ouellette, M.,Bergeron, M.G., 1998. Species-specific and ubiquitous-DNA-based assays for rapid identification of Staphylococcus aureus. J. Clin. Microbiol. 36 (3), 618–623. McCabe, K.M., Khan, G., Zhang, Y.H., Mason, E.O., McCabe, E.R., 1995. Amplification of bacterial DNA using highly conserved sequences: automated analysis and potential for molecular triage of sepsis. Pediatrics 95 (2), 165–169. Molenkamp, R.,van der Ham, A.,Schinkel, J.,Beld, M., 2007. Simultaneous detection of five different DNA targets by real-time Taqman PCR using the Roche LightCycler480: application in viral molecular diagnostics. J. Virol. Methods 141 (2), 205–211 (May). Paauw, A.,Caspers, M.P.,Schuren, F.H.,Leverstein-van Hall, M.A.,Deletoile, A.,Montijn, R.C., et al., 2008. Genomic diversity within the Enterobacter cloacae complex. PLoS One 3 (8), e3018. Pammi, M., Flores, A., Leeflang, M., Versalovic, J., 2011. Molecular assays in the diagnosis of neonatal sepsis: a systematic review and meta-analysis. Pediatrics 128 (4), e973–e985.

15

Peters, R.P.H., van Agtmael, M.A., Danner, S.A., Savelkoul, P.H.M., Vandenbroucke-Grauls, C.M.J.E., 2004a. New developments in the diagnosis of bloodstream infections. Lancet Infect. Dis. 4 (12), 751–760 (Dec). Peters, R.P.H., Mohammadi, T., Vandenbroucke-Grauls, C.M.J.E., Danner, S.A., van Agtmael, M.A., Savelkoul, P.H.M., 2004b. Detection of bacterial DNA in blood samples from febrile patients: underestimated infection or emerging contamination? FEMS Immunol. Med. Microbiol. 42 (2), 249–253. Peters, R.P.H., van Agtmael, M.A., Gierveld, S., Danner, S.A., Groeneveld, A.B.J., Vandenbroucke-Grauls, C.M.J.E., et al., 2007. Quantitative detection of Staphylococcus aureus and Enterococcus faecalis DNA in blood to diagnose bacteremia in patients in the intensive care unit. J. Clin. Microbiol. 45 (11), 3641–3646. Peters, R.P.H.,de Boer, R.F.,Schuurman, T.,Gierveld, S.,Kooistra-Smid, M.,van Agtmael, M.A., et al., 2009. Streptococcus pneumoniae DNA load in blood as a marker of infection in patients with community-acquired pneumonia. J. Clin. Microbiol. 47 (10), 3308–3312. Pierce, J.R., Merenstein, G.B., Stocker, J.T., 1984. Immediate postmortem cultures in an intensive care nursery. Pediatr. Infect. Dis. 3 (6), 510–513. Rood, I.G., Pettersson, A., Savelkoul, P.H.,de Korte, D., 2011. Performance and suitability of polymerase chain reaction for early detection of bacteria in platelet concentrates. Transfusion 51 (9), 2006–2011. Santo Domingo, J.W., Siefring, S.C., Haugland, R.A., 2003. Real-time PCR method to detect Enterococcus faecalis in water. Biotechnol. Lett. 25 (3), 261–265. Schelonka, R.L., Chai, M.K., Yoder, B.A., Hensley, D., Brockett, R.M., Ascher, D.P., 1996. Volume of blood required to detect common neonatal pathogens. J. Pediatr. 129 (2), 275–278. Squire, E., Favara, B., Todd, J., 1979. Diagnosis of neonatal bacterial infection: hematologic and pathologic findings in fatal and nonfatal cases. Pediatrics 64 (1), 60–64. Stoll, B.J., Gordon, T., Korones, S.B., Shankaran, S., Tyson, J.E., Bauer, C.R., et al., 1996. Lateonset sepsis in very low birth weight neonates: a report from the National Institute of Child Health and Human Development Neonatal Research Network. J. Pediatr. 129 (1), 63–71. Stoll, B.J., Hansen, N., Fanaroff, A.A., Wright, L.L., Carlo, W.A., Ehrenkranz, R.A., et al., 2002. Late-onset sepsis in very low birth weight neonates: the experience of the NICHD Neonatal Research Network. Pediatrics 110 (2 Pt 1), 285–291. Van den Hoogen, A., Gerards, L.J., Verboon-Maciolek, M.A., Fleer, A., Krediet, T.G., 2010. Long-term trends in the epidemiology of neonatal sepsis and antibiotic susceptibility of causative agents. Neonatology 97 (1), 22–28. Van Doornum, G.J., Guldemeester, J., Osterhaus, A.D., Niesters, H.G., 2003. Diagnosing herpesvirus infections by real-time amplification and rapid culture. J. Clin. Microbiol. 41 (2), 576–580. Wittwer, C.T., Herrmann, M.G., Gundry, C.N., Elenitoba-Johnson, K.S., 2001. Real-time multiplex PCR assays. Methods 25 (4), 430–442. Wu, Y.D.,Chen, L.H.,Wu, X.J., Shang, S.Q., Lou, J.T.,Du, L.Z., et al., 2008. Gram stain-specificprobe-based real-time PCR for diagnosis and discrimination of bacterial neonatal sepsis. J. Clin. Microbiol. 46 (8), 2613–2619.