HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Figures Only Full Text (PDF) Services Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (14) Citing Articles via Google Scholar Google Scholar Articles by Montenegro, R. Articles by Tonetti, M. Search for Related Content PubMed PubMed Citation Articles by Montenegro, R. Articles by Tonetti, M.

Quality of RCTs in Periodontology— A Systematic Review

¹ Departments of Periodontology and
² Oral Pathology, Eastman Dental Institute for Oral Health Care Sciences, University College London, 256 Gray's Inn Road, London WC1X 8LD, UK;

*corresponding author, i.needleman{at}eastman.ucl.ac.uk

	ABSTRACT

TOP ABSTRACT INTRODUCTION MATERIALS & METHODS RESULTS DISCUSSION REFERENCES

 
Randomization, concealment of treatment allocation, blinding, and patient follow-up are key quality components of randomized controlled trials (RCTs). The objective of this study was to assess the quality of RCTs in periodontology using these evidence-based components. Following a detailed search, screening and quality assessments of RCTs were conducted in duplicate and independently. The results showed that although 91% of trials were described as randomised, adequate methods for randomization and allocation concealment were found in 17% and 7% of studies, respectively. Blinding was adequate for the caregiver in 17% and for the examiner in 55% of studies. A clear accounting of all participants was present in 56% of reports. This rigorous systematic review revealed that the quality of RCTs in periodontology, judged by their publications, frequently does not meet recommended standards. If this quality is reflected in actual study conduct, fundamental errors could have a significant impact on the outcomes of these trials.

KEY WORDS: systematic review • clinical trials • periodontal diseases

	INTRODUCTION

TOP ABSTRACT INTRODUCTION MATERIALS & METHODS RESULTS DISCUSSION REFERENCES

 
The randomized controlled trial (RCT) provides the highest level of evidence for medical interventions (Sackett et al., 2000). The elements inherent in a carefully designed and conducted RCT will minimize bias, balance confounders, and hence produce the most reliable estimate of treatment effect. However, unraveling which components are most significant to trial quality is a recent development. Observational studies show these to be; randomization (the generation of a true randomization sequence [Roberts and Torgerson, 1998]), allocation concealment (concealing the randomization sequence during patient recruitment (Torgerson and Roberts, 1999), blinding (Day and Altman, 2000), and handling of withdrawals and losses to follow-up (May et al., 1981).

A meta-analysis of these observational studies showed that improper allocation concealment produced the largest impact on size of treatment effect (Juni et al., 2001b). Treatment was found to be 30% more beneficial if concealment was inadequate or unclear (combined odds ratio 0.70; 95% CI, 0.62, 0.80). While overall, an inadequate method for generating a random sequence did not produce an increased estimate, one study (Kjaergard et al., 1999) demonstrated a 51% greater treatment effect (OR 0.49; 95% CI, 0.3, 0.81). In addition, where allocation concealment was adequate but the method used to generate the randomization sequence was not, the treatment effect was exaggerated compared with adequate methods. In other words, with a sequence that is not truly random, the allocation of the treatment group for the next individual could be predicted, even if it is properly concealed.

Blinding is less straightforward to investigate, partly because it is dependent on the objectivity of the outcome assessed (e.g., tooth loss vs. gingival redness), and partly because it is not always achievable. The impact of blinding within the meta-analysis was variable, although three of the four included studies were in the direction of exaggerating the outcome. The fourth study showed the widest confidence interval and a negative estimate of effect (Moher et al., 1998), although it was based mainly on trials with objective outcomes such as mortality, where examiner blinding is irrelevant. Evidence that inadequate blinding can lead to larger treatment effects is also available from other studies (Colditz et al., 1989). Only two studies addressed the effect of losses to follow-up, and neither showed a significant effect of inadequate methodology on size of treatment outcome. Unfortunately, both studies used inadequate methods to assess patient attrition, which might explain their lack of effect (Juni et al., 2001b). Evidence for bias from incomplete follow-up is, however, available elsewhere (May et al., 1981). It should be noted that these studies are observational and cannot provide definitive evidence of effect. However, the consistency in direction of effect is notable, and a review of results provides compelling evidence for their importance to study quality.

Quality assessment of trials in periodontology has been reported before (Hayes et al., 1992; Antczak-Bouckoms, 1993; Knoll-Kohler, 1999). However, the use of checklists and scales as reported by these authors has become discouraged, partly because many of the included items have no evidence of an effect and partly because estimated quality can vary widely depending on which scale is used (Moher et al., 1996; Juni et al., 1999).

In summary, current best evidence highlights the importance of certain recognized aspects of trial quality. Assessing these components is now recommended in quality appraisal of clinical trials (Cochrane Reviewers Handbook, 2001; Juni et al., 2001a; NHS Centre for Reviews and Dissemination, 2001). Our objective was systematically to assess the quality of RCTs in periodontology using evidence-based criteria shown to have an impact on the size of treatment outcome, namely, randomization, allocation concealment, blinding, and handling of withdrawals.

	MATERIALS & METHODS

TOP ABSTRACT INTRODUCTION MATERIALS & METHODS RESULTS DISCUSSION REFERENCES

 
Protocol Development
This was a systematic review of randomized controlled trials in periodontology addressing intervention studies in humans. The first phase involved development of the review protocol detailing all methods, including: aim, search, eligibility criteria for study inclusion, methods for screening and data abstraction, and data analysis.

Search Strategy
A search of the Cochrane Oral Health Group (COHG) specialized register of randomized controlled trials of interventions (therapy) was conducted. This was limited to the Journal of Clinical Periodontology, Journal of Periodontology, and Journal of Periodontal Research from 1996-1998. These journals were selected as a convenience sample, since we were aware that the COHG register included both records from searching of electronic databases such as MEDLINE as well as complete hand-searching of these journals. This database should therefore provide as complete a record of RCTs in periodontology as could be found. The three-year period was selected to provide an adequate number of trials for analysis, and the results were pooled for the three years and for all journals. For searching the COHG register, the search terms "randomized controlled trial" and "RCT" were used. The type of study was limited to randomized controlled trials of interventions (therapy) and conducted on humans.

Quality Assessment
Quality of individual components was assessed. A key to definitions was derived from two guidelines for systematic reviews (Cochrane Reviewers' Handbook, 2001, and NHS Centre for Reviews and Dissemination, 2001):

• Randomisation: Adequate if generated by random number table (computer-generated or not), tossed coin, and shuffled cards. Unclear if the study refers to randomization but either does not adequately explain the method or no method was reported. Inadequate randomization methods include alternate assignment, hospital number, and odd/even birth date.
  
• Adequate allocation concealment methods included central randomization (e.g., by telephone to a pharmacy or trial office), pharmacy sequentially numbered/coded containers, and sequentially numbered, opaque envelopes. Concealment was unclear if the study referred to allocation concealment but either did not adequately explain the method or no method was reported. Inadequate concealment involved methods where randomization could not be concealed, such as alternate assignment, hospital number, and odd/even birth date.
  
• Blinding of patient, caregiver, and examiner was considered separately. These were recorded as adequate, inadequate, unclear, or, for examiner blinding, not applicable if the study design precluded the possibility of blinding. Handling of withdrawals and drop-outs was assessed by analysis of whether all patients who entered the trial were properly accounted for at the end. Where drop-outs occurred, the use of analyses to allow for losses (such as intention to treat) was noted.
  

Pilot Study
The pilot study consisted of two aspects: First, the lead reviewer (RM) was calibrated against another reviewer (IN) with experience in conducting systematic reviews. Each round of calibration consisted of duplicate, independent quality assessment of five publications previously used in another systematic review. After four rounds of calibration, a consistent level of agreement was found (unweighted kappa scores from 1st to 4th exercise: 0.55, 0.71, 1.0, 1.0). Second, all screening and data abstraction forms were piloted among all four reviewers for clarity and ease of use, and agreement re-checked. Following piloting, minor changes of forms were recommended, with excellent agreement at the fourth exercise (kappa 1.0).

Screening and Data Abstraction
Following completion of the search, all titles and abstracts were screened for possible inclusion in the study. Screening was based on the study being an RCT, that it was conducted on humans, that it was a treatment study, and that a full journal publication was available. Studies were excluded only if the screener could be sure that it did not fulfill the criteria. For all remaining RCTs, the full text of the trial was obtained for screening according to the same criteria. Study quality data were extracted for all eligible studies. Screening and quality evaluation were not performed blind to author/affiliation of study, since this has not been shown to result in meaningful differences (Moher et al., 1999).

Screening of titles and abstracts was conducted in duplicate by two reviewers independently (RM & IN). Agreement was measured and Kappa scores calculated. Disagreement was resolved by subsequent discussion. Screening and data abstraction of the full-text articles were also performed in duplicate. One reviewer screened all reports (RM) and three other reviewers were allocated samples of studies. Agreement was measured as before and disagreement resolved by discussion. One reviewer performed computer data entry, and analysis consisted of descriptive statistics for prevalence of quality components by means of SPSS Version 10.0 (SPSS Inc., Chicago, IL, USA).

	RESULTS

TOP ABSTRACT INTRODUCTION MATERIALS & METHODS RESULTS DISCUSSION REFERENCES

 
Search
The search revealed 283 articles for screening. The distribution among journals was: Journal of Clinical Periodontology, 155 (54.8%); Journal of Periodontology, 126 (44.5%); and Journal of Periodontal Research, 2 (0.7%). The flow of articles through the screening process is given in the Fig. Reasons for study exclusion are given in Table 1.

View larger version (19K): [in this window] [in a new window]   Figure. Flow of articles (number) through review.

View this table: [in this window] [in a new window]   Table 2. Inter-examiner Agreement for Study Screening and Quality Assessment ( scores)

Approximately half of the studies (56%) demonstrated proper accounting for all participants at the end of the trial. For the rest, accounting of participants was absent in 14% of studies and impossible to determine for 30% of trials. The use of analyses to take into account withdrawals (drop-outs, losses to follow-up, excluded patients) was not applicable for 43% of studies where follow-up was deemed complete. For the rest, only 11% of analyses took withdrawals into account, although a larger proportion (56%) provided inadequate information to determine whether this had been done.

	DISCUSSION

TOP ABSTRACT INTRODUCTION MATERIALS & METHODS RESULTS DISCUSSION REFERENCES

 
This systematic review has demonstrated that RCTs in periodontology frequently do not meet current recommendations on study quality. The randomization method could be judged adequate in only 17% of trials and the concealment of the allocation sequence adequate in 7%. While examiner blinding was more frequently reported (55%), both patient and caregiver blinding were not commonly used. We were surprised how often the accounting of patient follow-up was not given or unclear (44%). It is also evident that few analyses currently attempt to assess the impact of patients lost to follow-up. However, it is important to point out that such an analysis is only feasible due to the large numbers of RCTs published in periodontology compared with other dental specialties (Niederman et al., 2002). Attention to aspects of design and reporting will help to enhance the impact of this specialty in oral health care.

Several components of quality assurance were included in this review, since the possibility of reviewer bias must be minimized. Thus, screening of abstracts/titles and full-text articles was always performed in duplicate and independently. The same approach was taken for assessment of study quality. In this way, the possibility of bias from one reviewer has been reduced. In addition, the level of agreement was checked for each of these stages and is reported above. The level of agreement was very good, despite the frequent difficulty in finding data in trial reports. We attribute this level of agreement to several aspects. First, reviewers were carefully trained in the methods of the review. Second, piloting of the screening and data abstraction forms reduced ambiguities in language. Third, a clear key to categorizing the quality components was designed and provided to all reviewers.

It must be remembered that, in this study (and others in medicine), we have used the trial publication as a surrogate for actual study conduct. There may be reasons why this could be inaccurate. Most likely, authors may omit important data or be unaware of the need to include them. This could be particularly true for randomization, where researchers tend not to separate this process into (1) generation of the random sequence and (2) the concealment of the sequence (Roberts and Torgerson, 1998). Assessment of trial protocols or ethics committee submissions might provide a means of assessing this, although they too may lack detail. Contact with the original study authors to clarify unclear methods is a further method to check protocol details. We did not undertake this approach because of the numbers of trials involved in this review, and the conclusions should reflect this limitation. However, there is compelling evidence from two studies that poor reporting does reflect poor study quality (Liberati et al., 1986; Schulz et al., 1995). In one, the impact of inadequate allocation concealment was substantial, whether the methods were clearly inadequate or whether the reviewers were uncertain due to insufficient information (OR exaggerated by 41% and 30%, respectively (Schulz et al., 1995).

This study did not attempt to quantify the impact of trial quality on the outcome of clinical research, although this will be the subject of a future project. However, if we accept the findings from studies in other fields in medicine, we can speculate on the possible findings of such research. In the studies presented above, inadequate allocation concealment had the greatest potential to exaggerate the size of the treatment effect. In this study, adequate allocation concealment was found in only 7% of RCTs in periodontology. It is likely, therefore, that the effect of selection bias introduced by inadequate allocation concealment could engender an important overestimation of treatment effects in these trials. Furthermore, ascertainment bias (systematic distortion of results by knowledge of the intervention received by participants) could be introduced at several stages (Jadad, 1998). Blinding aims to minimize this bias, and the results of this study showed a poor level of blinding of the examiner and, particularly, the caregiver. Outcomes typically assessed in periodontology, such as probing changes, may be particularly prone to this type of bias. Therefore, an exaggeration of treatment effect is a likely consequence. While caregiver blinding was assessed only where we judged that it could be achieved, no such distinction was made for examiner blinding, and this might have underestimated the occurrence of appropriate examiner blinding.

With regard to bias from missing patient data (losses to follow-up and withdrawals), the current recommendation is that statistical analyses model the impact of missing data with intention-to-treat or worst-case-scenario approaches. It is clear that few studies in periodontology have attempted to examine this effect. As with the other quality components, failure to investigate this bias may lead to an inaccurate estimate of treatment effect. In experimental therapeutic studies, this could again lead to overestimation of treatment effect, since patients with adverse or poor outcomes might preferentially be lost to the study (Jadad, 1998).

Incomplete reporting of trials has been a common finding in medicine. To tackle this problem, investigators have proposed the CONSORT statement, which has recently been revised and updated (Moher et al., 2001b). CONSORT provides authors with a checklist and flow diagram to help improve the quality of reports of randomized controlled trials. The aim is to facilitate the appraisal of trial quality, although a further outcome might be an improvement in future study design. A recent study of the quality of reports before and after adoption of the CONSORT statement, and in comparison with those in a non-adopting journal, indicated a significant increase in reporting quality (Moher et al., 2001a). So far, the British Dental Journal (Needleman, 1999) and the Journal of Orthodontics are the only oral health journals to adopt this guideline. This study suggests the need for wider acceptance of CONSORT by oral health journals.

In conclusion, within the limitations of this study, we have shown that greater attention to quality aspects of design and reporting of RCTs in periodontology is needed, and it is likely that other fields in dentistry are similarly affected. Since many of these aspects of quality are likely to have an impact on the size of the reported treatment effect, the dental research community should view these results with concern. Further research is needed to determine the impact of quality on treatment outcomes in periodontology. Better reporting of clinical trials in periodontology is needed, and the adoption of the CONSORT statement should be a first step for many dental journals.

	ACKNOWLEDGMENTS

 
We are grateful to the Cochrane Oral Health Group for help with searching and for access to the specialized trials register. No external support was received for this study. University College London paid salaries to IGN, DM, and MT.

Received April 3, 2002; Last revision July 24, 2002; Accepted September 6, 2002

	REFERENCES

TOP ABSTRACT INTRODUCTION MATERIALS & METHODS RESULTS DISCUSSION REFERENCES

 
Antczak-Bouckoms A (1993). Meta-analysis of clinical trials in periodontal research. Periodontol 20002:140–149.

Cochrane Reviewers' Handbook (2001). The Cochrane Library. Clarke M, Oxman AD, editors. Oxford, UK: Update Software.

Colditz GA, Miller JN, Mosteller F (1989). How study design affects outcomes in comparisons of therapy. I: Medical. Stat Med 8:441–454.[Medline]

Day SJ, Altman DG (2000). Statistics notes: blinding in clinical trials and other studies. BMJ 321:504.[Free Full Text]

Hayes C, Antczak-Bouckoms A, Burdick E (1992). Quality assessment and meta-analysis of systemic tetracycline use in chronic adult periodontitis. J Clin Periodontol 19:164–168.[Medline]

Jadad AR (1998). Randomised controlled trials. London: BMJ Books.

Juni P, Witschi A, Bloch R, Egger M (1999). The hazards of scoring the quality of clinical trials for meta-analysis. J Am Med Assoc 282:1054–1060.[Abstract/Free Full Text]

Juni P, Altman DG, Egger M (2001a). Assessing the quality of randomised controlled trials. In: Systematic reviews in health care. Meta-analysis in context. Egger M, Davey SG, Altman DG, editors. London: BMJ Books, pp. 87-108.

Juni P, Altman DG, Egger M (2001b). Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 323:42–46.[Free Full Text]

Kjaergard LL, Villumsen J, Gluud C (1999). Quality of randomised clinical trials affects estimates of intervention efficacy. In: Proceedings of the 7th Cochrane colloquium, Universita St Tommaso D'Aquino. Rome: Centro Cochrane Italiano, Publishers, p. 57.

Knoll-Kohler E (1999). Metronidazole dental gel as an alternative to scaling and root planing in the treatment of localised adult periodontitis. Is its efficacy proved? Eur J Oral Sci 107:415–421.[Medline]

Liberati A, Himel HN, Chalmers TC (1986). A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 4:942–951.[Abstract/Free Full Text]

May GS, DeMets DL, Friedman LM, Furberg C, Passamani E (1981). The randomized clinical trial: bias in analysis. Circulation 64:669–673.[Abstract/Free Full Text]

Moher D, Jadad AR, Tugwell P (1996). Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 12:195–208.[Medline]

Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609–613.[Medline]

Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. (1999). Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess 3:1–98.[Medline]

Moher D, Jones A, Lepage L (2001a). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. J Am Med Assoc 285:1992–1995.[Abstract/Free Full Text]

Moher D, Schulz KF, Altman DG (2001b). The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials http://www.consort-statement.org/.

Needleman IG (1999). CONSORT. Consolidated Standards of Reporting Trials. Br Dent J 186:207.[Medline]

NHS Centre for Reviews and Dissemination (2001). Undertaking systematic reviews of research on effectiveness. CRD's guidance for those carrying out or commissioning reviews. CRD Report Number 4. Khan KS, ter Riet G, Glanville J, Sowden AJ, Kleijnen J, editors. York: University of York, http://www.york.ac.uk/inst/crd/report4.htm

Niederman R, Chen L, Murzyn L, Conway S (2002). Benchmarking the dental randomised controlled literature on MEDLINE. Evidence-Based Dentistry 3:5–9.

Roberts C, Torgerson D (1998). Randomisation methods in controlled trials. BMJ 317:1301–1310.[Free Full Text]

Sackett DL, Straus S, Richardson S, Rosenberg W, Haynes RB (2000). Evidence-based medicine: how to practice and teach EBM. 2nd ed. London: Churchill Livingstone.

Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J Am Med Assoc 273:408–412.[Abstract]

Torgerson DJ, Roberts C (1999). Understanding controlled trials. Randomisation methods: concealment. BMJ 319:375–376.[Free Full Text]

This article has been cited by other articles:

				D.R. Moles, I.G. Needleman, R. Niederman, and J. Lau Introduction to Cumulative Meta-analysis in Dentistry: Lessons Learned from Undertaking a Cumulative Meta-analysis in Periodontology J. Dent. Res., April 1, 2005; 84(4): 345 - 349. [Abstract] [Full Text] [PDF]

				Y.-K. Tu, A. Blance, V. Clerehugh, and M.S. Gilthorpe Statistical Power for Analyses of Changes in Randomized Controlled Trials J. Dent. Res., March 1, 2005; 84(3): 283 - 287. [Abstract] [Full Text] [PDF]

This Article Abstract Figures Only Full Text (PDF) Services Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (14) Citing Articles via Google Scholar Google Scholar Articles by Montenegro, R. Articles by Tonetti, M. Search for Related Content PubMed PubMed Citation Articles by Montenegro, R. Articles by Tonetti, M.