Voorzitter Raad van Bestuur; hoogleraar 'Patiëntveiligheid', Vrije Universiteit / Amsterdam UMC
Publicatie
The reliability of adverse events determination with patient record review does nort improve with additional reviewers.
Zegers, M., Bruyne, M.C. de, Wagner, C., Wal, G. van der, Groenewegen, P.P. The reliability of adverse events determination with patient record review does nort improve with additional reviewers. In: Book of abstracts. International Conference 'Patient Safety Research shaping the European Agenda', 24-26 september 2007, Porto. p. 159.
ABSTRACT
Introduction
In 2005, as first part of the Dutch Patient Safety Research Program, a study was initiated to determine the incidence, nature, type and impact of adverse events among hospitalised patients in the Netherlands. The method and instruments were based on a protocol originally developed by the Harvard Medica! Practice Study, which has studied the incidence of adverse events in New York state hospitals in 1984, based on analysis of information in patient records. This protocol, with modifications, was used in subsequent studies in Australia, the United Kingdom, New Zealand, the United States, Denmark, France and Canada. The protocol and instruments used in the Dutch study are based on the same protocol and instruments. However, in our pilot study we found moderate to poor interrater reliability for adverse events determination and their preventability. Also in the previous international studies inter-rater reliability appeared to be a major problem. Therefore, standing on the shoulders of our predecessors and keeping the method and instruments maximally comparable, we have tried to improve the reliability of the patient record review process for measuring (preventable) adverse events. In this study the reliability of the assessment of adverse events in our patient record review study is measured.
Methods
The study is a three-stage retrospective patient record study in which 7926 patient records of discharged or deceased patients in 2004 in 21 hospitals were reviewed. The records of randomly selected admissions were reviewed by nurses and physicians between August 2005 and October 2006. In the first stage, nurses reviewed records for the presence of triggers known to be sensitive to the occurrence of adverse events. To assess the reliability of the review process between nurses, a random sample of 5% of the records were independently reviewed by two nurses. In the second stage, the records screened positively by nurses were independently reviewed by two physicians for adverse event determination and their preventability. If the physicians disagreed they started a consensus procedure. If the consensus procedure failed a third reviewer gave a final judgment using all available review information. A random sample of 120 admissions was independently reviewed by a second pair of physicians followed by a third phase if applicable. The agreement in the second stage was measured for presence and preventability of adverse events. The inter-rater reliability was expressed as a kappa statistic with 95% confidence interval and as percentage of records for which there was agreement.
Discussion
The results of our study are still confidential and will be made public in April 2007. Preliminary results show that the kappa value for the first stage of the review process by nurses is good. This is equivalent to reliability assessed in studies using similar methodology. Kappa scores for agreement in the second stage by four physicians were fair. In contrast, the kappa statistic for reliability between the first to physician reviewers were good. The consensus procedure did improve standardisation within pairs of physicians, but not between pairs of physicians. Despite our efforts to improve the inter reviewer variation of the review process for adverse events determination, the kappa values of the second stage review process were not better than in other studies, which were fair to moderate.
This may be explained by several factors. Firstly, the variation in diagnosis and treatment of the included admissions is large and a gold standard on good clinical practice is often lacking. Especially because many patient suffer from multiple or complex diseases and need complex treatment. So even if the standardised review process was perfectly applied by al reviewers, there may still be disagreement about presence of adverse events. Secondly, the information in the patient records on which the review is based was not always complete and adequate for a good adverse events determination.
Introduction
In 2005, as first part of the Dutch Patient Safety Research Program, a study was initiated to determine the incidence, nature, type and impact of adverse events among hospitalised patients in the Netherlands. The method and instruments were based on a protocol originally developed by the Harvard Medica! Practice Study, which has studied the incidence of adverse events in New York state hospitals in 1984, based on analysis of information in patient records. This protocol, with modifications, was used in subsequent studies in Australia, the United Kingdom, New Zealand, the United States, Denmark, France and Canada. The protocol and instruments used in the Dutch study are based on the same protocol and instruments. However, in our pilot study we found moderate to poor interrater reliability for adverse events determination and their preventability. Also in the previous international studies inter-rater reliability appeared to be a major problem. Therefore, standing on the shoulders of our predecessors and keeping the method and instruments maximally comparable, we have tried to improve the reliability of the patient record review process for measuring (preventable) adverse events. In this study the reliability of the assessment of adverse events in our patient record review study is measured.
Methods
The study is a three-stage retrospective patient record study in which 7926 patient records of discharged or deceased patients in 2004 in 21 hospitals were reviewed. The records of randomly selected admissions were reviewed by nurses and physicians between August 2005 and October 2006. In the first stage, nurses reviewed records for the presence of triggers known to be sensitive to the occurrence of adverse events. To assess the reliability of the review process between nurses, a random sample of 5% of the records were independently reviewed by two nurses. In the second stage, the records screened positively by nurses were independently reviewed by two physicians for adverse event determination and their preventability. If the physicians disagreed they started a consensus procedure. If the consensus procedure failed a third reviewer gave a final judgment using all available review information. A random sample of 120 admissions was independently reviewed by a second pair of physicians followed by a third phase if applicable. The agreement in the second stage was measured for presence and preventability of adverse events. The inter-rater reliability was expressed as a kappa statistic with 95% confidence interval and as percentage of records for which there was agreement.
Discussion
The results of our study are still confidential and will be made public in April 2007. Preliminary results show that the kappa value for the first stage of the review process by nurses is good. This is equivalent to reliability assessed in studies using similar methodology. Kappa scores for agreement in the second stage by four physicians were fair. In contrast, the kappa statistic for reliability between the first to physician reviewers were good. The consensus procedure did improve standardisation within pairs of physicians, but not between pairs of physicians. Despite our efforts to improve the inter reviewer variation of the review process for adverse events determination, the kappa values of the second stage review process were not better than in other studies, which were fair to moderate.
This may be explained by several factors. Firstly, the variation in diagnosis and treatment of the included admissions is large and a gold standard on good clinical practice is often lacking. Especially because many patient suffer from multiple or complex diseases and need complex treatment. So even if the standardised review process was perfectly applied by al reviewers, there may still be disagreement about presence of adverse events. Secondly, the information in the patient records on which the review is based was not always complete and adequate for a good adverse events determination.
ABSTRACT
Introduction
In 2005, as first part of the Dutch Patient Safety Research Program, a study was initiated to determine the incidence, nature, type and impact of adverse events among hospitalised patients in the Netherlands. The method and instruments were based on a protocol originally developed by the Harvard Medica! Practice Study, which has studied the incidence of adverse events in New York state hospitals in 1984, based on analysis of information in patient records. This protocol, with modifications, was used in subsequent studies in Australia, the United Kingdom, New Zealand, the United States, Denmark, France and Canada. The protocol and instruments used in the Dutch study are based on the same protocol and instruments. However, in our pilot study we found moderate to poor interrater reliability for adverse events determination and their preventability. Also in the previous international studies inter-rater reliability appeared to be a major problem. Therefore, standing on the shoulders of our predecessors and keeping the method and instruments maximally comparable, we have tried to improve the reliability of the patient record review process for measuring (preventable) adverse events. In this study the reliability of the assessment of adverse events in our patient record review study is measured.
Methods
The study is a three-stage retrospective patient record study in which 7926 patient records of discharged or deceased patients in 2004 in 21 hospitals were reviewed. The records of randomly selected admissions were reviewed by nurses and physicians between August 2005 and October 2006. In the first stage, nurses reviewed records for the presence of triggers known to be sensitive to the occurrence of adverse events. To assess the reliability of the review process between nurses, a random sample of 5% of the records were independently reviewed by two nurses. In the second stage, the records screened positively by nurses were independently reviewed by two physicians for adverse event determination and their preventability. If the physicians disagreed they started a consensus procedure. If the consensus procedure failed a third reviewer gave a final judgment using all available review information. A random sample of 120 admissions was independently reviewed by a second pair of physicians followed by a third phase if applicable. The agreement in the second stage was measured for presence and preventability of adverse events. The inter-rater reliability was expressed as a kappa statistic with 95% confidence interval and as percentage of records for which there was agreement.
Discussion
The results of our study are still confidential and will be made public in April 2007. Preliminary results show that the kappa value for the first stage of the review process by nurses is good. This is equivalent to reliability assessed in studies using similar methodology. Kappa scores for agreement in the second stage by four physicians were fair. In contrast, the kappa statistic for reliability between the first to physician reviewers were good. The consensus procedure did improve standardisation within pairs of physicians, but not between pairs of physicians. Despite our efforts to improve the inter reviewer variation of the review process for adverse events determination, the kappa values of the second stage review process were not better than in other studies, which were fair to moderate.
This may be explained by several factors. Firstly, the variation in diagnosis and treatment of the included admissions is large and a gold standard on good clinical practice is often lacking. Especially because many patient suffer from multiple or complex diseases and need complex treatment. So even if the standardised review process was perfectly applied by al reviewers, there may still be disagreement about presence of adverse events. Secondly, the information in the patient records on which the review is based was not always complete and adequate for a good adverse events determination.
Introduction
In 2005, as first part of the Dutch Patient Safety Research Program, a study was initiated to determine the incidence, nature, type and impact of adverse events among hospitalised patients in the Netherlands. The method and instruments were based on a protocol originally developed by the Harvard Medica! Practice Study, which has studied the incidence of adverse events in New York state hospitals in 1984, based on analysis of information in patient records. This protocol, with modifications, was used in subsequent studies in Australia, the United Kingdom, New Zealand, the United States, Denmark, France and Canada. The protocol and instruments used in the Dutch study are based on the same protocol and instruments. However, in our pilot study we found moderate to poor interrater reliability for adverse events determination and their preventability. Also in the previous international studies inter-rater reliability appeared to be a major problem. Therefore, standing on the shoulders of our predecessors and keeping the method and instruments maximally comparable, we have tried to improve the reliability of the patient record review process for measuring (preventable) adverse events. In this study the reliability of the assessment of adverse events in our patient record review study is measured.
Methods
The study is a three-stage retrospective patient record study in which 7926 patient records of discharged or deceased patients in 2004 in 21 hospitals were reviewed. The records of randomly selected admissions were reviewed by nurses and physicians between August 2005 and October 2006. In the first stage, nurses reviewed records for the presence of triggers known to be sensitive to the occurrence of adverse events. To assess the reliability of the review process between nurses, a random sample of 5% of the records were independently reviewed by two nurses. In the second stage, the records screened positively by nurses were independently reviewed by two physicians for adverse event determination and their preventability. If the physicians disagreed they started a consensus procedure. If the consensus procedure failed a third reviewer gave a final judgment using all available review information. A random sample of 120 admissions was independently reviewed by a second pair of physicians followed by a third phase if applicable. The agreement in the second stage was measured for presence and preventability of adverse events. The inter-rater reliability was expressed as a kappa statistic with 95% confidence interval and as percentage of records for which there was agreement.
Discussion
The results of our study are still confidential and will be made public in April 2007. Preliminary results show that the kappa value for the first stage of the review process by nurses is good. This is equivalent to reliability assessed in studies using similar methodology. Kappa scores for agreement in the second stage by four physicians were fair. In contrast, the kappa statistic for reliability between the first to physician reviewers were good. The consensus procedure did improve standardisation within pairs of physicians, but not between pairs of physicians. Despite our efforts to improve the inter reviewer variation of the review process for adverse events determination, the kappa values of the second stage review process were not better than in other studies, which were fair to moderate.
This may be explained by several factors. Firstly, the variation in diagnosis and treatment of the included admissions is large and a gold standard on good clinical practice is often lacking. Especially because many patient suffer from multiple or complex diseases and need complex treatment. So even if the standardised review process was perfectly applied by al reviewers, there may still be disagreement about presence of adverse events. Secondly, the information in the patient records on which the review is based was not always complete and adequate for a good adverse events determination.