Measuring Patient Safety Events: Opportunities and Challenges By: Amy K. Rosen, PhD, Qi Chen, MD, PhD

NQMC Update Service: June 13, 2016

Expert Commentary

• Measuring Patient Safety Events: Opportunities and Challenges
  By: Amy K. Rosen, PhD, Qi Chen, MD, PhD

June 13, 2016

 

Measuring Patient Safety Events: Opportunities and Challenges

By: Amy K. Rosen, PhD, Qi Chen, MD, PhD

Increased pressure for public accountability and transparency makes efforts to improve and advance the measurement of patient safety even more critical than ever. Despite consensus that the related science is slowly maturing (1), most measurement methods are still evolving and several challenges remain.

First, although national and internal mandates to track, measure, and report hospital and provider performance on a multitude of patient safety measures have resulted in a growing number of quality metrics, the use of externally mandated efforts for measurement may do little to improve overall hospital performance. The profusion of publicly reported events may actually divert attention and resources from areas that may need improvement to ones in which hospitals are rated or financially penalized based on their performance. Further, there has been little movement on which of these metrics offer the best validity, reliability, and usability for public reporting and other safety-related purposes (1). Thus, publicly reported measures may not provide a more accurate estimate of safety than measures that are selected internally based on local priorities (2).

The second challenge facing safety measurement is that most approaches primarily focus on the retrospective reporting of patient safety events. Commonly used methods to measure a safety or "adverse event" in both clinical practice and research studies include incident reporting, administrative data screening, medical chart review, malpractice claims, and automated surveillance (via trigger tools—i.e., tools that use surveillance rules or algorithms derived from clinical logic to flag patient medical records for the presence of an adverse event). Progress in safety measurement using these methods has been limited by many factors, including the lack of an existing taxonomy and meaningful framework, lack of systematic or standardized measures, infrequent occurrences, inconsistencies in data elements across healthcare systems, and the lack of scientifically sound measures (3). While some of these methods can be used to prospectively capture real-time events, the infrastructure, technology, and resources necessary to carry this out are frequently unavailable. For example, trigger tools can be used prospectively to identify patients at high risk of adverse events; however, they have not yet evolved much beyond the research environment into real-world practice (4). Third, controversy exists over whether to use global measures to detect safety events (e.g., the Agency for Healthcare Research and Quality [AHRQ] Patient Safety for Selected Indicators—Patient Safety Indicator [PSI]-90) versus more domain-specific measures that pertain to fewer patients, such as surgical site infections (5). Finally, challenges still remain in incorporating the patients' perspective into metric development, detecting adverse events outside the hospital setting, and in determining their preventability. It is to these issues that we now turn.

The reliability of patient reporting of adverse events (i.e., injury due to medical care rather than the underlying condition of patient) is well established (6); physician review indicates that a substantial proportion of occurrences identified by patients qualify as safety events from the "physician perspective." However, few studies have examined this area. Knowing what patients experience would help to better understand the nature of safety events and also add value to the type of data typically used for this measurement. The related literature indicates that patients identify safety events that are unreported or absent from provider reporting systems (6), complementing other primarily provider-based events (7, 8, 9). For example, one study found that only 55% of patient-reported safety events could be found in hospital records, and none of these were present in the hospital incident reporting system (7). The literature suggests that the most common safety events reported by patients include medication errors, reactions to newly prescribed drugs, communication breakdowns between providers and patients, and complications after surgery; some studies also report the associated harm (8, 9, 10).

While importance of the patients' perspective on the safety of care has gained recognition, currently no comprehensive or standardized method for capturing patients' safety-related experiences exists. This is due to difficulties in defining a safety or adverse event (i.e., patients may have a different view of this than their providers) and in identifying and recruiting a sample of patients who have experienced adverse events. Interviewing patients upon discharge is one method commonly used. However, these data reflect only those patients willing to participate and those who could be located (unlocated patients may be the most vulnerable); the data are also subject to recall bias (9). Patient satisfaction surveys, another common method used to capture patients' beliefs/expectations, often fail to capture what happened to the patient during care and the factors that contributed to their positive/negative experiences (11). Thus, despite the importance of capturing the patients' perspective, challenges remain in standardizing the terminology on what patients report as safety events, establishing reliable and efficient methods for capturing patient reports, and identifying safety problems that occur in real-time (i.e., spontaneous reporting right after an event occurs) (6, 9).

Identifying adverse events that occur outside the hospital setting is also fraught with opportunities and formidable challenges. While the number of outpatient visits far outweighs the number of inpatient admissions, and the majority of care is now delivered in the ambulatory setting, patient safety measurement has not yet caught up with this trend. It focuses on the inpatient setting, due to the acuity of disease, high risk of complications, and ease of measuring an episode of care associated with a hospitalization (12). Compared to the multitude of available inpatient safety measures, the number of existing outpatient measures is relatively low. Because two to three patient safety incidents are estimated to occur for every 100 outpatient encounters, and 4% may result in severe harm (13), measuring patient safety in the ambulatory setting needs to advance significantly to reduce the number of adverse events that occur in ambulatory care.

There are several barriers, though, to improving measurement of safety in outpatient care. Because process measures are most frequently used, there is a need to develop outcome measures that relate directly to care processes in the outpatient setting (and that signify the end product of care). Patients are frequently seen by multiple providers in different outpatient settings, making assessment of the coordination of care difficult. Additionally, patients generally assume greater responsibility for their care in the outpatient setting. However, assessing the interpersonal aspects of these care exchanges can be problematic without the appropriate measurement tools. Also, real-time detection of outpatient safety events can prove even more challenging than within the inpatient setting, due to the longer between-visit times, heterogeneity of diseases managed, and the potential for error to be discovered in another setting (14).

Nonetheless, important strides in outpatient safety measurement are occurring in relatively understudied and unmeasured areas of patient safety. Detection of diagnostic errors, a major contributor to harmful patient outcomes, is moving forward through the use of both retrospective and prospective trigger tools and the Safer Dx Framework (4). For example, automated measurement and surveillance via the electronic medical record are being used to trigger medical records with evidence of potential delays in follow-up of cancer-related abnormal findings (15). However, these types of methods rely on the electronic medical record, which can lack the necessary documented information or vary in quality across systems (16).

Finally, determining the preventability of adverse events represents a wonderful opportunity to guide development of quality improvement initiatives and system changes that may reduce similar events in the future. Although it is now increasingly routine for patient safety managers to conduct root cause analyses in order to determine the cause of safety incidents identified through voluntary reporting systems, this process does not identify their preventability. Among the few safety measures that address preventability are the National Quality Forum's Serious Reportable Events (17) and the AHRQ PSIs (18). For example, many of the PSIs exclude occurrences that are unlikely to be avoidable by eliminating emergent cases (i.e., non-elective surgeries) (18). Although this method has strong face validity, no research to date has examined whether PSIs that occur among non-emergent cases are truly preventable. Identifying preventability of an adverse event is very challenging, since there is no "gold standard" for determining this. Preventability depends on the specific circumstances related to the occurrence of the event, and is not simply a "yes/no" phenomenon (18).

In conclusion, integrating the patients' perspective, improving measurement of outpatient safety, and identifying preventability are three important areas where we need to advance. Patient-reported events will add breadth and value to both inpatient and outpatient safety measurement; including the outpatient setting is critical for providing a better assessment of overall patient safety within a system of care. Given the era of public reporting, it is essential that we improve the science of safety measurement in order to ensure that we are measuring the "true" safety performance of providers, and also that we are targeting our quality improvement efforts at events that most likely can be prevented.

---

Authors

Amy K. Rosen, PhD 
VA Senior Research Career Scientist, Center for Healthcare Organization and Implementation Research (CHOIR), VA Boston Healthcare System 
Department of Surgery, Boston University School of Medicine

Qi Chen, MD, PhD 
Investigator, Center for Healthcare Organization and Implementation Research (CHOIR), VA Boston Healthcare System

Disclaimer

The views and opinions expressed are those of the author and do not necessarily state or reflect those of the Department of Veterans Affairs, the National Quality Measures Clearinghouse™ (NQMC), the Agency for Healthcare Research and Quality (AHRQ), or its contractor ECRI Institute.

Potential Conflicts of Interest

Dr. Amy Rosen and Dr. Qi Chen state no personal financial or family member conflict of interest with respect to this expert commentary. Dr. Rosen states business and professional interest as noted below:

• VA Boston Healthcare System

References

1. Pham JC, Frick KD, Pronovost PJ. Why don't we know whether care is safe?. Am J Med Qual. 2013 Nov-Dec;28(6):457-63.
2. Thomas EJ, Classen DC. Patient safety: let's measure what matters. Ann Intern Med. 2014 May 6;160(9):642-3.
3. Jha AK, Classen DC. Getting moving on patient safety–harnessing electronic data for safer care. N Engl J Med. 2011 Nov 10;365(19):1756-8.
4. Singh H, Sittig DF. Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework. BMJ Qual Saf. 2015 Feb;24(2):103-10.
5. Pronovost PJ, Goeschel CA, Marsteller JA, Sexton JB, Pham JC, Berenholtz SM. Framework for patient safety research and improvement. Circulation. 2009 Jan 20;119(2):330-7.
6. King A, Daniels J, Lim J, Cochrane DD, Taylor A, Ansermino JM. Time to listen: a review of methods to solicit patient reports of adverse events. Qual Saf Health Care. 2010 Apr;19(2):148-57.
7. Weingart SN, Pagovich O, Sands DZ, Li JM, Aronson MD, Davis RB, et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Intern Med. 2005 Sep;20(9):830-6.
8. Zhu J, Stuver SO, Epstein AM, Schneider EC, Weissman JS, Weingart SN. Can we rely on patients' reports of adverse events?. Med Care. 2011 Oct;49(10):948-55.
9. Fowler FJ, Jr., Epstein A, Weingart SN, Annas CL, Bolcic-Jankovic D, Clarridge B, et al. Adverse events during hospitalization: results of a patient survey. Jt Comm J Qual Patient Saf. 2008 Oct;34(10):583-90.
10. Weingart SN, Price J, Duncombe D, Connor M, Sommer K, Conley KA, et al. Patient-reported safety and quality of care in outpatient oncology. Jt Comm J Qual Patient Saf. 2007 Feb;33(2):83-94.
11. Walton M, Jorm C, Smith-Merry J, Harrison R, Manias E, Iedema R, et al. Using patients' experiences of adverse events to improve health service delivery and practice: protocol of a data linkage study of Australian adults age 45 and above. BMJ Open. 2014 Oct 13;4(10):e006599.
12. Spencer R, Campbell SM. Tools for primary care patient safety: a narrative review. BMC Fam Pract. 2014 Oct 26;15:166.
13. Panesar SS, deSilva D, Carson-Stevens A, Cresswell KM, Salvilla SA, Slight SP, et al. How safe is primary care? A systematic review. BMJ Qual Saf. 2015 Dec 29.
14. Lorincz CY, Drazen E, Sokol PE, Neerukonda KV, Metzger J, Toepp MC, et al. Research in ambulatory patient safety 2000-2010: a 10-year review. Chicago (IL): American Medical Association (AMA); 2011.
15. Murphy DR, Laxmisan A, Reis BA, Thomas EJ, Esquivel A, Forjuoh SN, et al. Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf. 2014 Jan;23(1):8-16.
16. Ricci-Cabello I, Goncalves DC, Rojas-Garcia A, Valderas JM. Measuring experiences and outcomes of patient safety in primary care: a systematic review of available instruments. Fam Pract. 2015 Feb;32(1):106-19.
17. National Quality Forum (NQF). Serious reportable events in healthcare–2011 update. A consensus report. Washington (DC): National Quality Forum (NQF); 2011.
18. AHRQ quality indicators. Guide to patient safety indicators [version 2.1]. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ); 2003 May 28. 143 p. (AHRQ Pub; no. 03-R203).