Safer Dx

VHA Directive 1088: Communicating Test Results to Providers and Patients

Dr. Singh co-led and worked with the Office of Primary Care on revising VHA Directive 1088 on communication of test results which requires VA providers to communicate normal test results to patients within 14 days after the result becomes available or within 7 days when results require follow-up action. This national policy released in July 2023 sets standards related to the timeliness of test results communication and informs a performance measurement and quality improvement program implemented across the entire VA healthcare system. This directive was informed by the Safer Dx team’s work on missed test results.

Computerized Patient Record System (CPRS) View Alert Notification Checklist

The Safer Dx team created this checklist for the VA to help clinicians and staff manage alerts in the EHR system. It was disseminated to all VISN directors and Chief Medical Officers in 2017. 

10 Strategies for Managing Notifications

This training document was created by the Safer Dx team and distributed to all VA facilities for CPRS training.

National and International Reports

The Safer Dx team’s work has informed:

• The American Medical Association (AMA) report Research in Ambulatory Patient Safety 2000-2010: A 10-Year Review (2011) was informed by 9 Safer Dx publications.
• The Institute of Medicine report Health IT and Patient Safety: Building Safer Systems for Better Care cited 4 publications from the team.
• The National Academies of Sciences, Engineering, and Medicine (NASEM, formerly the Institute of Medicine) report Improving Diagnosis in Healthcare (2015) cited 33 publications from the team.
• The National Health Service (NHS) guidelines Standards for the Communication of Patient Diagnostic Test Results on Discharge from Hospital (2016) cited 1 publication from the Safer Dx team.
• The World Health Organization (WHO) report Patient Safety In Primary Care (2016) was co-authored by Dr. Singh and cited 11 publications from the team.
• The ECRI and Partnership for Health IT Patient Safety report Health IT Safer Practices for Closing the Loop (2018) cited 13 publications from the team.
• The Organization for Economic Co-operation and Development (OECD) report The Economics of Patient Safety in Primary and Ambulatory Care (2018) cited 4 Safer Dx publications.
• The WHO report Patient Safety and Risk Management Service Delivery and Safety (2019) was informed by 2 publications from the Safer Dx team.
• The National Quality Forum (NQF) report Improving Diagnostic Quality and Safety/Reducing Diagnostic Error: Measurement Considerations (2020) cited 10 publications from the team.
• The National Academy of Medicine (NAM) special report Health Data Sharing to Support Better Outcomes: Building a Foundation of Stakeholder Trust (2021) cited 1 publication from the Safer Dx team.
• NASEM proceedings Achieving Excellence in the Diagnosis of Acute Cardiovascular Events: Proceedings of a Workshop in Brief (2021) cited 1 publication from the team.
• The NAM report Implementing High-Quality Primary Care (2021) was informed by 1 Safer Dx publication.
• The Joint Commission’s Sentinel Event Alert Diagnostic Overshadowing Among Groups Experience Health Disparities (2022) cited 2 publications from the team.
• The WHO report Implications of the COVID-19 Pandemic for Patient Safety: A Rapid Review (2022) was co-authored by Dr. Singh and cited 9 publications from the team.
• The AMA publication Reports of the Council on Science and Public Health (2022) cited 1 publication from the Safer Dx team.
• The NASEM report Achieving Whole Health: A New Approach for Veterans and the Nation (2023) cited 2 Safer Dx publications. 
• The NAM report Emerging Stronger from COVID-19: Priorities for Health System Transformation (2023) was informed by 1 publication from the Safer Dx team.
• The WHO Global Patient Safety Report (2024) was informed by 7 publications from the team and Dr. Singh contributed a featured story.
• Dr. Singh was invited to serve on the WHO Drafting and Review Task Force for the Global Patient Safety Action Plan 2021-2030. 
• Dr. Singh was invited to serve on the WHO World Patient Safety Day 2024 Steering Committee and the Planning Group. 
• The President’s Council of Advisors on Science and Technology released its finding on patient safety in a report to President Joe Biden entitled “A Transformational Effort on Patient Safety” that cited 2 publications from the Safer Dx team (PMID 30395508 and PMID 24742777). The report outlines recommendations to advance the nation's commitment to supporting robust safety solutions for patients and the healthcare workforce. 

Grant title: Diagnostic Safety Center for Advancing E-triggers and Rapid Feedback Implementation (DISCOVERI) [Singh]
Funding Agency: AHRQ – Agency for Healthcare Research and Quality
Description: The DISCOVERI project is one of eight AHRQ Diagnostic Centers of Excellence. The major goals of the DISCOVERI project are to implement existing e-triggers for real-time surveillance, implement existing e-triggers at new facilities, and evaluate Safety I and Safety II methods to provide clinicians with real-time diagnostic performance feedback. The project will create new knowledge, tools, and strategies through which these research methods can be better adopted by healthcare organizations for learning and action to improve patient safety.

Grant title: Diagnostic Safety Capacity Building   [Singh]
Funding Agency: AHRQ – Agency for Healthcare Research and Quality
Description: This project aims to develop novel resources and tools to reduce the frequency of diagnostic errors and the associated preventable harm to patients. The research teams at MedStar Health Institute for Quality and Safety and Baylor College of Medicine leverage cognitive science, informatics, human factors, and social science to test interventions and build diagnostic capacity using a multi-pronged approach by multidisciplinary teams.

Grant title: Measuring and Understanding Diagnostic Quality from Large-scale Data   [Singh]
Funding Agency: AHRQ – Agency for Healthcare Research and Quality
Description: This project focuses on developing an approach to measure diagnostic quality across providers using large-scale data in the setting of pneumonia and radiology. The research team aims to address three challenges fundamental to any data-driven approach to measuring diagnostic quality: (i) the lack of observable ground truth against which to benchmark diagnoses, (ii) the dependence on observed rates of diagnostic errors and the underlying disease prevalence, and (iii) small cases numbers per clinician that can complicate comparisons between clinicians. The project will also correlate measures of diagnostic quality with clinician characteristics.

Grant title: Developing e-Triggers to Detect Telemedicine Related Diagnostic Safety Events [Murphy, Singh]
Funding Agency: AHRQ - Agency for Healthcare Research and Quality
Description: The goals of this project are to identify risk factors that impact the tele-diagnostic process and develop guidance to avoid or mitigate these factors. The aims are to 1) evaluate factors that increase the risk of telediagnosis error using interviews with clinicians, patient safety personnel, telemedicine experts, and patients; 2) develop e-triggers to identify telemedicine-related diagnostic errors and identify potential contributory factors; and 3) develop a self-assessment tool to evaluate the risk of telemedicine-related diagnostic errors.

Grant title: Application of Novel Methods to Develop New Quality Measures of Gastrointestinal Cancer Diagnosis [Khalaf]
Funding Agency: Gordon and Betty Moore Foundation
Description: This project aims to develop and benchmark a quality measure of avoidable diagnostic delay for gastrointestinal cancers and to develop a machine learning-enhanced quality measure of avoidable emergency presentation of gastrointestinal cancer.

Grant title: Developing Safety Metrics for Safer EHR Transitions (VA Safety MEMO) [Davila, Singh]
Funding Agency: VA - Department of Veterans Affairs 
Description: The main goal of this project is to develop, validate, and implement safety metrics that VA can use to assess and monitor the safety of the new electronic health record (EHR) implementation during its EHR Modernization (EHRM) program. These safety metrics will focus on identifying and quantifying errors, unintended consequences, risks, workarounds, and near misses associated with the Oracle Health implementation at the VA.

Grant title: Diagnostically Challenging Conditions [Sloane]
Funding Agency: SIDM - Society to Improve Diagnosis in Medicine 
Description: This research project seeks to gain a better understanding of the diagnostic journey of patients with "diagnostically challenging conditions", such as celiac disease. The ultimate goal of this project is to develop, test, and validate electronic trigger(s) to proactively identify patients who have a high likelihood of having certain diagnostically challenging conditions.

Grant title: Evaluation of Calibrate Dx: A Resource to Improve Diagnostic Decisions in the Hospital Setting [Bradford, Meyer]
Funding Agency: AHRQ – Agency for Healthcare Research and Quality
Description: The major goals of this project are to create a methodology for evaluating diagnostic calibration in clinical practice, develop an implementation guide for use with Calibrate Dx (a resource aimed at helping clinicians gain insight into their diagnostic performance), and to assess whether use of Calibrate Dx leads to improved calibration and other attitudes and behaviors related to diagnostic safety.

Grant Title: The Diagnosis Improvement Safety Center (DISCovery) [Singh] 
Funding Agency: VA - Department of Veterans Affairs
Description: The goal of DISCovery is to develop a change package to facilitate the use of EHR-based measurement diagnostic errors in primary care by guiding VA facilities on how to implement e-triggers in their clinical environments. This project builds upon work accomplished during the previous 14 years of funding.

Grant Title: Understanding Physicians’ Diagnostic Accuracy in the EHR Era [Meyer]
Funding Agency: VA – Department of Veterans Affairs
Description: The goal of this project is to examine the relationships between diagnostic accuracy and characteristics of physicians’ decision-making and how they use the electronic health record in their decision-making.

Use of patient complaints to identify diagnosis-related safety concerns: a mixed-method evaluation (Giardina TD, Korukonda S, Shahid U, et al., 2021). This article was voted one of BMJ Quality and Safety's top articles of 2021. It evaluates the use of patient complaint data to identify safety concerns related to diagnosis. It demonstrates that patient complaint data and corresponding medical record reviews can identify patterns of failures in the diagnostic process. Therefore, data on patient complaints can be used to monitor diagnostic safety concerns and identify opportunities for learning and improvement.

Bridging the feedback gap: a sociotechnical approach to informing clinicians of patients' subsequent clinical course and outcomes (Cifra CL, Sittig DF, Singh H, 2021). This manuscript discusses challenges to developing systems for effective patient outcome feedback for physicians. Clinicians require effective feedback on individual patient outcomes to improve their diagnosis and management skills. They propose the use of health information technology (IT) to establish a feedback process that thoughtfully integrates into the sociotechnical aspects of the healthcare environment. The proposed strategies discussed can support clinician participation and advance the health IT structure and processes to enable feedback for improving patient outcomes.

Policies to Promote Shared Responsibility for Safer Electronic Health Records (Sittig DF, Singh H, 2021). This report highlights the need for a collaborative effort and shared responsibility between hospitals and developers in ensuring electronic health record (EHR) safety. Widespread adoption of best practices remains lacking. Therefore, the CMS mandate in collaboration with ONC now requires eligible hospitals to annually assess their EHR systems using SAFER Guides and mitigate EHR-related safety risks effectively. This effort underscores the need for further promotion and adoption of the SAFER Guides across the industry. As hospitals conduct self-assessments, developers must align their products with SAFER recommendations, reflecting a collective commitment to patient safety.

Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments (Vaghani V, Wei L, Mushtaq U, et al., 2021). This manuscript describes testing of a symptom–disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The e-trigger was applied to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed by trained clinicians. Of these, 31.2% had missed opportunities, 23.4% had no missed opportunity, and 40.7% were miscoded. Given the high frequency of coding errors and cases without errors, our findings underscore the need to validate the output of algorithmically identified diagnostic errors in large data sets. 

Applying Requisite Imagination to Safeguard Electronic Health Record Transitions (Sittig DF, Lakhani P, Singh H, 2022). In this report, our team applies principles of Requisite Imagination to offer 6 recommendations on how to proactively safeguard transitions from one EHR to another. The next decade will be dominated by such transitions, some forced by hospital acquisition and others by choice in search of a better EHR. EHR transitions cause major healthcare delivery disruptions. Proactive approaches using the Requisite Imagination principles outlined here can help ensure safe, effective, and economically sound EHR transitions.

Electronic Co-design (ECO-design) Workshop for Increasing Clinician Participation in the Design of Health Services Interventions: Participatory Design Approach (Savoy A, Patel H, Shahid U, et al., 2022). This manuscript details a web-based participatory design approach to conducting economical, electronic co-design (ECO-design) workshops with primary care clinicians. Traditionally, stakeholder-led design activities are time-consuming and make clinician participation difficult. This article demonstrates how the co-design workshop series concept was adapted for web-based delivery and to fit within a single 1-hour session. It was found that ECO-design workshops are viable, economical alternatives to traditional approaches. This approach fills a need for efficient methods to involve busy clinicians in the design of health care interventions.

Guidelines for US Hospitals and Clinicians on Assessment of Electronic Health Record Safety Using SAFER Guides (Sittig DF, Sengstack P, Singh H, 2022). The Centers for Medicare & Medicaid Services (CMS) issued a final rule on August 13, 2021, mandating eligible hospitals to annually attest to conducting a safety assessment of their electronic health record (EHR) systems using SAFER (Safety Assurance Factors for EHR Resilience) Guides. SAFER Guides is an essential tool, comprising of 146 recommended practices designed to enhance the safety and efficacy of electronic health record (EHR) systems. Tailored for clinicians, hospitals, and EHR vendors, these checklist-based risk-assessment tools delve into every aspect of EHR implementation and usage, from design and development to configuration and monitoring. These guidelines can help hospitals and clinicians systematically identify EHR-related features, functions, workflows, and organizational efforts to reduce safety risks to a level as low as reasonably achievable.                                                                         

Development and Usability Testing of the Agency for Healthcare Research and Quality Common Formats to Capture Diagnostic Safety Events (Bradford A, Shahid U, Schiff GD, et al., 2022). The Agency for Healthcare Research and Quality (AHRQ) has developed the Common Formats for Event Reporting for Diagnostic Safety Events (CFER-DS). This report describes our team's usability assessment of the CFER-DS in a sample of quality and safety personnel in 8 U.S. healthcare organizations. Participants shared generally positive feedback about the content, but they also shared concerns about user burden and identified opportunities to clarify several conceptual definitions and develop guidance to operationalize the use of CFER-DS.

Translating electronic health record-based patient safety algorithms from research to clinical practice at multiple sites (Zimolzak AJ, Singh H, Murphy DR, et al., 2022). In this article, the Safer Dx team and collaborators demonstrate how electronic trigger algorithms were successfully implemented at twelve sites (across two major hospitals) to proactively identify patients without timely follow-up of abnormal test results that warrant diagnostic evaluation for colorectal or lung cancer. This article describes a strategy and generalizable lessons learned to efficiently translate patient safety algorithms from research into practice.

Developing the Safer Dx Checklist of Ten Safety Recommendations for Health Care Organizations to Address Diagnostic Errors (Singh H, Mushtaq U, Marinez A, et al., 2022). Developed using a rigorous multimethod approach, the Safer Dx Checklist developed by Dr. Singh and the Safer Dx team, is a self-assessment tool with 10 recommended practices. Health care organizations can use this tool to understand the current state of diagnostic practices, identify areas to improve, and track progress toward diagnostic excellence over time.

Managing Interruptions to Improve Diagnostic Decision-Making: Strategies and Recommended Research Agenda (Sloane JF, Donkin C, Newell BR, et al., 2023). Interruptions are an inevitable occurrence in health care. Interruptions in diagnostic decision-making are no exception and can have negative consequences on both the decision-making process and well-being of the decision-maker. In this paper, we first present a modified model of interruptions to visualize the interruption process and illustrate where potential interventions can be implemented and then we consider several empirically tested strategies from the fields of health care and cognitive psychology that can lay the groundwork for additional research to mitigate effects of interruptions during diagnostic decision-making.

Diabetes Status and Pancreatic Cancer Survival in the Nationwide Veterans Affairs Healthcare System (Khalaf N, Kramer J, Liu Y, et al., 2023). This manuscript examines associations between diabetes status and pancreatic cancer outcomes. In this study, led by Dr. Khalaf, the investigators discovered that patients with long-standing diabetes had a 10% higher mortality risk compared to patients without diabetes after adjusting for sociodemographic factors and medical comorbidities.

Five Strategies for a Safer EHR Modernization Journey (Sittig DF, Yackel EE, Singh H, 2023). Transitioning to a new electronic health record (EHR) system is a formidable challenge. As EHRs intricately weave into every aspect of our healthcare system, unexpected challenges often arise during transitions, leading to disruptions in cost, quality, and efficiency measures for extended periods. In this article, our team recommends five actionable strategies to support the VA in a safer journey to modernize the EHR. While these strategies are not exhaustive, they will reduce the risk of patient harm, cost overruns, and workforce turnover.

Artificial Intelligence in Clinical Diagnosis: Opportunities, Challenges, and Hype (Kulkarni PA, Singh H, 2023). This manuscript details a realistic overview of generative artificial intelligence's (AI's) role in clinical diagnosis. This article discusses the hype, strengths, challenges, and future opportunities for AI integration into clinician workflow and how it can potentially improve the quality of care, including diagnosis. 

Developing Electronic Clinical Quality Measures to Assess the Cancer Diagnostic Process (Murphy DR, Zimolzak AJ, Upadhyay DK, et al., 2023). This manuscript discusses the development and implementation of two electronic clinical quality measures (eCQMs) at two sites: VA and Geisinger. The measures assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Results indicated that for CRC eCQM, the rate of appropriate follow-up was 36.0% in the VA and 41.1% at Geisinger (P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5%. Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157).

The Impact of Race on Pancreatic Cancer Treatment and Survival in the Nationwide Veterans Affairs Healthcare System (Khalaf N, Xu A, Nguyen Wenker T, et al., 2024). This publication describes the independent association between race and three endpoints related to pancreatic cancer (stage at diagnosis, receipt of treatment, and survival) while adjusting for sociodemographic factors and medical comorbidities. It was found that Black patients were 19% more likely to have late-stage disease and 25% less likely to undergo surgical resection. Black patients had 13% higher mortality risk compared with White patients.

Current State of Diagnostic Safety: Implications for Research, Practice, and Policy (Khan S, Cholankeril R, Sloane J, et al., 2024). The field of diagnostic safety has developed rapidly over the past decade. To accelerate progress in diagnostic safety science and improvement activities, we conducted a narrative review and interviews with diagnostic safety experts to identify major themes related to the current state and highlight key gaps in knowledge. The issue brief covers the past decade of progress in the field and identifies diagnostic safety work within 10 domains.

A Roadmap to Advance Patient Safety in Ambulatory Care. (Singh H, Carayon P, 2020). In the two decades since “To Err is Human: Building a Safer Health System” was issued by the Institute of Medicine (now the National Academy of Medicine-NAM), advances in patient safety have focused mainly on inpatient settings whereas outpatient settings have been overlooked. However, accumulated evidence leaves little justification to continue neglecting ambulatory safety.  The time to accelerate initiatives to reduce preventable harm in the outpatient setting has arrived. Key milestones related to scientific advances, practice improvements, policy changes, and strategies to partner with patients and families can accelerate meaningful advances to reduce patient harm in ambulatory care.   

A Sociotechnical Framework for Safety-related EHR Research and Reporting: The SAFER Reporting Framework. (Singh H, Sittig DF, 2020). Electronic health record (EHR)-based interventions to improve patient safety are complex and sensitive to who, what, where, why, when, and how they are delivered. Success or failure depends not only on the characteristics and behaviors of individuals who are targeted by an intervention, but also on the technical characteristics of the intervention and the culture and environment of the health system that implements it. Current reporting guidelines do not capture the complexity of sociotechnical factors (technical and nontechnical factors, such as workflow and organizational issues) that confound or influence these interventions. This article proposes a methodological reporting framework for EHR interventions targeting patient safety and builds on an 8-dimension sociotechnical model previously developed by the authors for design, development, implementation, use, and evaluation of health information technology. The Safety-related EHR Research (SAFER) Reporting Framework enables reporting of patient safety-focused EHR-based interventions while accounting for the multifaceted, dynamic sociotechnical context affecting intervention implementation, effectiveness, and generalizability. As an example, an EHR-based intervention to improve communication and timely follow-up of subcritical abnormal test results to operationalize the framework is presented. For each dimension, reporting should include what sociotechnical changes were made to implement an EHR-related intervention to improve patient safety, why the intervention did or did not lead to safety improvements, and how this intervention can be applied or exported to other health care organizations. A foundational list of research and reporting recommendations to address implementation, effectiveness, and generalizability of EHR-based interventions needed to effectively reduce preventable patient harm is provided. The SAFER Reporting Framework is not meant to replace previous research reporting guidelines, but rather provides a sociotechnical adjunct that complements their use.

COVID-19 and the Need for a National Health Information Technology Infrastructure. (Sittig DF, Singh H, 2020). The need for timely, accurate, and reliable data about the health of the US population has never been greater. Critical questions include the following: (1) how many individuals test positive for severe acute respiratory syndrome coronavirus (SARS-CoV-2) and how many are affected by the disease it causes—novel coronavirus disease 2019 (COVID-19) in a given geographic area; (2) what are the age and race of these individuals; (3) how many people sought care at a health care facility; (4)  how many were hospitalized;(5)within individual hospitals, how many patients required intensive care, received  ventilator support, or died; and (6) what was the length of stay in the hospital and in the intensive care unit for patients who survived and for those who died. The privacy, legal, and ethical trade-offs warrant further consideration, even though in an era of eroded trust, some discussions will be difficult. Recently, HHS issued limited waivers to facilitate the nation’s ability to care for patients during the COVID-19 pandemic. These changes show how regulations can be modified during extraordinary times. With a sharp focus on maximizing benefits of scarce resources, treating everyone equally, and prioritizing efforts to save lives while maintaining trust and confidentiality, a national health IT infrastructure could meet the highest ethical standards. It is time to make some difficult decisions and exploit and enhance existing technical capability to build and deploy these solutions. Given the severity and immediacy of theCOVID-19 pandemic, the US should no longer rely on outdated laws, social norms, or potentially inaccurate modalities to obtain timely, accurate, and reliable health information essential to save lives.

Operational Measurement of Diagnostic Safety: State of the Science. (Singh H, Bradford A, Goeschel C, 2020) This review outlines the state of the science of diagnostic safety measurement, with a focus on practical strategies that healthcare organizations can use to begin identifying and learning from diagnostic errors.

The Path to Diagnostic Excellence Includes Feedback to Calibrate how Clinicians Think. (Meyer AND, Singh H, 2019) Improving diagnosis in health care is considered the next imperative for patient safety. Rapidly evolving diagnostic tests and treatments and competing priorities and pressures encountered by clinicians to deliver high-quality, low-cost health care make this a major challenge. Clinicians frequently balance undertesting, possibly missing a diagnosis, with pursuing overzealous diagnostic testing, which could be harmful and costly. Rigorous multidisciplinary research and innovation from cognitive psychology, human factors, informatics, and social sciences are needed to stimulate previous efforts to reduce diagnostic errors.

Application of Electronic Trigger Tools to Identify Targets for Improving Diagnostic Safety. (Murphy DR, Meyer AND, Sittig DF, et. al., 2018) Electronic trigger (e-trigger) tools, which mine vast amounts of patient data to identify signals indicative of a likely error or adverse event, offer a promising method to efficiently identify diagnostic errors. The authors present a knowledge discovery framework, the Safer Dx Trigger Tools Framework, that enables health systems to develop and implement e-trigger tools to identify and measure diagnostic errors using comprehensive electronic health record (EHR) data. The authors outline key future research, including advances in natural language processing and machine learning, needed to improve effectiveness of e-triggers. Integrating diagnostic safety e-triggers in institutional patient safety strategies can accelerate progress in reducing preventable harm from diagnostic errors.

Beyond Dr. Google: The Evidence about Consumer-Facing, Digital Tools for Diagnosis. (Millenson ML, Baldwin JL, Zipperer L, et. al., 2018) Direct-to-consumer (DTC), interactive, diagnostic apps with information personalization capabilities beyond those of static search engines are rapidly proliferating. While these apps promise faster, more convenient and more accurate information to improve diagnosis, little is known about the state of the evidence on their performance or the methods used to evaluate them. The authors conducted a scoping review of the peer-reviewed and gray literature and found that apps vary widely in functionality, accuracy, safety and effectiveness, although the usefulness of this evidence was limited by a frequent failure to provide results by named individual app.

Methods for Patient-Centered Interface Design of Test Result Display in Online Portals. (Nystrom DT, Singh H, Baldwin J, et. al., 2018) Patients have unique information needs to help them interpret and make decisions about laboratory test results they receive on web-based portals. However, current portals are not designed in a patient-centered way and little is known on how best to harness patients’ information needs to inform user-centered interface design of portals. The authors designed a patient-facing laboratory test result interface prototype based on requirement elicitation research and used a mixed-methods approach to evaluate this interface. They found that requirement elicitation studies can inform the design of a patient-facing test result interface, but considerable user-centered design efforts are necessary to create an interface that patients find useful. To promote patient engagement, health information technology designers and developers can use similar approaches to enhance user-centered software design in patient portals.

Adherence to recommended electronic health record safety practices across eight health care organizations. (Sittig DF, Salimi M, Aiyagari R, et. al., 2018) The Safety Assurance Factors for EHR Resilience (SAFER) guides were released in 2014 to help health systems conduct proactive risk assessment of electronic health record (EHR)- safety related policies, processes, procedures, and configurations. The extent to which SAFER recommendations are followed is unknown. The authors conducted risk assessments of 8 organizations of varying size, complexity, EHR, and EHR adoption maturity. They found that, despite availability of recommendations on how to improve use of EHRs, most recommendations were not fully implemented. They suggest new national policy initiatives are needed to stimulate implementation of these best practices.

Evaluating a mobile application for improving clinical laboratory test ordering and diagnosis. (Meyer AND, Thompson PJ, Khanna A, et. al., 2018) Mobile applications for improving diagnostic decision making often lack clinical evaluation. The authors evaluated if a mobile application improves generalist physicians' appropriate laboratory test ordering and diagnosis decisions and assessed if physicians perceive it as useful for learning. They found that a mobile app, PTT Advisor, may contribute to better test ordering and diagnosis, serve as a learning tool for diagnostic evaluation of certain clinical disorders, and improve patient outcomes. Similar methods could be useful for evaluating apps aimed at improving testing and diagnosis for other conditions.

Electronic health record reviews to measure diagnostic uncertainty in primary care. (Bhise V, Rajan SS, Sittig DF, et al., 2018) Diagnostic uncertainty is common in primary care. Because it is challenging to measure, there is inadequate scientific understanding of diagnostic decision-making during uncertainty. The authors' study found that, while current diagnosis coding mechanisms (ICD-9 and ICD-10) are unable to capture uncertainty, review of EHR documentation can help identify diagnostic uncertainty with moderate reliability.

Patient perceptions of receiving test results via online portals: a mixed-methods study (Giardina TD, Baldwin J, Nystrom DT, et al., 2017) Online portals provide patients with access to their test results, but it is unknown how patients use these tools to manage results and what information is available to promote understanding. A mixed-methods study was conducted to explore patients’ experiences and preferences when accessing their test results via portals. The author's findings suggest that online portals are not currently designed to present test results to patients in a meaningful way. They found that simply providing access via portals is insufficient, and additional strategies are needed to help patients interpret and manage their online test results.

Electronic Triggers to Identify Delays in Follow-Up of Mammography: Harnessing the Power of Big Data in Health Care (Murphy DR, Meyer AND, Vaghani V, et al., 2017) The authors conducted a study to evaluate the effectiveness of an electronic trigger to flag delayed follow-up on mammography. Flagged records were reviewed to determine the trigger's performance characteristics. The frequency of delays and patient communication related to abnormal results, reasons for lack of follow-up, and whether patients were subsequently diagnosed with breast cancer were also assessed. The authors concluded that clinical application of mammography-related triggers could help detect delays in follow-up of results.

An electronic trigger based on care escalation to identify preventable adverse events in hospitalised patients (Bhise V, Sittig DF, Vaghani V, et al., 2017). The authors refined the methods of Institute of Healthcare Improvement’s Global Trigger Tool (GTT) application and leveraged electronic health record (EHR) data to improve detection of preventable adverse events. The EHR data-based trigger and modified review process were able to efficiently identify hospitalized patients with preventable adverse events, including diagnostic errors. Such e-triggers can help overcome limitations of currently available methods to detect preventable harm in hospitalized patients.

The Burden of Inbox Notifications in Commercial Electronic Health Records. (Murphy DR, Meyer AND, Russo E, et. al., 2016) With wider use of electronic health records (EHRs), physicians increasingly receive notifications via EHR-based inboxes (e.g., Epic’s “In-Basket” and GE Centricity’s “Documents”). Types of notifications include test results, referral responses, medication refill requests, and messages from clinicians, among others. Information overload is of emerging concern because new types of EHR-based notifications and ‘FYI’ messages can be easily created (versus in paper-systems) and this additional workload remains uncompensated despite reimbursement reductions. Moreover, EHRs make it easier to measure information load. The authors quantified notifications physicians received via inboxes of commercial EHRs to estimate their burden.

Challenges in Patient Safety Improvement Research in the Era of Electronic Health Records. (Russo E, Sittig DF, Murphy DR, et. al., 2016) Electronic health record (EHR) data repositories contain large volumes of aggregated, longitudinal clinical data that could allow patient safety researchers to identify important safety issues and conduct comprehensive evaluations of health care delivery outcomes. However, few health systems have successfully converted this abundance of data into useful information or knowledge for safety improvement. In this paper, the authors use a case study involving a project on missed/delayed follow-up of test results to discuss real-world challenges in using EHR data for patient safety research. To leverage EHRs and their abundant data for patient safety improvement research, many current data access and security policies and procedures must be rewritten and standardized across health care organizations. These efforts are essential to help make EHRs and EHR data useful for progress in our journey to safer health care.

Computerized Triggers of Big Data to Detect Delays in Follow-up of Chest Imaging Results (Murphy DR, Meyer AND, Bhise V et al., 2016). A “trigger” algorithm was used to identify delays in follow-up of abnormal chest imaging results in a large national clinical data warehouse of electronic health record (EHR) data. The report found that the application of triggers on “big” EHR data may aid in identifying patients experiencing delays in diagnostic evaluation of chest imaging results suspicious for malignancy.

The global burden of diagnostic errors in primary care (Singh H, Schiff GD, Graber ML, 2016). The authors discuss the global significance, burden and contributory factors related to diagnostic errors in primary care. They also summarize interventions based on available data and suggest next steps to reduce the global burden of diagnostic errors. They recommend that the World Health Organization (WHO) consider bringing together primary care leaders, practicing frontline clinicians, safety experts, policymakers, the health IT community, medical education and accreditation organizations, researchers from multiple disciplines, patient advocates, and funding bodies among others, to address the many common challenges and opportunities to reduce diagnostic error.

Measuring and improving patient safety through health information technology: The Health IT Safety Framework (Singh H, Sittig DF, 2015). In response to the fundamental conceptual and methodological gaps related to both defining and measuring health IT-related patient safety, the authors propose a new framework, the Health IT Safety (HITS) measurement framework, to provide a conceptual foundation for health IT related patient safety measurement, monitoring and improvement. The framework proposes to integrate both retrospective and prospective measurement of HIT safety with an organization's existing clinical risk management and safety programs. It aims to facilitate organizational learning, comprehensive 360-degree assessment of HIT safety that includes vendor involvement, refinement of measurement tools and strategies, and shared responsibility to identify problems and implement solutions.

Improving Diagnosis in Health Care – The Next Imperative for Patient Safety (Singh H, Graber M, 2015). Based on a 1999 Institute of Medicine report, the authors discuss the history of diagnostic error, previous work and research in the area, current policies and recommendations regarding the reduction diagnostic error and future actions that can be taken to further improve diagnosis and reduce patient harm from diagnostic error.

Editorial: Helping Health Care Organizations to Define Diagnostic Errors as Missed Opportunities in Diagnosis (Singh H, 2014). An editorial discussing an article from Graber et al. In the editorial, Dr. Singh touches on challenges that healthcare organizations face regarding diagnostic errors and missed opportunities, providing a conceptual model of missed opportunities in diagnosis.

Ebola US Patient Zero: lessons on misdiagnosis and effective use of electronic health records (Upadhyay DK, Sittig DF, Singh H, 2014). The authors discuss the first travel-associated case of U.S. Ebola on Sept. 30, 2014, to highlight the public health challenge of diagnostic errors and discuss the effective use of EHRs in the diagnostic process. We analyze the case to discuss several missed opportunities and outline key challenges and opportunities facing diagnostic decision-making in EHR-enabled healthcare.

The frequency of diagnostic error in outpatient care: estimations from three large observational studies involving US adult populations (Singh H, Meyer AND, Thomas, EJ, 2014). The authors estimated the frequency of diagnostic errors in the U.S. adult population by synthesizing data from three previous studies of clinic-based populations that used conceptually similar definitions of diagnostic error. Their estimate suggests that diagnostic errors affect at least 1 in 20 U.S. adults. This foundational evidence should encourage policymakers, healthcare organizations and researchers to start measuring and reducing diagnostic errors.

Types and Origins of Diagnostic Errors in Primary Care Settings (Singh H, Giardina TD, Meyer AND et al., 2013). The authors reviewed medical records of diagnostic errors detected at two sites through EHR–based triggers. It was found that most errors were related to process breakdowns in the patient-practitioner clinical encounter. It was found that the diagnostic errors identified in the study involved a large variety of common diseases and had significant potential for harm. Most errors were related to process breakdowns in the patient-practitioner clinical encounter.

Electronic health record-based surveillance of diagnostic errors in primary care (Singh H, Giardina TD, Forjuoh SN et al., 2012). Diagnostic errors in primary care are harmful but difficult to detect. The authors tested an electronic health record (EHR)-based method to detect diagnostic errors in routine primary care practice. The report found that while physician agreement on diagnostic error remains low, an EHR-facilitated surveillance methodology could be useful for gaining insight into the origin of these errors.