Question
What are emergency department physicians’ views on using a new clinical decision support system designed to show risk predictions, gather patient information, and suggest treatments for acute heart failure?
Key Findings
This qualitative study, involving 58 emergency physicians, revealed several suggestions to improve the usability and integration of clinical decision support (CDS) tools into their workflow. Most doctors believed CDS could enhance heart failure treatment in the emergency department and lead to better patient outcomes.
Implications
Emergency department physicians recognize the value of CDS interventions that offer risk assessments, patient data compilation, and treatment advice for acute heart failure. Their feedback provides crucial insights for refining these tools to ensure successful implementation and maximize benefits. This study highlights the importance of physician input in designing effective CDS tools for managing complex conditions like heart failure in acute care settings.
This qualitative research investigates how emergency department (ED) physicians perceive the usability and workflow integration of clinical decision support in their daily practice.
Abstract
Significance
Clinical decision support (CDS) systems have the potential to significantly aid emergency department (ED) physicians in managing heart failure (HF) patients. By providing risk estimates, compiling relevant patient history, and assisting with medication prescriptions, CDS can improve decision-making. However, the design and implementation of these tools must be informed by the perspectives of the physicians who will use them.
Objective
This study aimed to evaluate the usability and workflow integration of a CDS tool from the viewpoint of ED physicians who are the end-users in a clinical setting.
Methods
This mixed-methods qualitative study involved semi-structured interviews with ED physicians from two community EDs within Kaiser Permanente Northern California in 2023. The interview guide was developed based on established usability heuristics and sociotechnical environment models. The themes identified from these interviews were then used to create an electronic survey, which was distributed to all ED physicians at the participating sites.
Main Outcomes and Measurements
The primary focus was to assess physicians’ perceptions regarding the use of CDS to support clinical decision-making, its usability, and its integration into the ED workflow.
Results
Seven key informant physicians were interviewed (5 [71.4%] female, median [IQR] 15.0 [9.5-15.0] years of experience), and 51 physicians responded to the survey (23 [45.1%] female, median [IQR] 14.0 [9.5-17.0] years of experience) from EDs piloting the CDS intervention. The survey response rate was 67.1% (51 out of 76). Physicians proposed modifications to improve CDS accessibility, functionality, and workflow integration. A majority agreed that CDS could enhance patient care, but fewer than half expressed concerns about consistently following CDS recommendations due to workload pressures. Physicians indicated a preference for a passive prompt system that encourages, but does not force, interaction with the CDS.
Conclusions and Relevance
This qualitative study, focusing on physicians utilizing a novel CDS tool for managing acute heart failure in the ED, identified key opportunities for enhancing usability and highlighted both barriers and facilitators to CDS implementation. The findings emphasize the importance of user-centered design in developing CDS tools that effectively support clinical practice and improve patient care in the emergency department.
Introduction
Background
Integrating prediction models and decision support tools into the fast-paced environment of an emergency department (ED) presents significant challenges. [1, 2, 3, 4] Successful implementation requires a deep understanding of potential obstacles to usability, effective integration, and physician acceptance. [1, 2, 5, 6, 7, 8] Gaining insight into how physicians perceive technology as a decision-making aid is particularly crucial when introducing interventions for high-stakes conditions like acute heart failure (HF). Several attempts to incorporate electronic health record (EHR)-integrated clinical decision rules have faltered because they didn’t adequately consider user needs and existing workflows. [9, 10, 11, 12] The REVEAL-HF trial, for instance, implemented an EHR alert showing 1-year HF mortality risk, but it failed to change physician decisions or improve patient outcomes. The study authors attributed this to the intervention’s lack of specific, actionable recommendations. [13]
Heart failure patients represent a complex population medically and socially. Social determinants of health, such as unemployment, education, and disability, disproportionately affect these individuals. [14] They frequently visit the ED, often leading to hospitalization. [15] Accurately assessing the short-term risk of severe events (death, resuscitation, intubation, myocardial infarction, etc.) in ED patients with acute HF is difficult. This challenge may contribute to both high hospital admission rates and adverse event rates among discharged patients. [16, 17] To address these care gaps and improve patient outcomes, clinical decision support (CDS) interventions have been developed to risk-stratify ED patients with HF and guide disposition decisions. [16, 18, 19, 20] CDS encompasses various technologies that deliver patient-specific information to clinicians at opportune moments, including alerts, reminders, and disease-specific order sets. [21] Combining risk prediction models with CDS may improve disposition decisions for ED patients with acute HF, as recommended by the American College of Emergency Physicians’ guidelines on heart failure syndromes. [22, 23, 24]
Our research team recently created an acute HF risk prediction model using over 60 variables to estimate the 30-day risk of serious adverse events. [16] We integrated this risk model and a corresponding CDS into the EHR and conducted a pilot study in two EDs within our large, integrated healthcare system from January to March 2023.
CDS tools driven by risk prediction models offer the potential to enhance ED care for complex, high-risk patients. However, understanding the clinical environment where the CDS will be used and addressing usability concerns are essential for successful implementation and long-term sustainability.
Our primary objective was to evaluate CDS usability, identify barriers and facilitators to its workflow integration, and assess the perspectives of ED physicians using the intervention to manage acute HF patients. This understanding is crucial for optimizing CDS tools and ensuring they effectively support clinical practice.
Methods
Study Design and Setting
We employed a mixed-methods qualitative study design to gather both quantitative and qualitative feedback from community ED physicians. These physicians were from two medical centers of Kaiser Permanente Northern California (KPNC) participating in a pilot study of our risk prediction model and CDS. KPNC is a large, integrated healthcare delivery system with 21 hospitals, serving nearly 5 million members with demographics representative of the regional population. [25] Compared to national averages, KPNC hospitalizes a lower percentage (57%) of ED patients with acute HF, yet maintains comparable 30-day mortality rates for discharged ED patients. [16, 26, 27, 28] The two EDs in our study are located in urban areas, operate as a single medical center with shared physician staffing and specialty services, and together handle approximately 120,000 patients annually with a 12% hospital admission rate. Both EDs use the Epic EHR system and serve as training sites for residents and medical students. One facility has a catheterization lab and an ED observation unit, facilitating transfers for patients from the other ED needing these services.
Physicians participated in one-on-one usability sessions and semi-structured interviews, which informed the design of an electronic survey. This survey was then distributed to all ED physicians at the study sites and included ordinal, categorical, and free-text response questions. All full-time physicians received an email invitation to participate in the survey and were offered a $75 gift card as an incentive. The KPNC institutional review board approved the study, granting a waiver for informed consent for the data-only portion and a waiver for documented consent (signature not required) for physician interviews. This qualitative study adhered to the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines. [29, 30]
Study Population and Selection
We used a convenience sampling method to select physicians working at the two EDs involved in the CDS pilot study for usability sessions and interviews. During the pilot, 76 full-time physicians staffed these EDs. All full-time physicians were invited to participate in usability interviews via email and announcements at monthly department meetings. An electronic survey link was sent to all full-time ED physicians, with three subsequent email reminders to encourage participation.
Clinical Decision Support Tool
The design and performance of our 30-day serious adverse event risk prediction model have been previously detailed. [16] This model incorporates over 60 variables, including clinical, laboratory, and sociodemographic factors. [16] Prior evaluations demonstrated strong discrimination using both logistic regression-based (AUC, 0.80 [95% CI, 0.79-0.82]) and machine learning-based (AUC, 0.85 [95% CI, 0.83-0.86]) approaches.
We integrated the risk prediction model into an EHR-embedded CDS tool, informed by physician preferences (eFigure 1 in Supplement 1). [31] The CDS was designed to: (1) consolidate relevant patient-specific clinical information in one location (e.g., recent cardiac studies, lab results, vital signs, cardiac medications); (2) provide ED medical management recommendations, including diuretic strategies and adherence to Guideline Directed Medical Therapy (GDMT) [32]; and (3) display patients’ 30-day serious adverse event risk and corresponding ED disposition recommendations. Based on initial physician surveys, we opted for a passive physician prompt to encourage, but not mandate, CDS interaction. [31]
ED physicians at pilot study sites received training on CDS use through two 30-minute virtual and one 30-minute in-person session, supplemented by monthly email updates. Patients meeting eligibility criteria (based on HF history, relevant chief complaint, and HF-related orders) during the pilot were flagged in the EHR via a passive prompt at the top of a screen commonly used by ED physicians to view patient information.
Research Team and Reflexivity
The usability study and semi-structured interviews were conducted by three practicing ED physicians (S.D.C., D.R.S., C.H.L.) and a human-centered design expert (J.G.). Two of the physicians (D.R.S., C.H.L.) had existing professional relationships with participants, and one (D.R.S.) had prior experience with qualitative research. The remaining team members (S.D.C. and J.G.) had no prior professional connections with participants, although all were employed by the same health system.
Data Collection
Usability sessions lasted approximately 60 minutes and were conducted in March 2023. Interviews took place in a hospital conference room or at an investigator’s home. We used a piloted interview guide (eFigure 2 in Supplement 1), refined through testing with three ED physicians (S.D.C., D.R.S., C.L.). Usability studies employed a deductive “think aloud” approach in “near live” clinical scenarios [33, 34], allowing for emergent themes (inductive approach). After obtaining verbal consent, investigators explained that the study aimed to understand physician interactions with the EHR in acute HF patient care, their perspectives on CDS use, and to identify barriers and opportunities for CDS adoption. We collected physician demographics including self-identified gender, years in practice, and years in their current position. Field notes were recorded during interviews, and sessions were video and audio recorded and transcribed using Microsoft Teams.
For usability testing, participants reviewed triage information, arrival mode, and chief complaint of a sample HF patient on the ED trackboard within the EHR. They were asked to “think aloud,” verbalizing their usual process for reviewing medical data, placing orders, assessing risk, and determining ED disposition. If participants did not independently access the CDS, a team member prompted them to locate it and describe how they might use it for clinical decision support. Following this, semi-structured interview questions explored the sociotechnical environment domains: (1) internal policies, (2) human-computer interface, (3) workflow, (4) people, and (5) clinical content.
Data Analysis
Usability themes were identified through content analysis, using heuristics for user interface design [35] and thematic analysis based on the Sociotechnical Framework for health information technology assessment. [36] Two team members (S.D.C. and D.R.S.) independently reviewed transcripts to develop an initial coding framework through comparison and consensus before independently coding transcripts. The principal investigator reviewed all transcripts for coding discrepancies, which were then discussed with the study team. The codebook was iteratively reviewed and refined as new codes emerged or existing codes were clarified (eFigure 3 in Supplement 1). Interviews continued until information power was reached. [37] Dedoose version 9.0.62 qualitative data management software was used for coding and transcript analysis. Qualitative rigor was ensured using methods described by Lincoln and Guba (eTable in Supplement 1). [38] Survey data was graphically presented using Microsoft Excel version 2105. Data analysis occurred between May 1 and June 30, 2023. Participants received a $75 gift card and a copy of their interview transcript for feedback.
Themes from physician interviews informed the design of an electronic survey instrument with categorical, ordinal, and Likert scale responses. The survey was pretested by four ED physicians, and iterative changes were made for clarity before distribution to all full-time ED physicians at the CDS pilot study sites.
Results
After conducting seven interviews (average duration 50 minutes, range 35-63 minutes) with physicians (5 [71.4%] female; median [IQR] 15.0 [9.5-15.0] years in practice) from the two EDs piloting the CDS, we achieved sufficient information power. [37] Physician usability sessions and interviews yielded several themes, with representative quotes illustrating key domains of the Sociotechnical Environment and CDS usability frameworks (Table 1 and Table 2; eFigure 4 in Supplement 1). Of 76 physicians invited to complete the electronic survey, 51 responded (67.1% response rate) (Table 3). Among these 51, 36 (70.6%) preferred voluntary CDS access from multiple locations, and 23 (45.1%) were open to an involuntary opt-out CDS prompting approach (Figure); 48 physicians (94.1%) agreed CDS would improve patient outcomes, and 31 (60.8%) believed it would save time. Thirty-one physicians (60.8%) reported using CDS during the 3-month pilot, while fewer than half (21 [41.2%]) found CDS easily locatable. Most physicians (36 [70.6%]) preferred accessing CDS from multiple EHR locations. During the CDS pilot, 703 patients (median [IQR] age 76 [66-84] years; 374 [53.2%] female, 214 [30.4%] White, 260 [37.0%] Black, 103 [14.7%] Hispanic, 121 [17.2%] Asian) met criteria for CDS prompting.
Usability Domains
The Usability and Sociotechnical Environment models converged significantly in the human-computer interface domain. Physicians valued flexibility in their typical ED workflow, allowing them to review information, access results, and place orders at different times during patient encounters. Some found CDS-collated data useful early in visits, others for diagnosis and treatment recommendations (initial diuretics, lab interpretation), and some after initial ED treatment (disposition decisions, shared decision-making, mortality estimates, outpatient medication adjustments for dischargeable patients). Despite this varied timing of use, all physicians indicated they were more likely to access CDS if prompted early in the ED visit with a passive, non-mandatory prompt. Dissatisfaction with CDS prompt location and limited access points in the EHR highlighted an area for usability improvement.
While most physicians appreciated centralized HF data, most desired greater customization in data display. Specific requests included an interface allowing data expansion/collapse (drop-down lists) and organization by relevance. Many also suggested aesthetic changes to the CDS, such as bolder colors for better visibility and more concise text to reduce information overload.
Physicians noted that direct order entry from CDS or access to the order entry field while CDS was open would minimize information recall needs and improve workflow efficiency. They also recognized CDS risk estimates could expedite disposition decisions for HF patients.
Sociotechnical Environment Domains
Internal Policies
All interviewed physicians acknowledged that adhering to CDS recommendations would expand their current role in HF management. Several noted that GDMT recommendations [32] and required outpatient lab monitoring (e.g., creatinine, potassium) for discharged patients would add complexity to ED discharges. Some suggested these tasks could be handled by other clinicians (pharmacists, cardiologists, primary care providers) outside the ED.
Some physicians, in interviews and surveys, reported that CDS adherence would necessitate additional clinical tasks (medication initiation, patient education, prognosis/care goals discussions), and workload constraints might prevent them from taking on these responsibilities during busy shifts. One physician considered initiating new GDMT medications in the ED unrealistic and a barrier to CDS adoption. However, others valued tailored CDS medication recommendations as an opportunity to improve patient outcomes. Most surveyed physicians indicated willingness to consider starting GDMT medications for patients discharged from the ED.
Workflow
Many interviewed physicians reported considering multiple data points from various parts of their clinical workflow when deciding to hospitalize HF patients. These include current weight, vital signs, breathing effort, ED treatment response, disability level, social factors, and shared decision-making conversations. Many felt CDS data collation would improve efficiency by presenting this important information centrally. All agreed CDS risk estimates would be useful for disposition decisions and efficiency. Nearly all interviewed physicians stated they would likely reassess patients initially considered for discharge but flagged as “high risk” by CDS. Furthermore, many reported that risk estimates would influence patient communication, shared decision-making, and outpatient follow-up planning. Some physicians noted CDS risk estimates would also be valuable in confirming appropriate outpatient management for low-risk patients.
People
Physicians showed openness to incorporating CDS risk estimates into their clinical practice, viewing them as valuable for patient and consultant discussions. While a few expressed concern about potential machine learning model bias, none indicated this would deter CDS use. Physicians generally viewed machine learning models favorably in healthcare, despite limited understanding of their mechanics. Trust in local ED leadership’s adoption of technological innovations was cited by several physicians. Personal relationships with study researchers also fostered comfort with practice changes.
Clinical Content
Multiple physicians highlighted the helpfulness of quantitative risk—a specific number communicable to patients and consultants—in HF management. Physicians reported that CDS provided useful recommendations to translate this quantitative risk into appropriate management decisions.
Discussion
Our mixed-methods study revealed broad physician support for the consolidated, HF-specific information and risk estimates provided by a CDS intervention. Physicians identified usability concerns, barriers, and facilitators to CDS adoption, which informed iterative design improvements.
The human-computer interface domain emerged as the most significant area for maximizing CDS adoption and continued use. Workflow integration, concerning task timing and intervention usage [1], varied among physicians. Usability testing and survey responses indicated physicians used (or would use) CDS for multiple purposes throughout their workflow. Earlier prompting and CDS design adjustments to accommodate diverse physician-EHR interactions could enhance use. A key barrier identified was CDS placement in a single, often overlooked EHR location. Survey data confirmed physicians’ desire for CDS access from multiple EHR points. The BETTER CARE-HF trial [39] similarly found that prompts in multiple EHR locations (medical record opening, medication ordering, refill requests) doubled GDMT compliance compared to standard care.
Physicians offered concrete, actionable suggestions to improve CDS prompts and content, including visual display modifications (color, text length) and greater control over displayed CDS content. Our intervention employed a passive prompt to encourage CDS interaction, similar to BETTER CARE-HF. This passive approach aimed to minimize alert fatigue, a known issue leading to desensitization from prolonged electronic alerts. [40] While a mandatory opt-out approach might increase CDS use, it could worsen alert fatigue, hindering long-term implementation and sustainability. [40, 41] The PROMPT-HF trial recently demonstrated an association between an opt-out physician prompting strategy and increased GDMT prescribing compliance in ambulatory HF patients. [42] However, PROMPT-HF’s ambulatory population and short study duration limit direct comparison and long-term alert fatigue assessment. [40]
Physicians noted the current CDS limitation in seamless workflow integration. Physicians had to exit CDS, remember recommendations, and separately enter orders, increasing cognitive load and workflow duration. [1] In a busy ED, this was a substantial barrier.
Physicians expressed concerns about their capacity to meet CDS expectations, citing workload as a potential adoption barrier. While recognizing GDMT prescribing cues as an opportunity to improve HF care, they worried about consistent guideline adherence due to heavy workloads. Furthermore, some viewed ED initiation of GDMT for discharged patients as outside traditional ED practice, more appropriately managed by cardiology, primary care, or HF specialty programs. These workload and role considerations are crucial, as interventions increasing physician task load can pose psychological and behavioral barriers to ED intervention adoption. [2, 43] Encouragingly, survey results showed most physicians believed CDS would improve guideline-concordant treatment and patient outcomes. Future physician education is vital to emphasize the link between GDMT prescribing and improved long-term outcomes.
Distrust of machine learning models is often cited as a barrier to ED tool deployment. [5, 44] However, physicians in our study did not identify machine learning as a CDS use barrier. Personal relationships with ED physician advocates likely mitigated these concerns. Trust emerged as a key factor in physician willingness to adopt CDS risk estimates, despite limited understanding of the model’s origins. Prior physician experience with other EHR-embedded CDS tools for different ED conditions may have also increased their receptiveness to the HF CDS.
Physicians found CDS clinical content helpful for HF management. Despite extensive experience (median 15 years) and established HF practice patterns, they found CDS clinical recommendations potentially useful. Though CDS risk estimates were novel, physicians recognized their value for both low-risk (reassurance for safe discharge) and high-risk populations (shared decision-making, consultant discussions, care transitions for home patients), potentially facilitating adoption and long-term use.
CDS Redesign
Based on usability findings, we made several changes to improve CDS accessibility, content, and workflow. Passive prompts were added to two more EHR locations. We included additional HF-specific data (recent ejection fraction, weight trends, cardiology notes) and reordered data by relevance based on physician feedback. We also implemented individualized medication recommendations based on current outpatient medications to aid GDMT ordering. These recommendations are now clickable text links directly to the EHR ordering module. Links to guidelines from the American Heart Association, European Society of Cardiology, and Kaiser Permanente were also added.
Our mixed-methods approach assessed novel CDS usability in the ED and identified implementation barriers and opportunities. Usability testing revealed real-time physician-EHR interactions and the need to tailor CDS implementation to varied physician workflows. Physician feedback on content, display, information organization, and technical barriers informed CDS redesign to enhance user satisfaction, adoption, and patient outcomes.
Limitations
This study has limitations. The interviewed physicians had significant experience (mean 15 years), and usability themes may not generalize to less experienced physicians. Early-career physicians might find CDS content more useful, be more open to risk prediction tools, and identify fewer implementation barriers. Physicians in our study were also familiar with other ED CDS tools, which may influence their perspectives. Physicians without prior CDS exposure might report different or more numerous barriers. The clinical practice environment (lower admission rates, high outpatient follow-up, high PCP access) may also limit generalizability. Furthermore, many survey respondents (39%) had not used the tool clinically, potentially limiting survey interpretation.
Conclusions
This mixed-methods qualitative study of ED physicians using CDS in practice identified key usability factors, barriers, and facilitators to implementation. Physicians valued CDS risk estimates, data collation, and medication management, believing CDS could improve HF patient outcomes. Findings informed CDS redesign and will guide system-wide implementation. Future usability and implementation studies are needed to maximize physician use and clinical impact.
Supplement 1. eFigure 1. Original CDS Used in the Pilot Study; eFigure 2. Semi-Structured Interview Guide; eFigure 3. Codebook Used for Qualitative Analysis; eTable. Methods Used to Ensure Qualitative Rigor; eFigure 4. Intersection of Theoretical Frameworks Used in Thematic Analysis of ED Physician Interviews and Usability Testing Sessions
Supplement 2. Data Sharing Statement
References
[References from the original article would be listed here, maintaining the original numbering and links if possible]
Associated Data
Supplementary Materials
Supplement 1. eFigure 1. Original CDS Used in the Pilot Study; eFigure 2. Semi-Structured Interview Guide; eFigure 3. Codebook Used for Qualitative Analysis; eTable. Methods Used to Ensure Qualitative Rigor; eFigure 4. Intersection of Theoretical Frameworks Used in Thematic Analysis of ED Physician Interviews and Usability Testing Sessions
Supplement 2. Data Sharing Statement
