A PTS Clinical Usability Paper

Clinical Usability
                     

Reprint of paper delivered at
UPA 99, sponsored by the
Usability Professionals’ Association

How Usability Engineering 
Can Improve Clinical Information Systems

Deborah Hinderer, Tec-Ed, Inc.
Phillip Scarborough, Tec-Ed, Inc.
Stephanie Rosenbaum, Tec-Ed, Inc.

Abstract

This paper describes a usability engineering program of integrated laboratory and ethnographic studies for collecting user data about a clinical information system. The authors evaluated MIRACLE (Medical Information Retrieval Application for Clinical Enhancement), developed by Philips Medical Systems; physicians and allied health personnel can access MIRACLE from their offices to obtain data about their patients who use hospital services. After an initial heuristic evaluation, we conducted two usability tests and weekly ethnographic interviews with physicians and hospital staff during the software alpha test. Our experience resulted in guidelines for conducting usability programs with medical professionals.

Introduction

Successful user interfaces for computer-based systems used in patient care (clinical information systems) are heavily dependent on acceptance by their user communities: physicians, nurses, and a wide variety of allied health personnel. These user communities are conservative about clinical information systems, for several reasons:

• Errors have the potential to be life-threatening to patients
• The health care profession as a whole has been less computer-literate than many other industries, although this literacy gap is slowly closing.
• The primary focus of patient care professionals is on the patient, so computer-based systems receive only secondary attention

User-interface design for clinical information systems must consider the entire context of use, not just the computer screen. Although clinical information systems in hospitals and free-standing facilities are proliferating rapidly, their usability depends on how well they can be integrated into the work processes of the clinical staff.

This paper describes a usability engineering program combining laboratory and ethnographic methods. The usability program was designed to inform Philips Medical Systems’ development of MIRACLE (Medical Information Retrieval Application for Clinical Enhancement).

MIRACLE is a clinical information system for referring physicians whose patients are using hospital services, either as inpatients (for services that require a hospital stay) or as outpatients (for services not requiring an overnight hospital stay). Historically, physicians have encountered delays obtaining lab results about their patients, yet they need this information promptly to specify appropriate treatment. Physicians can access MIRACLE from their offices to obtain data about their patients who use hospital services.

The authors worked with Philips Medical Systems to improve the usability of MIRACLE. Although the study design was created for a specific piece of software, the methodology is transferable to other medical and health-care products. The concerns we addressed about physicians, nurses, and hospital staff are encountered consistently by usability professionals and developers working on products for the medical environment.

Project Goals and Challenges

The authors conducted the usability program during the alpha test of MIRACLE at a major U.S. hospital. The initial goals of the usability program were to:

• Uncover ease-of-use problems with the MIRACLE user interface
• Assess the usefulness of the MIRACLE system to healthcare professionals (MDs and some hospital staff)

Overall, the software development team wanted to answer the following questions:

• How quickly and successfully could the audience learn to use MIRACLE?
• Was the speed of information access acceptable?
• Did the system display information in a clinically appropriate manner?
• Did the system present information in a format that made the users’ work easier?

In the study designs, the authors identified many specific issues related to the software that would help answer these questions. (Because the software is not yet released, this paper does not include detailed descriptions of the MIRACLE functionality or user interface, but rather concentrates on usability methodology.)

We correctly anticipated that motivating busy physicians to participate in the usability study would be a major challenge. In fact, participation was an ongoing challenge throughout the project, not just at the project start.

Methodology for Integrated Laboratory and Ethnographic Studies

In the project plan for the usability program associated with the alpha test, we defined six activities:

• Informal heuristic evaluation to inform usability test designs and physician interview questions
• Initial “out-of-box” usability testing for learnability, conducted onsite at each physician’s office
• Weekly ethnographic interviews with the participants
• Review of audiotaped diaries maintained by the physicians as they used the software
• Review of automated system usage logs
• A second usability test, of complex tasks and less-used features, conducted near the end of alpha test

The usability program began with the domain expert on the MIRACLE team and the three Tec-Ed usability specialists independently performing informal heuristic evaluations of the user interface. These heuristic evaluations identified immediately obvious usability problems and recommended changes to the user interface, several of which were made before the first usability test.

Both usability tests were exploratory, although structured; they identified problems and issues of concern, as well as simple measures for tabulation. The usability test administrator worked from a script to ensure consistency andreduce bias. To make the best use of the physicians’ time, we designed each usability test to address as many issues as practical in 60-minute sessions. We prepared a summary report of the results of each usability test.

 Testing learnability was important because many physicians in their normal context of work are unwilling to spend valuable time receiving training. However, we were concerned that if the physicians had unpleasant experiences during initial usability testing, they would be less likely to continue with the usability program. Therefore, we planned a short coaching session at the end of the first usability test, to answer questions and build skills.

The authors conducted regular ethnographic interviews with the participants, using a checklist of questions. These interviews compared the participants’ perceptions with the usability goals for the software. For continuity and progress monitoring, we planned to interview each participant weekly for about 30 minutes. Due to the physicians’ patient schedules, we expected that the interviews might vary in length and some physicians might skip a week occasionally. We maintained an “interview log” for each participant, which contributed to our final project report.

 During our ongoing work with the participating physicians, we explained the importance of keeping audiotape diaries, as well as offering encouragement and reminders to improve the regularity and completeness of the diary entries. We hoped that both the physicians’ diaries and the system audit trail reports would provide additional data to inform our findings and recommendations.

The realities of the physicians’ work processes and the alpha-test situation affected both our activities and their results. Every protocol required some adjustment during the course of the usability program, primarily to maximize data collection in the face of the physicians’ limited availability. In fact, almost all the usability data collected during the alpha test came from the two usability tests and the ethnographic interviews. Only one participant used the audiotape diary, and problems with the system server and patient data available during the alpha test meant that most participants did not use the software extensively outside the usability sessions. Thus the system audit trail reports were less relevant to our findings.

Participants In The Usability Program

The alpha-test hospital initially recruited six physicians: four internal medicine specialists, one general surgeon, and one pediatrician. Four were fairly experienced computer users, one was intermediate, and one was quite inexperienced. Although we would have preferred a higher proportion of novice users, it is consistent with our observations from other projects that some interest in computer-based systems was a prerequisite in motivating physicians to participate. Such interest is likely to be associated with greater computer experience.

Throughout the project, recruiting and scheduling participants with limited availability posed a major challenge. fter the first usability test, session scheduling and follow-up passed from hospital IT staff to the authors and their colleagues. Despite many calls, management discussion of the project’s importance, and highly flexible scheduling, only one participant took part in all scheduled activities. Two physicians participated in only the first usability test. Therefore, after the first ethnographic interview, we added two members of the hospital staff to the study: a unit clerk from an in-patient ward and a registered nurse. The jobs of both these people would involve extensive use of a major component of MIRACLE. Table 1 identifies the usability activities in which participants took part; Participant #7 was the unit clerk and Participant #8 was the nurse.

Participant Orientation And Training

Before the alpha test began, the authors and our colleagues in the MIRACLE development team agreed on an approach to ongoing training and orientation for the participants. Philips Medical Systems invited the participating physicians to a “kick-off” dinner to thank them for joining the alpha test and to provide orientation. At this dinner, the chief medical officer of Philips Medical Systems described the usability program to the participants and introduced the usability team.

We then implemented several orientation and training activities:
 

• Each participant was provided with an introductory/orientation packet describing MIRACLE, the usability program, and their participation in the program.
• The last 15 minutes of the first 60-minute usability test session were designated for training physicians on specific features and answering their questions about the interface.
• Each participant was provided with an updated copy of the user’s guide that reflected the interface of the MIRACLE prototype installed in physicians’ offices.
• The authors’ 1/2-hour interviews with participants each included 10 to 15 minutes of our interview questions and 10 to 15 minutes of training physicians on specific features and answering their questions about the interface.
• Throughout the alpha study, IT staff at the hospital acted as a Help line for participants.
• All these training activities both helped the participating physicians and nurses stay involved in the usability program and contributed to the feedback we received from them. We observed their behavior and noted their comments during the training as well as during the formal usability activities.
   

Findings And Recommendations

Although confidentiality prevents us from describing the specific problems we identified with MIRACLE, our findings covered the following areas:

•  Information Retrieval: The alpha user interface offered three different ways to access different kinds of new patient information. We recommended approaches for clarifying the distinctions.

• Symbols and Colors: The alpha user interface used five symbol shapes in five different colors, and participants were not clear about their meanings. We suggested ways of reducing the number of symbols and colors.
• Abbreviations: Participants were bothered by unfamiliar abbreviations found in the user interface. Because of regional variations in clinical terminology, we recommended eliminating abbreviations whenever possible.
• Expanded Access: Several physicians wanted to view the records of patients of other physicians for whom they “cover” regularly. This feedback led to further consideration of realistic usage scenarios before the beta test.
• User Guide: Only two of eight participants consulted the user guide; the others said they didn’t have time to read it. The hospital experience was typical of what we observe when busy professionals begin to use a computer system: only a few turn willingly to the documentation for help. However, research has shown that minimalist documentation design can improve the effectiveness of manuals for such audiences. We recommended a minimalist Getting-Started booklet for the software.

The usability program results also included other findings directly tied to specific MIRACLE functions. For all of the usability program findings, the authors provided both short-term recommendations and redesign recommendations the software development team could consider over the long term.

Lessons Learned

Based on the usability program the authors conducted during this hospital-based alpha test, we recommend the following guidelines for usability engineering of clinical information systems:

• Focus user requirements-gathering on user goals and tasks; that is, investigate exactly what the target health-care audiences do during their daily activities and how they do these activities. Use site visits and interviews (and contextual inquiries, if possible) as predecessor projects to performance-based studies.
• Perform heuristic evaluation of the software to identify and correct obvious problems before usability testing. This iterative approach always improves productivity and is especially valuable when dealing with medical professionals.
• Budget extra time and resources for participant recruiting, scheduling, and orientation, both early and throughout the project. Obtain support from respected domain experts for initial contacts with candidates and for participant orientation.
• Provide the most accurate, up-to-date data possible, even in the test setting. If necessary and feasible, delay some—but not all—usability studies until databases are populated with current information; health-care audiences are especially sensitive about the timeliness of data.
• Use methodologies that permit some flexibility in study design or implementation. Plan on more postponements and drop-outs from participants than are typical for usability projects with business or consumer systems.

Overall, our experience from this usability program has underlined the need to be especially conscientious in following best professional practices with potentially difficult target audiences. Health-care personnel have many good reasons to be demanding usability participants. If we want to improve clinical information systems and user acceptance of them, usability professionals must expect to “go the extra mile” to collect data from this audience.

Acknowledgments

The authors wish to thank Beverley Kane, MD, Chief Medical Officer of Philips Medical Systems MedGRID Program, for permission to describe the usability studies Tec-Ed performed during the recent MIRACLE alpha test, and for her ongoing expertise and support throughout the project. Thank you also to Amy Crownover, MSN, CS, FNP-C, ARNP (Assistant Professor, Tennessee State University School of Nursing) for background information on clinical practice; and to the Tec-Ed Support Services Team for participant scheduling, data compilation, and report production support during the project.